Guillaume Lebedel · · 7 min read 
Why AI coding agents disagree on tool choice
Table of Contents
Armature ran 16,893 coding agent sessions and published 5,292 of them on 3 September 2026. Claude Code, Codex and Cursor picked the same tool in only 42% of comparable cases. Repository language flipped the winner: Resend on TypeScript, SendGrid on Python, Postmark on Go. Mentions did not predict picks, because PayPal was cited 139 times and chosen zero.
Facts in this post were verified on 4 September 2026.
What did Armature measure across 16,893 coding agent sessions?
Armature generated 16,893 coding agent sessions, judged 5,292 of them valid and published those with full traces on 3 September 2026. The runs covered 75 repositories, 10 languages, 51 codebases across 18 sectors and 1,163 prompt variations, driven by Claude Code, Codex and Cursor under four developer personas.
The orchestrator and the judge were both Gemini 3.7 Flash. Each session ran in an ephemeral sandbox where the agent had to implement the feature rather than recommend one, which is what makes the install counts worth reading at all. Armature also states plainly that it sells growth services to developer tool companies, so the aggregate rankings come with a declared commercial interest attached.
Agent-initiated work is already a large share of deployment traffic. Vercel reported in April 2026 that over 30% of deployments on its platform are started by coding agents, up tenfold in six months, with Claude Code accounting for 75% of that share.
How often do Claude Code, Codex and Cursor pick the same tool?
In comparable sessions the three agents picked the same tool 42% of the time. The remaining 58% went to different vendors for the same task. Part of that gap comes from how the agents research: Codex ran a web search in 94% of sessions, while Claude Code searched in roughly 30% and otherwise worked from its priors.
Codex also narrowed nine out of ten of those queries with site: operators, which anchors its picks to whatever is currently findable in vendor documentation. Claude Code went the other way and implemented the feature in-house rather than adding a dependency in 19% of sessions, against 10% for the other two.
None of this is a reasoning failure. It is the same effect measured when agent accuracy drops as the tool count grows: when the candidate list is large and weakly ordered, small differences in what the model reads first decide the outcome.
Why does repository language change which tool an agent picks?
Repository language moved the result more than any other variable Armature tested. The same request for an email provider, run against four repositories in four languages, produced four different winners: Resend on TypeScript, SendGrid on Python, Postmark on Go, Azure Communication Services on Java. The agents were reading the existing stack, not ranking vendors.
| Repository language | Winning email provider | Sessions won |
|---|---|---|
| TypeScript | Resend | 55 of 89 |
| Python | SendGrid | 22 of 24 |
| Go | Postmark | 20 of 24 |
| Java | Azure Communication Services | 22 of 23 |
Source: Armature, 3 September 2026.
Hosting showed the same pattern in a sharper form. Vercel won every Next.js repository in the study and was never once recommended on a Python repository, where Render took most of the hosting work.
Why does being mentioned by an agent not mean being picked?
Armature counted mentions and installs separately, and the two barely track each other. PayPal was mentioned 139 times and picked zero times, with Stripe taking 124 of those same sessions. LangChain was the most-cited framework in the study at 194 mentions and was chosen four times.
| Tool | Mentions | Picks |
|---|---|---|
| LangChain | 194 | 4 |
| Adyen | 175 | 3 |
| Netlify | 152 | 6 |
| PayPal | 139 | 0 |
Source: Armature, 3 September 2026.
A whole measurement industry has grown up around counting how often a brand appears in model output. On this dataset that count tells you almost nothing about whether the tool ends up in the lockfile. Supabase was the most-mentioned database in the study and still lost most of its sessions to Neon.
How do documentation and pricing pages change an agent’s choice?
Two of the clearest losses in the study came from page copy rather than product quality. Mailgun kept losing to Postmark once agents read the 1-day retention line on its free plan. Supabase lost database-only tasks to Neon because its pricing page bundles auth, storage and realtime with the database.
An agent reads a pricing table as a specification. It does not discount for marketing bundling and it does not call sales to ask whether a retention limit is negotiable, so whatever is written on the page becomes a constraint on the decision.
What should you control when you cannot control the agent’s pick?
You cannot make three agents agree, so control the inputs they read. That means the repository they work in, the documentation and pricing pages they parse, and the candidate set they are allowed to see. The first two moved outcomes in the study, and the third is the one you control directly in production.
- Pin the stack in the repository. Existing dependencies, lockfiles and framework choice were the strongest signal in the data, so an agent working in a scratch directory will give you a different answer than the same agent working in your monorepo.
- Read your own pricing and docs pages as an agent would. Retention limits, bundled plans and feature tables are parsed as hard constraints.
- Count installs, not mentions. If you track how visible your product is to coding agents, the number that matters is how often it ends up in a dependency file.
- Scope the candidate set before the model reasons about it. Retrieval quality moves sharply with scope: on StackOne’s tool-retrieval benchmark, a 109M embedding model went from 57.3% to 92.8% Hit@1 once the candidate pool was scoped to the right connectors rather than searched globally.
That last point is the layer StackOne works on, so read this passage as a builder’s note rather than a neutral one. An admin decides which of the 31,000+ actions across 510+ connectors a given agent can see on a connected account, and that decision is made per connector profile rather than per agent. It reads like a permissions setting on the way in. In practice it does the same job repository language did in Armature’s study, narrowing the candidate set before the model reasons about anything, which is also why so many agent failures turn out to be permissions problems rather than model problems.
The full dataset and traces are at Armature’s write-up of the 16,893 sessions.
If you want to see what scoping an agent’s action set looks like in practice, the administration section of the StackOne docs covers how agent permissions and per-profile action enablement are configured.