Guillaume Lebedel · · 8 min 
Natural Language Tools vs function calling: 14 models tested
Table of Contents
Natural Language Tools present an agent’s tools as plain prompt text answered YES or NO, instead of JSON schemas. In a July 2026 replication across 14 models and 8,560 trials, NLT picked the right tool 62.3% of the time against 47.4% for structured function calling, with 93% fewer critical errors and 25.2% fewer tokens.
That average is the weakest number in the paper. Eleven models improved, three got worse, and the spread between best and worst case is 77 percentage points. The usable finding for anyone building agents is that tool format behaves like a per-model setting that has to be measured, and most teams have never measured it once.
What are Natural Language Tools for AI agents?
Natural Language Tools, or NLT, are tool definitions written as ordinary prompt text rather than machine-readable schemas. The model reads a list of capability names and answers YES or NO to each one. The framework comes from Johnson et al. (2025), and a July 2026 replication study by Somma, Plante and Premji validated it independently across a wider model set.
No new decoding mode is involved, and no fine-tuning. The tool list is text in the prompt, the answer is text in the completion, and the runtime is unchanged.
How do Natural Language Tools differ from structured function calling?
Structured function calling asks the model to emit a JSON object matching a declared schema, so a missing brace or an invented field name fails before any tool runs. NLT asks for a word from a two-word vocabulary, which makes the parse step close to unfailable. The structured arm produced 755 critical errors and the NLT arm produced 51.
| Structured function calling | Natural Language Tools | |
|---|---|---|
| Wire format | JSON object matching a declared schema | Prompt line answered YES or NO |
| Typical failure | Malformed JSON, hallucinated field name, empty call | Wrong tool selected, output still parses |
| Critical errors in the study | 755 | 51 |
| Token use | Baseline | 25.2% lower |
| Parameters | Supported | Not tested |
| Accuracy across 14 models | 47.4% | 62.3% |
The authors argue the error gap compounds in recursive workflows where agents chain calls across sub-agents: a structured parse failure triggers a retry, then fallback routing, then coordination overhead, while a wrong-but-parseable answer costs one turn.
What did the July 2026 replication measure, and how?
The design was 2 approaches by 2 scenarios by 16 inputs by 2 perturbations by 5 replicates, giving 640 trials per model. Each trial was a single-turn selection from seven tools and a finish option, with no parameters, graded on exact match. Tools reached the model as prompt lines it answered one at a time.
Two models have partial coverage, Gemini 2.5 Pro at 6 of 8 conditions and Qwen3-VL at 5, which is why the total is 8,560 trials and not the 8,960 the design implies. The literal format:
Thinking: (insert_thinking)
Recap of previous conversation - YES/NO
Website information - YES/NO
Recent social media posts - YES/NO
Available discounts - YES/NO
List of upcoming events - YES/NO
Past Purchases - YES/NO
Talk to a Human - YES/NO
Assessment finished.That is the whole intervention. Reading it next to the equivalent JSON schema block makes the accuracy gap easier to believe than the headline number alone.
Which models gained the most, and which got worse?
Eleven of 14 models scored higher with NLT and three scored lower. Claude Sonnet 4 went from 18.8% to 61.9%, a 43.1 point gain. Mistral 7B went from 0.0% to 39.4%, unusable to usable. Gemini 2.5 Pro lost 33.7 points, GPT-OSS-120B lost 6.4, and Kimi-K2 was flat.
| Model | Structured | NLT | Change |
|---|---|---|---|
| Claude Sonnet 4 | 18.8% | 61.9% | +43.1pp |
| Mistral 7B | 0.0% | 39.4% | +39.4pp |
| Qwen3-VL | 0.0% | 33.8% | +33.8pp |
| DeepSeek-R1 | 31.0% | 55.0% | +24.0pp |
| DeepSeek-V3 | 69.7% | 90.0% | +20.3pp |
| GPT-5-nano | 59.4% | 79.1% | +19.7pp |
| Llama 3.1 8B | 32.9% | 47.8% | +14.9pp |
| Gemini 2.5 Flash Lite | 63.1% | 73.1% | +10.0pp |
| Gemini 2.0 Flash | 79.5% | 85.0% | +5.5pp |
| GPT-OSS-20B | 39.3% | 42.7% | +3.4pp |
| GPT-5 | 80.3% | 81.9% | +1.6pp |
| Kimi-K2 | 67.8% | 67.2% | -0.6pp |
| GPT-OSS-120B | 49.0% | 42.6% | -6.4pp |
| Gemini 2.5 Pro | 82.1% | 48.3% | -33.7pp |
Treat the two partial-data rows with more caution, and note the Qwen3-VL structured score is corrected for survivorship bias. The paper rounds accuracies and deltas independently, so the Gemini 2.5 Pro row subtracts to 33.8 while the paper reports 33.7.
Why do Natural Language Tools help Claude Sonnet 4 and hurt Gemini 2.5 Pro?
The paper’s explanation is training rather than architecture. Models tuned hard on structured tool calling through reinforcement learning have that path reinforced, so taking the schema away removes the thing they were optimised for. Models without native tool calling, reasoning models and smaller models gained between 24.0 and 43.1 points.
Read that way the result stops being surprising. A frontier model is asked to work in a mode it saw less of, while a 7B model with no native tool calling gets a task it can express in ordinary language for the first time, which is how Mistral 7B moves off a 0.0% floor. The pattern matches the losses that appear when a catalogue grows past what a model can discriminate between: whichever format asks the model to hold less in working memory wins.
What can the Natural Language Tools study not tell you about production agents?
Three limits, all stated by the authors. The task was single-turn, so chaining and error recovery are untested. Tool calls carried no parameters, so argument extraction, the part most production agents get wrong, is out of scope entirely. Each scenario offered seven tools, well below the catalogue sizes where selection accuracy usually breaks down.
That third limit is the gap most worth closing next. Seven tools is small enough for a model to read the whole list, and the interesting case is 2,000 actions, where the definitions are the dominant token cost in the context window. NLT’s token advantage likely stops deciding anything at that size, but the paper does not test it.
The reliability number survives those caveats better than the accuracy number. A 93% reduction in unparseable output is a property of the format rather than of the task size.
How should you choose between Natural Language Tools and function calling?
Test both formats against the models actually in use, on the real tool set, and keep the result per model. A format that gains 43 points on one model and loses 34 on another is not a best practice, and a company-wide default is a bet rather than a decision.
A version of this that fits in an afternoon:
- Take the 20 requests the agent gets most often and label the correct tool for each by hand. This set is the whole experiment, so it is worth an hour of care.
- Run them through the current structured tool calling. Record selection accuracy and parse failures separately, because they have different fixes.
- Run the same 20 with the tool list flattened into prompt lines and a YES/NO answer per tool.
- Compare per model, never averaged, which is what makes the headline 14.9 points misleading.
- Re-run whenever the model or model version changes. A gain measured on one release is not inherited by the next.
Step 5 is the one people skip. Model swaps get treated as like-for-like, and tool-selection accuracy is not something teams notice degrading until an agent starts choosing the wrong action in production.
StackOne treats tool exposure the same way, which is why it is a query parameter on the StackOne MCP server rather than fixed behaviour. tool-mode=individual registers one tool per enabled action, which our docs recommend for action sets under about 50 tools. tool-mode=search_execute registers two meta-tools instead, a search over the catalogue and an execute by action id, so prompt cost stays flat whether the catalogue holds 50 actions or the 30,000+ on our platform. Individual is the default, and a default is not a recommendation for every model. We published the benchmark numbers for the embedding model that runs action search for the same reason.
How tools reach the model is a variable, and it is cheaper to measure than to argue about. To compare the two exposure modes on a given catalogue, what each mode registers is documented in the StackOne MCP reference.
Facts in this post were verified on 31 August 2026 against the published paper and StackOne’s live documentation.