Function calling is how a language model asks a program to do something for it. The model writes a short piece of structured text, such as {"name": "get_forecast", "arguments": {"city": "Oslo", "days": 3}}. The program around the model reads that text, runs the function, and hands the result back. A person forgives a missing comma, but a program does not. So every agent needs a precise contract for what a call looks like, a way to check each call against it, and ideally a way to make broken calls impossible to write.

Here is why this matters, in numbers. An agent, as §24.1 set it up, runs a loop. The model writes text, and a small program called a parser reads an action out of it, either a tool call or a final answer. The tool runs, and its result is added to the model's context for the next turn. When the parser cannot read the text, it returns a special symbol $\bot$, meaning "malformed". Suppose a model writes a malformed call once in every 50, and a task needs 30 calls. Then at least one call of the task is broken with probability $1 - 0.98^{30} = 0.45$. Almost half of all tasks hit a broken step before any reasoning has gone wrong.

There are three lines of defense, and this section builds all three.

1. State the contract. Describe each tool's arguments exactly, in a standard language called JSON Schema, and train the model on calls that follow it.
2. Check and repair. Check each call after it is written, and send the list of what is wrong back to the model so it can fix the call.
3. Make malformed output impossible. While the model writes, remove every token that would break the contract before the next token is chosen. This is called constrained decoding.

The third defense gets the most space, because it is an exact algorithm with a proof. It is built on finite automata, small machines that read text one character at a time, and for nested formats like JSON on automata that also keep a stack. It has a cost that most practitioners never notice, because it changes what the model says, and we will measure by how much. The section ends with how models are trained to call tools and how that skill is measured.

A tool is a function with a contract

A tool is a function that the model may ask the surrounding program to run. The model never runs anything itself. It writes a request, and the program decides what to do with it. To write a good request, the model needs the function's name, a description of what it does, and an exact description of its arguments.

Here is a tool definition as you would send it to a model provider. It is an ordinary function signature written in JSON. The description tells the model when to use the tool, and the schema says which arguments are acceptable.

{
  "name": "place_order",
  "description": "Place an order for in-stock items. Quantities are whole numbers from 1 to 99.",
  "input_schema": {
    "type": "object",
    "properties": {
      "customer_id": {"type": "string"},
      "items": {
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "sku": {"type": "string"},
            "qty": {"type": "integer", "minimum": 1, "maximum": 99}
          },
          "required": ["sku", "qty"],
          "additionalProperties": false
        }
      },
      "shipping": {"enum": ["standard", "express"]},
      "gift_note": {"anyOf": [{"type": "string"}, {"type": "null"}]}
    },
    "required": ["customer_id", "items", "shipping"],
    "additionalProperties": false
  }
}

Read it as a function signature. customer_id is a required string. items is a required list of objects, and each object has a string sku and a whole-number qty from 1 to 99, and nothing else. shipping must be one of two words. gift_note may be left out, and when present it is a string or null. No other argument is allowed.

The schema is written in JSON Schema, a standard language for saying which JSON values are acceptable (JSON Schema 2020-12, the current version). Anthropic's API puts the schema in a field called input_schema, and OpenAI's in one called parameters. Open-weight models receive the same object written into their prompt. JSON Schema is large, with string patterns, conditional rules, and references between schemas. Tool definitions use a small core of it. That core is worth knowing exactly, because the exact rules are where implementations disagree.

In plain words, a schema is a list of independent checks, and a value is valid when it passes all of them. Each check speaks only about values of its own type. That last rule is why real schemas pair every keyword with a type. On its own, {"minimum": 1} would let a string through.

Two of these rules collide with Python, and both cause real bugs. JSON has only one kind of number, so json.loads turns 3.0 into a Python float even though JSON Schema calls it an integer, and it turns 1e2 into the float 100.0. And Python's bool is a subclass of int, so isinstance(True, int) is True. A validator written with isinstance therefore accepts true as a quantity without complaint. The cell shows both.

The check never converts a Python int to a float. Python integers have no size limit, so json.loads turns a 401-digit number into an int, and float() of it raises OverflowError. A validator must not crash on any input a model can write.

A validator should not answer only yes or no. It should return a list of everything that is wrong, each error tied to a path, the address of the bad value inside the arguments. That list is what goes back to the model when we ask it to repair the call. A model told $.items[2].qty: expected integer can fix the one thing that is wrong. A model told "invalid" can only guess. In an interview, "what does your validator return?" is the question that separates a toy from a production design.

def validate(instance, schema, path="$"):
    return []      # placeholder: reports no errors, so it accepts everything

One call, end to end

A tool call passes through six steps between the schema and the function, and every production bug lives in one of them. One term first. A chat template is the fixed recipe that turns a conversation (system message, user turns, assistant turns, tool results) into one string of tokens for the model. Special tokens mark where each message starts and ends (§14.3).

1. Render. The serving stack writes the tool definitions into the prompt, usually inside the system message, using the chat template. Anthropic's API does the same with its own wording. Its documentation says the API "constructs a special system prompt from the tool definitions", which is why tool definitions cost input tokens on every request.
2. Generate. The model samples tokens as usual (§6.4). It may write some text first, then one call, several calls, or none at all.
3. Stop. The turn ends at the end-of-turn token. In many open models the markers that open and close a call are single tokens added to the vocabulary, so opening a call is one decision, learned like any other token (§2.5).
4. Parse. The server finds the calls in the text and decodes their JSON. This is the parser of the agent loop, and it can fail. The JSON may be malformed, the name may be missing or unknown, or the token limit may have cut the output off in the middle of a call.
5. Validate. The arguments are checked against the tool's schema, with the validator you just wrote.
6. Execute and return. The function runs. Its result, or its error, goes back into the context as a tool message tied to the call by an id, a short unique string attached to the call. The model reads the result as its next observation, and the loop continues.

Hosted APIs hide steps 1 and 4 behind structured objects, but the six steps are the same everywhere. The table maps them onto the two largest APIs and the Qwen2.5 template.

Two rows of the table hide design decisions. The first is forcing a call. The tool_choice setting can make the model call some tool (any or required) or one named tool. Anthropic implemented this with a prefill. Its documentation says that with any or a named tool "the API prefills the assistant message to force a tool to be used". To prefill is to write the start of the model's reply for it, so the turn begins inside the call. The model then writes no explanation and no reasoning before the call. The newest Claude models (Opus 5.5, Sonnet 5.5, and Fable 5.1, as of October 2026) reject forced calls with an error. The documentation recommends auto, a strict schema, and an instruction in the prompt instead.

The second is parallel calls, several calls in one turn. Their results may come back in any order, and the id is the only thing that binds a result to its call. Templates without ids, like Qwen2.5's, match results to calls by position, which breaks as soon as a result is dropped or reordered.

Here is the whole loop against a real SDK. It cannot run in your browser, which has no network, but every line is one of the six steps. The runnable cells in this section replace the model with a short script that has the same interface, messages in and text out. The loops are real, and only the decisions are hand-written.

The strict flag asks the provider to constrain decoding to the schema, which is the subject of most of this section. Notice the range check inside the function body. Anthropic's strict mode, as documented in October 2026, does not support minimum and maximum, and a raw request that uses them is rejected. Its SDK helpers remove such keywords before sending and copy them into the field's description, and the helpers that parse the response check them on the client afterward. So some rules must still be checked after the fact, by your own code.

With open-weight models you write step 4 yourself, and it is a classic interview problem. The model's output is raw text. You must find every call in it, survive every way it can be broken, and never pass a bad call through.

import json, re

def parse_tool_calls(text, tool_names):
    m = re.search(r"<tool_call>(.*)</tool_call>", text, re.S)     # greedy, and only one call
    if not m:
        return []
    call = json.loads(m.group(1))
    return [{"name": call["name"], "arguments": call.get("arguments", {})}]

Validation, repair, and the arithmetic of retries

The second line of defense treats a validation error as an ordinary observation. The server does not crash or give up. It sends the list of errors back to the model as the tool's result, and the model tries again. Protocols now say so explicitly. The Model Context Protocol is the open protocol for connecting models to tool servers, and §24.6 takes it apart. Its 2025-11-25 revision makes invalid arguments a tool execution error, returned to the model so it can correct itself, rather than a protocol error that ends the request.

Here is a repair loop. The policy is a scripted stand-in for a model. Its first call has two mistakes, since days is the string "3" and there is an extra units argument. On each retry it fixes whatever the error list names.

How much does repair buy? §24.1 split error handling into two probabilities. The chance that an error is detected is $d$, and the chance that a detected error is repaired is $q$. A validator catches every schema violation, so for this kind of error $d = 1$, and only the repair rate matters. The new question is what a budget of several retries buys.

In plain words, each allowed retry multiplies the leftover failure rate by $1-q$, while the cost in turns levels off at $N_c\varepsilon/q$. The first retry buys the most. Later retries are cheap, because they are rarely needed.

Two cautions keep this honest. First, repairs are not independent in practice. A model that misread the schema once tends to misread it again, so the real failure rate falls more slowly than $(1-q)^k$. Second, a validator can only detect what the schema can express. A call to place_order with the right types and the wrong customer is valid, and for that kind of error $d$ is far below 1. Repair is worth exactly as much as the error messages are specific, which is why the exercise insisted on paths. An interviewer who asks "how many retries would you allow?" wants this formula, and then these two caveats.

Making malformed output impossible

Repair fixes mistakes after they are made, at the price of a turn. The third defense prevents them. Here is the idea in plain words. At every step the model gives a probability to every token in its vocabulary. Before the next token is chosen, we cross out every token that would make a valid output impossible, and we choose only among the tokens that remain. If we cross out exactly the tokens that lead to dead ends, the output is valid by construction.

Crossing out is easy. Each token has a logit, the score the model computes for it before the softmax turns scores into probabilities. Setting a token's logit to $-\infty$ makes its probability exactly zero. This is the same masking trick that §6.4 uses for top-k sampling. The hard part is deciding, fast, which tokens to cross out. For that we need a description of the valid strings that a program can step through one character at a time. The right description is a finite automaton, a small machine with a few states and edges labeled by characters. Here is one, small enough to run by hand.

Now the general picture. Let $\Sigma$ be the set of characters (in practice bytes, as in §2.3). Let $\mathcal{V}$ be the vocabulary, each token a nonempty string over $\Sigma$, and let EOS be the end token.

In plain words, $\delta$ says where one character takes you, $\delta^*$ says where a whole string takes you, and the language is the set of strings that end in an accepting state. A live state is one from which acceptance is still reachable. In the yes-or-no example every state is live.

Two notes on symbols. The subscript in $\mathcal{L}_G$ (for "grammar") keeps the language apart from a loss $\mathcal{L}$. And from here on a plain $q$ always denotes a state. The repair probability $q$ of the previous part does not appear again, and $q_{\text{loc}}$, defined below, is a distribution, not a state.

One identity does the work in everything that follows. Reading a string $st$ is the same as reading $s$ and then $t$, so $\delta^*(q, st) = \delta^*(\delta^*(q, s), t)$, by induction on the length of $t$. That is why we may feed the automaton one token at a time, even though tokens are several characters long.

In plain words, the index answers "which tokens keep the output on track from here?" for every state, in advance. Decoding zeroes out every other token and rescales the rest to sum to one. $Z_t$ is the probability the model put on allowed tokens, so $1 - Z_t$ is the probability the mask removed. Engines implement the mask by adding $-\infty$ to every disallowed logit before the softmax. We write $p_\theta$ for the model so that $p$ alone keeps its §24.1 meaning of a per-step success rate.

The key design choice is to compute the index once, before any decoding, for each pair of a pattern and a vocabulary. Each decoding step then costs a table lookup instead of a scan of the whole vocabulary. This idea, compiling a regular expression to an automaton and precomputing its allowed tokens per state, is the core of the Outlines library (Willard and Louf 2023). The widget runs it on three patterns, a date, a JSON object whose one field must be one of three words, and a signed integer.

def live_states(dfa):
    return set(dfa["delta"])                      # placeholder: calls every state live

def build_index(dfa, vocab):
    E = len(vocab)
    return {q: {i for i, s in enumerate(vocab) if s[0] in dfa["delta"][q]} | {E}   # first character only, EOS always
            for q in dfa["delta"]}

The yes-or-no example showed that a correct index can still strand the decoder. The fix is one extra assumption, that every single character is also a token. With it we can prove that constrained decoding does what it promises.

In plain words, if single characters are always there as a fallback, the decoder can always take one more step toward a valid string, and it can only stop when the string is valid.

Byte-level vocabularies satisfy the assumption automatically, since all 256 bytes are tokens (§2.3). Without it, the index must also be pruned by what tokens, not characters, can reach. Call a state token-live if some sequence of allowed tokens leads from it to EOS. The token-live states are found by a fixpoint computation, one that repeats an update until nothing changes. It starts from the states that allow EOS and keeps adding states that have an allowed token into the set. Then every token that leads to a state outside the set is removed. The first cell builds the index for the yes-or-no example, and the second prunes it.

State 2 is the one token-dead state, and pruning removes ye from the start, exactly the trap of the hand example. What remains is the price of the index.

So the index turns the mask from a bottleneck into noise. That is the answer an interviewer wants to "why precompute?", together with its price, one stored mask per automaton state.

The automaton offers a second saving, in forward passes rather than in mask work. A tool call is full of text the model has no choice about, such as the braces and key names of its arguments. Masked decoding still spends one forward pass per forced token. Jump-forward decoding, built in §20.7, appends the whole forced string in one pass instead, as a small prefill chunk, and retokenizes the seam with the text already written so the model never sees a split it was not trained on. The SGLang serving system measured 1.6 times the throughput on JSON decoding from this (Zheng et al. 2024). For an agent, whose every step emits a call with fixed key names, that saving recurs on every turn of the loop.

The next exercise is the decoding loop itself. It is a favorite in interviews, because each line has a classic bug.

import numpy as np

def constrained_decode(logits_fn, dfa, vocab, index, max_len, greedy=True, rng=None):
    out, q = [], dfa["start"]
    for _ in range(max_len):
        v = int(np.argmax(logits_fn(out)))          # argmax before masking
        out.append(v)
        if v == len(vocab):

Why JSON needs a stack

A language that some finite automaton recognizes is called regular. Dates, fixed sets of words, and integers are regular. JSON is not, because objects and arrays nest to any depth, and a machine with finitely many states cannot keep count of how many brackets are open. Here is the precise statement.

In plain words, an automaton's only memory is its current state, and remembering which brackets are open takes a stack. A depth limit makes JSON regular again. But then the automaton must spell out every possible stack of open brackets as its own state, and their number grows exponentially with the depth. Real schemas also nest strings, numbers, and keys inside each level, which multiplies the count. In an interview, "why not just a regex for JSON?" is answered by this pigeonhole argument.

The fix is to give the machine a stack, a list it can add to at the top (push) and remove from at the top (pop). A pushdown automaton is a finite automaton that also keeps a stack. At each character it looks at its state and the top symbol of the stack, moves to a new state, and may push or pop. Here is one that checks a small piece of JSON.

For reference, here are the two formal objects behind this example, a grammar that generates the valid strings and a machine that recognizes them.

In plain words, a grammar says how to build valid strings piece by piece. A pushdown automaton checks a string from left to right while keeping on its stack the pieces it has opened and not yet closed. This machine moves only when it reads a character, one move per character, which is what a decoder wants.

The same construction scales to real formats. The subset of JSON Schema that tool definitions use (type, properties, required, enum, items, anyOf, and $ref, a reference to another schema) compiles to a context-free grammar. There is one rule per subschema, and recursion wherever a $ref points back to an enclosing schema. Engines usually also fix the order of an object's keys to the order in the schema, as OpenAI's Structured Outputs does. The reason is that a grammar allowing every order of $n$ required keys grows exponentially in $n$.

Some keywords do not fit. uniqueItems, not, and rules that tie fields together (dependentRequired, and if with then and else) are not context-free in general, or they blow up the grammar. So do large minItems, maxItems, and maxLength bounds, which must be unrolled into counted copies of a rule. Engines reject such schemas, drop the keywords silently, or check them after decoding. A benchmark later in the section measures how large this gap is.

Constrained decoding with a grammar keeps a configuration (a state and a stack) in place of a single DFA state, and the theory so far carries over. The proof of soundness and progress used three facts. Allowed tokens lead to live states, every single character is a token, and $\delta^*$ composes over concatenation. None of them needs finitely many states, so the theorem holds word for word with configurations in place of states.

What fails is the index. A pushdown automaton has infinitely many configurations, one for every stack, so $\operatorname{allow}$ cannot be tabulated in advance.

Liveness, at least, needs no search here. For the machine above, every configuration that a valid prefix reaches is live, because closing every open array finishes the output. So a token is allowed exactly when the walk through its characters survives. Exact prefix parsers for any grammar behave the same way, since they fail as soon as the text stops being the start of some valid string. The cell implements the machine and walks tokens through it. These include ]] and ],, which pop and keep reading, and [[, which pushes twice.

Look at ]] and ], after [1 and after [[1. Both times the state is num and the top of the stack is [. Yet the tokens are rejected the first time and allowed the second. Each token pops the open array and then reads one more character. That character is checked against the symbol beneath the old top, and only the full stack says whether that symbol is another [ or the bottom marker.

This splits the vocabulary in two. Call a token context-dependent at a configuration if reading it pops the symbol that was on top when it began and then reads another character. Call it context-independent otherwise. The verdict on a context-independent token depends only on the state and the top symbol. So it can be precomputed once per pair of a state and a top symbol, exactly like the DFA index. Only the context-dependent tokens need the full stack while decoding. The widget lets you build a nested array token by token and see how few they are.

import numpy as np

def init_config():
    return ""                                   # placeholder: the text so far

def advance(config, s, max_depth):
    return config + s                           # placeholder: never rejects anything

The same split, at the scale of a real grammar and a real vocabulary, is how the fastest engines work. The XGrammar engine (Dong et al. 2024) compiles the grammar to a pushdown automaton over bytes, whose states are positions inside grammar rules. It sorts tokens into two kinds.

• A token that enters a child rule, or moves forward within the current rule, is context-independent. Its verdict is precomputed and stored in a mask cache, keyed by the position on top of the stack.
• A token that reaches the end of the current rule and must continue in the parent rule is context-dependent. Only these are checked against the full stack while decoding.

OpenAI's announcement of Structured Outputs (August 6, 2024) gave the reason for grammars directly. Its example is a recursive schema for a user interface, whose components contain components, which a finite-state approach cannot express. Not every provider goes that far. Anthropic's strict mode, as of October 2026, rejects recursive schemas.

Constrained decoding changes what the model says

Constrained decoding guarantees that the output parses. It does not guarantee that the output follows the distribution you would get by sampling the model freely and throwing away every invalid output. Those are two different distributions, and the gap between them can be large. §20.7 met this gap from the serving side, in a hand example where the mask made the model's least favorite valid string its most likely one. Here we measure it, prove when it appears, and correct it, starting from the smallest example where the two distributions come apart.

The general statement needs one quantity, the probability that the model, continuing from a prefix, ends up with a valid output.

In the example, $\operatorname{acc}(a) = 0.01$, $\operatorname{acc}(b) = 1$, and $\operatorname{acc}(\langle\rangle) = 0.109$. Park et al. (2024) call this quantity the expected future grammaticality. By the law of total probability over the next token (§0.4), it satisfies a one-step recursion, $\operatorname{acc}(y_{<t}) = \sum_{v} p_\theta(v \mid y_{<t})\,\operatorname{acc}(y_{<t}v)$.

In plain words, exact conditioning weights each next token by how likely the model is to finish validly after it. Local masking only asks whether finishing validly is possible, a yes-or-no stand-in for a probability. When some allowed tokens lead into paths the model itself is unlikely to complete, local masking gives them too much weight. In the example, a had acceptance probability 0.01 and b had 1, and local masking treated them as equals.

Two consequences matter in practice. The first says when the distortion is small. Suppose $\operatorname{acc}(y_{<t}v) \ge 1 - \eta$ for every allowed token $v$ at every reachable prefix, for some small $\eta$. Then $\operatorname{acc}(y_{<t})$ lies between $(1-\eta)Z_t$ and $Z_t$, so each local factor is within a factor $1/(1-\eta)$ of the exact one in either direction. The distortion is small exactly when the model already writes valid output almost always.

So constrained decoding is safest on models trained to follow schemas, and distorts most on the models that need it most. OpenAI did both, training gpt-4o-2024-08-06 on schemas and then constraining it, which is the combination this result favors.

The second consequence is an exact correction. Weight each locally masked sample by $\prod_t Z_t$, a number the decoder computes anyway, and resample (or average) with those weights. The result targets the true conditional. This is importance sampling, drawing from an easy distribution and reweighting the samples to stand for a hard one, here with local masking as the easy distribution. An interviewer who asks "does constrained decoding change the model's answers?" wants the elevenfold example, the acceptance ratio, and this reweighting.

The widget is the hand example with its probabilities on sliders. The third slider is new. It sets the chance that the model stops after b instead of writing something outside the language. Move the sliders and watch local masking and the exact conditional separate or agree. They agree exactly when $p_\theta(b \mid a) = p_\theta(\text{EOS} \mid b)$, because those are the acceptance probabilities of a and b, and the theorem asks them to be equal.

Let us measure the distortion at a scale where the hand calculation is out of reach. The model is a bigram language model, one whose next-token distribution depends only on the previous token, with random parameters. Its vocabulary has 30 tokens, among them digits, punctuation, two-digit tokens like 12, hyphen tokens like -3, two word pieces, and EOS. The format is a date-like DD-DD. The first cell sets this up, and the second decodes greedily with and without the mask.

Unconstrained, the model rambles through digits and does not stop within eight tokens. Constrained, it writes 30-32 in three tokens and EOS. That string parses perfectly, and it is not a date anyone has seen, since there is no month 30 and no day 32. At two of the steps the mask removed 86% and then 91% of the model's probability, a sign that the model and the format disagree badly there.

Now the exact conditional. For a bigram model, the acceptance probability depends only on the previous token, the automaton state, and the number of tokens left. So it obeys a recursion we can memoize, caching each value the first time it is computed. With it we can list every output that local masking can produce and compare the two distributions exactly.

With a neural model, the recursion for $\operatorname{acc}$ cannot be computed, because it sums over all continuations of every prefix. That is why exact sampling from the conditional remains a research problem, and why production engines mask locally.

import numpy as np

def exact_next(P0, P, dfa, vocab, prefix, max_len):
    raise NotImplementedError

def kl_local_vs_exact(P0, P, dfa, vocab, index, max_len):
    raise NotImplementedError

In PyTorch

In a real engine the index is a boolean tensor of shape (states, V), and constraining a step is one masked_fill on the logits. Here a character-level bigram model is trained on a few sentences that mention dates. It is then decoded greedily, with and without a mask for the pattern DDDD-DD-DD.

Unconstrained greedy decoding falls into the repetition loop 2020... that §6.4 warned about. The mask forces the hyphens in, and the model fills in plausible digits around them. Production engines (XGrammar, Outlines, and the grammar backends of the vLLM and SGLang servers) do exactly this. They build the mask from a compiled grammar and apply it to a batch of logits on the GPU.

Token boundaries, and what format does to reasoning

The index above judges whole tokens against the automaton. So it allows a token that spans two parts of the grammar, such as a space followed by a quotation mark. Many systems do not work that way. A template engine writes the fixed parts of the output itself (the braces, the key names) and lets the model fill in only the values. It tokenizes the fixed text separately, and that produces token sequences the model never saw in training.

Beurer-Kellner et al. (2024) measured this. They used perplexity, the exponential of the average negative log-probability per token, which is low when the model finds a text natural. Constraining Mistral-7B with a fixed template raised the perplexity of its output from 4.17 unconstrained to between 24.50 and 26.75. On a JSON version of the GSM8K math benchmark, such misaligned constraining cut accuracy from 41.5% to 30.8%, while their aligned method reached 41.8%. Two fixes are in use.

• Token healing (§2.6) backs the prompt up by its last token and constrains the first generated token to start with the removed text. The model can then choose the merged token itself.
• Subword-aligned decoding precomputes how the grammar's pieces and the vocabulary's tokens overlap, so that tokens bridging two pieces of the grammar stay available. The authors' algorithm, DOMINO, does this at little or no cost in speed.

A related question is whether forcing a format hurts reasoning. The studies disagree at first sight, and the deeper box sorts them out. The reading that fits all of them is that format constraints are not free, but their cost depends mostly on how you prompt and on where the format puts the reasoning.

Training models to call tools

Constrained decoding guarantees syntax. Choosing the right tool with the right arguments, and knowing when not to call a tool at all, has to be learned. The data recipes have converged on four steps. Collect real APIs, write user requests for them, generate candidate calls and check them, and fine-tune on the calls that pass.

Three projects show the recipe growing up.

• Training with the documentation in the prompt. Gorilla (Patil et al. 2023) fine-tuned LLaMA on 1,645 machine-learning APIs from Hugging Face, TorchHub, and TensorFlow Hub, with ten generated instructions for each. During training the relevant documentation was retrieved and placed in the prompt, so the model learned to follow documentation that may change at test time. It scored calls by comparing their abstract syntax trees, the parsed structure of a call with its function name and arguments, against the reference. That comparison also catches calls to APIs that do not exist.
• Real APIs at scale, with search. ToolLLM (Qin et al. 2023) collected 16,464 real web APIs in 49 categories from the RapidAPI marketplace and had ChatGPT write single-tool and multi-tool instructions. It found solution paths by a depth-first search over trees of calls, which can back up from a failed call, instead of following a single greedy chain.
• Checking at the center. APIGen (Liu et al. 2024) generated candidate calls for 3,673 executable APIs in 21 categories. Each candidate must pass three checks in order, its format, an actual execution, and a semantic check that the result answers the query. A 6.7B model trained on the checked data ranked sixth on the Berkeley Function Calling Leaderboard at the time, above GPT-4o and Gemini-1.5-Pro. The authors released 60,000 of the checked examples.

Two details separate good data from bad. The first is negative examples. A model trained only on conversations that end in a call learns to call something every time. So the data must include requests that no available tool can serve, where the right output is a plain answer, and the evaluation must measure this (the relevance tests below). The second is the loss mask of §14.3, the per-token weight that decides which tokens are trained. Train the tokens the model writes, including the call and its end-of-turn token. Mask the tool definitions, the user turns, and above all the tool results, which come from the environment. A model trained on tool results learns to invent them.

Reinforcement learning sharpens what supervised data starts. The key design choice is the reward, and the lesson so far is to give graded credit for partly right calls rather than all or nothing. ToolRL (Qian et al. 2025) trained Qwen2.5 and Llama 3.2 models this way with GRPO, the policy-gradient method that scores each sample against the others drawn for the same prompt (§18.4). Its models gained about 17% over their base models and 15% over supervised fine-tuning. Multi-turn agent RL, where whole trajectories of calls and observations are rewarded and the observation tokens are masked from the gradient, is the subject of §24.3.

Evaluating function calling

A function-calling benchmark has to decide what "correct" means for a call, and there are two answers.

• An AST check parses the call and compares its abstract syntax tree with a set of acceptable answers, covering the function name, every required argument, and each value's type and content. It is cheap, deterministic, and needs no running services.
• An execution check runs the call and compares the result. It is expensive and can be flaky, but it catches calls that are well formed and wrong in ways a reference answer did not anticipate.

The Berkeley Function Calling Leaderboard (BFCL, from the Gorilla team) uses both. Its first release sorted single-turn calls into these categories.

• Simple, one function and one call.
• Multiple, choosing one of 2 to 4 functions.
• Parallel, several calls to one function from one request.
• Parallel multiple, several calls to several functions.
• Relevance detection, where none of the functions fit and the right output is no call at all.

Its AST rules are strict on types, and one of them contradicts JSON Schema. For Python functions it accepts an integer where a float is expected, but never a float where an integer is expected. So 3.0 for an integer parameter is wrong in BFCL and valid in JSON Schema. Later versions added live functions contributed by users (V2), multi-turn and multi-step interactions (V3), and agentic tests of web search, memory, and sensitivity to format (V4, last updated April 12, 2026). The overall score is the unweighted average of the subcategories.

Structured output engines have their own benchmark. JSONSchemaBench (Geng et al. 2025) collects nearly 10,000 real JSON Schemas and measures six engines (Guidance, Outlines, llama.cpp, XGrammar, OpenAI, and Gemini) on efficiency, coverage, and output quality. Coverage is the fraction of schemas for which an engine produces valid output, and it is the weak point.

Here are the options side by side. JSON mode is the weakest constraint, valid JSON of any shape.

In practice the layers stack. Fine-tune (or choose a model trained for calls), constrain decoding where the engine supports the schema, and validate anyway. The schema subset that decoding enforces is smaller than the set of rules your tool needs, so the final check belongs to your own code.

Read more

• Willard and Louf (2023). Efficient Guided Generation for Large Language Models. The vocabulary index over finite-state machines, and its extension to context-free grammars, as implemented in Outlines.
• Dong et al. (2024). XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models. Context-independent and context-dependent tokens, the persistent stack, and overlapping mask computation with the GPU.
• Park et al. (2024). Grammar-Aligned Decoding. Proves that local masking distorts the distribution, names expected future grammaticality, and gives the ASAp sampler.
• Beurer-Kellner, Fischer, and Vechev (2024). Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generation. The token misalignment problem, its measured cost, and the subword-aligned DOMINO decoder.
• Geng et al. (2025). JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models. Nearly ten thousand real schemas, and why coverage, not speed, is the weak point of today's engines.
• Tam et al. (2024). Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models, with the rebuttal Say What You Mean: A Response to 'Let Me Speak Freely'. Read both to see how much prompts and parsing decide.
• Patil et al. (2023). Gorilla: Large Language Model Connected with Massive APIs, and the Berkeley Function Calling Leaderboard. Training with retrieved documentation, AST matching, and the categories of function-calling evaluation.
• Liu et al. (2024). APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets. The three-stage checking recipe for tool-calling data.

Watch next

▶

▶

▶