---
title: "Tool calling on open-weight models"
canonical: "https://router.xark.io/blog/tool-calling-and-structured-output"
description: "The API validates the shape of your tools block and forwards the rest untouched. What that guarantees, what it cannot, and the failures a good SDK will not catch."
section: "blog"
updated: "2026-09-08"
source: "https://router.xark.io/blog/tool-calling-and-structured-output.md"
---

# Tool calling on open-weight models

*Published 2026-09-08. Topics: api-contract, openai-compatible, agents, sdk.*

The API validates the shape of your tools block and forwards the rest untouched. What that guarantees, what it cannot, and the failures a good SDK will not catch.

The contract this API enforces is the shape of the request, not the behaviour of the model: tools must be an array, tool_choice must be none, auto, required or a tool object, and response_format must be an object whose type is text, json_object or json_schema. Beyond that the body is forwarded to the model byte for byte -- there is no constrained decoding here, so a model that ignores your schema is not something the gateway can correct. Five of the twelve catalogued text models are callable today, and two of the three the catalogue recommends for schema-constrained generation are not among them.

## What is checked, and where it is checked

Four parameters carry the whole of tool calling and structured output, and each is validated for shape before anything leaves this API. A wrong shape is a 400 that names the parameter, the allowed values and what you actually sent, so it fails in a test rather than in production.

What is not checked is meaning. Whether a tool is ever called, whether the arguments parse, whether the JSON matches your schema -- none of that is visible from here, and the next section explains why that is a consequence of a decision made for a different reason entirely.

| Parameter | Accepted | Rejected with |
| --- | --- | --- |
| tools | An array. The contents are not inspected | 400, param tools |
| tool_choice | none, auto, required, or a tool object | 400, param tool_choice, listing the three strings |
| response_format | An object whose type is text, json_object or json_schema | 400, param response_format.type |
| reasoning_effort | minimal, low, medium or high | 400, param reasoning_effort |
| reasoning (responses endpoint) | An object, e.g. { effort: medium } | 400, param reasoning, naming the shape |

## Why the envelope is validated and the behaviour is not

The relay in front of the inference gateway hands your request body upstream as a stream and never reads it. That is a streaming decision rather than a policy one -- parsing and re-serialising would mean buffering, and a buffered stream and a real one are indistinguishable in a status check and instantly obvious to a human watching a cursor. Exactly one endpoint reads a body, and only to check a video duration against the price on file.

So a relay that cannot read your request also cannot repair it. There is no constrained decoding here, no grammar enforcement, no retry-until-parseable. Your response_format is forwarded and the model honours it or it does not, and the honest way to describe that is as a limitation rather than as neutrality.

Two things follow for anyone building on this. Test schema adherence per model rather than assuming it from the parameter being accepted. And treat the availability question as the first one: five of the twelve catalogued text models are callable, and the catalogue's own recommendations for JSON-mode and schema-constrained generation -- Qwen3 Max Instruct and MiniMax M2 -- are both among the seven that are not.

## Two response shapes, and only one of them is flat

On chat completions the answer is choices[0].message, and a tool call arrives as tool_calls on that message. That shape is stable and every OpenAI-compatible client already handles it.

On the responses endpoint the answer is an array. A client is expected to walk output[], switch on output[].type -- message, and later reasoning or function_call -- and then switch again on output[].content[].type. Reading a fixed path like output[0].content[0].text works today and breaks the first time a reasoning item is emitted ahead of the message, which is exactly the ordering a tool-using model produces.

The envelope also echoes your own settings back: tools as sent, tool_choice defaulting to auto, and parallel_tool_calls. That is deliberate -- a caller who stores responses can reconstruct precisely how each one was produced, which is the difference between an audit and an argument.

```javascript
// Chat completions: flat, and the tool call is on the message.
const call = res.choices[0].message.tool_calls?.[0];
if (call) run(call.function.name, JSON.parse(call.function.arguments));

// Responses: an array. Switch on the item type, never index into it.
for (const item of res.output) {
  if (item.type === "function_call") run(item.name, JSON.parse(item.arguments));
  if (item.type === "message") {
    for (const part of item.content) {
      if (part.type === "output_text") text += part.text;
    }
  }
  // A reasoning item can precede the message. output[0] is not the answer.
}
```

## Three failures that survive a good SDK

The first is truncation. A response cut short by the output-token ceiling returns finish_reason length on chat, and status incomplete with incomplete_details.reason max_output_tokens on responses. A JSON object truncated mid-string is not a parse error you can distinguish from a model that formats badly -- it is a valid prefix of valid JSON. Check the finish reason before you parse, not after the parse fails.

The second is a stream that ends without its terminator. Chat and completions terminate on data: [DONE]; responses terminates on response.completed, or response.incomplete when generation hit the ceiling, and has no [DONE] at all. A socket closing without one of those is a failed request, not a short answer, and a tool call assembled from a truncated argument delta is the worst version of that mistake because it often still parses.

The third is smaller and catches everyone once: arguments arrive as a JSON string, not as an object. It has to be parsed separately, and it can be invalid JSON from a model that got it wrong. Parse it in a try, and treat a failure as a retry rather than as an exception.

On the responses endpoint there is one extra tool for the second problem: every event carries a sequence_number that increments by one. A gap is a dropped frame, and nothing else will tell you.

## What being wrong costs

A tool call is a lopsided request. The tool definitions, the schema and the conversation are the input; the output is a name and a short arguments object. On a reference shape of 1,500 input and 300 output tokens, that is $0.001995 on GLM-5.2 and $0.000189 on DeepSeek V4 Flash.

The number that matters is the retry rate, because a call that comes back unparseable is a second billed request for the same work. One retry in ten adds ten per cent to the whole line, and a model that needs one retry in three is thirty-three per cent more expensive than its rate card says before you have improved anything.

This is the argument for measuring schema adherence per model rather than reasoning about it. The cheapest model with a 30% retry rate is not obviously cheaper than the model above it at 2%, and the only way to know which you have is to run the two against your own schema and count.

| Model | One tool call | 100,000 calls | With one retry in ten |
| --- | --- | --- | --- |
| Kimi K3 | $0.005475 | $547.50 | $602.25 |
| GLM-5.2 | $0.001995 | $199.50 | $219.45 |
| Kimi K2.6 | $0.001515 | $151.50 | $166.65 |
| DeepSeek V4 Pro | $0.000585 | $58.50 | $64.35 |
| DeepSeek V4 Flash | $0.000189 | $18.90 | $20.79 |

## The retry is the line nobody budgets

Every response carries usage.cost_usd -- the actual charge for that request, not an estimate against a rate card -- so the retry line is measurable rather than modelled. Sum it across a run including the retries and you have the true cost of your adherence rate in one number.

If your agent streams, ask for the usage explicitly. Without stream_options.include_usage a streaming caller never learns what any call cost, and an agent harness that streams for progress output and reports zero spend is the version of this problem that stays hidden longest.

Log it per call rather than per run. A schema problem is almost never uniform across your tools; it is one tool with an awkward argument shape, and that is visible in a distribution and invisible in a total.

## Caching, and the one ordering rule that matters here

Tool definitions are the ideal cached prefix. They are large, they are identical on every turn, and cached input here is billed at a fifth of the uncached rate. An agent resending a 40,000-token prefix over 50 turns pays $0.35 on GLM-5.2 instead of $1.64 on the input side.

The rule is that a prefix cache matches from the first token forward and stops at the first divergence, so the tool definitions have to come first and they have to be byte-stable. Re-serialising a JSON tool definition with different key ordering produces different tokens and therefore a cache miss, even though the semantic content is identical. Serialise once and reuse the string.

The commonest way this is engineered away by accident is a timestamp, a session id or a request id injected near the front of the system prompt. One volatile token at the top voids the entire prefix on every single request.

## Where somebody else is the better answer

If you need a guarantee that output matches a schema rather than a strong tendency, you want a provider doing constrained decoding, and we are not doing it. Forwarding response_format is not the same promise and we are not going to describe it as one.

If you want tools that already exist rather than tools you wire up, Groq ships Compound and Compound Mini with web search and code execution built in. That is a different product from an API that forwards your own tool definitions, and for some agents it is straightforwardly the right one.

If your architecture depends on failing over when a model degrades mid-run, OpenRouter routes across many providers per model and we are single-provider per model, with one upstream and no automatic second path. And if the model you want is one of the seven catalogued here without an upstream, a provider with a broader live catalogue is the answer today rather than a roadmap.

```bash
# What can actually run a tool call right now. No key required.
curl -s "https://router.xark.io/api/v1/models?type=text" | jq -r '.data[] | .id'

# One model, with its context window and licence alongside the rates.
curl -s https://router.xark.io/api/v1/models/z-ai/glm-5.2 \
  | jq '{id, license, context_window, pricing}'
```

## A checklist before you ship an agent on this

Measure schema adherence per model against your own schema, and record the retry rate next to the rate card. The second number changes which model is cheapest.

Check finish_reason and status before parsing. A truncated JSON object is a valid prefix and will not announce itself.

Terminate on the terminator. [DONE] for chat and completions, response.completed or response.incomplete for responses, and a sequence_number gap on responses means a frame was dropped.

Put tool definitions at the very front of the prompt and keep the serialisation byte-stable, or the cached rate never applies to the largest fixed thing you send.

Parse the arguments string in a try. It is a string, it comes from a model, and it can be wrong.

Check availability before you write the matrix. Five text models are callable today, and the models endpoint answers that without a key rather than making you trust a page.