What it costs to run an evaluation suite

The accounting, before any numbers

Eval spend is the line teams forget because it does not look like production. It has no users, it runs on a laptop or in CI, and it is mentally filed under development rather than inference. It is inference, and it is often the largest single consumer of tokens in a project's first three months.

The formula has five factors and most cost estimates use two of them. Items in the suite, times input plus output tokens per item, times samples per item, times candidate configurations, plus the judge's own tokens if you are using one. Dropping the last three is how a $75 afternoon gets budgeted as $1.68.

Everything below is computed against a single reference item: 800 input tokens -- a system prompt, a rubric and the case -- and 400 output tokens of answer. Substitute your own shape; the structure of the arithmetic does not change.

One pass, five models

The five text models callable today, priced against one 1,000-item pass at 800 in and 400 out. The cost-per-item column is the useful one to keep, because it is what you multiply by everything in the next section.

Note that the spread across models is a factor of 35, from $0.144 to $5.08 for identical work. Choosing the eval model is a bigger cost lever than any prompt optimisation you will do afterwards.

One pass is not the bill

Nobody runs an eval once. The reason the suite exists is to compare things, and every dimension you compare on multiplies the whole run.

A realistic prompt-engineering afternoon is five prompt variants, three configuration settings each, and three samples per cell so you can see variance rather than noise. That is 45 passes. On GLM-5.2 it is $75.42; on Kimi K3 it is $228.60; on DeepSeek V4 Flash it is $6.48. The point of the table is not which is cheapest -- it is that the ratio between the model you evaluate on and the model you ship on is a decision you can make separately.

Then there is CI, which is the spend that never stops. Twenty merges a week, one pass each, is $33.52 a week on GLM-5.2 and $1,743 a year. That is a small number for a team and a surprising one for a solo developer, and either way it belongs in a forecast rather than in a monthly statement.

The judge is a second model with a second bill

LLM-as-judge doubles the accounting, and almost every published eval-cost estimate omits it. The judge reads the original prompt, the candidate's answer and a rubric, then writes a verdict -- so its input is larger than the candidate's was, even though its output is much smaller.

Take a judge reading 1,400 input tokens and writing 150. Judged on GLM-5.2, that is $1.5305 per thousand items against $1.676 to generate the answers: judging costs 91% of what generating cost. Judged on DeepSeek V4 Flash it is $0.153, which is a tenth of the candidate cost.

This is the clearest place in an eval pipeline where mixing models is obviously correct. The candidate model has to be the one you are actually evaluating. The judge does not -- and a judge on a cheap model that you have separately validated against human labels is a far better use of the budget than a symmetric setup nobody chose deliberately.

Caching helps an eval, right up until the sweep you wanted

An eval suite looks like the ideal cached-input workload. A fixed rubric and system prompt sit at the front of every item, so the prefix repeats a thousand times, and cached input here is billed at a fifth of the uncached rate.

The arithmetic is good. A 1,200-token fixed prefix across 1,000 items on GLM-5.2 is $0.984 uncached. With the first item paying full rate and the remaining 999 hitting the cache, it is $0.193 -- an 80% reduction on that portion of the bill.

And then the sweep destroys it. The whole point of a prompt sweep is to vary the system prompt, and a prefix cache matches from the first token forward and stops at the first divergence. Change the rubric and every item in that variant is a cache miss. So the discount applies fully to a regression run over a fixed prompt, and not at all to the exploratory run that costs the most.

The lever that does survive a sweep is ordering. Put whatever is genuinely fixed across all variants -- the tool definitions, the output schema, the shared instructions -- at the very front, and the varying text after it. Then the sweep only invalidates the tail.

Wall clock is the other budget

Chat endpoints are limited to 60 requests per minute per key, so a 1,000-item pass takes about 17 minutes at the limit no matter how many workers you run. Forty-five passes run serially is 12.5 hours -- an overnight job, not an afternoon one, unless you parallelise across keys or accept a longer feedback loop.

This is the argument for keeping a small suite alongside the full one. A 100-item smoke pass is under two minutes and $0.17 on GLM-5.2, which is a loop you can actually iterate against; the 1,000-item suite is what you run before merging.

If your eval streams responses, ask for usage explicitly. Without stream_options.include_usage a streaming caller never learns what any call cost, and an eval harness that streams for progress output and then reports zero spend is a harness that will surprise you at the end of the month rather than at the end of the run.

Measure it rather than estimating it

Every response carries usage.cost_usd -- the actual charge for that request, computed on the same rates the pricing page prints, not an estimate reconstructed from a token count. Summing that field across a run gives you the true cost of the run, and it takes one line.

Do this on the first hundred items rather than trusting the arithmetic above, because tokenisation rarely matches a character-count estimate and the reference shape at the top of this page is a reference, not your workload. If the measured per-item cost is double the estimate, that is worth knowing before pass number 45.

Log it per item, not per run. A cost regression in an eval is almost always one class of item generating far more output than the rest, and that is visible in a distribution and invisible in a total.

total = 0.0
per_item = []

for case in suite:
    r = client.chat.completions.create(
        model="z-ai/glm-5.2",
        messages=build(case),
        max_completion_tokens=512,     # the cap that bounds the worst item
    )
    per_item.append(r.usage.cost_usd)  # the real charge, not an estimate
    total += r.usage.cost_usd

print("run cost", round(total, 4), "worst item", round(max(per_item), 6))

The ceiling that stops a broken harness

An eval harness with a retry-on-error loop is exactly the runaway shape that prepaid credit exists to bound. A malformed case that always fails, a judge that always returns unparseable output, a retry that never gives up -- all of them consume tokens indefinitely, and on a postpaid arrangement the first anyone knows is the invoice.

Here the balance is a hard ceiling. At zero the API returns 429 with code insufficient_credits and the refused request is not charged. That bounds the incident at whatever you loaded, which is a number you chose.

The exception is auto-recharge, and it is worth knowing before you leave a suite running overnight. With it enabled the ceiling is the balance plus up to three $50 charges per rolling day. That is a cap rather than an absence, but it is $150 a day higher than the number you thought you had set.

What to actually budget

Count the multipliers before the rates. Items times samples times configurations is the factor that decides the order of magnitude; the per-token rate only decides the digit in front.

Price the judge separately, and pick a cheaper one deliberately. On the reference shape, judging on the candidate's own model costs 91% as much as the candidate run.

Cap max_completion_tokens. It is the only mechanism that bounds the worst case of a single item, and an eval suite exists precisely to find inputs that behave badly.

Keep a 100-item smoke suite for the loop and a 1,000-item suite for the gate. Seventeen minutes is not an iteration cycle.

Only five text models are callable today, so a cross-model comparison here spans five rather than the full catalogue. Check the models endpoint before you write the matrix -- it is unauthenticated, and it is the honest answer rather than a marketing page.