Engineering blog
What running an inference gateway actually costs, what breaks, and the numbers behind both. Every post carries at least one figure that comes from operating this service rather than from a vendor’s marketing page.
- pricingtokenscost-control
What a context window actually costs
K means 1,024 here, so 200K is 204,800 tokens. What filling each window costs once, what it costs every turn, and where retrieval wins.
- api-contractopenai-compatibleagents
Tool calling on open-weight models
The API validates the shape of your tools block and forwards the rest untouched. What that guarantees, what it cannot, and the failures a good SDK will not catch.
- cost-controlbillingcredits
What a spending ceiling protects you from
The balance is the only hard limit here. Four things that look like spend controls and are not, and the exact arithmetic of how fast a ceiling can go.
- billingtokenspricing
Reconciling an API bill, token by token
From usage on the response, through the ratios the gateway actually multiplies, to the integer on the ledger row -- and the three places rounding is allowed to land.
- billingcreditsapi-contract
Why a payment cannot credit you twice
Two Stripe events describe one payment. Which one credits, what a redelivery does, and the append-only ledger that makes a second credit unrepresentable.
- api-contractopenai-compatiblestreaming
Inside the request path
Sixteen header operations, a body that is never read, two cancellation paths and exactly two upstream errors we rewrite. The relay, hop by hop.
- open-weightsprocurementlicensing
Open-weight or frontier: how to decide
Four questions that settle it, seven cases where a closed frontier model is the right answer and we are not, and the one property no price cut can replicate.
- evaluationcost-controlpricing
What it costs to run an evaluation suite
One pass over a 1,000-item benchmark is $1.68 on GLM-5.2. The suite you actually run is forty-five passes, and the judge nearly doubles it.
- embeddingspricingcost-control
Embeddings: dimensions and corpus cost
Matryoshka truncation is a free quarter of your storage bill; changing model is not. Worked corpus costs, and the storage table nobody prints.
- rate-limitsapi-contractagents
Rate limits, concurrency and backoff
Five independent per-endpoint buckets, a fixed window that allows 120 requests in two seconds, and the field that says whether retrying will help.
- securityapi-contractcredits
What a leaked API key can actually do
Keys are shown once because the plaintext is fetched once. What rotation has to do, and why a leaked key's blast radius is your balance plus auto-recharge.
- streamingapi-contractsdk
Streaming: who pays when nobody listens
The SSE frames in order, the one frame that carries usage, and what actually happens to the bill when a client hangs up mid-completion.
- billingpricingcredits
Prepaid credits vs postpaid invoicing
A prepaid balance is a hard ceiling a runaway agent cannot exceed. What that protects you from, what it costs at procurement, and the fees across the market.
- pricingvideoopen-weights
What open-source video generation costs
Per-second rates for six open-weight video models, the three incompatible billing units this market uses, and why our calls bill a fixed five seconds.
- pricingcachingagents
Cached input pricing and what an agent costs
Cached input is a fifth of the uncached rate here. On a 50-turn agent loop with a 40,000-token system prompt that is $1.64 against $0.35.
- licensingopen-weightscompliance
Open-weight licences and what you can ship
Can you use Llama commercially? Yes, with conditions Apache 2.0 does not impose. Four licence families across 24 models, and the ones we refuse to resell.
- migrationopenai-compatibleapi-contract
What OpenAI compatibility does not change
Changing base_url is the whole migration. Three things still behave differently, and each one fails on the unhappy path in production rather than during the swap.
- pricingtokensbilling
How LLM API pricing is calculated
Input, cached input and output are three separate rates, not one. A worked example on a real request, and the integer money grid underneath the bill.