Every LLM invoice is denominated in tokens, and almost every pricing page lists two numbers — a price for input tokens and a higher price for output tokens — with no explanation of why they differ or how to make the total smaller. The gap between those two numbers, and everything that drives it, is the whole economics of running these models. Once you understand what the hardware is actually doing during a request, the pricing stops looking arbitrary and starts looking like a set of levers you can pull. Let me build it up from the metal.
Prefill and decode are two different jobs
A generation request has two phases, and they stress the GPU in completely different ways.
Prefill processes your entire prompt at once. Every input token is pushed through the model in a single parallel pass — the GPU's matrix units love this, because there's a big batch of work with no dependencies between tokens. Prefill is compute-bound and fast per token.
Decode generates the output one token at a time. Each new token depends on all the previous ones, so the model runs a fresh forward pass per token, and there's no parallelism to exploit within a single sequence. Worse, each of those passes reads the entire model's weights from GPU memory to produce a single token. Decode is memory-bandwidth-bound, and it's slow: the GPU spends most of its time shuttling weights, not computing.
That asymmetry is the answer to the question everyone asks.
Why output tokens cost more
Output tokens cost several times what input tokens cost — commonly 4x to 5x. On Claude Opus 4.8, for instance, input runs $5 per million tokens and output $25, exactly 5x; Sonnet-class and Haiku-class models keep a similar ratio at lower absolute prices. This isn't a margin decision. It's the hardware. An input token is processed during compute-friendly, parallel prefill; an output token requires a full, memory-bound, sequential decode pass that hogs the GPU for far longer. You're paying for the GPU-seconds each token actually consumes, and a decode token consumes many more than a prefill token. The practical consequence: if you want a cheaper application, the highest-leverage move is almost always to generate less — ask for structured, terse outputs — not to feed the model less.
The KV cache, and why long context gets expensive
To avoid recomputing the entire prompt for every new output token, the model caches the intermediate attention state — the "keys" and "values" — for every token seen so far. This KV cache is what makes decode tractable, but it lives in GPU memory and it grows with sequence length. Its size is linear in the number of tokens: double the context, double the cache.
The quadratic-ish reputation of long context comes from the interaction of two effects. The KV cache memory grows linearly with length, but attention compute grows with the square of length (every new token attends to every prior one), and the memory pressure squeezes out the batch parallelism that keeps GPUs efficient. A 100,000-token context doesn't cost proportionally more than a 10,000-token one — it costs more than proportionally, because the KV cache eats the memory you'd otherwise use to serve other requests concurrently, and throughput collapses. Long context is a real tax, and it's why million-token windows are a capability, not a default you should reach for casually.
Batching is where throughput (and margin) comes from
A single decode request leaves the GPU mostly idle, waiting on memory. The fix is batching: run many sequences through the same forward pass, so the one expensive weight-read serves dozens of tokens instead of one. Continuous batching (adding and retiring sequences mid-flight rather than waiting for a fixed batch to finish) keeps utilization high under real, ragged traffic. This is the single biggest reason a hosted API is cheaper per token than your own idle GPU: the provider is batching your request with thousands of others and amortizing the memory-bandwidth cost across all of them. When you self-host at low utilization, you pay for that bandwidth per request with nobody to share it with.
Quantization: fewer bits, more throughput
Model weights default to 16-bit precision. Quantizing them to 8-bit (fp8/int8) or 4-bit halves or quarters the memory the weights occupy and — because decode is bandwidth-bound — roughly proportionally speeds up generation, since there's less to read per token. It also lets a larger model fit on a smaller, cheaper GPU. The cost is a small, measurable quality drop that varies by model and task; 8-bit is often nearly free in quality, 4-bit more of a real trade. For self-hosters, quantization is the main dial that turns an unaffordable GPU footprint into an affordable one.
Prompt caching and its real discount
If you send the same large prefix on many requests — a long system prompt, a fixed knowledge base, few-shot examples — prompt caching lets the provider store the computed KV state for that prefix and skip re-running prefill on it. The discount is steep: cache reads typically bill at roughly a tenth of the normal input price. The catch is the cache write, which costs a premium (around 1.25x the input price for a short-lived cache, up to 2x for a longer-lived one), because populating the cache is extra work.
That premium sets the break-even. With a standard short TTL, you come out ahead after just two requests hit the cached prefix (1.25x to write plus 0.1x to read beats 2x to process it twice uncached). A longer-TTL cache costs 2x to write and needs three or more reads to pay off, but it survives gaps in bursty traffic. Caching is a genuine cost lever precisely because it attacks the compute-heavy prefill phase — but only when the prefix is truly stable. A timestamp or a per-request ID anywhere in the cached prefix silently invalidates it, and you pay the write premium with no reads to earn it back. Verify with the cache-hit counters in the API response; if the read count is zero across identical-prefix requests, something in your prefix is changing.
Self-hosting break-even: SageMaker and GPUs vs. the API
The recurring question is whether to run the model yourself. The math is a utilization story. A per-token API price bundles the provider's batching efficiency; a self-hosted GPU bills by the hour whether or not it's busy.
Take a SageMaker real-time endpoint on a GPU instance. A single ml.g5.xlarge (one A10G) runs on the order of ~$1.40/hour, and the big multi-GPU boxes — ml.p4d/ml.p5 class, eight A100s or H100s — run into the tens of dollars per hour (all region- and commitment-dependent). That instance costs the same at 3am with no traffic as it does at peak. To beat a hosted API, you need enough sustained, batched throughput that your hourly cost divided by your tokens-per-hour lands below the API's per-token price. For a steady, high-volume workload on an open-weights model you're comfortable operating, self-hosting can win handily. For spiky or low-volume traffic, the idle hours destroy the economics, and a per-token API is almost always cheaper — you're renting someone else's batch.
SageMaker Serverless Inference for spiky workloads
The awkward middle case is a workload that's real but bursty — busy for an hour, dead for six. A provisioned GPU endpoint bleeds money during the dead hours; a per-request API is fine but gives up control over the model and data path. SageMaker Serverless Inference targets exactly this shape: you're billed for compute only while a request is actually being processed, and the endpoint scales to zero in between, so idle time is free. The trade is cold starts (the first request after a quiet period waits for capacity to spin up) and ceilings on memory and concurrency that make it unsuitable for the largest models or the tightest latency SLAs. For intermittent, self-hosted inference where you can tolerate an occasional cold start, it collapses the idle-cost problem that makes provisioned endpoints uneconomical.
The through-line: LLM cost is GPU-seconds in disguise. Output tokens, long context, and low utilization are all ways of spending more GPU-seconds per useful result, and caching, quantization, batching, and scale-to-zero are ways of spending fewer. Price the tokens, but understand the seconds.