How we handle 1M-token requests without latency spikes — the sharding strategy, parallel decode, and context compression pipeline.
A 1M-token request is not a big small request. Naive handling produces latency spikes that break every downstream timeout you have. Three techniques keep p99 flat as context grows.
Sharding the prefill
Prefill is the dominant cost at long context and it parallelizes well. We split the prompt at semantic boundaries, prefill shards concurrently across devices, and stitch the KV cache before decode begins. Above 200k tokens this cuts time-to-first-token by roughly 60%.
Parallel decode
Decode is sequential by nature, but speculative decoding with a small draft model recovers much of the loss. The draft proposes several tokens, the target verifies in one pass, and acceptance rates on structured output run above 70%.
Context compression
- Deduplicate repeated boilerplate across documents before the request leaves the gateway
- Drop retrieval chunks below a relevance floor rather than padding to a fixed k
- Summarize turns older than the last twenty in long conversations
Most 1M-token requests do not need to be 1M tokens. Median compressible waste in customer traffic is 23%, and removing it is both faster and cheaper than serving it well.
Token budgets
Set a hard per-request ceiling and a per-tenant hourly budget. Without both, one runaway agent loop can consume a day of capacity in twenty minutes — the most common incident class we see in long-context workloads.