Our data from 140M daily calls shows that semantic caching eliminates redundant requests. Here's how the hit rate scales with traffic.
Across 140M daily calls on the Starsg gateway, 31% of prompts are near-duplicates of a request we served in the previous 24 hours. Exact-match caching catches barely a third of those. Semantic caching catches the rest — and it is the single biggest line-item reduction most teams can make without changing a model.
What counts as a cache hit
We embed each incoming prompt, search the tenant's own vector index, and treat a result as a hit when cosine similarity clears a per-route threshold. The default is 0.97 for factual lookups and 0.99 for anything the caller marks as generative. Below the threshold the request goes upstream and the response is written back to the index.
- Support and FAQ traffic: 52% hit rate — the same twenty questions dominate
- Product catalog enrichment: 38% hit rate, driven by repeated SKU descriptions
- Code assistance: 19% hit rate; prompts carry unique file context
- Open-ended chat: 6% hit rate — caching is not the lever here
How the hit rate scales with traffic
Hit rate is a function of prompt density, not raw volume. A workload at 10k calls/day and one at 10M calls/day converge to the same rate once the index has seen roughly 50k distinct prompts. What volume changes is the time to reach that point: large tenants warm up in hours, small ones in weeks.
Averaged across all cache-enabled tenants, semantic caching removed 41% of billable upstream tokens last quarter. Median added latency on a hit: 11ms.
Turning it on
client.chat.completions.create(
model="claude-sonnet-4-5",
messages=messages,
extra_body={
"amplink": {
"cache": {"mode": "semantic", "threshold": 0.97, "ttl": 86400}
}
},
)
Set the threshold conservatively, watch the served-from-cache header for a week, then loosen it route by route. Teams that tune per route rather than globally land 8-12 points higher on hit rate at the same quality bar.