Case study · 2 August 2026
Ninety-nine per cent of the input was the same thing, sent again.
We instrumented one of our own products and found a 39-tool schema being re-prefilled on every single message. Fixing it cut input cost by 67.7%. The interesting part is not the number — it is that nothing in the application was broken, no bill had spiked, and no amount of reading the code would have found it.
One
A ratio that should not have been possible.
Kodokyo is an AI-native project tool. Four consecutive turns of one chat conversation, measured:
| Turn | Input tokens | Output tokens | Ratio |
|---|---|---|---|
| 1 | 8,932 | 152 | 59:1 |
| 2 | 9,092 | 81 | 112:1 |
| 3 | 9,719 | 105 | 93:1 |
| 4 | 10,459 | 174 | 60:1 |
| Total | 38,202 | 512 | 75:1 |
Thirty-eight thousand tokens of input bought five hundred tokens of answer. Input rose turn on turn while output stayed flat.
Nobody had noticed, and there was no reason anyone would have. The feature worked, the answers were good, and the monthly invoice was unremarkable. Per-request this looks like an ordinary chat call.
Two
Our first diagnosis was wrong.
Input climbing while output holds steady is the signature of conversation history being re-sent whole each turn — the growth curve fits, and it is a known agentic waste pattern. We said so.
Counting the tokens instead of recognising the shape gave a different answer:
| Component of turn 1 | Tokens |
|---|---|
| 39 tool definitions | 7,567 |
| Static system prompt | 1,351 |
| Cacheable static prefix | 8,918 |
| Everything else — the user’s actual message | 14 |
| Measured turn 1 input | 8,932 |
99.8% of the first turn was a static prefix. The growth we had latched onto — 8,932 climbing to 10,459 — was conversation history, and it was the small part: about 1,500 tokens across four turns. The other ~8,900 per turn was the same tool schema, sent four times.
That reframes the problem entirely. Trimming conversation history would have addressed 4% of it. And it makes the fix far better than the one we had proposed, because a fully static prefix is the ideal case for prompt caching, where a growing one is not.
Three
Two things would have silently prevented the fix.
Prompt caching matches a prefix exactly, from the first byte. Two pieces of volatile content sat inside what would have been the cached region:
- The user’s name and current page rendered at the start of the system prompt. Cached as-is, the entry is keyed per user and per navigation — it would have re-written on every page change and shared across nobody.
- Recent activity gathered from the database appended at the end. Highly volatile, and inside the cached region it poisons every hit.
Both are the textbook anti-pattern and both are invisible from inside the application: caching that silently does nothing returns no error, no warning, and cache_creation_input_tokens: 0.
The fix is the standard shared-prefix / varying-suffix split — static preamble and tool schema carrying the cache breakpoint, then user, page and activity data as an uncached block after it. Verified against the live API before deploying:
| Request | Cache write | Cache read |
|---|---|---|
| Cold | 8,726 | 0 |
| Same user, same page | 0 | 8,726 |
| Different user, different page | 0 | 8,726 |
That third row is the invalidator fix demonstrated rather than asserted. Before the split it was a guaranteed fresh write every time.
Four
The result, and the lower of two numbers.
After the change, a turn sent 511 fresh tokens and read 8,450 from cache. Across a comparable four-turn conversation on Claude Sonnet 4.6:
| Input cost, four turns | USD | Reduction |
|---|---|---|
| Before — nothing cached | $0.11461 | — |
| After — as our engine priced it when this was measured | $0.03044 | 73.4% |
| After — as actually billed | $0.03699 | 67.7% |
We publish 67.7%, and we published it before our own engine agreed. At the time of measurement the engine reported 73.4%, because it did not model the cache-write premium: Anthropic bills a write at 1.25× input and the collector counted it as ordinary input. That understated the “after” side and inflated the apparent saving, so the honest figure was the lower one.
Fixed on 3 August 2026. Cache writes now travel in their own field and are priced at the premium, which closes the gap exactly: 8,791 write tokens at the missing 0.25× of $3/M is $0.0066, and $0.03044 + $0.0066 is the billed $0.037. The two rows converge on 67.7% by construction.
The figure above is still the billed one, not a re-measurement. This capture predates the field, and a write already summed into ordinary input cannot be separated again — so re-running the engine over this trace still returns 73.4%. The correction applies to what is captured from here. We will restate the table when the same conversation has been measured on a collector that carries the field, and not before: arithmetic showing what a number would have been is a derivation, and this page is about measurements.
Quoting the number that flatters us, in a document arguing for honest measurement, would refute the argument it is making.
Five
We found a bug in our own engine on the way.
The first figure we produced was 79%. It was wrong, and the cause was ours: the engine subtracted cached tokens from input tokens, treating a cache read as a subset of input. Our own collector has always sent them as separate quantities, and said so in a comment.
The consequence was one-directional. Any turn reading more from cache than it sent fresh clamped to zero uncached input and lost the fresh tokens entirely — and that is the normal case once caching works, not an edge case. Kodokyo’s 511 fresh tokens were priced at zero.
There were already tests for cached pricing. They passed, and they encoded the same wrong assumption — which was not carelessness. While no sender in the world was writing a cached-token count, “reads are separate from input” and “reads are part of input” produce identical answers on every input a test could construct. The assumption became falsifiable at the exact moment it started to matter.
That is the uncomfortable class of latent error: it activates when the optimisation lands, so the first data seen through it is also the first data anyone wants to quote. Fixed in @kumokodo/tetrameter-core 0.6.0, recorded in the restatement log, and the headline revised downward as a result.
Six
The same finding did not transfer.
A second product, AI Colosseum, showed a similar shape: a review pass sending 1,766 input tokens to produce 46 of output, 38:1. The obvious move was to apply the same fix.
Checking rather than assuming killed it, for three separate reasons:
- The prompt is too small. Claude Haiku 4.5 has a 4,096-token minimum cacheable prefix — the highest tier. At 1,766 tokens, caching does nothing and reports nothing.
- The prefix is unstable by design. The reviewing model is chosen at random per turn, and the prompt names it in the first block. Eight distinct models appeared across ten turns, so consecutive turns almost never share a cache line.
- The cap was the wrong lever anyway. Measured across that product, 88% of the review call’s cost is input; a maximum-output limit governs the other 12%.
Trimming what the reviewer sees was the real lever. Reviewer input fell from 65,566 tokens to 24,209 across a comparable run — 63% — and turn cost from $0.179 to $0.111.
A finding that is measured, then ruled out elsewhere on the same evidence, is worth more than one that generalises by assertion. Nobody’s spend was optimised on a guess.
What this does not show
The limitations, stated rather than discovered.
- Tier 1 evidence throughout. Model energy is a class average and grid intensity an annual average, because no commercial provider discloses per-request energy. Cost figures are exact — they come from published list prices against measured token counts — but the carbon bands are wide, and honestly so.
- Region is the caller’s, not the provider’s. An application reports where it runs. Inference happens on the provider’s hardware, wherever that is. It is the best available proxy and it is why Tier 2 is the ceiling for third-party API usage.
- We measured the cost, not the quality. Trimming a review pass buys cost with judgement. We can show the input curve flatten; we cannot show whether the reviewer still catches what it used to, because that requires reading the completions our architecture is built never to hold. That half belongs to the customer, and it is still owed here.
- One product, one week. These are our own applications, instrumented by us. Independent methodology review is scheduled and has not happened.
What it cost to find.
One afternoon of instrumentation, and a chat conversation. The waste had been running since the feature shipped, in an application whose authors wrote every line of it.
The engine that produced every number here is open source, Apache-2.0 (opens in a new tab), including the restatement log recording the bug above. Argue with the arithmetic — that is what it is published for.