The Context Window Is Not the Problem. Re-reading It Is.

Bigger windows made long sessions slower, not better. A simple cost model explains why, and why the right compaction policy is a hysteresis rule rather than a size limit.

The obvious response to an agent that forgets is a bigger context window. We tried it. A 65k window made our long sessions slower and the answers worse. The reason is a cost that window size does not change: the price of reading the window.

Two costs, not one

Each step of an agent pays twice. Generation is the part people watch: tokens appearing one by one, 15 to 60 per second depending on hardware. Prefill is the part nobody sees: the model reads the whole prompt before the first token appears. On an Apple M4, an 8-billion-parameter model prefills at about 190 tokens per second; a 27-billion one on a laptop GPU at about 76. The window does not change those rates. It only raises the ceiling on how much there is to read.

With a cache, prefill is paid once and later steps read only what is new. Without one, every step re-reads everything, and the cost grows with the session: a 50k-token context costs 11 minutes per step on the slower model. That is the whole difference between "long sessions are fine" and "long sessions are unusable", and it has nothing to do with the window.

Generation slows down too

Attention cost grows with context. The same 27B model that generates 63 tokens per second with 3k tokens of context generates 5 per second at 50k. So the bigger window hurts both halves: longer prefill when the cache misses, slower generation always. Past the model's trained length, quality drops as well. The window we settled on is 32k for local models, with the model's trained maximum as a hard cap.

What compaction has to preserve

If the window is fixed, something has to give when the history outgrows it. The usual answer is "drop the oldest messages". The non-obvious constraint is that the dropping must be stable, because the cache only helps if the prefix does not move. Three rules cover it:

Cut at user boundaries. A tool call and its result must stay together; dropping one without the other confuses the model more than losing both.

Pin the cut. Once the agent decides to keep the history from message 118 onward, it keeps using 118 on the next steps as long as that still fits, even if 112 would also fit. Stability beats maximal context.

Cut with hysteresis. When the pinned cut no longer fits, cut down to about 70 percent of the budget, not to just under the line. Otherwise the next message pushes it over again and the cut moves every turn, which is the sliding window in disguise.

Tool results are the real bulk

In our sessions, tool results are over 80 percent of the tokens. A shell command that dumps 20 kilobytes is read in full by the model once, which is right, and then would be re-read on every later step, which is waste. The rule we use: results from earlier turns are cut to 400 characters, results from the current turn stay whole, and the cut is deterministic so the prefix stays byte-identical. If the current turn alone overflows, the oldest results in it are cut one by one and the cut is remembered for the rest of the turn.

A number to watch

Serving logs report the cached prefix length on every request. If it is not within a few hundred tokens of the previous request's total, the policy is broken somewhere. We put that number in the product's status line, and the smoothness test fails when it regresses. The context window stopped being a topic after that.

Resources