Context Is Not Free: Generation Speed Against Window Size on Local Models

The default answer to an agent that forgets is a bigger context window. We tried it: 65,536 tokens on a 27-billion-parameter model. Sessions got slower and answers got worse, and the reason is a cost that window size does not reduce, it only raises the ceiling on. This paper reports what the serving logs measured at three context lengths and the window we settled on.

Two costs are paid on every step. Prefill reads the prompt before the first token; generation produces the answer one token at a time. The claim under test: both costs grow with context, and generation, which people assume is constant, grows enough to matter.

What came back

On the laptop (Apple silicon, 48 GB), the 8B model at 2,799 tokens of context prefilled at 654 tokens per second and generated at 63.3. The 27B model at 9,844 tokens prefilled at 187 per second and generated at 18.3. The same 27B at 49,614 tokens prefilled at 76 per second and generated at 5.3. Generation fell 3.4× between 10k and 50k of context on identical weights, and prefill fell 2.5×. A step that produces 300 tokens takes 16 seconds at 10k context and 57 at 50k, before any prefill.

Generation speed in tokens per second against context length, from llama-server eval time lines on the laptop: the 8B at 2.8k tokens, the 27B at 9.8k and 49.6k. The first bar is a different, smaller model, shown for the shape of the curve; the second and third are the same 27B weights. Nothing is authored for the figure.

Model	Context (tokens)	Prefill tok/s	Generation tok/s	Cold prefill of that context
8B, laptop	2,799	654	63.3	4 s
8B, Mac mini	8,184	187	16.3	44 s
27B, laptop	9,844	187	18.3	53 s
27B, laptop	49,614	76	5.3	651 s

The quiet result

The headline is the 11-minute cold prefill at 50k. The quieter number is that generation at 50k is slower than the mini's 8B at 8k. A bigger window on a bigger model, loaded with a long session, produced a worse experience than a small model with a short one, on every axis a user can feel. Attention cost grows with the number of tokens attended to, and on a laptop GPU that growth is not hidden by headroom.

Past the trained length

There is a second cost the logs do not show. The 8B model's trained context is 40,960 tokens. Asked for 65,536, the server silently clipped the prompt from the front, which removes the system prompt and the tool definitions first. The model then answered with no instructions, which reads as the model losing its mind. Sentinel now asks the server for the trained length once and caps the window to it. A window the model was not trained for is not more memory; it is a prompt with the beginning cut off.

How far this is from a general result

• Four measurements on two machines; the curve between them is interpolated, not measured. Three of the four are single runs from production logs rather than repeated benchmarks.
• Apple silicon with a unified-memory GPU. Data-centre GPUs have far more bandwidth and the generation slope is shallower there.
• Flash attention and an 8-bit KV cache were on for all runs; without them memory, not speed, is the first limit.
• Quality past the trained length was observed, not scored: we saw truncated-prompt answers, we did not measure a benchmark drop.

What it means

The window is a budget, not a feature. Sentinel runs local models at 32k tokens (24k per slot when two sessions share a 16 GB machine), compacts history with a stable cut when it overflows, and keeps the prefix deterministic so that prefill is paid once. For the operator the decision rule is: pick the window from the generation speed you can accept at the top of it, not from the largest number the model card allows.

The determination

Same 27B weights, two context lengths, generation 18.3 against 5.3 tokens per second and prefill 187 against 76. The numbers are from production logs on one laptop. The direction is plain. Context is paid for on every token generated, not just the ones read, and a window chosen for its size rather than its speed makes the agent feel slower than a smaller model would.