Prefix Stability: Why Long Agent Sessions Re-read Themselves

Every agent product is sold on the model. When a session gets slow, the reflex is a bigger model, a bigger window, a faster machine. This paper tests a narrower claim that we kept hearing in our own logs: that the time an agent spends silent before its first token is mostly the serving layer re-reading a prompt it has already read. Read the absolute numbers first: these are one machine, one model family and our own loop. What holds up is the comparison, and the comparison is not close.

Sentinel's loop sends the model a prompt on every step: system prompt, tool definitions, the conversation so far, the new tool results. Inference servers cache the key-value state of that prompt and reuse it when the next prompt starts with the same tokens. The claim under test is simple. Is the slowness in the model, or in whether the prompt is byte-identical from one step to the next?

The test isolates that one variable. Same model (an 8-billion-parameter Qwen3, 4-bit), same server (llama-server behind Ollama on an Apple M4 with 16 GB), same session, same tool call. The only thing that changes is how the loop assembles the prompt: the original sliding-window trim, which cuts tool results older than the last six messages, against a deterministic view, which cuts results from earlier turns and leaves the current turn intact.

What came back

With the sliding window, step two of the turn reported cached n_tokens = 0 and a prompt evaluation of 43.8 seconds for 8,184 tokens, about 187 tokens per second. With the deterministic view, the same step reported cached n_tokens = 8180 and a prompt evaluation of 0.51 seconds for the 61 tokens that were actually new. That is an 88× ratio on identical weights. Generation speed was the same in both runs, 16 to 18 tokens per second, which is the point: the model did not get faster. It stopped re-reading.

Prompt evaluation time for step two of the same turn, same weights, same context. Before: the prefix diverged at the first trimmed tool result and the server re-read all 8,184 tokens. After: 8,180 tokens served from cache, 61 read. Numbers are copied from the llama-server log lines cached n_tokens and prompt eval time; nothing is authored for the figure.

The quiet result

The dramatic number is the 88×. The important one is what the cache matched before the fix: on a 201-message session with a 27-billion-parameter model, the log read cached n_tokens = 9216 against a 49,614-token prompt. The first 9,216 tokens are the system prompt and the tool definitions, which never change. Everything after them was re-read every step, 651 seconds per step at 76 tokens per second. The cache was working perfectly. It was being handed a different prompt each time.

Two mechanisms produced the divergence, and both were deliberate design. The sliding window trims tool results relative to the end of the history, so each new message moves the boundary and rewrites one old result from long to short. The compaction step, when the history outgrows the window, recomputed its cut point from scratch, dropping a slightly different set of early messages each step. Each is reasonable alone. Together they guarantee that the prefix changes at the earliest point either touched.

Not a cache bug. A determinism bug.

A sceptic's first objection is that this is the server's cache being too small or too eager to evict. It is not. The cache held 49,614 tokens of the previous request and matched the first 9,216 of the next one; the mismatch is in the bytes, not the storage. The second objection is that the fix just keeps more context. It keeps less: results from earlier turns are cut to 400 characters, which is the same cut as before. What changed is that the cut no longer moves. A result that was short on step four is short on step five, so the prefix is identical and the server reads only the tail.

Three rules make the prefix stable. Cut tool results by turn, not by distance from the end. Pin the compaction point in the session and reuse it while it fits. When it must move, move it to 70 percent of the budget so it does not move again next step. The third rule is the one that turns a one-time win into a steady state.

How far this is from a general result

• One model family (Qwen3, 8B and 27B, 4-bit), one server (llama-server via Ollama 0.34 and 0.40), Apple silicon only. CUDA servers with paged attention cache differently and may re-read less.
• The 88× is step two of a two-step turn. Across the ten-turn benchmark the median time to first output fell from tens of seconds to 1.3 s and the longest silence to 7.2 s; those are the numbers that describe a session, the 88× describes one step.
• The cache is per server slot. A second session with a different prefix evicts the first unless the server keeps two slots, which is why Sentinel now runs two.
• Prompt sizes here are 8k to 50k tokens. Below about 2k tokens the re-read costs under a second and none of this is visible.

What it means for anyone building an agent loop

Treat the prompt as a cache key. Any logic that rewrites earlier messages, injects a timestamp, reorders tools or recomputes a cut each step is paying a full prefill for the privilege, and on a local GPU that is minutes. The decision rule: log cached n_tokens on every request, and if it is not within a few hundred tokens of the previous request's total, the loop is not deterministic. Fix the loop before buying the GPU.

The determination

Same weights, same session, same tool call. The deterministic prefix was served from cache in 0.5 seconds; the sliding-window prefix was re-read in 44 seconds. On the long session the cache matched only the fixed header and re-read the other 40,000 tokens every step. The numbers are from one machine and we have said so. The direction is not ambiguous. An agent that feels frozen is usually an agent reading its own conversation again, and the cure is a prompt that does not change shape.