Why Your AI Agent Freezes Mid-Task

We traced every "it just stopped" report to four mechanisms in the serving stack, none of them the model. Here is what the logs showed, with the numbers, and the five-line rule that fixed it.

Every agent product gets the same complaint: it stops halfway through an answer. The session looks dead, the user hits refresh, the work is gone. We had the complaint too. Here is what we found when we stopped guessing and read the serving logs.

Symptom: eleven minutes of silence

On a session with 201 messages, a 27-billion-parameter model took 11 minutes and 11 seconds to produce its first word. The log line is unambiguous: prompt eval time 651,337 ms for 49,614 tokens, about 76 tokens per second. Then it generated 42 tokens at 5 tokens per second and the request ended. From the chair, that is a freeze followed by a half sentence.

slot operator(): new prompt, n_ctx_slot = 65536, task.n_tokens = 49614
slot operator(): cached n_tokens = 9216, memory_seq_rm [9216, end)
prompt eval time = 651337.77 ms / 49614 tokens (76.17 tokens per second)

The second line is the whole story. The server had 49,614 tokens of the previous request in its cache, and matched only the first 9,216. Everything after that was re-read from scratch. On a laptop GPU that is ten minutes, every step, for the rest of the session.

Cause one: the prompt changed shape between steps

Inference servers cache the key-value state of the prompt prefix. The cache is byte-exact: the new prompt must start with the same tokens as the old one. Our agent was changing the prefix every step in two places. It cut old tool results with a sliding window ("keep the last six messages intact"), so each new message pushed one more old result from long to short. And when the history no longer fit, it recomputed the compaction point from scratch, dropping a slightly different set of early messages each time. Both are reasonable on paper. Both guarantee a cache miss at the point of change, and everything after it is re-read.

The fix is a rule about determinism, not about size: the view of earlier messages must be identical from one step to the next. Tool results from earlier turns are cut; the current turn's stay intact. Once a compaction point is chosen it is pinned in the session and reused while it still fits, and when it has to move, it moves far enough that it will not move again soon. After the change the same step reports cached n_tokens = 8180 and a prefill of 0.5 seconds.

Cause two: the server kept reloading the model

Three different pieces of our code talked to the same model: the chat loop, a keep-warm job every four minutes, and an after-turn reflection step. Only one of them specified the context size. The server treats a different context size as a different model: it unloaded the runner and loaded it again with the new size, and a reload empties the cache. So the four-minute keep-warm job, whose whole purpose was to avoid cold starts, was causing one on every turn.

Rule: every request to a model carries the same context size and the same keep-alive. We also learned that a keep-alive of "-1" must be sent as a number; as a string the server answers missing unit in duration and the request fails.

Cause three: asking for a bigger window than the model has

The configured window was 65,536 tokens. The 8-billion-parameter model's trained maximum is 40,960. The server does not refuse; it silently drops tokens from the front of the prompt, which is where the system prompt and the tool definitions live. The model then answers with no instructions and no tools, which looks like the model has lost its mind. The fix is one request at startup: ask the server for the model's trained context length and cap the window to it.

Cause four: a queued side request

Before each turn the agent looked up relevant notes in its knowledge base, which needs an embedding from a small model. When the GPU is full, the server queues the load of that small model until a big one expires, which can be an hour. The turn waited on it. Lookups now have a 20-second deadline and the turn starts without them if they are late.

What it looks like after

None of this required a faster machine or a different model. It required treating the prompt as a cache key and the serving layer as part of the product. The test that guards it now runs a real ten-turn session and fails if any turn goes silent for longer than 45 seconds.

Resources