A Cure Ladder for Agent Failures

Retrying everything is how agents loop forever. Classifying failures first, then applying the cheapest cure that fits, with a budget and a memory of what already failed, is how a turn survives.

The naive error handler in an agent loop is a retry. The slightly less naive one is a retry with backoff. Both produce the same failure mode in practice: the agent repeats the thing that did not work, burns its step budget, and ends the turn with a stack trace. The alternative we settled on is boring and it works. Classify first, then cure, cheapest cure first, with a budget.

Classify before you act

Every exception, timeout and bad model output is mapped to one of a dozen fault classes with a dependency attached: model_unreachable on the local server, context_length on the model, tool_timeout on a browser, invalid_tool_json, empty_answer, agent_bug, and so on. The classifier is a list of rules over the error text and the state of the step. It is the single most useful piece of code in the loop, because every other decision keys off the class, not the raw error.

A ladder of cures, cheapest first

Each class has an ordered list of cures. For a model that stopped mid-answer: ask it to continue once; then compact the history harder; then move to a different model. For invalid tool JSON: repair the JSON if it is merely truncated; then re-ask with a larger output cap; then move down the model ladder. For an unreachable local model: wait two seconds and retry once; then restart the model server; then switch to a remote model. The order is by cost, in seconds and in dollars, and the agent stops at the first cure that works.

Budget and memory

Two guards keep the ladder from becoming a loop. A heal budget caps how many cures a turn may spend in total and per class, so a turn cannot spend forty steps healing. And an incident memory records each cure attempt with its signature; a cure that failed recently for the same signature is skipped, which is what a person would do.

Circuit breakers for dependencies

If the local model server fails ten times in six hours, something is wrong with it, not with the prompts. A breaker per dependency opens after repeated failures, routes around it for a while, and lets a probe through later. When it opens, the user gets one line: self-heal: local model unavailable, using the remote one. Not a wall of errors, and not silence.

The model ladder

Cures that change the model walk a ladder: the current model, then the local 8B, then a hosted frontier model. The history is kept in a canonical format and converted to each backend's wire format at request time, so a session can change models mid-turn without losing its tool calls. The expensive model is the last rung on purpose; most faults are cured for free before it is reached.

Gated self-patching

The agent_bug class is special. When the fault is in the agent's own code, the agent may propose a patch, but only on a git branch, only if the test suite passes, and only once per signature per day. After restart a health check runs; if it fails, the patch is reverted automatically and the incident is written to memory. We have had it revert itself. That is the correct outcome.

What it looks like in the ledger

cls=model_unreachable dep=ollama cure=retry_once_same ok=1 ms=2002
cls=context_length    dep=model  cure=compact_history ok=1 ms=0
cls=tool_timeout      dep=mcp    cure=restart_server  ok=1 ms=4210

Three incidents, three one-line notices to the user, zero dead sessions. The ledger is also what the agent reads every night to write its lessons.

Resources