When a stage fails: what is mechanical and what is the LLM?

People assume that an AI orchestrator's failure-recovery loop is "the model figures out what went wrong and tries again." That is half true, and the half that is false is the interesting half.

In my build system, when a stage fails, the recovery path is a deliberate sandwich: cheap deterministic machinery on the outside, exactly one model judgment call in the middle, and deterministic execution again on the way out. The model is in the loop for precisely one decision. Everything around it -- detecting the failure, capping the retries, dispatching on the decision, resetting the workspace -- is plain bash and config with no model anywhere near it.

This post traces one failure from the moment it happens to the moment it is resolved, and labels every step as either mechanical or LLM. The thesis is that the line between them is much sharper, and the LLM's slice much smaller, than the "the agent recovers" story implies.

If you want the happy path -- clone, implement, review, ship -- that is the orchestration lifecycle post. This post is what happens when a stage in that lifecycle exits non-zero instead.

The failure event: a watchdog and exit 124

Start with the most common way a stage dies: it does not die on its own. It gets killed.

Each stage runs under a timeout wrapper, the workflow scripts. The wrapper launches the stage command in the background, starts a sleeper subshell, and if the sleeper wins the race it sends SIGTERM, waits five seconds, then sends SIGKILL. When that happens the wrapper returns exit code 124 -- the same convention GNU timeout uses -- so the rest of the orchestrator can tell "the agent exceeded its budget" apart from "the agent exited with an error on its own."

There is no model in any of this. It is a sleep, a kill, and a flag file under .state/scratch/ that records whether the watchdog fired. The whole primitive is about 150 lines of bash.

The timeout value itself is also mechanical, and the way it is computed is worth a sentence because it is a place people get the defaults wrong. The implement stage (the workflow scripts) reads the ticket's ## Time budget block and uses max(stuck, expected * watchdog.expected_to_stuck_floor) as the timeout, falling back to watchdog.implement_seconds from the review configuration when the ticket does not declare one. The floor exists because authored tickets sometimes set "expected" and "stuck" to the same number by mistake, which would make the watchdog fire at expected wall-clock -- killing healthy runs that were simply taking their expected time. The floor is a deterministic guard against a deterministic authoring mistake. Still no model.

This is the boundary the spiral-detection watchdog sits next to but does not overlap with: that post is about detecting a run that is busy-but-spinning before the timeout fires. This post picks up after a stage has already exited non-zero -- whether from a spiral that ran out the clock, a lint failure, or a transient upstream error -- and asks what the recovery loop does with it.

The mechanical outer loop

A stage exited non-zero. Now the orchestrator has to decide whether to try again, and it makes most of that decision without consulting any model.

The caps fire first. Two config keys in the review configuration bound the whole loop:

• diagnostic.max_diag_invocations_per_ticket -- how many times the diagnosis step may run at all, across every stage of the ticket. As of this writing it is 2.
• diagnostic.max_retries_per_ticket -- the total budget for retry-like actions (retry, escalation, engine switch). As of this writing it is 2.

The order matters and it is deterministic: the diag-invocation cap is checked before the model is ever called, and the retry cap is checked after the verdict comes back. If the diag cap is already spent, the orchestrator terminates the run as a failure and never spins up the diagnostic agent. The cheapest possible exit is the one where the model is not invoked at all. (the workflow docs has the exact dispatch table.)

The dispatch is a case statement. Once a verdict exists, routing it -- increment the attempt counter, re-run the stage, write sub-tickets, or terminate -- is a bash case on a string. The model picked the string; bash does everything after.

The clone reset is the detail people get wrong. Each retry re-runs the failed stage from the top. For the implement stage, "from the top" includes the clone setup, and the clone setup is destructive:

rm -rf "$CLONE_DIR" mkdir -p "$(dirname "$CLONE_DIR")" git clone "$CLONE_SOURCE" "$CLONE_DIR"

So a retry does not inherit the timed-out attempt's working tree. The half-finished edits, the throwaway debug scripts, the partial diff -- all of it is wiped and re-cloned fresh from origin. What survives a retry is exactly two things: the branch name and the run's state file in .state/runs/. Everything else starts over from a clean checkout.

This is easy to assume backwards. "Retry" sounds like "resume," and resume implies continuity of the workspace. It is the opposite here: retry means start the stage again in a fresh clone, with only the branch name and the external state record carried across. If you are building something like this, name that behavior explicitly in your own docs, because the intuition runs the other way. (The companion doc ticket docs-retry-fresh-clone-per-attempt owns making this airtight in the canonical stage docs; I am describing the behavior, not the doc fix.)

The synthetic short-circuit. There is one place the recovery loop decides "abandon, get a human" without any model in the loop at all. The diagnosis step reads the worker clone's git diff to give the model useful context. If state.clone_dir points at a directory that is gone -- the clone never got persisted, or it was already cleaned up -- there is nothing to diff and nothing safe to retry, so the workflow scripts writes a synthetic abandon-human verdict and returns immediately. It looks like a diagnosis. It is a mechanical safeguard wearing a diagnosis's clothes. The model is never asked.

That short-circuit is a good tell for the whole design philosophy: when the deterministic layer can tell that asking the model is pointless, it does not ask.

The single LLM call

Here is the one place a model actually decides something.

When a stage fails, the caps pass, and there is a real clone to inspect, the orchestrator invokes the diagnostic agent through the workflow scripts. It gathers the failure context -- the run's state JSON, the ticket, the run record if one exists, a capped tail of the failed stage's log, and a capped git diff from the clone -- and asks the model exactly one question: given this failure, what should happen next?

The model must answer with a single JSON object and nothing else. Prose before or after the object is rejected as malformed. The documented action vocabulary (the workflow docs) is:

Action	What it means
retry	Looks transient or environmental; re-run the same stage as-is
escalate-model	Approach looks valid but under-powered; re-run with more capacity
switch-engine	A different engine is a better fit; re-run there
split	The ticket bundles independent concerns; propose sub-tickets
abandon-human	Broken in a way automation should not keep retrying
mechanical-fix	A known-shape lint violation with a deterministic repair recipe

That is the entire surface of the model's authority in the recovery loop: it reads evidence and emits one of these strings (plus structured fields like a repair recipe or a tier recommendation). It does not get to re-run the stage, edit files, write sub-tickets, or override a cap. Those are all downstream of the verdict, and they are all mechanical.

And even the verdict is validated mechanically before anyone acts on it. The output is parsed; an unknown action, missing required field, malformed JSON, a watchdog timeout on the diagnostic call itself, or a credential failure all degrade to a synthetic abandon-human. The model proposes; the deterministic validator disposes.

Deterministic execution on the way out: mechanical-fix

The cleanest illustration of the sandwich -- and the highest-value path in practice -- is mechanical-fix. It is the one action where you can watch the handoff from model to machine happen in a single step.

The setup: a lint verb failed in a known, mechanical shape. A bundled commit that should have been split. A fixture file in the wrong place. The kind of failure where the fix is not a judgment call, but recognizing which fix applies is.

So the model does the recognizing and nothing else. When it returns action="mechanical-fix", it includes a repair_plan: a list of shell commands plus the verification_check -- the exact the project dev wrapper lint verb that originally failed. The diagnose stage persists that recipe to .state/runs/<run-id>.mechanical_fix.json and hands control back to the orchestrator.

From there it is pure bash. The orchestrator runs the recipe's commands in the recorded clone, then re-runs the same failed lint verb. If the check goes green, it commits the result. If a command fails, times out, or the check stays red, it preserves the artifact and falls back to a normal LLM retry. The mechanical execution plus any fallback together count as one attempt against the cap.

That is the sandwich in one path. The LLM decides ("this is a bundled-commit violation, here is the recipe to unbundle it"). Bash executes ("run the recipe, re-run lint-bug-commit-order, commit if green"). The decision is judgment; the execution is deterministic and re-verifiable. Nobody asks the model to run anything, and nobody asks bash to decide anything. (the workflow docs and the workflow docs describe the dispatch; the action itself shipped as a deliberate "let the model hand work to a deterministic executor" feature, not as a general "let the model fix it" escape hatch.)

The honest-edges beat: the routing is nearly binary

Six actions are defined. The model has a six-way menu. So here is the uncomfortable observation: in practice the menu is closer to three items, and the load-bearing decision is closer to one bit.

Two layers of evidence, in increasing order of how reproducible they are from a fresh clone.

At the code layer, the live menu is already smaller than the documented one. The dispatch tables in the workflow docs and the workflow docs list six actions, and switch-engine is among them -- it has orchestrator-level test fixtures and a documented validation path. But the actual output validator in the workflow scripts accepts a set of five: retry, escalate-model, split, abandon-human, mechanical-fix. A switch-engine verdict coming back from the diagnostic agent today would fail that validator and degrade to a synthetic abandon-human. (I verified this against the validator at write time; it is the kind of doc-vs-code drift the companion doc ticket exists to reconcile, and I am only noting it, not fixing it here.) So one of the six is, at the live diagnosis seam, not actually reachable.

At the run-history layer, the firing distribution is narrower still -- but I want to be honest about what I can and cannot recompute here. Which action fired on which run is recorded in .state/runs/*.json under diagnosis.action, alongside the terminal state. That directory is gitignored and local to the orchestrator host; a fresh ticket clone (which is where this very post was drafted) does not carry it, so I cannot stand here and recompute exact per-action counts without overstating my evidence. What the investigation that prompted this post found, qualitatively, is that only three of the six actions have ever been observed firing in this project's history: mechanical-fix, abandon-human, and retry. escalate-model, split, and switch-engine are schema-described and test-fixtured but have not been seen chosen on a real failure. Treat that as a dated, qualitative claim -- "three observed, three not" -- not as a precise tally, and re-derive it against your own state files if you want numbers.

And of the three that do fire, the free-form retry has the weakest record. Most of the time a bare retry is the verdict on a transient upstream error -- a provider 529, a flaky network hop -- and retrying does not fix the class of problem where the approach was wrong; it just spends an attempt against the cap. The durable wins cluster on the path that ends in deterministic execution: mechanical-fix, where the model's judgment is immediately cashed out into a re-verified bash run, and abandon-human, where the honest move is to stop and get a person.

Put those together and the lesson is sharper than the six-way schema suggests. The load-bearing judgment the model actually makes in the recovery loop is close to binary: is this mechanically fixable, or is this a human's problem? The rich routing the schema allows is mostly unused, and the parts that earn their keep are the parts where the model hands the work straight back to a deterministic executor or hands it off to a human. The design lesson, if you are building one of these: do not invest in a wide diagnosis taxonomy before you have run data showing the wide branches fire. Mine has six branches and leans on two.

What this means if you are building agent infrastructure

The instinct when a stage fails is to give the model more rope: let it inspect, let it decide, let it try things. The shape that has actually held up for me is the opposite -- give the model exactly one decision, wrap it in deterministic machinery on both sides, and make every other part of the recovery loop a cap, a case statement, or a git clone that a human can read and a test can pin.

The model is genuinely good at the one thing it is asked to do here: read a messy failure and say "this is the bundled-commit lint, here is the recipe" or "this premise is broken, get a human." It is not asked to be good at the surrounding bookkeeping, because the surrounding bookkeeping should never be non-deterministic in the first place. Detection is mechanical. Enforcement is mechanical. Execution is mechanical. Only diagnosis is the model -- and even diagnosis spends most of its value routing toward the deterministic exits.

If you build this, the part worth copying is not the diagnosis prompt. It is the discipline of drawing the mechanical/LLM line on purpose, and keeping the LLM's side of it as small as the problem allows.

---

Or: subscribe to the newsletter for more posts on building this stuff.