The LLM sandwich: one model decision between two deterministic layers¶

Most "AI agent" designs put the model in charge of a loop. It inspects a situation, decides what to do, does it, looks at the result, and repeats until it thinks it is done. That works in demos. It is hard to test, hard to bound, and hard to debug, because the part that can be wrong in an interesting way -- the model -- is touching every step.

The pattern I keep coming back to instead is the opposite, and I call it the LLM sandwich:

Deterministic pre-processing gathers inputs, checks preconditions, and decides whether the model should even be called.
Exactly one model call makes exactly one decision and returns it as structured data -- not actions, not edits, not tool calls.
Deterministic post-processing validates that decision, executes it, and verifies the result with a check that does not depend on the model being right.

Plain code on the outside, one bounded judgment call in the middle. The model is the filling, not the cook. This post defines the pattern and then traces one real end-to-end example through all three layers, with the actual inputs and outputs.

Why put the model in the middle and nothing else¶

An LLM has one property no other component in your system has: given the same input, it can produce a different output, and one of those outputs can be subtly wrong in a way no type checker will catch. That is exactly why it is valuable -- it can read a messy, half-structured situation and produce a judgment -- and exactly why you want its blast radius small.

So the sandwich draws a hard line. The model is allowed to decide. It is not allowed to act. Everything it might want to do -- run a command, edit a file, retry a step, give up -- is done by deterministic code on the far side of a validated, typed decision. That single rule buys four things:

Testability. The two bread layers are pure functions of their inputs. You can feed them fixtures and assert their behavior with no model in the loop. The only non-deterministic step is one call with one typed output.
Bounded cost and blast radius. One decision means one model call, capped in code. The model cannot spin a 40-turn loop, cannot touch files you did not hand it, cannot exceed the retry budget, because it never holds the tools.
Verifiability. The model's decision is cashed out into a check that passes or fails on its own terms. If the check fails, the decision is discarded. The model proposes; the check disposes.
Debuggability after the fact. The one uncertain step is captured as a small structured object you can read later, next to the deterministic inputs that produced it and the deterministic result it led to.

None of this requires trusting the model less. It requires putting the model only where its strength pays off, and keeping everything a machine can do reliably on the machine.

A worked example: recovering from a failed build stage¶

Here is the pattern in a system I actually run: an orchestrator that turns a written ticket into reviewed, merged code by running stages -- clone the repo, implement the change, run the review checks, open the pull request. Stages fail. When one does, a recovery loop decides what to do about it. That loop is a sandwich, and it is a good example because the middle is genuinely a judgment call while everything around it is not.

The scenario: an "implement" stage just finished work on a bug-fix ticket, but a review check called lint-bug-commit-order failed. That check enforces a rule this project cares about -- on a bug-fix ticket, the commit that changes production code must land before any commit that only touches tests, so the fix is proven to move the bug before a test is written to pin it. The agent squashed the code fix and its test into a single commit. The check exits non-zero. Now what?

Layer 1: deterministic pre-processing¶

No model has been consulted yet, and most of the decision is already made.

Detect the failure. Each stage runs under a timeout wrapper: it launches the stage, starts a sleeper, and if the sleeper wins the race it sends SIGTERM, waits, then SIGKILL, and reports exit code 124 -- so "ran out of time" is distinguishable from "exited with an error." Here the stage exited non-zero on its own because the lint failed. Either way, a stage failed, and that is a plain exit-code check.

Check the caps before spending anything. Two limits, read from config, bound the whole loop:

max_diag_invocations_per_ticket = 2 -- how many times the model may be asked to diagnose, across the entire ticket.
max_retries_per_ticket = 2 -- the total budget for retry-like actions.

The diagnosis cap is checked before the model is called. If it is already spent, the run terminates as a failure and the model is never invoked. The cheapest exit is the one where you do not ask the model at all.

Check a precondition that can make the model call pointless. Diagnosis needs the failed stage's working directory (the clone) so it can show the model the git diff and the failing check's output. If that directory is gone -- never persisted, or already cleaned up -- there is nothing to inspect and nothing safe to retry. So the code writes a synthetic "abandon, get a human" verdict and returns, again without calling the model. It looks like a diagnosis. It is a deterministic safeguard wearing a diagnosis's clothes.

Assemble the context. The caps passed and the clone exists, so the code gathers a fixed, capped bundle: the run's state JSON, the ticket text, the run record if one exists, a bounded tail of the failed stage's log, and a bounded git diff from the clone. Bounded matters -- it keeps the prompt size, and thus the cost, predictable regardless of how big the failure was.

Everything above is a sleep, a kill, some integer comparisons, a directory existence check, and file reads. It is a few hundred lines of shell and config that a human can read and a test can pin. No model has run.

Layer 2: the single model call¶

Now, and only now, the model is asked exactly one question: given this failure, what should happen next? It must answer with a single JSON object and nothing else -- prose before or after the object is treated as malformed. The decision is drawn from a closed vocabulary of actions:

Action	What it means
`retry`	Looks transient; re-run the same stage unchanged
`escalate-model`	Approach is sound but under-powered; re-run with more capacity
`switch-engine`	A different model/engine is a better fit; re-run there
`split`	The ticket bundles independent concerns; propose sub-tickets
`abandon-human`	Broken in a way automation should not keep retrying
`mechanical-fix`	A known-shape violation with a deterministic repair recipe

For this failure the model recognizes the shape -- a bug-fix ticket with the code and test squashed into one commit, tripping a commit-order rule -- and returns mechanical-fix with a recipe:

{
  "action": "mechanical-fix",
  "summary": "bug-fix ticket squashed the code fix and its test into one commit; lint-bug-commit-order needs the non-test commit to land first",
  "repair_plan": {
    "verification_check": "bash scripts/dev.sh lint-bug-commit-order",
    "commands": [
      "git -C <clone> reset --soft HEAD~1",
      "git -C <clone> restore --staged -- tests/",
      "git -C <clone> commit -m 'fix: correct the off-by-one in the deal'",
      "git -C <clone> add -- tests/",
      "git -C <clone> commit -m 'test: pin the corrected deal'"
    ]
  }
}

That is the entire scope of the model's authority. It read the evidence and emitted one action string plus a couple of structured fields. It did not run any of those commands. It did not re-run the stage, edit a file, write a sub-ticket, or override a cap. The commands list is a proposal, expressed as data.

Note what the closed vocabulary does: it collapses an open-ended "what should I do" into one of six known tokens. Anything the model might say that is not one of those tokens is, by construction, not a valid decision -- which is what makes the next layer possible.

Layer 3: deterministic post-processing¶

The model's JSON is now input to plain code, and it is treated as untrusted input.

Validate. Parse the object. The action must be one of the known tokens. For mechanical-fix, summary must be a non-empty string, verification_check must be a non-empty string, and commands must be a non-empty list of non-empty strings. Any failure here -- malformed JSON, an unknown action, a missing field, even a timeout on the diagnosis call itself -- degrades to the same synthetic abandon-human from Layer 1. The model cannot crash the loop by returning garbage; garbage routes to "get a human."

Persist the decision. The validated recipe is written to .state/runs/<run-id>.mechanical_fix.json, so the exact thing that ran is on disk for later inspection.

Execute, then verify with an independent check. The code runs the recipe's commands in the recorded clone, and then -- this is the load-bearing step -- re-runs the exact check that originally failed: bash scripts/dev.sh lint-bug-commit-order. The verifier is not the model's opinion that it fixed the problem. It is the same deterministic gate that flagged the problem in the first place.

If the check goes green, the result is committed. The recovery worked.
If a command fails, times out, or the check stays red, the artifact is preserved and the loop falls back to a normal retry.

Either way, the mechanical execution plus any fallback together count as one attempt against the retry cap. The model got one decision; the machine got the whole execution and the final say on whether it worked.

The same picture as pseudocode¶

# ---- bottom bread: deterministic ----
if stage_exit_code == 0: return SUCCESS
if diagnosis_invocations >= MAX_DIAG: return TERMINATE      # never call the model
if clone_dir_missing():   verdict = abandon_human()          # never call the model
else:
    context = gather_bounded(state, ticket, run_record, log_tail, diff)

    # ---- filling: exactly one model call, one typed decision ----
    verdict = model_decide(context)      # returns one JSON object, no side effects

# ---- top bread: deterministic ----
verdict = validate(verdict) or abandon_human()   # bad output -> safe default
persist(verdict)
if verdict.action == "mechanical-fix":
    run(verdict.repair_plan.commands)
    if rerun(verdict.repair_plan.verification_check) == GREEN:
        commit()
    else:
        fall_back_to_retry()
# ... other actions dispatch the same way: a case statement, no model

Read top to bottom, the model appears on exactly one line. Everything above it is a guard; everything below it is a dispatcher and a verifier.

Contrast: the loop you are tempted to build instead¶

The obvious alternative to all of this is shorter to write: give the agent a shell tool, hand it the failing check, and say "fix it." Let it look around, try things, and re-run the check until it passes.

It will often work. But compare it against the sandbox above:

Cost is unbounded. "Try things until it passes" has no natural stopping point; a confused agent can burn a large, variable number of turns. The sandwich spends one model call per diagnosis, capped at two per ticket.
Blast radius is unbounded. An agent holding a shell can edit files you did not mean to expose, or "fix" the check by disabling it. In the sandwich the model never holds the shell; it hands over a typed decision, and the code runs a recipe against a check the model cannot weaken.
It is not reproducible. Two runs of the free agent can take different paths to different diffs. The sandwich's bread layers are deterministic, so a given failure plus a given verdict always produces the same execution.
You cannot tell success from a masked failure. "The agent says it is fixed" is not evidence. "The same gate that failed now passes" is. The sandwich makes verification a property of the system, not a claim by the model.

The free-agent loop optimizes for looking capable. The sandwich optimizes for being checkable.

How to apply it¶

When you are about to let a model drive something, try to fold it into a sandwich first:

Name the single decision in one sentence. "Given this failure, what should happen next?" If you cannot state it that tightly, the filling is too big -- split it into more than one sandwich.
Make the output a closed vocabulary or typed schema. A fixed set of action tokens, or a JSON shape you validate. Not free-form prose, and not tool calls the model gets to execute.
Put a deterministic verifier after the model. A check that passes or fails on its own terms -- ideally the same check that detected the problem. If you cannot verify the decision without asking the model again, you do not yet have a sandwich.
Define the malformed-output path up front. Any parse or validation failure should degrade to a safe default -- usually "stop and get a human," never "guess and continue."
Cap the invocations in code. One decision, a small fixed number of retries, enforced by the outer layer -- not by a sentence in the prompt asking the model to be careful.
Keep the bread pure. Pre- and post-processing should be functions of plain data, so a test can pin them with fixtures and no network.

If a task cannot be shaped this way -- if it genuinely needs open-ended exploration with the model in the loop -- that is worth knowing before you build it, and it is a different, more expensive kind of system to operate. Most of the places I reached for "an agent" turned out to be a decision I could isolate, wrap in deterministic bread, and verify. The model got smaller. The system got easier to trust.