Skip to content

How do you test an orchestrator whose job is to call an LLM?

If you have never built one, an "agent orchestrator" sounds like the hardest thing in the world to test. Its whole job is to call a large language model, take whatever that model produced, open a pull request on a real VCS host, and ping a human on their phone. Every one of those is something you actively do not want a unit test to do for real. You cannot afford to spend tokens and minutes calling a real model in a loop, you do not want test runs opening real PRs, and you certainly do not want your test suite buzzing your phone at 2am.

So new builders get stuck on a false question: "how do I test whether the LLM wrote good code?" You cannot, and you should not try. The model is non-deterministic; "good code" is not a property a test can assert reproducibly. That question is a trap.

Here is the reframe the rest of this post is about. You do not test the model's judgment. You test the orchestrator's behavior given a scripted world. You fake the model, the host, and the notifier so each returns a fixed, scripted response, and then you ask one reproducible question: given this exact model reply, this exact host result, and this exact crash, did the orchestrator do the right thing? That is a question with a yes/no answer that holds on every run.

This repo does exactly that. The harness lives under the workflow test harness, and every claim below points at a real file in it.

Beat 1: fake the external world, not the agent's judgment

To test a system whose job is to call out to four external things, you build four fakes -- one per external seam:

  • The model -- the workflow test harness
  • The VCS host -- the workflow test harness
  • The notifier -- the workflow test harness
  • The browser harness -- the workflow test harness

A fake is not a mock framework with matchers and expectations. It is a tiny script that reads a scenario JSON file and replays whatever that file tells it to: a chunk of stdout, a chunk of stderr, an exit code, an optional delay, and sometimes a few files written into the worker clone. That is the whole trick. The scenario is the script; the fake is the actor that reads its lines.

fake-claude.sh is the clearest example. Hand it a scenario and it prints the scenario's stdout, writes the scenario's files, sleeps for the scenario's delay_ms, and exits with the scenario's exit_code. It can also walk a list of sequential_responses so that the first call returns one thing and the second call returns another -- which is how you script a multi-turn flow like "the diagnosis agent says retry, then on the next call says merge" without a real model anywhere in sight. The point is that the orchestrator thinks it called Claude. It called a script that returned a constant.

The other three follow the same shape with their own twist:

  • fake-gh.sh dispatches on the subcommand, so pr create and pr merge can return different scripted results. Its most useful trick is emitting informational text on stderr while still exiting 0 -- because real gh does that during a merge, and a naive orchestrator that treats any stderr as failure would abort a merge that actually succeeded. The fake lets a test pin "stderr must not abort the pipeline" forever.
  • fake-ntfy.sh mimics the small slice of curl the notifier uses to post to ntfy.sh. Instead of making an HTTP request, it appends every "sent" message to a capture file you can read back and assert on.
  • fake-playwright returns pass or fail depending on which scenario you select, so the orchestrator's "did the browser tests pass?" branch can be exercised both ways without launching a browser.

How do the orchestrator's calls to claude and gh reach the fakes instead of the real binaries? Through PATH. The shared fixtures in the workflow test harness prepend the fixtures directory to PATH so a bare claude or gh lookup resolves to the fake first. (There are two-line shim files named claude and gh next to the fakes whose only job is to forward to fake-claude.sh / fake-gh.sh, so the on-PATH name matches the real CLI exactly.) conftest.py also goes belt-and-suspenders to guarantee no real notification or provider call ever escapes during a test run: it forces the notifier to the fake binary, points the topic at a no-op value so even a misconfigured send hits nothing real, and clears provider API keys so no test can accidentally probe a live provider. You can see the notifier fake exercised directly in the workflow test harness and the browser fake in the workflow test harness.

The mental shift for a new builder is this: none of these tests assert that the model's output was correct. They assert what the orchestrator did with a known output. The model's reply is an input you control, the same way you control any other test fixture.

Beat 2: inject faults where the orchestrator can actually fail

Faking the seams lets you test the happy path deterministically. But the failures that matter most for an orchestrator are not "the model returned bad JSON." They are "the run died partway through and nobody noticed." A crash you can see is an annoyance. A crash you cannot see -- a run that silently stops while its records still say "running" -- is the dangerous one, because the human stops trusting the system the first time a run vanishes without a trace.

So the fault you inject is a mid-stage kill, and the thing you assert is observability, not success.

The mechanism reuses the same scenario trick. A scenario can tell a stage to hang -- the implement stage in the workflow test harness just sleeps -- which gives the test a wide, reliable window to kill the orchestrator while a specific stage is mid-flight. the workflow test harness backgrounds a real orchestrator run, waits until the state file shows the stage it wants, and then sends a signal. It kills mid-implement and mid-merge, two different points in the pipeline. The whole file is built on one discipline, stated at the top of the module: every test asserts on the contents of the state file on disk, and no test scrapes a log. The state file is the observability contract; if the invariant lives anywhere, it lives there.

The invariant has a name worth remembering: no silent termination. A killed run must always leave a durable record that it was killed, and a human must always be told. Concretely, after a mid-stage kill the test asserts the state file landed in terminal_state = killed, that a cause was recorded, and that a notification fired -- either sent, or written to the fallback queue if the notifier was unreachable. "It crashed" is acceptable. "It crashed and the records still claim it is running" is the bug the whole beat exists to forbid.

There are two genuinely different kill paths, and it is worth being precise about them rather than waving at "the orchestrator handles signals":

  • SIGTERM is catchable. The orchestrator installs a trap (on_terminate, then exit 143) so that when a SIGTERM arrives mid-stage, its own handler runs: it flips the state file to killed, records the cause, releases the lock so the next run is not blocked, and fires the notification before exiting. the workflow test harness pins the specific fields -- the killed state, the cause value, the released lock, and the exit code -- so a future change that silently drops any of them fails loudly.
  • SIGKILL is not catchable. Nothing runs when a process is SIGKILL-ed; no trap, no handler, no notification. So the run dies with running still on disk, and the invariant has to be restored after the fact by a separate liveness check. the workflow test harness covers it: the reconciler notices the recorded process is gone -- and, crucially, compares a process signature rather than just a PID, so a recycled PID belonging to some unrelated program cannot masquerade as the still-living run -- and flips the orphaned state to killed. The two paths reach the same destination by different roads: the trap handles the kill you can intercept, the liveness check cleans up the kill you cannot.

The dropped-notification fault rounds out the beat. The notifier fake can be told to fail as if ntfy.sh were unreachable, and the test asserts the orchestrator sets a ntfy_fallback flag and writes the undelivered message to a pending-notifications queue on disk, to be retried at the start of a later run. The notification is allowed to fail to send; it is never allowed to vanish. (The notification doc the workflow docs describes this fallback queue and its retry policy; the orchestrator design reference the orchestrator design docs describes the stage contract that makes the state file the source of truth.)

Beat 3: a checklist you can reuse, and the edge it does not cover

Strip the specifics away and you are left with a pattern any reader can lift into their own orchestrator. Three columns:

Which seams to fake. Anything the orchestrator reaches out to and cannot control:

  • the model it calls,
  • the VCS host it opens PRs against,
  • the notifier it pages a human through,
  • the browser/test harness it shells out to.

Each becomes a small script that replays a scenario file -- a scripted, deterministic response -- so the orchestrator's behavior, not the dependency's, is what your test measures.

Which faults to inject. Not "bad model output" -- the failures that hurt are the ones that can go unobserved:

  • a mid-stage crash (hang a stage, then kill it),
  • a signal kill on both paths: the catchable one your handler intercepts, and the uncatchable one a later liveness check must reconcile,
  • a dropped notification (make the notifier fail and confirm the message survives in a queue).

What to assert. Observability invariants, written against durable state, not against logs or outputs:

  • the run is recorded as killed (or failure) with a cause -- never left claiming running,
  • the human was notified, by send or by fallback,
  • shared resources (locks) were released so the system can move on.

The phrase to keep is: assert that the run was observable, not that it succeeded. A test that says "the orchestrator survived a crash and the record proves it" is worth ten that say "the happy path was happy."

And now the honest edge, because a checklist that oversells itself is worse than none. Fakes prove the spine, not the live provider. Every test described above runs against a scripted Claude and a scripted GitHub; by design the suite never invokes the real binaries. That is the right default -- it is fast, free, and deterministic -- but it means a fully green fake suite tells you your orchestrator does the right thing given a well-behaved dependency. It does not tell you the real claude CLI still takes its prompt the way you think, or that real gh still phrases its merge output the way your fake does. Those contracts drift, and when they drift your fakes drift with them and keep passing.

The repo keeps one deliberately separate, opt-in smoke test for exactly this -- the workflow test harness -- which makes a real model call and is skipped unless you explicitly turn it on. Its own docstring says it plainly: the rest of the suite uses the fake and never invokes the real binary, so nothing else pins the actual CLI contract. That is the division of labor to copy. Fakes are your fast, deterministic spine coverage and you should have a lot of them. A small number of live shakeouts are how you catch the day the real provider changes shape under you. You need both, and you should never let a green fake suite talk you out of the second.

What this means if you are building agent infrastructure

If you are new to this and the orchestrator feels untestable, the unlock is realizing you were aiming at the wrong target. You are not grading the model. You are pinning your own control flow against a world you script down to the exit code.

Fake the four seams so the dependencies are constants. Inject the faults that can go unseen -- the mid-stage kill, both signal paths, the dropped page. Assert the observable record, not the outcome: a killed run is recorded as killed and the human hears about it. Then keep a few live shakeouts on the side so the day a real provider changes its contract, something other than production is the first to find out.

That is the entire strategy. The deterministic part is large and cheap and should carry almost all of your confidence. The live part is small and precious and should carry the rest.

Or: subscribe to the newsletter for more posts on building this stuff.