Skip to content

Parity-first: replacing the system that builds your software without trusting the rewrite

Here is a rewrite problem most posts skip, because it is the scariest one.

You can rewrite a feature and feel the bug the moment you ship it. You can rewrite a parser and watch the test suite go red. But what do you do when the thing you want to rewrite is the system that builds your software -- the orchestrator that runs every ticket, clones every workspace, opens every pull request, decides what merges? A wrong swap there does not throw an exception. It silently corrupts every future build, and you find out three features later, when you can no longer tell which of your last twenty merges the new machine got subtly wrong.

That is the situation this project is actually in. The orchestrator that runs every ticket in this repo is a bash workflow under the orchestrator dispatcher -- the dispatcher described in the companion lifecycle post, The orchestration lifecycle: from phase to ticket to retro. There is now a Go reimplementation, tempo, that aims to replace it. So the question is concrete and unavoidable: how do you switch out the machine that builds your software when a wrong switch is invisible until it is expensive?

The answer this project used is a method, not a leap of faith, and it generalizes well past this one repo. It has a name worth keeping: parity-first. Run both systems over identical inputs, diff the normalized output, and -- the part that separates discipline from theater -- be relentlessly honest about what the diff does and does not prove.

Run both over identical inputs, then diff

The core of the method is a harness, and the harness is small enough to describe in one breath. It lives in the Tempo implementation, entry point RunParity. Given nothing but a fixture project, it does this:

It copies a frozen fixture project -- a tiny, hand-built sample repo with a known backlog and a known run history -- into a scratch directory, twice. One copy is fed to the old system, one to the new. Copying matters: each system gets a pristine, byte-identical starting point, so any difference in their output is a difference in them, not in the input or in some mutation one made to the other's files.

Then it runs both. The old system runs through runBashReference, which shells out to the frozen bash reference -- the very same the orchestrator dispatcher that runs real tickets today -- pointed at the first copy. The new system runs its own Go code over the second copy. Same command, same input, two implementations.

Then it diffs. Not a raw text diff -- that would fail on cosmetic differences that do not matter, like the two systems formatting a table differently. Instead the harness normalizes each side into a small structured summary and compares meaning. Two functions carry this:

  • compareBacklogList asks both systems to list the backlog, parses each one's output into the same shape (which tickets are active and in what order, how many shipped, with what outcomes), and checks the shapes match.
  • compareStatus does the same for run status: it asks both systems for the active runs and the recent terminal runs, normalizes both into the same record, and compares them field by field.

Each comparison returns a per-surface verdict -- a pass or a fail with details -- and the harness collects them into one report. If any surface fails, the whole run fails. You do not get a vibes-based "looks close enough"; you get a named surface that diverged and the two normalized values that disagreed, which is exactly what you need to go fix the port.

This is the whole load-bearing idea, and it is almost embarrassingly simple: two implementations are at parity on a surface when, given identical input, they produce the same normalized output on that surface. Everything else is bookkeeping around that sentence.

The honest coverage admission

Now the part that makes this trustworthy instead of reassuring.

It would be very easy to run that harness, watch every surface go green, and announce "tempo is at parity with the orchestrator." That sentence would be a lie, and the harness itself refuses to let you tell it.

RunParity builds its report with an explicit KnownGaps list -- uncovered work, written down in the same artifact that reports the passes, so nobody reading the green checkmarks can miss the asterisks. The gaps are blunt. The harness compares read surfaces only: listing the backlog, reporting status. It does not exercise the surfaces that actually change the world. In the lifecycle post's vocabulary, those are the live stages -- ticket, review, ship, merge -- where an agent writes code, a reviewer runs the gates, a pull request opens, and a branch merges. None of that is touched here. The KnownGaps entry says so directly: live ticket / review / ship / merge execution is not exercised, and the harness does not invoke agents, gh, or notifications.

That admission is not a disclaimer bolted on at the end. It is structural. A frozen fixture project has no live agent to call, no GitHub to push to, no network at all -- so the only thing this harness can compare is the deterministic read-side output both systems compute from the same files on disk. Confirming the backlog and status surfaces match is real, and it is necessary. It is just nowhere near the whole orchestrator, and the report says which part it is.

This is the same honesty posture as the review-guardrails post, which makes a parallel admission about its lint wall: the gates check shape, not always correctness. Here the parity harness checks read surfaces, not the whole lifecycle. In both cases the system's credibility comes from naming its own ceiling out loud rather than letting a row of green checkmarks imply more than it earned. A parity report that hid its gaps would be worse than no report, because it would license exactly the false confidence the method exists to prevent.

One real ticket, before you switch anything

So fixture parity over read surfaces is necessary and not sufficient. What closes some of the gap?

You run one real ticket through the new system and watch what comes out.

By default, the new system's ticket verb does not call a live model. It uses a deterministic static runner, on purpose, so the default test suite never spends money or depends on a provider being up -- the same instinct the lifecycle keeps everywhere it can. But there is an opt-in path, documented in the Tempo design memo, that swaps the static runner for a live engine adapter. You select it explicitly (you have to ask for it by naming an engine), and it runs one genuine ticket end to end through the new orchestrator: it hands the stage prompt to the real agent, captures its output, keeps the run heartbeat alive while the agent works, and enforces the same timeout and stale-output limits the real system would.

The clever part is how it knows the run actually finished, and how it ended. The adapter requires the engine to print, as its final line, a structured terminator: a line beginning with TEMPO_RESULT, carrying terminal_state, outcome, and cause fields. That single contracted line is the seam. It turns "the agent said a bunch of things and then stopped" into a machine-readable verdict the orchestrator can compare against what the old shell orchestrator would have recorded for the same ticket.

That is what makes a real-ticket comparison possible before any switchover. You do not have to cut the whole repo over to the new system and pray. You run one real ticket through it, read its TEMPO_RESULT, and check that the new machine reached the same terminal state and outcome the old machine would have. One real run is a far narrower claim than "every future build will be correct" -- but it is a real run, against a live agent, with a verdict you can diff, and it costs you exactly one ticket instead of your whole pipeline.

The discipline, stated plainly

Strip away the Go and the fixtures and the terminator line, and the method is four moves you can apply to any rewrite of a system you cannot afford to corrupt:

  1. Freeze the old system as a reference. Not "the old system as I remember it" -- the actual, runnable, frozen thing. Here that is the bash the orchestrator dispatcher, invoked unchanged.
  2. Feed both the same input. Identical, copied, pristine on each side, so a difference in output can only mean a difference in the systems.
  3. Diff normalized output, surface by surface. Compare meaning, not bytes; report a named verdict per surface; fail loudly on any divergence.
  4. Write down what you did not cover. The gaps go in the report next to the passes, so green never gets read as done.

And then the edge that the whole post has been walking toward, because leaving it out would make me guilty of exactly the overclaim the method guards against:

Parity over fixtures is not parity over reality. A frozen fixture is a controlled, offline, deterministic slice of the world. It cannot exercise a live agent's judgment, a flaky network, a gh call that times out, a notification that does or does not fire, or any of the live lifecycle stages the harness itself lists as uncovered. Fixture parity tells you the two systems agree on the deterministic, observable surfaces a fixture can drive. It tells you nothing about the surfaces a fixture cannot reach. That is why the method pairs the fixture diff with the single live ticket: the diff covers breadth cheaply and offline; the one real run probes depth against reality, expensively, once. Necessary and sufficient are different words, and a parity-first cutover earns trust precisely by never confusing them.

The reason this matters for someone new to building this way is that the temptation runs the other direction. The exciting demo is "I rewrote the orchestrator and it works." The trustworthy engineering is "I ran both over the same inputs, here are the surfaces that match, here in writing are the ones I have not covered yet, and here is the one real ticket I ran to probe past the fixtures." The first sentence is how you corrupt a build system without noticing. The second is how you replace one and still sleep.

If you want the machine being rewritten here -- how a phase becomes tickets, how a ticket moves through staged hops in a disposable clone -- that is the lifecycle post. If you want the wall of gates that keeps each individual ticket honest, that is the review-guardrails post. This post is the layer above both: how you swap the whole apparatus out, for a faster one, without ever having to take the swap on faith.

Or: subscribe to the newsletter for more posts on building autonomous AI development infrastructure.