Review surfaces and the AI-first guardrails¶

The companion to this post -- The orchestration lifecycle: from phase to ticket to retro -- opened one of the two boxes under my AI build loop. It walked how a phase decomposes into contract-shaped tickets, how a dependency graph schedules them, how a single ticket runs through staged hops in a disposable clone, and how a retro turns run records into changes to the system. It deliberately left one box closed: the review stage and the lint wall around it. This post opens that box.

The question this post answers is the one a skeptical engineer asks the moment I say "an autonomous agent writes the code." Namely: how does it not fall apart? An autonomous agent will happily ship code that compiles, passes the tests that happen to exist, and is quietly wrong -- a hidden-information leak with no test pinning the part that must not leak, a bug "fix" that proves the agent's premise instead of moving the real symptom, a stale comment that lies to the next agent that reads it. A human reviewer feels the hair on their neck stand up at those. The agent feels nothing.

So the project's answer is not "trust the model." It is a wall of machine-checked contracts. The load-bearing idea of this entire post is one sentence: AI-first development is making the right thing mechanically enforced and the wrong thing fail a gate. Everything below is a worked example of that sentence.

Review is a stage with surfaces, not a vibe check¶

In the lifecycle post, review was one verb in the dispatcher (the orchestrator dispatcher) -- a stage between IMPLEMENTED and READY. Here is what is actually inside it.

Review runs a preflight: an ordered list of named check verbs, every one of which must pass before a ticket can advance. They live in the review configuration under three buckets:

review.always_checks -- the unconditional wall. Every ticket, every time. This is where lint, test, docs-build-strict, and the whole row of lint-* guardrails this post is about live.
review.soft_checks -- checks that report drift but do not, on their own, block the merge (today: docs-nav freshness, codemap-trace freshness, test-map freshness, blast-radius drift, dedup drift, and library-replacement drift).
review.client_changed_checks -- the bucket reserved for checks that should fire only when client code changes.

On top of the preflight verbs sit the review surfaces, documented in the workflow docs. A surface is a glob plus a rule pack: engine rules apply to the relevant engine module, server rules to the relevant server module, client rules to the private source tree. When a ticket's diff touches a surface, /dev-review pastes that surface's rule pack into a review subagent's prompt and categorizes findings into fix-in-place, escalate, or defer. The mechanism is generic; the rules are per-project, because "flag any non-stdlib import" only means something in a module that is supposed to be dependency-free. The server surface also carries triggers_security_review: true, so a change there independently spawns the security-review skill -- the mechanism that has historically caught the real bugs (a prompt-injection issue, an SDK base-URL pinning bug) that the rule packs alone did not.

The part worth dwelling on is how a check verb is allowed to exist. A CLAUDE.md hard rule -- "Process-contract changes must land atomically" -- says any ticket that adds, removes, or renames a preflight verb must update all four surfaces in the same ticket: the config bucket entry, the case arm in the project dev wrapper, the help block in the project dev wrapper, and at minimum one guard test under the workflow test harness. A partial update is itself a review-blocking defect, and there is deliberately no silent "skip if missing" fallback to tolerate drift. The reason is scar tissue: config/runner/help/test drift has, in this project's own history, blocked review preflight on perfectly healthy work. The contract that defines the gates is itself gated.

That is the frame for everything that follows. Each guardrail below is a verb in review.always_checks, backed by a script, and -- the recurring move -- paired with a standing CLAUDE.md rule that tells the agent to keep the surface current so the gate stays meaningful.

AI-first code annotations, enforced¶

Post #9 in this series, Code comments written for the robot, not the human, describes the annotation system in full: inline codemap: blocks with provenance tiers (auto facts the tool derives, ai semantic judgment an agent supplies, human policy a person owns), a closed vocabulary in the controlled vocabulary, and a hash: line that stamps the annotated unit so a stale judgment becomes machine-detectable. Read that post for the why of the shape.

This post is the and here is the gate framing. The annotation surface would be just another comment convention -- the kind that rots silently because nothing depends on it being current -- if not for two enforced halves that turn it into a guardrail.

The first half is a CLAUDE.md hard rule: "Annotations are part of the code -- update both in the same edit." Any edit to an annotated unit must re-stamp its ai:/human:/risk: judgment fields with the unit's new hash: in the same diff. The second half is the gate that makes the rule bite: lint-annotations, a verb in review.always_checks (and runnable directly via the project dev wrapper). It validates that every token comes from the closed vocabulary, that the auto: lines match what the annotator would regenerate, and -- the load-bearing check -- that no unit's hash: is stale relative to the code it sits above.

That hash-currency check is the thing conventional comments can never have: a mechanical way to know a comment is lying. A stale judgment is not a code-review nit you might catch if you are paying attention. It is a hard error that stops the merge. The comment cannot quietly drift away from the code, because the build will not let it.

Tests that declare what they are, enforced¶

Post #10, The test that knows what it is, describes the test taxonomy: the coarse unit/ / integration/ / e2e/ split plus a closed mark vocabulary (speed:, touches:, covers:, kind:, evidence:) that lets an agent select, sweep, and second-guess the suite mechanically. Again: read that post for the why.

The guardrail framing is the same two-part shape. The standing rule is "Test marks are part of the test -- update both in the same edit": any edit that crosses a taxonomy boundary must re-mark in the same diff. Adding a time.sleep() flips speed:fast to speed:slow. Swapping a handcrafted payload for a generated fixture flips evidence:handcrafted to evidence:fixture. Turning a behavior check into a regression backreference flips kind:behavior to kind:regression-pin and forces a pins-bug:<ticket-slug> mark.

The gate that makes the rule enforceable is lint-test-marks, a verb in review.always_checks (runnable via the project dev wrapper). It validates the mark vocabulary and the currency rules, so a test whose marks no longer match its body is a blocked merge rather than a label that quietly went stale. Why this matters for an autonomous build specifically: the automation layer selects tests by these marks -- diff-driven selection maps changed paths to touches: tokens and runs the overlap. Stale marks do not just look untidy; they make the machine confidently run the wrong tests. The evidence:handcrafted mark is the sharpest example -- it keeps agent-invented inputs visible to a sweep, and handcrafted inputs are exactly where an AI most easily encodes a premise the real system never produces.

There is a sibling gate worth naming here because it enforces the same instinct at the architecture level: lint-no-handcrafted-uistate (also in always_checks) fails any stage-component test that hand-builds a UIState object literal instead of deriving it from a server-shaped fixture. The whole family of checks is aimed at one failure: an agent testing its own imagination instead of the system.

Bug fixes must prove themselves¶

This is the strongest example of "make the agent prove the premise," and it is worth a concrete walkthrough because it is five gates composing into one contract.

The problem is specific to autonomous agents. Ask one to fix a bug and the path of least resistance is to write a test that encodes what the agent believes the bug is, make that test pass, and declare victory -- never once running the actual user repro against the actual server. The fix proves the agent's premise. The premise can be wrong. The bug is still there.

The project's defense applies to any ticket whose source: matches a bug-report heuristic (a whole-word match against bug, broken, regression, still fails, doesn't work, and friends, unless the ticket sets bug_heuristic: false). For those tickets, five gates fire:

lint-bug-ticket-shape is the upstream gate that makes the rest meaningful. It checks two things on a bug-heuristic ticket. First, the ticket actually carries the bug-report sections -- a root-cause hypothesis, observed-vs-expected, anti-patterns to avoid, and behavioral acceptance criteria -- so the agent has to commit to a falsifiable claim before it touches code. Second, a production-diff gate: if the branch changed only test files and zero non-test lines, the gate fails. You cannot "fix" a bug by editing tests alone, because a fix that touches no production code is, by definition, not a fix.
lint-bug-commit-order enforces that the first commit touching a non-test file lands before any test-only commit. You fix the production code, check out HEAD, re-run the user repro by hand, and only then -- once the fix demonstrably moves the symptom -- author the test that pins the corrected behavior. The lint walks git log <primary>..HEAD --reverse --name-only and fails if a test-only commit precedes the first code commit. (Squashing both into one commit is allowed; the check only fails when an earlier commit touches tests and no earlier commit touches code.)
lint-bug-repro-evidence requires a durable evidence artifact under the evidence directory -- at minimum one non-empty *.webm / *.mp4 / *.png / *.har / *.log / *.txt file. The preferred path harvests the real .runtime/ backend and client logs from the dev server across the repro window (before-*.log and after-*.log), so the artifact comes from the real server path, not from a test that might encode the wrong premise.
lint-bug-multi-device-spec-order is the client-specific one. For a bug-heuristic ticket whose diff touches the private source tree, the private source tree, or the private source tree, the first commit on the branch must be a multi-device Playwright spec, containing zero production-code changes, that fails against the pre-fix tree. The spec runs against a real dev-up backend -- mocked WebSockets, stubbed /room/create routes, and hand-crafted game_state props are all banned inside it. A spec that passes on a clean checkout is itself a review-blocking defect, because it proves the agent never actually reproduced the bug.
lint-bug-playwright-real-server closes the obvious escape hatch in the gate above. The multi-device spec has to run against a real server -- but an agent could still satisfy "a spec exists" with a spec set that quietly mocks the WebSocket layer. This gate scans the Playwright specs the bug branch added or changed and fails if every one of them uses a known mocking pattern (mock-socket, vi.mock, MockWebSocket, a class extends WebSocket, a hand-rolled new WebSocket("ws://localhost...")). At least one changed spec must exercise the real WS layer, so the proof of reproduction cannot turn out to be a proof against a mock harness.

All five are verbs in review.always_checks, each invocable directly (the orchestrator dispatcher, ... lint-bug-commit-order, ... lint-bug-repro-evidence, ... lint-bug-multi-device-spec-order, ... lint-bug-playwright-real-server) and each skipping silently on non-bug tickets. Each has a documented bypass (an env var plus a written justification paragraph in the run record) for the genuine-refactor case that trips the heuristic by accident -- and a bypass without the justification paragraph is itself a defect. Together they encode a single sentence the agent cannot talk its way out of: reproduce the failure for real, fix it, show the symptom moved, then pin it.

The always-on rules¶

The gates above are mechanical, but they sit on top of a layer of standing CLAUDE.md contracts that constrain every agent regardless of which ticket it is running. These are the cultural guardrails -- the ones enforced by being the first thing every agent reads, and in several cases by a lint as well.

Ticket-first: never hand-edit main from chat. Any change to repo content from a chat session defaults to drafting a ticket under the backlog, not editing main directly. This is the rule that keeps every change inside the lifecycle -- run record, review, dependency graph, retro -- instead of bypassing it. (This very blog post was written by an agent running a ticket, under exactly this rule.)
ASCII-only in code, commits, shell, and tickets. No character above codepoint 127 in anything the project touches programmatically, enforced by lint-ascii in always_checks. The reason is unglamorous and real: non-ASCII has already broken parsed workflow artifacts and shell tooling. A guardrail does not have to be clever to be load-bearing.
No backward-compatibility shims. This is a pre-production project with zero installed users, so the rule is: delete the old code, rename in place, update every call site in the same commit. Recovery is git revert, not a parallel codepath. It exists because an autonomous agent's instinct to "leave the old path just in case" is how a codebase silently doubles.
Doc-propagation. When an edit changes a named concept (a role name, a message type, a route), the rule requires updating every related doc in the same turn -- it explicitly overrides ticket scope. A concept rename is a global edit; stopping at the requested file leaves stale architecture guidance that the next agent then trusts.

The pattern across all four is the same one this post keeps returning to: the rule is stated where the agent will read it, and wherever it cheaply can be, it is also a gate. The rules constrain judgment; the gates catch the cases where judgment slipped.

The honest edges¶

I would be selling you something if I stopped at "and then the wall holds." Here is where it does not.

Gates check shape, not always correctness. lint-test-marks proves a test's marks match its body; it cannot prove the test asserts the right thing. lint-bug-repro-evidence proves an artifact exists and is non-empty; it does not -- and explicitly says it does not -- inspect the artifact's content, codec, or whether it actually shows the bug. The gates verify that the agent did the form of the right thing. Whether the form was filled with substance is still a human-review concern.

A determined wrong premise can satisfy every lint and still be wrong. This is the deepest edge. An agent that has convinced itself of a false model of the bug can write a real repro of the wrong thing, fix the wrong thing, harvest real logs of the wrong thing, and pass commit-order, evidence, and multi-device-spec gates -- all green, all wrong. The gates raise the floor dramatically; they do not raise the ceiling to certainty. They make the wrong premise expensive and visible, not impossible.

The rule wall has a real cost, and it can ossify. Every gate is code someone authored, tests someone maintains, and a help block someone keeps current -- the atomic four-surface contract above is itself overhead. A wall of forty checks is forty things that can false-positive on healthy work and block it (the project has the scar tissue to prove it). Rules that stop finding bugs but keep firing become tax, and the retro is supposed to retire them -- but a rule that feels safe is psychologically hard to delete, and a guardrail nobody dares remove is how a process calcifies.

What the guardrails actually buy is narrower than "correct code" and more valuable than it sounds: they shrink the blast radius of a bad agent run. A bad run still happens. But it is reproducible, evidenced, annotated, marked, and recorded -- which means when it goes wrong, a human can see exactly where and decide what to do. That is the same thesis as the lifecycle post: this is an instrumented development system, not a self-driving one. The gates are the instruments. They do not replace judgment. They make the places where judgment is still required visible, bounded, and cheap to inspect -- which, with an autonomous agent on the keyboard, is the most you can honestly ask of them.

If you came here from the builder's-eye post, Tickets in, a game I can play out, this is the internals behind the gates that post mentioned but refused to open. And if you want the other half of the machine -- how a phase becomes tickets, how the dependency graph schedules them, how a run record becomes memory and a retro becomes a process change -- that is the companion deep-dive, The orchestration lifecycle: from phase to ticket to retro.

Or: subscribe for more posts on building autonomous AI development infrastructure.