The test that knows what it is

A test that an AI agent can write is a test that tells the agent what it is. Not in a comment a human skims -- in a small, closed vocabulary the agent can grep, filter, and second-guess. Same for the logs the test reads, and same for the live UI state the agent dumps to find out what actually happened before it writes a single assertion.

This is the test-and-logging half of a two-part system. The other half -- code annotations addressed to the agent that edits the file next -- is the subject of the companion post, Code comments written for the robot, not the human. That one owns the source-annotation control plane: provenance tiers, the three-tier dependency graph, risk-vs-complexity scoring. This one is about the seam where an agent has to write and debug a test -- which turns out to need three things designed agent-first: a closed test taxonomy, a closed log-event vocabulary, and a UI-state dump aligned to that log. The owner's words for the last one were that it was "a huge help in test creation." That is the part I most want to explain.

The closed test taxonomy is AI-first

Most test-tagging schemes are a human filing system. Ours, specified in the overview docs, is built so an AI agent can reason about the suite mechanically without re-reading every file.

It has two layers. First a coarse structural split -- unit/ / integration/ / e2e/ -- which answers "how expensive and how broad is this test?" before the agent looks at any individual case. Then a closed mark vocabulary on top:

• speed:fast|slow -- the wall-clock budget (fast means under 100 ms in isolation).
• touches:<subsystem> -- which project subsystem the test exercises, from a fixed list (engine, server, auth, ws, db, protocol, shell, stage:<game>, lobby, recap, lint, docs). No ad-hoc touches:frontend.
• covers:<area>:<capability> -- the capability the test protects.
• kind:behavior|contract|invariant|regression-pin|property -- what kind of claim the test makes.
• evidence:replay|fixture|handcrafted -- where the test's input came from.

Because those tokens are a fixed, grep-able set rather than free text, an agent can do three things deterministically. It can do diff-driven selection: the project dev wrapper maps changed paths to touches: tokens and runs only the overlapping pytest and Vitest files. It can run drift sweeps: grep for marks that no longer match the body -- a speed:fast test that now sleeps, an evidence:handcrafted input that should be a fixture. And it gets wrong-mental-model detection nearly for free, because evidence:handcrafted keeps inline, agent-invented inputs visible to a sweep -- and handcrafted inputs are exactly where an AI most easily encodes a premise the real system never produces. Regression backreferences ride along too: a kind:regression-pin test must also carry pins-bug:<ticket-slug>, so a fixed bug's pin always points at the ticket that fixed it.

The bridge to the companion post is one token. The capability half of covers: is not free text -- it reuses the exact same capability vocabulary from the controlled vocabulary that the source-side annotations stamp on each unit. That shared word is what lets an agent join "which units have this capability" (from the annotations) to "which tests protect it" (from covers:) with a single token, instead of maintaining two parallel naming schemes that drift apart. The source annotations and the tests speak one subsystem language on purpose.

Closing the loop, a CLAUDE.md hard rule -- "Test marks are part of the test -- update both in the same edit" -- requires any edit that crosses a dimension boundary to re-mark in the same diff. Adding a time.sleep() flips speed:fast to speed:slow. Swapping a handcrafted payload for a generated fixture flips evidence:handcrafted to evidence:fixture. The mark is data the tooling can check, not a label that quietly goes stale.

The log words are AI-first too

A test is only as good as what it can read back. So the logs are a closed vocabulary the same way the marks are.

the server docs is the single catalog of every server and client event name, and it carries an explicit rule at the top: "Do not invent new event names without updating this file." That sentence is the whole design. A closed event vocabulary is what lets an agent treat the logs as a stable, machine-readable contract instead of best-effort printf output. The CLAUDE.md rule "UI logging is a test/debug contract" is the agent-facing statement of the same idea: browser-facing work must preserve or extend the documented events and then use those events in tests and failure artifacts.

Several events exist specifically so a machine -- or an LLM doing post-session audit -- can reconstruct what happened without replaying the engine:

• game.session_summary (INFO) -- emitted once per game at terminal state; the guide calls it "the primary anchor record for post-session LLM audit" (game id, outcome, action count, duration, players, referee-override / rewind flags).
• game.state_invariant (DEBUG) -- a compact structural snapshot per apply_action site for LLM post-session audit, deliberately carrying no hidden-information values.
• action.invalid_attempt (INFO) -- logged before the MSG_ERROR reply when a requested action is excluded from valid moves, explicitly for the LLM audit trail.

The spine that makes cross-device timelines line up is the causation_id. Action handling derives it as client-msg-<client_msg_seq>:server-seq-<seq> (the client segment is client-msg-none when there is no client sequence) and threads it through the whole round trip: client send, server action.apply, room.broadcast_state, the outbound state_update payload, and the client receive events. Because every hop carries the same correlation id, an agent can stitch one logical action across the TABLE, PLAYER, and REFEREE logs into a single ordered story -- which is exactly what the multi-device timeline specs assert against.

The vocabulary is closed in the other direction too: there is a secrets policy. auth.fail logs at most a 4-character token prefix; llm.config may log provider and model but never the key, a prefix, or a hash; broadcast.view_filtered exists to make hidden-information filtering observable while logging only a redacted_field_count and a view_shape_hash, never the hidden values themselves. A log that leaks the private hand it was supposed to protect is not a debugging aid -- it is the bug. Treating logs as a contract means the redaction is part of the contract.

Dumping UI state aligned to the log

Here is the part that earned the "huge help in test creation" line.

When you write a timeline assertion, the failure mode is guessing event names. You think the flow emits client.surface_rendered; it actually emits something adjacent; the spec is green against your imagination and red against reality, or worse, green against both and meaningless. The fix is to never guess -- to read the real events out of a real session first, and only then write the spec from that real data.

The browser gives you a surface for exactly this: window.__bgui, installed by the private source tree. It is dev/test-only -- installDiagnosticsGlobal is a no-op when import.meta.env.PROD is true, and a no-op in non-browser environments -- so it never ships to a production bundle. Its members:

Member	What it returns
state(stageName)	The current UIState for a mounted stage host, e.g. state('memory_match'), or null when that stage is not mounted.
events(n)	The last n ui.* events from the in-memory client log (ui.screen_rendered, ui.control_clicked, ui.event_dispatched). Secrets are already redacted by logClientEvent before storage.
screen	Current active screen name from the data-screen DOM attribute.
role	Current role, read from the most recent client log event.
route	Current pathname + search.
controls()	All data-control elements currently in the DOM.
simulate(event)	Dispatches an event into mounted stage reducers without a click.

The reason this is more than a debug toy: the ui.* events and the state(...) dump come from the same running session, so the agent can correlate "what the user or system did" (the events) with "what the UI now holds" (the state). The dump is aligned to the log. That alignment is the whole trick -- it turns "I think the component is in this state" into "here is the state, next to the event timeline that produced it."

From there the loop is mechanical -- the discover-then-specify loop documented in both LOGGING_GUIDE.md and the playwright-timeline-spec skill:

1. Dump real events. Run the flow against a real the project dev wrapper backend and read what actually fired -- getClientLog(page) inside a Playwright spec (from client/tests/playwright/_support/diagnostics.js), or window.__bgui.events() in the browser console.
2. Read them. Confirm the exact event names, the ordering, and the compact role-safe fields each event actually carries. The Client-side events table in the logging guide is the catalog you check against.
3. Specify the timeline. Encode the ordered (and forbidden) events as a named timeline in client/tests/fixtures/expectedLogTimelines.js, and assert it with expectLogTimeline(page, diagnostics, expectedTimeline).

A test built this way carries the evidence:replay mark from the taxonomy above -- and that is the point where the three pieces snap together. evidence:replay is a claim about provenance: the input came from a real recorded session, not from an agent's mental model. The taxonomy gives the agent a word for "this came from reality." The logging contract gives it a stable timeline to record. The aligned state dump is how it reads that reality back before writing anything. Tie those together and an agent can produce a regression test that pins the behavior that actually happened, instead of the behavior it assumed -- which is the failure that the evidence:handcrafted sweep exists to catch in the first place.

The other side of the same system

The taxonomy, the log vocabulary, and the aligned state dump are the test-and-logging half of one machine. The other half -- the annotations that tell the agent who authored each fact about a unit, how far to trust it, and whether it is still true; the dependency graph that knows about convention and execution edges, not just imports; and the risk number that is distinct from complexity -- is the subject of Code comments written for the robot, not the human. The bridge between the two halves is the shared capability vocabulary: the same token that an annotation stamps on a unit is the token a covers: mark uses to say it protects that unit. One language, two surfaces.

The hard part of either half was never the code. It was deciding that a test mark, a log event name, and a state dump should be typed, closed, and aligned -- so the agent writing the next test can read what the system actually did, and write a test that knows what it is.

---

Or: subscribe to the newsletter for more posts on building this stuff.