Why this project is a strong AI training substrate¶

Most repositories keep only the final answer. You can read the merged patch and maybe the tests, but the useful middle is usually gone: what the agent was asked to do, what it read, where it got stuck, which local rules it tripped, what review corrected, and whether the same kind of work stayed stable later.

This project keeps more of that middle than a normal codebase. It is not because the app is special. It is because the development loop is built from local, parseable artifacts:

Tickets say what work was requested, what files to read, what is in scope, what is out of scope, what paths may be touched, and what commands verify the result.
Run records sit next to tickets and record the outcome, dead ends, judgment calls, review corrections, context-manifest summaries, and optional audit pointers.
The codemap gives source files closed-vocabulary capabilities, risk flags, maturity, affinity, hashes, dependency edges, and review read-sets.
The orchestrator and Tempo write state, lint results, watchdog signals, audit events, and history-index rows around ordinary ticket work.

That is the reason this repo is a useful AI training and eval substrate. It does not just ask whether the final diff passed. It can ask whether the agent understood the local system before it acted.

The ticket is the prompt and the rubric¶

A ticket in this repo is a structured work contract, not just a paragraph of chat. The frontmatter carries state, source, priority, grouping, agent-routing hints, and prediction fields such as expected complexity, files, and minutes. The body carries the goal, why, dependencies, required context, scope, allowed touches, acceptance criteria, verification commands, and definition of rejected.

That gives later analysis objective questions:

Did the run read the files listed under ## Context before editing?
Did it stay inside ## Scope -- in and avoid ## Scope -- out?
Did it edit only the paths listed under ## Touches?
Did it run the verification commands from the ticket?
Did the actual file count and time match the prediction?

The same data also helps the scheduler. ## Depends on and ## Touches let the orchestrator serialize related work instead of launching colliding tickets in the same wave. The ticket is therefore both the model prompt and the machine rubric for checking the attempt.

Run records keep the result honest¶

Every completed ticket needs a sibling *.run.md file. The run record captures facts that git cannot show: surprises, judgment calls, dead ends, review corrections, project-rule changes, feedback-artifact changes, outcome, prediction echoes, and measured fields when available.

That matters because a clean final branch can hide a messy process. If an agent first used the wrong command, learned that a path was generated instead of canonical, or had review fix a missing section, the final diff alone would make the attempt look cleaner than it was. The run record preserves that distinction. For training data, this is valuable: a run that passed after review is not the same example as a first-pass correct run.

Run records also carry compact pointers into larger stores. The audit exporter can stamp audit_store_ref, audit_blob_hashes, and audit_event_count into frontmatter after staged audit artifacts are centralized. The committed markdown stays small, while later tools can still find the durable JSONL trajectory and content-addressed blobs when those artifacts are available.

Codemap makes the codebase legible¶

The codemap is the control plane for source context. Inline codemap: blocks record closed-vocabulary capabilities, human and AI judgment fields, risk flags, affinity, maturity, and a hash over the annotated unit. The closed vocabulary lives in codemap/vocabulary.json; agents are not supposed to invent near-synonyms such as a new capability token just because it sounds right.

The codemap tooling turns those annotations into useful machinery:

codemap/deps.py builds import, protocol, security, pure-engine, and test edges, then computes blast radius from incoming edges.
codemap/review_context.py emits a compact JSON read-set for changed source files, including changed units, pinning tests, dependency-reached files, warnings, and a byte/line size estimate.
codemap/maturity.py and the maturity snapshot flow separate raw churn from interface churn, so stable interfaces are not demoted just because their internals changed.

For eval, this creates concrete grounding checks. If a ticket touched a high risk view-filter path, did the agent read the related filtering and test surfaces? If it changed a source file, did it keep the annotation hash current? If review needed broader context, did the codemap read-set already point there?

The audit trail captures what survives review¶

The audit trail is the high-volume layer. The current contract records trajectory events with a fixed envelope: run_id, ticket_stem, stage, step_index, timestamp, event type, and payload. The type set is closed: prompt, model_response, tool_call, tool_result, lint_result, review_verdict, preference_pair, and reward.

Large or sensitive payloads go through the content-addressed store. Text is redacted before hashing and writing. Worker clones can stage JSONL under .state/audit/<run-id>/; the ship/merge path centralizes those staged files into the configured audit_store_dir, writes trajectories/trajectory-<run-id>.jsonl, and stamps the run record with compact pointers.

The important honesty point is what the project does not claim. A fresh clone may have many committed run records and no reachable audit-store artifacts. In that case the history index does not fabricate trajectory rows. Collectors for trajectories, lint_gate_vector, grounding, spiral_labels, and preference_pairs use committed audit pointers or local .state/audit staging when available; otherwise those derived tables can be empty. The producer-health manifest treats that as an insufficient-samples case for the audit-derived table, not as proof that every run has rich trajectory data.

That captured-vs-dropped boundary is what makes the data usable. The repo keeps compact durable joins in git. It keeps bulky trajectories out of git. When the bulky side is absent, the tooling records absence instead of pretending.

Lint gates become reward signals¶

The review stage runs local gates that encode project conventions. Examples include run-record shape, ASCII-only artifacts, DevBlog mirrors, doc tags, ticket-shape checks, prompt-builder golden coverage, direct-random bans, test marks, annotation currency, screen-layout coverage, and producer registration.

Those gates are useful during development, but they are also useful as labels. The audit lint-vector path records each configured review.always_checks gate as a lint_result event with the gate name, gate index, pass/trip booleans, exit code, source, and command text. The history index flattens those events into lint_gate_vector rows.

That gives a dense local reward surface. Tripping lint-devblog-mirror is a different failure from tripping lint-run-records or lint-no-direct-random. A model that clears the first-pass gate vector has shown it can obey this repo's written and executable rules before a review agent fixes anything.

The reward layer is intentionally bounded. run_rewards can combine known lint exit codes, first-pass review approval, review-iteration count, and a configurable post-merge stability window. If a historical run has no structured lint events, acceptance_known is false instead of guessed.

Tempo makes the control plane portable¶

Tempo is the Go port of the orchestrator. It is not a separate product story in this post; it is evidence that the control plane is being made portable.

The current Tempo tree already mirrors important surfaces: codemap commands and embedded defaults, lint-gate implementations, history-index support scripts, audit eval, audit-store sync, R2 round-trip verification, status and backlog queries, and audit centralization. For example, tempo/review.go writes lint_result trajectory events during review preflight, and tempo/audit_export.go centralizes staged audit artifacts into the same objects/ and trajectories/ store shape while stamping run records with the same pointer fields. Tempo's parity tests and golden files check ported command surfaces against expected behavior.

That matters for training because the substrate should not depend on one shell script forever. A portable orchestrator can carry the same ticket, run-record, codemap, lint, audit, and history contracts into other projects without copying this app's game code.

The game-authoring loop is the payoff¶

The project is a board-game platform, so the concrete long-term use case is game authoring. A verified game build leaves behind a structured episode: ticket, required context, code diff, tests, lint vector, review notes, run record, and sometimes audit trajectory. The next game can retrieve that episode as a worked example. An eval harness can replay a representative ticket against a candidate patch and compare the result with the historical run.

The near-term claim is modest: retrieval, few-shot examples, and private evals over the existing catalog can make future authoring attempts easier to judge. The rulebook-PDF-to-engine experiment is one narrow rung: ask whether a model can translate a specified rulebook and design contract into a candidate GameEngine plus tests, then grade it with the same local checks. A fine-tuned model that writes whole games is a future rung, not a current capability.

That is the flywheel:

Build a game through the ticket workflow.
Keep the verified work and the process trail.
Use the trail as retrieval, eval, and training material.
Improve the next authoring attempt.
Ship more verified examples.

The project is a strong substrate because every step leaves inspectable artifacts. The ticket says what should have happened. The codemap says what the agent should have understood. The lint vector says which conventions held. The run record says what really happened. The audit trail preserves the high-volume middle when the store is available. The history index makes the durable parts queryable.

That does not make the repo perfect training data. It makes the failures visible enough to learn from.

Or: subscribe for more posts on building autonomous AI development infrastructure that can inspect its own work.