Building an audit trail for AI-assisted development

The audit trail is not one log file. It is a chain.

In this project the chain starts before a worker agent ever sees a ticket. A chat produces a ticket. The ticket enters the backlog. The orchestrator runs an implementation. Review checks the branch. Merge or failure closes the run. Each step can teach a future model something useful, but only if the project keeps the evidence in a shape that can be joined later.

The useful unit is still the trajectory triple:

1. What context did the model have?
2. What did it choose next?
3. What happened after that choice?

The old version of this post mostly described the implementation run. That was too narrow. It skipped the authoring leg, overstated what failure capture kept, and described the history index as if it flattened everything into one vague vector. The current system is more concrete than that.

It has a chat-to-ticket authoring capture, a per-run trajectory capture, terminal centralization into a durable store, and dedicated DuckDB tables for the parts that analytics and evals actually query.

The first leg is authoring

A ticket is not just an input prompt. It is the product of a conversation. The owner asks for work, the agent reads the authoring rules, maybe splits scope, wires dependencies, and writes a markdown ticket. If the ticket is malformed, over-broad, missing dependencies, or dishonest about scope, the downstream implementation run is already damaged.

That makes the authoring leg part of the audit trail.

The implementation lives in workflow/audit/authoring.py. When authoring capture is enabled, the hook looks at staged ticket markdown files. Added tickets and modified tickets are handled differently:

• For an added ticket, it selects a JSONL chat transcript from the configured transcript root, writes the redacted transcript text through AuditBlobStore, writes an authoring/authoring-<ticket-stem>.jsonl sidecar in the audit store, and stamps three pointer fields into the ticket: authoring_audit_ref, authoring_audit_blob_hashes, and authoring_audit_event_count.
• For a modified ticket, it records an operator_ticket_edit row with a redacted diff summary: changed sections, changed frontmatter fields, line counts, and hunk count. It does not store the full edited ticket body as a second raw transcript.

The sidecar records metadata such as the ticket stem, transcript session id, capture timestamp, blob hashes, byte count, and truncation flag. The ticket frontmatter keeps only compact pointers. That is the same design as the run trajectory: large text stays out of git, while the committed artifact keeps a join key.

This capture is deliberately best-effort. Disabled config, a missing transcript directory, an empty transcript, or a store error does not block the commit. The hook emits a compact message and moves on. That is important because audit producers are not merge gates in this project. They must be observable when broken, but product work should not be held hostage by a recorder.

What is captured: the redacted authoring transcript when the hook can resolve one, or a compact operator-edit summary for staged ticket edits.

What is dropped: raw chat text in git, raw transcript text in sidecars, secrets matched by the store redactor, and any transcript the hook cannot find. Missing authoring capture is absence, not something the system fabricates later.

The second leg is the implementation run

Once a ticket runs, the worker has a different shape of evidence. It reads the ticket, project rules, context files, source files, and tool output. It chooses commands, patches, tests, and run-record prose. The tools and review gates answer back.

The run trajectory code is in workflow/audit/trajectory.py. It normalizes agent-log records into closed event types:

• prompt
• model_response
• tool_call
• tool_result
• lint_result
• review_verdict
• preference_pair
• reward

The closed set matters. If every feature invents a new event type, the audit trail becomes another pile of logs. Stable event names let downstream tools write ordinary queries: show me first-pass lint trips, show me review verdicts, show me tool calls that opened files, show me rewards for merged runs.

The payloads are intentionally compact. Small redacted values can stay inline. Large text moves through AuditBlobStore and becomes a blob pointer with a sha256, byte count, and media type. The redactor handles authorization headers, API-key shapes, provider error bodies, and role-private values such as hidden cards. The history index is a catalog of pointers and summaries, not a database of raw prompts.

For example, a Tempo parity ticket can produce ordinary tool events for file reads and verification commands. Review can then add lint_result events for each review.always_checks gate. If tempo-check trips before the reviewer repairs anything, the trail keeps that first-pass failure instead of letting the final clean diff erase it.

That is the point of the run leg. The final patch says what changed. The trajectory says whether the model found the right files, used the right wrapper, tripped a local contract, recovered, and wrote the required receipt.

Batch preflight is its own failure surface

Some failures happen before there is a worker clone.

A backlog batch can reject a ticket because frontmatter is malformed, a group is invalid, ## Depends on is not parseable, a predecessor is still draft, a predecessor is quarantined, or a top-N selection would cut the dependency graph in the wrong place. Those are not implementation failures. They are ticket authoring failures.

The recorder in workflow/audit/batch_preflight.py captures those as lint_result events with stage: batch-preflight. For a ticket-specific failure, ticket_stem is the failing ticket. For a batch-wide refusal with no single offending ticket, ticket_stem is the batch id and the payload uses scope: "batch".

The payload carries the check kind, outcome, pass/trip booleans, exit code, closed failure category, scope, redacted reason, and predecessor stem when a dependency caused the rejection.

That gives the project a direct authoring-quality query:

SELECT category, COUNT(*) AS failures
FROM batch_preflight_failures
GROUP BY category
ORDER BY failures DESC, category;

Without that event, "the agent wrote an unbuildable ticket" would disappear as stderr between batches. With it, invalid authoring becomes visible beside implementation quality.

Terminal capture is no longer success-biased

The old audit-trail story was too optimistic about failures. It implied that failed and recovered runs were captured as naturally as successful runs. That was not true until the terminal centralization path changed.

The shell orchestrator now drains staged audit artifacts before normal success, before failure, and before escalation. The success path calls the same centralization helper before state_terminate success. Failure branches call it before state_terminate failure. Split-style escalation calls it before state_terminate escalated. Batch preflight has a separate batch-level export path because those failures happen from the project root, before a worker clone exists.

There is one important distinction: terminal centralization preserves staged audit artifacts that exist. It does not manufacture missing provider events, missing stage logs, or missing authoring transcripts. If an implement stage dies before a harvester wrote a trajectory, the terminal export can only drain what was already staged. That is honest. The run state and run record can still show the failure, but the trajectory store cannot pretend it saw events it did not see.

The merge boundary remains stricter. Audit export failure at the merge boundary can fail the run because that is where the normal successful trajectory should be durable. Non-merge terminal export is best-effort so it does not overwrite the original failure cause.

So the accurate captured-vs-dropped rule is:

• Successful runs: staged trajectory data is centralized and the run record can be stamped with audit_store_ref, audit_blob_hashes, and audit_event_count.
• Failed or escalated runs: staged trajectory data is drained before terminal state when possible, but export failure is warning-level so the original cause stays intact.
• Batch preflight failures: batch-scoped staged events are exported from the project root.
• Missing capture inputs: absent transcripts, absent logs, disabled capture, or unharvested events remain absent. The system records pointers and counts only for data it actually has.

That boundary matters for training data. A missing event is different from a successful step with no error. The trail should preserve that difference.

The durable store is outside git

Git keeps tickets, run records, code, docs, and compact pointers. It does not keep raw prompts, model responses, tool output, or large blobs.

The durable store is configured by audit_store_dir. Its layout is simple:

<audit_store_dir>/
  objects/
    <first-two-sha-hex>/
      <remaining-sixty-two-sha-hex>
  trajectories/
    trajectory-<run_id>.jsonl
  authoring/
    authoring-<ticket-stem>.jsonl

During a run, a worker clone can write staging artifacts under .state/audit/. That clone is disposable. At the terminal or merge boundary, the orchestrator copies the staged trajectory into the durable store, dedupes blob content by sha256, and stamps the compact run-record pointer when there are artifacts to stamp.

That split keeps the repository reviewable. A reviewer can read the ticket and run record without pulling a giant provider transcript into git. A later eval exporter can follow the pointer and fetch the exact redacted blobs it needs.

An optional remote mirror can reconcile the local audit store to R2-compatible storage. The local store remains authoritative; the remote is a mirror and fetch fallback. Remote failures are non-fatal and visible, not merge-blocking.

The history index uses real tables

The old post described "vectors" too loosely. The current history index has dedicated tables, and each table has a different job.

tickets stores the authoring pointer fields: authoring_audit_ref, authoring_audit_blob_hashes, and authoring_audit_event_count.

authoring_audit catalogs the reachable authoring sidecar: ticket stem, sidecar path, blob hashes, event count, capture timestamp, and session id.

run_records stores the run-record pointer fields: audit_store_ref, audit_blob_hashes, and audit_event_count, plus measured run metadata such as model, tokens, tool calls, actual files, actual minutes, review iterations, and failure mode when available.

trajectories catalogs reachable trajectory JSONL files. It stores path, event count, schema version, stage, generated timestamp, and blob hashes. It does not store raw event lines or blob bytes.

batch_preflight_failures stores the pre-run authoring failures that would otherwise vanish before a worker run exists.

lint_gate_vector stores one row per (run_id, gate) from lint_result events. That is the first-pass convention signal: which local review gates tripped before review corrected the branch.

run_rewards stores the compact reward vector: known acceptance exits, acceptance pass status, first-pass approval, review iterations, post-merge stability status, and an overall score computed only from known components.

grounding compares files read in tool-call provenance with the run's gold edit set, using a captured patch when available and ticket touches as a fallback. That makes "patched a file the agent never opened" queryable.

spiral_labels stores weak negative process labels attached by the spiral watchdog to suspicious tool-call regions, such as repeated tool use.

preference_pairs stores review-correction pairs when the preference CLI emits them: rejected blob, chosen blob, correction text, and step metadata. It is usually empty, and that is valid. Empty is not a failed producer when no review correction pair was emitted.

The important design choice is that these tables are query surfaces, not the source of truth. Tickets, run records, trajectory JSONL, and blobs remain the source artifacts. DuckDB makes them searchable.

What this buys

The full chain lets the project ask better questions.

For authoring:

• Did the agent write a ticket that batch preflight could launch?
• Did it wire dependencies?
• Did later operator edits change scope, frontmatter, or acceptance criteria?
• Can a failed ticket be joined back to the conversation that created it?

For implementation:

• Did the worker read the ticket's context files?
• Did it use the project wrapper instead of hunting for executables?
• Which lint gates tripped on the initial review pass?
• Did review approve on the first pass?
• Did the run become unstable after merge?

For training and eval:

• Can a new model replay the same ticket from the same base commit and make fewer local-process mistakes?
• Can a training exporter select clean demonstrations and reject runs with missing context or review-heavy recovery?
• Can a process-reward model learn that editing canonical workflow/docs/ is a good move and editing generated site-src/ is a bad one?

That is why the audit trail is a chain. The final diff is useful, but it is only the last artifact. The ticket shows intent. The authoring sidecar shows how that intent became a ticket. The trajectory shows the model's path through the project. Review and reward tables show how the project judged that path.

Together they turn ordinary development work into replayable evidence: not just "what merged," but "what the model saw, what it did, what the project verified, and what the system honestly failed to capture."

---

Or: subscribe for the next posts on what researchers can do with this trail and why this project is a useful training substrate.