Skip to content

The run record that grades itself

A normal audit trail answers what happened. This project needed a trail that also answers whether the work was estimated, routed, and reviewed well.

The basic artifact pair did not change. A ticket under tickets/ says what an agent was asked to do. A sibling .run.md file says what happened. Git carries the diff. The history index rebuilds .state/history.duckdb from those local files. That is already enough to debug many failures.

The new part is calibration. The ticket now carries optional predictions made before execution. The run record carries measured fields and review outcomes after execution. The history index turns both into columns, so the repo can ask ordinary SQL questions about forecast, cost, context, and review churn.

The Prediction Is Captured Before The Run

Prediction fields live on the ticket because that is the only honest place to write them. After a branch is finished, the diff has already biased everyone's memory of how large the work looked.

The current optional ticket fields are:

predicted_complexity: medium
predicted_files: 4
predicted_minutes: 45

workflow/docs/authoring/TICKET_AUTHORING.md defines the contract. The lint checks shape, not accuracy: predicted_complexity is low, medium, or high; predicted_files and predicted_minutes are non-negative integers. Wrong predictions are allowed. They are the data the scheduler and future ticket authors need.

The run-record contract in workflow/docs/authoring/RUN_RECORD.md echoes those fields into the run record when they exist. That echo is deliberate. The run_records table can answer calibration questions without joining back to the ticket table:

SELECT
  ticket_stem,
  predicted_minutes,
  actual_minutes,
  actual_minutes - predicted_minutes AS minutes_delta
FROM run_records
WHERE predicted_minutes IS NOT NULL
  AND actual_minutes IS NOT NULL
ORDER BY abs(minutes_delta) DESC
LIMIT 10;

That query is useful because both halves are recorded at the right time: expectation before the work, measurement after the work.

The Run Record Carries The Measured Outcome

Tempo and the workflow tooling stamp measured fields into run records instead of asking the implementer to guess them. The current run_records schema in workflow/scripts/history_index/schema.py includes:

  • started_at and ended_at
  • actual_files and actual_minutes
  • predicted_complexity, predicted_files, and predicted_minutes
  • base_commit and model_exact
  • tokens_in, tokens_out, tokens_total, provider-native token fields, and normalized token_* fields
  • tool_calls and review_iterations
  • failure_mode, production_diff_lines, escalation_reason, surprises_text, and dead_ends_text

The measured fields are not all equally available. A degraded harvest may miss token usage. A legacy run record may not have timestamps. A fresh clone may not have local sidecars under .state/. The contract prefers nulls and explicit omissions over invented values. That matters because this data is meant to train future decisions, not make a dashboard look complete.

The same distinction applies to time. wall_time_estimate still exists as a fallback for older records, but current timed runs use started_at and ended_at as the measured source. Retros can still query old rows, while new rows do not have to parse prose to know how long the run took.

Context Is A First-Class Signal

The run record says what happened, but training also needs to know what context the agent had when it acted.

There are two surfaces for that. During a run, the context-manifest harvester can write a gitignored sidecar such as .state/runs/<run-id>.<stage>.context.manifest.json. The history index ingests those sidecars into context_manifests when they exist. The current table stores the run id, stage, sidecar path, generated_at, source_log, files_read, token rollups, tool_calls, base_commit, git_ref, and file freshness fields.

The sidecar is useful local telemetry, but it is not the durable review artifact. The committed artifact is the run record's ## Context manifest section. That section carries a compact summary, for example:

## Context manifest
- files_read: 12
- tokens_in: 18234
- tokens_out: 2410
- tokens_total: 20644
- tool_calls: 19
- base_commit: abc1234

If the harvester cannot parse the stage log, the section records that instead of pretending it has data:

- harvest unavailable: stage log missing

workflow/scripts/history_index/context_manifests.py also indexes committed summary sections when they expose numeric fields. Sidecar rows win for the same (run_id, stage) pair, and summary fallback rows make fresh review clones queryable even when .state/ is absent.

Review Corrections Are Not Implementation Work

The run-record contract also has a ## Review corrections section. That keeps a clean boundary between the first implementation and what review repaired.

Without that boundary, the final diff hides an important training label. A branch that passes only after review fixes a stale heading, missing evidence, or wrong field name is different from a branch that passes on the first review iteration. Both may end with outcome: merged, but they should not teach the same lesson.

The numeric companion is review_iterations. The prose companion is ## Review corrections. Together they let future analysis ask whether a ticket shape, model route, context size, or rule surface tends to need review repair.

The Useful Queries Are Boring

The point is not clever SQL. The point is that the facts now exist in stable columns.

To find underestimated work:

SELECT
  ticket_stem,
  predicted_files,
  actual_files,
  predicted_minutes,
  actual_minutes
FROM run_records
WHERE predicted_files IS NOT NULL
   OR predicted_minutes IS NOT NULL
ORDER BY COALESCE(actual_minutes - predicted_minutes, 0) DESC;

To compare context size with review churn:

SELECT
  rr.ticket_stem,
  cm.files_read,
  cm.tool_calls,
  rr.review_iterations,
  rr.outcome
FROM run_records AS rr
LEFT JOIN context_manifests AS cm
  ON cm.run_id = rr.ticket_stem
 AND cm.stage = 'run-record'
WHERE cm.files_read IS NOT NULL
ORDER BY cm.files_read DESC;

To group outcomes by exact model when the runner harvested it:

SELECT
  model_exact,
  outcome,
  count(*) AS runs
FROM run_records
WHERE model_exact IS NOT NULL
GROUP BY model_exact, outcome
ORDER BY model_exact, outcome;

These are plain operations over local project memory. They do not require a SaaS tracker, a transcript search, or a human reconstructing intent after the fact.

The Feedback Loop Is Local

This is why the run record now grades itself. Not with a score, and not with a claim that the agent did a good job. It grades itself by preserving the fields needed to compare:

  • what the ticket author expected
  • what the run actually touched
  • how long it took
  • what context was read
  • which model ran
  • how much review had to fix
  • which failure mode, if any, was recorded

That is enough for a future agent to learn from the run without trusting memory. It is also enough for a human maintainer to improve the next ticket. If a class of tickets is always underpredicted, split it earlier. If a rule is always missed despite being in context, improve the implementation path. If a context manifest is constantly absent, fix the harvester without blocking product work.

The important design choice is modest: keep the source of truth in committed markdown, derive the DuckDB index from it, and let missing metadata stay missing. A durable training signal is not the same as a perfect log. It is the set of facts that would be expensive or impossible to recover later.

Or: subscribe to the newsletter for more posts on building this stuff.