Watching code risk trend over time, with no metrics datastore¶
In the companion post, Code comments written for the robot, not the human, I described the codemap: blocks I write above risky functions: small typed records addressed to the AI agent that edits the file next, not to a human who might read it. That post was about what the annotations are. This one is about what they are for once they pile up across history.
Here is the claim, and it is the whole post: because the annotations live INLINE in the source they describe, every commit is already a snapshot of the codebase's risk profile. So git history is itself a risk time series -- with no separate metrics pipeline, no datastore, no nightly job writing rows into a table. The trend was already in the repo the whole time. We just had to replay it.
Most "code health over time" tooling needs a datastore. This one does not.¶
The usual way to chart code health over time is to stand up a metrics pipeline: a job runs on every commit (or every night), computes some numbers, and writes them into a time-series store. Now you have a second system to keep alive, and -- the subtle part -- the history in that store is only as trustworthy as the pipeline was on the day each row landed. Change how you compute the metric and your old rows quietly stop meaning the same thing.
The codemap: blocks dodge all of that for one structural reason: the data is co-located with the code. The risk flags, the maturity label, and the affinity level for a file are sitting in a comment block at the top of that file, in git, at every revision. To reconstruct the trend you do not query a datastore -- you replay commits and re-read the blocks. The repo is the datastore.
That is what codemap-trend does. It walks a range of git history, and for each sampled commit it reads the inline blocks without checking out the working tree -- git ls-tree to list files at that revision, git show <rev>:<path> to read each one -- parses the codemap: block with the same parser the rest of the system uses (the codemap subsystem calls annotate.parse_existing_block), and rolls up the categorical counts. No working-tree mutation, no git checkout, no scratch state left behind.
The concrete command lives in the private project wrapper for this workflow.
Here is real output from this repo (trimmed; --format json is the default and emits the full per-sample objects):
```text date commit files risk_flags maturity affinity 2026-06-14 4033adc2932a 0 hidden-info=0,security=0,race-prone=0,wasm-target=0,view-filter=0 experimental=0,active=0,stable=0 high=0,medium=0,low=0 2026-06-15 73ddb71ac2dd 356 hidden-info=83,security=28,race-prone=33,wasm-target=11,view-filter=80 experimental=14,active=327,stable=15 high=240,medium=100,low=16 2026-06-16 42c6d77f682d 435 hidden-info=90,security=42,race-prone=52,wasm-target=11,view-filter=85 experimental=17,active=400,stable=18 high=263,medium=139,low=33 2026-06-19 5f8b1e2d5cb6 464 hidden-info=99,security=43,race-prone=56,wasm-target=11,view-filter=95 experimental=17,active=429,stable=18 high=281,medium=150,low=33 2026-06-21 704e57c8cf45 508 hidden-info=104,security=47,race-prone=56,wasm-target=14,view-filter=103 experimental=18,active=469,stable=21 high=295,medium=180,low=33
The 2026-06-14 row of zeros is not a bug. The annotation backfill landed on 2026-06-15; before that, no file carried a block, so the snapshot is honestly empty. The step from 0 to 356 annotated files in one day is the backfill landing. After that you can watch the real curve: hidden-info flags climbing from 83 to 104 as more games with private state land, view-filter from 80 to 103, the active maturity bucket growing as the catalog grows. Those are categorical facts read out of the files, not derived numbers. That distinction is about to become load-bearing.
Two cuts: by functional area, and by module¶
A single repo-wide count per day answers "is the codebase getting riskier overall." The more useful questions are narrower, and the inline data answers both because every block carries more than just the flag.
By functional area. The capability vocabulary in the controlled vocabulary -- engine, server, auth, ws, hidden-info, view-filter, race-prone, state-machine, and the rest -- is a closed set of subsystem tokens shared across the source annotations, the test taxonomy, dependency selection, and risk scoring. Because each codemap: block tags its unit with those tokens, "is the hidden-info surface getting riskier as we add games?" is a rollup over the capability vocabulary, not a project you have to instrument. The HEAD-snapshot version of this cut already ships as codemap-report --by area, and the history index stores per-unit capabilities as a queryable list column (codemap_units.capabilities in .state/history.duckdb, a DuckDB TEXT[] you can filter with list_contains(capabilities, 'hidden-info')). Replaying that same rollup across commits is the same git-walk codemap-trend already does, pointed at the capability field instead of the global counts.
By module. Every block belongs to a file, so the same data rolls up by path. "Which module accumulated the most race-prone flags this month?" is a path rollup over the identical git-resident records -- codemap-report --by component is the HEAD-snapshot form. You are not joining against a separate table keyed by file; the file is the key, because the annotation lives in it.
Both cuts fall out of one property: the categorical risk data is a function of git-resident inline fields, so any slice you can express over those fields -- by capability token, by path, by maturity bucket -- is reconstructable from history alone. The trend verb ships the repo-global cut today; the by-area and by-module slices are the same inline data read with a different group-by, and the HEAD-snapshot rollups (codemap-report) are already in the tree to prove the shape.
The trap: numeric scores are weight-dependent¶
Everything above is categorical -- counts of flags, maturity labels, affinity levels. Those are inputs to the risk model. They are never invalidated by a change to how you score, because they are facts in the files, not outputs of a formula.
The numeric risk and complexity scores are a different animal, and this is the one subtle lesson the whole post is built around.
Those numbers come out of the codemap scorer, which multiplies the categorical inputs by a big table of weights in the codemap subsystem: capability weights, severity values, maturity multipliers, blast factors, coverage discharge multipliers, risk bonuses. A hidden-info capability without a covering test contributes its severity (8.0 today); a stable, high-blast unit takes a maturity bump; an uncovered capability adds a coverage-gated term. The score is a weighted function of the categorical inputs.
So here is the trap. Suppose you persisted those numeric scores into a time series the naive way -- one row per commit, just the float. Then someone tunes weights.json (today it is at version: 2; it will move again as the catalog of shipped changes teaches us what is actually dangerous). The moment that weight change lands, every old row and every new row stop being comparable. The line on your chart jumps -- not because the code got riskier, but because the ruler changed length. That discontinuity is an artifact of the scoring definition moving, and a naive numeric series cannot tell it apart from a real risk change. You would be charting your own weight edits and calling it code health.
The categorical trend never has this problem, because counting flags does not depend on any weight. The numeric series has it acutely, because the number is the weighting.
The fix is to refuse to treat a persisted score as a source of truth, and treat it instead as a recomputable cache stamped with the exact scoring definition that produced it. Every score_change() result now carries a provenance block:
```python def scorer_provenance(weights: dict[str, Any]) -> dict[str, Any]: return { "weights_version": int(weights["version"]), "weights_hash": weights_hash(weights), "scorer_version": SCORER_VERSION, }
weights_version is the human-bumped integer in weights.json. weights_hash is a sha256 over the canonically serialized weights content (json.dumps(weights, sort_keys=True)), so a whitespace-only reformat of the file leaves the hash unchanged while a real numeric edit moves it -- the hash tracks the meaning of the weights, not their byte layout. scorer_version is a constant in score.py that bumps when the formula itself changes, independent of the weights. With all three stamped on every point, a numeric series is segmentable: you can draw separate line segments per (weights_version, weights_hash, scorer_version), or -- because the score is a pure function of git-resident inputs plus a known weights definition -- recompute the whole history under one current ruler and get a single comparable line. The provenance stamp is what makes either move possible instead of a silent lie.
What is shipped, and what is not¶
I want to be precise about what exists today, because the honest split is the point.
The categorical trend works now. codemap-trend is a real verb (the project dev wrapper, backed by the codemap subsystem); the output above is from this repo, not a mockup. It reconstructs the flag/maturity/affinity composition over any git range, with no datastore.
The provenance stamp is in place. Every numeric score the codemap scorer emits carries the (weights_version, weights_hash, scorer_version) block.
The numeric series got its persistence layer recently -- the named follow-up codemap-numeric-risk-series-persistence landed a provenance-stamped sample table (codemap_numeric_samples in the disposable .state/history.duckdb index, with weights_version/weights_hash/scorer_version columns alongside risk/complexity, populated by the history-index implementation). It samples every commit that changes a scorer input plus daily snapshots, exactly so the provenance boundaries stay exact without scoring every commit in the repo.
The dashboard now exists. codemap-dashboard renders the categorical trend and the numeric risk/complexity line over PR-cadence or hourly axes. The numeric default is segment: the dashboard reads .state/history.duckdb and splits the line whenever (weights_version, weights_hash, scorer_version) changes, so persisted scores never pretend to share a ruler they do not share. Recompute remains available as an explicit mode for scoring displayed snapshots under the current the codemap subsystem ruler, but it is expensive enough on this repo that it is not the default page-render path. If you build something like this, ship the categorical trend first -- it is honest from day one and needs no ruler -- and do not let a numeric line out the door until every point either shares one ruler or is visibly segmented by ruler.
Why this generalizes¶
The reusable idea is smaller than the tooling: put the metric where the thing it measures lives, and version the ruler separately from the readings. Co-locating the categorical risk data with the code turned git into a free time-series store and made two whole categories of slicing -- by subsystem and by module -- fall out of a group-by instead of a pipeline. And separating the weight-independent inputs (counts you can trust across any scoring change) from the weight-dependent outputs (scores you must stamp or recompute) is what keeps the trend honest when the definition of "risky" inevitably moves.
The companion post, Code comments written for the robot, not the human, covers the annotation surface itself -- the provenance tiers, the closed vocabulary, the hash-currency lint that makes a stale judgment a build error. This one was about what those blocks become once history accrues: a risk time series that was hiding in the commits all along.
Or: subscribe to the newsletter for more posts on building AI-first dev infrastructure.