What a numeric risk dashboard would have to be, and why I have not built it yet¶
The companion post, Watching code risk trend over time, with no metrics datastore, used to end on a deliberate cliff. It showed that because the codemap: annotations live inline in the source they describe, git history is already a risk time series -- you replay commits, re-read the blocks, and roll up the counts, with no datastore. Then it admitted the one thing it could not do yet: render the numeric risk and complexity scores as a chart. That gap is now closed. The categorical trend still ships as the weight-independent baseline, and the numeric dashboard now renders a ruler-aware line instead of pretending old scores and new scores are always comparable.
This post is about that cliff and the implementation that finally crossed it. The useful lesson was never "draw a line chart." It was the measurement question hiding underneath: how do you draw a weighted score over time without turning a scoring-rule change into fake code movement? The answer stayed grounded in machinery that was already in this repo. The single transferable habit, stated up front so you can carry it past this codebase: separate the weight-independent facts you can chart freely from the weight-dependent scores you must version-pin or recompute.
The direction the cliff points at¶
The categorical trend already supports two cuts, and they are the right two.
By functional area. Every codemap: block tags its unit with capability tokens from the closed vocabulary in the controlled vocabulary -- engine, server, auth, ws, hidden-info, view-filter, and the rest. So "is the hidden-info surface getting riskier as we add games?" is a rollup over that vocabulary, not a project you have to instrument. The HEAD-snapshot form ships as codemap-report, which buckets the current tree by area.
By module. Every block belongs to a file, so the same data rolls up by path. "Which module accumulated the most race-prone flags this month?" is a path rollup over the identical git-resident records. codemap-report buckets by component too; the file is the key, because the annotation lives in it.
A numeric risk dashboard is those same two cuts -- by functional area over the capability vocabulary, by module over the file path -- but rendered over the numeric risk and complexity scores instead of categorical flag counts. The shape is identical. The data source is the same git history that the codemap subsystem already walks with git ls-tree and git show <rev>:<path>, no working-tree checkout. You would point that same walk at the numeric score instead of the flag counts and group by area, then by module.
If the shape is identical and the walk already exists, why did this need a separate implementation step? Because the numbers being charted are a different kind of thing from the counts, and one property of them breaks the whole exercise if you ignore it.
The hard problem: the ruler changes length when the weights change¶
The categorical counts are facts read out of the files. A hidden-info flag is in the block or it is not; counting it does not depend on any policy. That is why the categorical trend is honest the moment you compute it: nothing about how you score can invalidate a count.
The numeric risk and complexity scores are not facts. They are outputs of a formula. the codemap scorer takes the categorical inputs -- the capabilities, the maturity label, the affinity level, the coverage marks, the blast radius from the dependency graph -- and multiplies them by a large table of weights in the codemap subsystem: capability weights, severity values, maturity multipliers, blast factors, coverage-discharge multipliers, risk bonuses. The score is the weighting. Today that table is at version: 2, and it will move again -- the weights are hand-tuned judgment about what is dangerous, and the catalog of shipped changes keeps teaching us where that judgment was wrong.
Here is the trap, and it is the whole reason this was a research direction before it was a dashboard. Suppose you persisted the numeric score the naive way: one row per commit, just the float, drawn straight onto a line chart. The day someone tunes weights.json, every old point and every new point stop being comparable. The line jumps -- not because the code got riskier, but because the ruler changed length. A naive numeric series cannot tell that discontinuity apart from a real risk change. You would be charting your own weight edits and calling it code health. That is not a rough edge you can ship around; it is a measurement error baked into the axis. The shipped dashboard handles that by making segment mode the default and keeping recompute available as the cleaner but more expensive option.
The categorical trend never has this problem, because counting flags does not depend on any weight. The numeric series has it acutely, because the number is the weighting. Same git history, same two cuts, but the numeric version inherits a constraint the categorical version is immune to.
The two honest fixes, and the stamp that makes either possible¶
There are exactly two correct moves, and score.py is already built to support both. Every score it emits carries a provenance block:
def scorer_provenance(weights: dict[str, Any]) -> dict[str, Any]: return { "weights_version": int(weights["version"]), "weights_hash": weights_hash(weights), "scorer_version": SCORER_VERSION, }
weights_version is the human-bumped integer in weights.json. weights_hash is a sha256 over the canonically serialized weights content, so a whitespace-only reformat leaves it unchanged while a real numeric edit moves it -- the hash tracks the meaning of the weights, not their byte layout. scorer_version is a constant in score.py that bumps when the formula itself changes, independent of the weights. With all three stamped on every point, a numeric series is no longer a bare float per commit. It is a sequence of (risk, complexity, weights_version, weights_hash, scorer_version) tuples, and that changes what an honest dashboard can do:
- Segment. Draw separate line segments per distinct
(weights_version, weights_hash, scorer_version). The chart shows where the ruler changed, instead of pretending it never did. A reader sees "the definition of risky moved here" as a labeled boundary, not as a mystery spike. - Recompute. Because the score is a pure function of git-resident inputs plus a known weights definition, you can recompute the entire history under one current ruler and get a single comparable line. The provenance stamp tells you which points already match the current ruler and which need recomputing.
Either move is correct. A naive persist-the-float series can do neither, because it threw away the one piece of information -- which ruler measured this point -- that the whole problem turns on. The fix is to refuse to treat a persisted score as a source of truth and treat it instead as a recomputable cache stamped with the exact scoring definition that produced it.
This is the same discipline the Code comments written for the robot, not the human post argues for at the annotation layer: a hash: stamp that makes a stale judgment machine-detectable instead of a silent lie. Here the stamp is (weights_version, weights_hash, scorer_version), and what it keeps honest is a numeric trend instead of a single annotation. Same idea -- version the ruler separately from the readings -- one level up.
What is shipped, and what is not¶
I want to be precise, because the honest split is the point.
The provenance stamp is in place. Every numeric score score.py emits already carries the (weights_version, weights_hash, scorer_version) block, via scorer_provenance.
The persistence layer landed. The named follow-up the companion post pointed at shipped: the history-index implementation collects provenance-stamped numeric samples into the codemap_numeric_samples table in the disposable .state/history.duckdb index, carrying risk and complexity alongside weights_version, weights_hash, and scorer_version. It samples every commit that changes a scorer input plus daily snapshots, exactly so the provenance boundaries stay exact without scoring every commit in the repo.
The dashboard now exists. codemap-dashboard renders categorical counts plus numeric risk and complexity over PR-cadence or hourly axes. The default numeric mode is segment: it reads the provenance-stamped rows from .state/history.duckdb and splits the SVG paths whenever (weights_version, weights_hash, scorer_version) changes.
The recompute mode remains available but is not the default. codemap-dashboard --series numeric --numeric-mode recompute scores displayed historical snapshots under the current the codemap subsystem ruler, which is the cleanest comparable line. On this repo it is expensive enough that the default dashboard path uses segmented persisted samples instead of recomputing the world on every page render.
The hard part was never plotting points -- any library does that. The hard part was the measurement question: how do you draw a numeric risk line that does not lie when the definition of "risky" moves underneath it? The dashboard now picks a default answer, labels it, and preserves recompute for the cases where paying the scoring cost is worth the single-ruler line.
Why this generalizes¶
The reusable idea is smaller than this repo and outlives it: put the metric where the thing it measures lives, and version the ruler separately from the readings. Co-locating the categorical risk data with the code turned git into a free time-series store, and both cuts -- by subsystem and by module -- fell out of a group-by instead of a pipeline. That is the categorical half, and it is honest the moment you compute it.
The numeric half adds one rule you cannot skip: a weighted score is only comparable across time if every point knows which weighting produced it. Stamp the score with its scoring definition, and you can either segment the chart at every ruler change or recompute the whole history under one ruler. Skip the stamp, and your trend line is a chart of your own weight edits wearing the costume of code health.
So if you build something like this, ship the categorical trend first -- it needs no ruler and is honest from day one -- and do not let a numeric line out the door until every point on it knows which ruler measured it. This repo now takes the segmented path for the default dashboard and keeps the recompute path for explicit current-ruler inspection. The constraint is the lesson: the chart is only as honest as its treatment of the ruler.
Or: subscribe to the newsletter for more posts on building AI-first dev infrastructure.