The project that remembers itself¶
Most project memory lives where the agent cannot reach it. The ticket was in Jira. The postmortem was in Slack. The reason a refactor was abandoned was in a pull request comment, or in one person's head, or in a standup nobody recorded. Then an AI agent lands in the repo and asks the only question it can answer from local facts: what does the code say right now?
That is not enough for autonomous work. A codebase is not just files; it is a trail of decisions, reversals, bug reports, failed attempts, verification commands, and judgment calls. If the agent cannot query that trail, it repeats old mistakes. If it has to scrape a web tracker or ask a human to remember, the loop is broken before it starts.
This project took a different path almost by accident: tickets are markdown files in the repo, and every ticket leaves a sibling run record beside it. The source of truth is not a SaaS tracker. It is the backlog, the ticket archive, the run-record files next to the tickets, and git history. That makes the project memory cloneable, reviewable, branchable, and most importantly for an AI agent, queryable.
The implementation is documented in the history-index docs. This post is the field report on why that shape matters: the repo builds a disposable DuckDB database from its own history, layers BM25 full-text search over tickets and run records, then derives deterministic local theme clusters from the prose. The payoff is simple: when the next agent wonders "have we hit this before?", it runs a query instead of grepping around in the dark.
The memory stays in markdown¶
The load-bearing choice is not DuckDB. It is the fact that the authoritative memory is plain text already committed with the code.
A ticket describes what was asked for: the goal, scope, acceptance criteria, context files, and verification commands. A run record describes what actually happened: surprises, judgment calls, dead ends, outcome, and wall time. The pair sits in the same directory with matching stems, so a future reader does not need an external ID mapping to join intent to execution.
That matters because project memory has two different readers. Humans need a narrative they can inspect in review. Agents need a structured enough surface to parse and query. Markdown with frontmatter and fixed headings is the compromise: readable as a document, regular enough for tooling. The run-record format in the run-record contract is not elaborate, but it gives the index stable fields like outcome, complexity, wall_time_estimate, ## Surprises / judgment calls, and ## Dead ends.
External trackers make that harder. Even when their APIs are good, the tracker is not part of the branch. A ticket edit does not show up in git diff. A run note cannot be reviewed in the same change that produced it. A clone does not carry the whole memory of the project unless the agent also has credentials, network access, and a second integration path. That is fine for many teams. It is a bad fit for a repo that expects autonomous agents to work from local context and leave auditable artifacts behind.
Keeping the memory in-repo flips the default. A ticket branch carries the code, the instructions, and the retrospective evidence together. The index can be rebuilt from scratch at any time because it is derived from files already in the tree.
The database is disposable¶
the orchestrator dispatcher rebuilds the local DuckDB history index from the repo's durable sources: ticket markdown, sibling run records, git history, optional cached PR JSON, and orchestrator state JSON.
The database is deliberately not canonical. Delete the generated history database, run history-refresh, and it comes back. That distinction keeps the system honest. If a query result looks wrong, the fix is not to hand-edit a database row; it is to fix the ticket, run record, or history source that produced it, then rebuild.
The schema is boring in the best way. tickets stores one row per ticket with fields such as stem, path, state, source, goal, why, touches_prefixes, and depends_on_stems. run_records stores one row per *.run.md, including ticket_stem, outcome, complexity, wall_time_minutes, surprises_text, dead_ends_text, and escalation_reason, plus the newer audit-trail columns started_at, ended_at, actual_files, actual_minutes, predicted_complexity, predicted_files, predicted_minutes, base_commit, model_exact, tokens_in, tokens_out, tool_calls, and review_iterations. commits and commit_files connect shipped work back to touched paths. prs imports cached pull request metadata when present. state_events gives the orchestrator's own run state a place in the same query surface. index_meta records rebuild metadata.
Those newer columns are the subject of the companion draft The run record that grades itself, which explains how the same project memory became predicted-vs-actual training feedback.
That gives an agent SQL over the project memory without making SQL the authoring format:
The concrete command lives in the private project wrapper for this workflow.
The query command opens the database read-only and accepts a single SELECT, or a WITH ... SELECT, through a simple guard. It is not an adversarial SQL sandbox; it is a local inspection tool. That is exactly the right posture for a derived database sitting beside the repo.
The more important point is that the index turns history into a joinable surface. "Which tickets touched workflow scripts and later reported preflight drift?" is no longer a scavenger hunt through commit subjects and markdown blobs. It is a query across tickets, run_records, commits, and commit_files.
Search is the first agent reflex¶
SQL is useful when you know the shape of the question. Most history lookups start fuzzier than that.
An agent usually has a symptom, a concept, or a half-remembered phrase: "preflight", "websocket reconnect", "stale fixture", "docs nav", "missing run record". That is why history-refresh also builds DuckDB full-text search indexes over selected prose columns. Tickets contribute title, goal, and why. Run records contribute surprises_text, dead_ends_text, and escalation_reason.
The agent-facing command is:
The concrete command lives in the private project wrapper for this workflow.
The output includes a stem, BM25 score, and snippet, ordered by score. That turns old run records into a ranked memory surface. Instead of running git log --grep and hoping the commit subject used the same word, the agent can search the prose written at the moment the work happened.
This is especially useful because run records preserve negative information. A commit says what landed. A run record says what failed first, what judgment call was made, and what was explicitly left alone. That is the information an agent needs before it repeats a dead end.
There is a subtle governance benefit too: search rewards writing good run records. If "Surprises / judgment calls" contains the real reason a design changed, that reason becomes findable later. If the run record is empty theater, future agents get empty search results. The index does not replace discipline; it makes the value of discipline visible.
Themes are local and deterministic¶
Keyword search answers "where did this phrase show up?" The next question is different: "what keeps recurring even when the words vary?"
That is the job of history-themes. During history-refresh, the index tokenizes run-record prose, builds normalized TF-IDF-style vectors, links records whose cosine similarity crosses a fixed threshold, and writes connected components to themes and run_record_themes.
The command is intentionally plain:
The concrete command lives in the private project wrapper for this workflow.
It prints a cluster id, size, mechanical label, and example stems. The label is built from top weighted terms, not authored prose. The cluster id is stable because it is derived from the sorted member stems. Rebuilding the same tree twice produces the same cluster ids, labels, and memberships.
That determinism is the point. This layer does not call an LLM, embedding provider, API, or network service. It does not decide that two tickets are spiritually related. It groups the local prose with a reproducible algorithm, good enough to tell a retro, "these run records keep talking about preflight drift," or "this cluster is really about websocket reconnect behavior."
It is not semantic search in the embedding sense, and the section above does not pretend otherwise. BM25 ranks shared words and the theme clusters group prose by TF-IDF cosine similarity; both are lexical, so neither connects "phone drops the game after backgrounding" to "suspended-app reconnect" when the words diverge. That honest gap is exactly what motivates an embedding layer rather than papering over it.
Closing that gap is the job of a fourth query surface, history-semantic, which sits alongside history-query, history-search, and history-themes. It embeds the same ticket and run-record prose through a configured provider and ranks rows by true vector similarity using DuckDB's array_cosine_similarity. It is deliberately the optional layer: it is gated behind embedding environment configuration, makes network calls, consumes provider quota, and asks DuckDB to download the vss extension the first time it runs. With no provider configured, history-refresh skips embeddings, prints one note, and every other surface keeps working. The cheap, local, deterministic, and disposable layers are the baseline that always works in a clone; history-semantic is the vector-ranking layer you opt into when shared terms are not enough. See the history-index docs for the provider config and skip behavior.
The agent rule changed¶
The system becomes real when it changes the agent's first move.
The project agent rules now say that history lookups start with tickets, run records, and the generated history index before falling back to git log. They name the exact verbs:
The concrete command lives in the private project wrapper for this workflow.
That rule is small, but it changes the loop. An agent investigating a repeated failure should first ask the memory surface, not the raw commit graph. If it knows the ticket stem, it reads the ticket and sibling run record. If it knows the concept, it uses history-search. If it wants recurring pain, it uses history-themes. If those do not answer the question, then git log and git show remain available as fallback.
This matters because git is a poor project-memory database. It knows snapshots and commit messages. It does not know which acceptance criterion was ambiguous, which browser scenario was skipped, which setup command failed, or why the reviewer accepted a partial result. The run record knows that. The index lets the agent find it.
What this buys¶
The practical payoff is not a dashboard. It is a better next edit.
Before touching workflow code, an agent can ask which prior tickets touched the orchestrator dispatcher, which run records mention preflight failures, and which theme clusters keep accumulating around runner drift. Before writing a docs ticket, it can search for past docs-build-strict failures and learn which links broke before. Before changing a game engine, it can find old run records about stale client fixtures and remember why the fixture regeneration rule exists.
None of that requires a central service. It works in a clone. It works offline after the DuckDB extension is available. It is reviewable because the source facts are markdown and git. It is disposable because the database can be rebuilt. It is agent-friendly because the commands are stable and the schema is documented.
The deeper lesson is that AI-first infrastructure is often less exotic than it sounds. The uncommon part here is not "we used DuckDB." It is that the project arranged its ordinary workflow artifacts so a machine could read them later: tickets as files, run records as siblings, fixed headings, documented verbs, and a rebuildable index. The memory was already being written. The index made it queryable.
That is the shape I want in agentic codebases: not a chatbot taped to a tracker, but a repo that carries its own memory and gives the next agent a first-class way to ask, "what happened last time?"
Or: subscribe to the newsletter for more posts on building this stuff.