Skip to content

The AI fluency trail guide

Two and a half months. That is the gap between the first commit in this project -- a board-game app scaffold -- and the day I wrote this. In that window, working solo with no prior web-UI experience, I merged roughly 1,300 pull requests to main (the latest is #1340). Each one was authored, checked against a wall of automated gates in a separate context, and reviewed before it merged -- by a pipeline of AI agents I built as I went.

I want to be honest about that number, because a raw count is easy to inflate. The repo has around 2,900 commits, but roughly 870 of them are pure ticket bookkeeping -- creating, editing, and archiving the markdown files the system runs on. That is paperwork, not work, and I am not counting it. The real throughput is the reviewed PRs.

And the PRs tell a story a single total hides. Here is the merge rate, week by week:

Week of Merged PRs
Apr 20 42
Apr 27 69
May 4 67
May 11 36
May 18 31
May 25 54
Jun 1 68
Jun 8 74
Jun 15 231
Jun 22 506
Jun 29 (partial, 3 days) 121

For the first eight weeks the rate sits between 30 and 75 PRs a week -- the pace of a human in every loop, reading every change. Then, in mid-June, it jumps almost 8x and peaks above 500 in a single week. Nothing about me got 8x faster in mid-June. What changed is where I was standing in the loop. That inflection is the whole story of this post, and it happens at a specific, nameable stage.

The climb from "better Google" to "a machine that builds machines" is not a leap. It is a trail, walked one surprised discovery at a time, in an order that turns out to be nearly universal -- I keep meeting other builders stuck at the exact camps I got stuck at. So this is a trail guide: eight stages, each folding in a cluster of "aha" moments, each grounded in a real artifact from this project, and each ending with the question that gets you unstuck when you have been camped too long.

One caveat: the early stages predate the project's artifacts, so I illustrate those from the journey itself. By Stage 5 the project stops being an example and becomes the exhibit.

Stage 1 -- The Oracle

AI as a better Google.

You stop typing keywords into a search box and start asking a model in full sentences. Then one day you stop asking and start instructing -- you tell it what you want, with context and a goal, and it does the thing. That is your first real prompt, and it changes the relationship: the model is no longer a reference book, it is a worker.

The plateau: you treat it as a lookup box, one shot at a time, and you get quietly burned by a confidently wrong answer, so you trust it less instead of asking it better.

Your next breadcrumb: What if I told it what I want, not just what I am searching for?

Stage 2 -- The Prompter

You produce real work, and learn the hard lesson about context.

You vibe something into existence -- a script, a page, a whole small project -- faster than you could have written it, even if it is messy and hard to reuse. Then you hit the wall every beginner hits: hallucinations. And you find the counterintuitive fix. The problem was not the model's intelligence; it was that you were talking to it in one endless thread, like a coworker with memory, until the context rotted. The fix is many small, fresh chats -- one clean context per task, thrown away when done.

This is the single most important lesson on the entire trail: context is a resource you spend, not a history you accumulate.

In the project: this beginner habit grew up into architecture. The whole orchestrator is built to hand each AI agent a small, purpose-built context and nothing else -- tickets are deliberately sized to fit one context window, and a wide job that would blow past that budget is split into a batch. If you want the same honesty boundary told from the keyboard rather than the architecture, Shipping a React client I have not read is the account of driving an AI-built frontend through tickets, gates, and review without pretending to have learned the code.

The plateau: you hand-craft every prompt from scratch, every time.

Your next breadcrumb: Why am I still typing these by hand -- can the AI write the next prompt?

Stage 3 -- The Customizer

You stop hand-crafting and start building tools around the model.

Four discoveries arrive close together, and they are all the same move -- stop repeating yourself. You ask the AI to help write your next prompt. You save a good prompt as a slash command so you never rewrite it. You learn you can hand the tool a memory file -- a CLAUDE.md -- full of standing rules it reads on every task. And you start asking the AI to refactor not just your code, but your prompts and your rules.

Underneath that habit is a discovery that reframes what the rulebook is for. You start writing rules to stop repeating yourself; you keep writing them because the model gets distracted. It does not hold a to-do list the way a person does. Raise three loose ends in one reply, watch someone pick up the first, and the other two are not deprioritized -- they are gone, because nothing was ever holding them. Notice a second problem worth fixing while you are deep in the first, and by the next turn that noticing has evaporated too. The reflex is to tell it to remember harder, which never works: remembering is not something you can ask for, it is something you have to build. So you write the rule that makes remembering the machine's job instead of yours -- track every open thread and keep raising it until it is explicitly settled, write down every follow-up the moment you notice it instead of promising it for later, and when a whole class of these lapses shows up, kill the class with one rule instead of swatting each instance. The rulebook stops being a list of your preferences and becomes a set of guardrails bolted onto the model's own failure modes.

In the project: this stage is visible as raw scale. The repo has 32 slash commands and a memory file grown to 40 hard rules across ~1,600 lines -- with a sibling file, AGENT_RULES_HISTORY.md (another ~1,600 lines), that records the incident behind every single rule and why it exists. That second file is the tell: someone who learned to refactor their rules like code, complete with a changelog of what each rule defends against. A whole cluster of those rules exists to fight the distraction directly: one demands you "author every ticket you identify in the same turn -- never suggest-and-defer," so a noticed follow-up becomes a file on disk instead of a promise, and another makes you "propose the class-level prevention, not just the instance fix," so you stop patching lapses one at a time. A third -- an open-threads follow-through rule that forces the agent to re-surface every parked question until it is explicitly resolved -- is drafted and waiting in the backlog, not yet merged. From surprise to rule to skill walks exactly this graduation -- a one-off surprise becoming a durable rule and then a reusable skill -- and Don't fold it in is that same instinct distilled to a single habit: defer the second problem into its own ticket instead of smuggling it into this one.

The plateau: you have a drawer full of commands and a growing rulebook, but every task is still you, one prompt at a time, holding the steps together by hand.

Your next breadcrumb: Which sequence of prompts do I always run in the same order -- that sequence wants to be one thing.

Stage 4 -- The Agent Wrangler

You stop running steps and start running agents.

You build something that takes several steps on its own. You learn the models come in tiers -- a big expensive one for hard reasoning, a small fast one for grunt work -- and you deploy them deliberately. And you learn to lift a prompt out of one chat and carry it to another as a portable spec, usually with a little front-matter header describing what it needs and produces.

In the project: tiering is now a field on every unit of work. Each ticket carries front-matter hints -- agent_tier, agent_effort, agent_context, even agent_engine -- that let the orchestrator route each job to the right-sized model. The portable-prompt-with-front-matter instinct became the ticket format itself.

The plateau, and it is the big one: your multi-step agent works, until it does not. When it fails, you have no idea why. It made a different choice than last time on the same input. You cannot reproduce it, you cannot debug it, and you have started re-running it and praying.

Your next breadcrumb: Which of these steps actually needs judgment -- and which am I handing to the LLM only because it is there?

Stage 5 -- The Pipeline Engineer

You stop hoping and start engineering.

This is the longest, richest stage -- where using AI becomes engineering with AI. The core revelation: a reliable pipeline is mostly boring, deterministic glue, with the LLM reserved for the few genuinely soft decisions. Everything else follows from that:

  • A controlled vocabulary. Force the AI to pick from a closed list of allowed values and the whole chain gets dramatically more reliable. Free text is where pipelines rot.
  • Deterministic gates. Put mechanical checks between AI stages that reject anything off-spec, so a bad step cannot propagate.
  • Separated review. If the same context both writes the work and reviews it, the review is a rubber stamp -- so the reviewer gets a fresh, narrow context that never sees the author's rationalizations.

In the project this stage is not described, it is executed. The controlled vocabulary is a file: the controlled vocabulary pins a closed ~25-token list of capabilities plus fixed enums for risk, maturity, and failure modes, and a lint rejects any annotation that invents a token. The gates are real: about two dozen named lint-* checks plus the test suite and a strict docs build must all pass before any AI-written change can merge. And the separated review is architecture -- implementation runs in one throwaway clone, review runs as a distinct stage against surface-specific rule docs, so the honest critique happens in a context that did not write the code. The test that knows what it is is the vocabulary idea applied to tests; Review surfaces and the AI-first guardrails is the separated-review stage in detail; and When a stage fails: what is mechanical and what is the LLM? is the determinism-first split as a recovery loop.

The plateau: you have a beautiful, reliable pipeline -- and you are running it by hand on toy problems. It is a demo, not a system.

Your next breadcrumb: What real, recurring stream of work could I point this at, end to end?

Stage 6 -- The System Builder

You stop running the pipeline and let the pipeline run the work.

You wrap the pipeline around real, recurring work. Here that became a ticket orchestrator: you write a markdown ticket describing what you want, and the machine picks it up, spins a clean clone, implements it, runs the gates, reviews it in a separate context, and ships the PR. Work that was a day is now a paragraph and a wait.

In the project: this is where the weekly curve at the top of this post starts to climb. The orchestrator has run more than 1,200 tickets to date; the acceleration is not a metaphor, it is a merge history you can scroll. The orchestration lifecycle opens the box on phase-to-ticket-to-retro, and Tickets in, a game I can play out is what shipping through it feels like day to day.

The plateau: the system now produces faster than you can think. You are approving things you barely read, and the "done, awaiting your review" queue keeps growing. You feel busier than ever, and throughput has stopped climbing.

Your next breadcrumb -- the most important question on the trail: When the system stalls, who is it always waiting on? The answer is you.

Stage 7 -- The Bottleneck, and the Flip

You stop optimizing the machine and start optimizing the process -- for the machine.

Look back at the weekly table. The eight-week plateau at 30-75 PRs, then the jump past 500. That jump is this stage, and it has two parts.

First, the realization: the human is the bottleneck. The AI is not the scarce resource anymore -- your attention is. So you instrument the system to find where work piles up and route it, spending your limited human judgment only where it changes the outcome.

Then the heresy: you stop bolting AI onto a workflow designed for humans and start redesigning the workflow for the machine. Most people never ask this question. It is the difference between paving the cow path and drawing a new road, and it is exactly what un-plateaued the merge rate -- once a person no longer had to be present in every loop, the ceiling was the machine's, not mine.

In the project: the flip is everywhere once you look. A spiral watchdog watches every running agent for the signatures of a stuck loop and pings a human only when a run that is actually in trouble needs one. The time budgets that decide "is this run stuck?" are calibrated on observed agent wall-clock, not human hours -- a rule in the memory file literally warns that older human-anchored estimates were inflated ~10-20x. That correction is the flip: the process re-baselined around the machine's clock. Risk-routed review is the "spend human attention where it earns its keep" idea in full, and Spiral-detection watchdog is the watchdog itself.

The plateau: everything is instrumented, routed, machine-legible -- and it dawns on you that all this exhaust is the most valuable dataset you own.

Your next breadcrumb: I have been instrumenting to observe the system -- what if I instrument to teach one?

Stage 8 -- The AI-Native Architect (closing the loop)

The machine starts building the machine.

You now design artifacts for the AI first -- structured so a machine can parse and find them, not just so a person can skim. You obsess over findability, because an agent that cannot locate the right context cannot do the job. Then the last turn: the structured record you built to debug the system is exactly the record you would use to train one, and the loop closes.

In the project: the findability layer is the codemap system -- code carries inline annotation blocks (capability, risk, maturity, reachability) so an agent can mechanically answer "what does this touch and what breaks if I change it" -- backed by a ~24 MB queryable index of every ticket, run record, and commit, so project history is a database query instead of an archaeology dig. The ~1,300 run records are not development throughput; they are the memory: each is a structured, controlled-vocabulary account of how a job actually went -- what surprised the agent, what dead-ended, estimate versus actual -- which is exactly what you would train the next orchestrator on. The project that remembers itself covers the queryable memory; The run record that grades itself shows that corpus becoming predicted-vs-actual training feedback; and Why this project is a strong AI training substrate is the synthesis. The loop-closing move is already visible: a from-scratch Go re-implementation of the whole orchestrator, told in Two rewrites, one strategy and The orchestrator leaves home. The system is being rebuilt, with the system's help, into something faster and portable.

The honest bit: I will not pretend this stage is mastered. This is the frontier, standing at the edge looking up. Which is the right place for the personal half of this guide to end -- here is as far as I have climbed. But the map above this line is not entirely blank. Other climbers and the wider field have been up there, and they left blazes. So before the summary, let me do the thing this whole post has done for you, in reverse: leave the breadcrumbs for me.

Above the treeline: the breadcrumbs left for me

There is a symmetry in how I gathered these. This whole post is breadcrumbs I left for people behind me on the trail. This section is the reverse -- breadcrumbs left for me, by the climbers ahead and by the wider field. Fittingly, I did not write it from memory: I asked an AI to scout the trail above my treeline and report back. An AI leaving a human breadcrumbs to climb by is, itself, the Stage 8 loop closing one more time.

Each one keeps the shape of the others: the discovery, why it is probably your next step from where you are standing, and the smallest first action. Sources for the field claims are collected at the end of the section.

Measure quality, not just rule-breaks. Your two-dozen-odd gates answer "did this break a rule?" They do not answer "is this any good, and is the pipeline getting better or worse over time?" The field's name for the missing layer is eval-driven development: write the eval cases first (input, expected trajectory, success criteria), grade with an LLM-as-judge calibrated against your own past review decisions, and gate merges on the score, not just pass/fail. You already own the hardest ingredient -- a golden set drawn from real failures -- because you have thousands of run records. First action: turn twenty of your ugliest past failures into scored eval cases and run them on every prompt change.

Your gates are already a reward signal. You named "train models to do the process" as the summit and then treated it as far away. It is closer than you think, because you are sitting on the exact ingredients: your deterministic gates are verifiable rewards, and your run records are trajectories. The field calls training on this RLVR -- reinforcement learning with verifiable rewards -- and its current state of the art is a hybrid of rule-based checks for hard constraints and an LLM for soft ones, which is word-for-word your Stage 5 philosophy. First action: export your passing trajectories as fine-tuning data and pick one repetitive, well-gated stage -- ticket triage, say -- to distill into a small, fast, cheap model.

Standardize the seams you hand-built. Your findability layer -- the codemap and the queryable history index -- is bespoke. While you were building it, the field converged on a standard for exactly this job: the Model Context Protocol (MCP), a common way to expose tools and context to any model or agent. It went from nonexistent to ubiquitous in under two years and is now stewarded by the Linux Foundation. First action: wrap history-query and the codemap as MCP servers, so any agent -- not just yours, not just one vendor's -- can read your project's memory.

Use more of the orchestration shapes. Your pipeline is a chain plus a separated review. The published pattern catalog names three more you probably underuse: evaluator-optimizer (a generate -> critique -> revise loop inside a single stage), orchestrator-worker (a lead agent that decomposes a task whose sub-steps it cannot predict in advance), and parallelization / best-of-N (sample several attempts, judge them, keep the best). First action: add a best-of-N-with-a-judge step to your highest-value ticket type and measure whether three cheap attempts plus a judge beats one expensive attempt.

Enforce the vocabulary at the moment of generation. At Stage 5 you close your vocabulary with a lint that rejects off-list tokens after the model emits them. The stronger version stops the model from emitting them at all: constrained decoding and schema-enforced structured outputs make the closed vocabulary a property of generation, not a post-hoc check. First action: move one closed-vocabulary field from "the lint rejects it later" to "the decoder can only produce legal values."

Context engineering is a real discipline now -- and it has a warning for you. You discovered "context is spent, not accumulated" by instinct at Stage 2. The field turned it into a named practice -- context engineering -- with real tooling for compaction and sub-agent isolation, and a counterintuitive headline finding: past a point, more context makes agents worse. There is also a caution aimed straight at your ~1,600-line rulebook: research on "governance decay" shows that when a long context gets compacted, safety and governance rules are silently among the first things dropped. First action: measure which of your 40 rules actually survive into each agent's working context, and stop assuming a rule in the file is a rule in the room.

Harden the pipeline itself. Your single best instinct -- "the server is authoritative, clients are untrusted" -- is exactly the instinct this step needs, pointed at a new target: your own agents. Prompt injection is now widely treated as a structural, possibly unpatchable flaw, and your pipeline feeds agents untrusted input (rulebook PDFs, web pages, ticket text) while handing them real tools. The defense is the one you already believe in: least-privilege tool scoping, and treating every retrieved document and tool output as data, never instructions. First action: pick one autonomous path, threat-model it as if its input were hostile, and scope that path's tool permissions to the minimum it needs.

None of these are summit flags. They are just the next blazes up the trail -- the ones I could not see from where I am standing, that other people already have.

Sources for this section

How to read the map

  • You are probably one stage lower than you think. Owning a stage's tools is not living in it. Having a CLAUDE.md does not make you a Customizer if it never changes how you work; building one agent does not make you a Pipeline Engineer if you cannot debug it.
  • The plateaus are the whole point. Nobody gets stuck because the next step is hard -- they get stuck because the current stage is comfortable and works, and nothing forces the move. Every breadcrumb above is the same shape: a question that makes the plateau visible, and the smallest action that ends it.
  • Two ideas do most of the work. Context is spent, not accumulated (Stage 2), and make it deterministic wherever you can, reserve the model for the soft decisions (Stage 5). Internalize those two and the rest is elaboration.
  • Climb for the problem, not the altitude. Every stage solves a specific pain. If you do not have the pain yet, you do not need the stage yet. This whole system exists because a board-game side project hit each wall in turn, not because a higher number is better.

The receipts are real, and they are the weekly curve at the top: a merge rate that sat flat while a human was in every loop, then jumped almost 8x the week the human stepped out of it. All of it started with someone deciding to ask a chatbot a question instead of Google. Watch for the breadcrumbs.