Skip to content

A watchdog that catches AI agents going in circles

Last week I watched my own coding agent burn 30 minutes debugging a problem it had created for itself. The heartbeat detector said the run was healthy. Tool calls landed in the log every 10-30 seconds. The agent was busy. It just wasn't progressing.

This is how I learned that "is the agent still alive?" and "is the agent still making progress?" are two different questions, and that almost everyone I've talked to is only answering the first one.

The 30-minute spiral

The setup: my orchestrator launches Claude in a fresh git clone with a ticket as input. The agent works for 10-90 minutes depending on ticket size, then opens a PR. A cheap heartbeat detector watches each run's log file: if no new output appears for 1800 seconds, kill the process and notify me.

The ticket that day was a medium-sized refactor of a merge-conflict diagnostic library. Expected wall-clock: 30 minutes.

What actually happened: the agent introduced a Windows gh.cmd shim as a test fixture, hit a cmd.exe argument-reparsing problem on >/</comma-list args, and spent 30 minutes creating tmp_debug_mc.py, then tmp_debug_mc2.py, then re-invoking each one with slightly different arguments and adding more print statements. The actual library it was supposed to fix had been correct since minute 5. The spiral lived entirely in test-fixture land.

The heartbeat detector saw a perfectly healthy run. Tool calls every 10 seconds. Log file growing. Process responsive.

I caught it manually because I happened to ask "is that ticket doing anything?" If I'd been asleep it would have run another two hours before the stuck-threshold fired.

A 90-minute test loop

About a week after the gh.cmd case, a second spiral landed on a different kind of ticket.

The task: extend the orchestrator's batch-overlap check to count terminal_state=stalled runs as still-blocking -- so a queued ticket touching the same files as a stalled run would wait instead of launching. Medium bucket, expected 30 minutes.

The agent got the orchestrator logic right in the first half of the run. Then it got stuck on one test: test_batch_parallel.py::test_overlap_queuing_emits_queued_event. For the last 30 minutes of the run (which reached 90 minutes total), the agent executed that test variant roughly 15 times in succession -- read the output, edit orchestrator.sh, run the test again, tail the log, edit again. Nine of the last 20 tool calls in the log hit test.sh. Signal 2 (tool-call repetition) was firing clearly.

The agent also created debug_overlap.sh, debug_out.txt, and debug_overlap_check.py during those 30 minutes. Three files with debug_ prefixes -- the exact pattern Signal 1 watches for.

Both signals fired. The spiral warning was never pushed.

The reason: the spiral detector runs when dev.sh status is called -- specifically inside the status_active_runs function in lib-status.sh. During the run's soft-band window (roughly 45 to 90 minutes for a medium ticket), I was not actively polling status. The implement-stage watchdog hit its 5400-second hard cutoff before the detector got another chance to evaluate the log. The run is captured at the workflow test harness.

The recalibration work that followed (the ticket archive) added a fourth signal: sustained-bash-repetition (>=10 bash calls on the same target in the last 30 events). Signal 4 covers a pure test-debug loop where the agent never writes a throwaway script -- it just reruns an existing test file. A no-tmp-file variant of this spiral would have tripped only Signal 2, leaving suspicion at 1 (below the 2-signal threshold). Signal 4 closes that gap by making strong bash repetition alone sufficient to push suspicion past the threshold alongside Signal 2.

Why heartbeat detection misses this

Heartbeat is a liveness check. "The process exists, the log is being written, the OS thinks it's running." It catches three failure modes:

  • The process actually crashed but the parent didn't notice (Windows gh.cmd PID lying about its child, MSYS2 fork weirdness).
  • The agent is genuinely wedged on a syscall that never returns (rare but happens).
  • The agent is waiting for input that will never come.

It misses the failure mode that actually happens: the agent is busy but spinning. Reading the same file. Editing the same file. Running the same script with marginally different arguments. Producing log output that looks like progress but isn't.

You cannot catch this with a liveness check. The process is alive. You have to look at what kind of work it's doing and ask "is this work plausibly leading toward the goal, or is it self-similar to the work the agent was doing 10 minutes ago?"

Five cheap signals

I built a detector that combines cheap signals -- four to start, then a fifth once a real polling-spiral turned up that the first four missed. None of them require an LLM. None of them require parsing English. All of them run in milliseconds against a JSONL log file plus git status --porcelain.

Signal 1: tmp-file accumulation

Every spiral I've seen starts with the agent creating a throwaway script. tmp_debug.py, scratch_test.py, repro_v2.py, try_again_3.py. The pattern is unmistakable in retrospect: the agent has decided that the way to understand a problem is to write a small program that exhibits it, run that program, and iterate.

That's a perfectly fine instinct as a human. As an agent, it's a tarpit, because the agent has no automatic check that "this script is now answering the wrong question." So it writes tmp_debug_v2.py. Then tmp_debug_v3.py. Each iteration adds prints, tweaks args, and stays one cognitive step away from the actual fix.

The detector watches the working directory for files matching ^(tmp_|debug_|scratch_) or trailing _v[0-9]+$ / _test[0-9]+$. If two or more such files exist and none of the ticket's declared output files have been modified in the last 10 minutes, that's signal 1.

The "and" matters. If the agent created a tmp file but is also editing the real target, it's not spiraling -- it's doing legitimate exploratory work that happens to leave debris behind. The signal only fires when the debris exists instead of progress.

Signal 2: tool-call repetition

The agent's log is JSONL, one tool call per line. Look at the last 20 entries. If 8 or more are Bash calls executing the same script (modulo arguments), or 6 or more are Read/Edit/Bash against the same file, that's signal 2.

This catches the pattern of "run the script, see it fail, edit the script to add more prints, run it again." A legitimate build-watch loop wouldn't trip this signal because the script being run pre-exists in git ls-files -- the detector cross-checks against the repo's tracked files, so pytest, npm test, cargo run, etc. don't fire it. The signal only fires on scripts the agent itself created in this run.

Signal 3: file-mutation entropy

Snapshot git status --porcelain every 30 seconds. Track the set of touched-but-not-tmp files over time. If that set has stopped growing and the elapsed time has crossed 1.5x the expected wall-clock, that's signal 3.

This is the hardest one to tune because some real work consists of polishing one file (a README, a single function refactor). The signal works because it only fires in the soft band -- between expected and stuck thresholds -- when "I'm still polishing one file" is no longer a plausible explanation for the wall-clock.

Signal 4: sustained-bash-repetition

Added after the 90-minute test loop above. Signals 1-3 require either throwaway-file debris (Signal 1) or file-mutation stall (Signal 3) to push suspicion past the 2-of-N threshold; on their own, tool-call repetition (Signal 2) only contributes one vote. The batch-parallel test loop was a pure rerun-the-same-test-file spiral with no throwaway script, so Signal 1 and Signal 3 both stayed quiet and Signal 2 sat alone below the threshold.

Signal 4 closes the gap: if 10 or more of the last 30 events are Bash calls targeting the same script or test file, that's an independent vote on its own. The threshold is deliberately higher than Signal 2 (which fires at 8-of-20) -- Signal 4 is the "I am very sure this is a loop" backstop for cases where the agent never leaves debris behind. In the corpus, Signal 2 + Signal 4 together fire on the spiral-002 run and stay quiet on the matched healthy run.

Signal 5: verify-polling-loop

Added after a third corpus case -- and the one that finally pushed the count past four. The fix-tv-pair-qr-target-role-defaults-to-observer run (2026-06-03, captured as the spiral-003 corpus case) launched a background test correctly, landed its code change in the first 30 events, and then spent 30+ minutes in a tight cat/sleep loop reading the Bash-tool task-output file under /tmp/.../tasks/. The edit was already done. The polling was pure wasted wall-clock.

None of the first four signals caught it. There was no throwaway script, so Signal 1 stayed quiet. There was no file-mutation stall of the kind Signal 3 watches. And the command being repeated was a shell builtin (sleep) or a cat of a /tmp/ path -- not a tracked script -- so the script-name counters in Signals 2 and 4 never locked onto a single target.

Signal 5 matches on the full command string instead of an extracted script name. It looks at the last 20 Bash events and counts how many match a background-task polling pattern: sleep N for N up to 60, a cat or tail of a /tmp/.../tasks/*.output file, or an until ... grep / until ... [ -s ... ] wait loop. If 6 or more of those last 20 Bash calls match, that's signal 5. The 30-minute spiral above tripped it with six sleep 5 calls and six cat /tmp/.../tasks/ reads in its final window.

Signal 5 is still a cheap, deterministic check -- a few regexes and a counter, no model call. It is not the LLM-as-judge idea from the next section; it just happens to be the fifth pattern.

The soft-band timing model

I bucket every ticket into small / medium / big based on what the agent is being asked to do. Each bucket has an expected wall-clock and a stuck threshold:

Bucket Expected Soft-band start Stuck
Small (cleanup, small UX, small retro) ~20 min ~30 min ~35 min
Medium (new infra, AI variant, protocol + one surface) ~45 min ~70 min ~85 min
Big (new engine, multi-surface subsystem) ~90 min ~135 min ~2.5 hr

These numbers aren't guesses anymore. They come out of a recalibration over 708 successful runs that re-anchored every budget from human-engineer hours -- where the first estimates lived, and where they ran 10-20x too high -- to observed agent wall-clock.

Three regions:

  • Healthy (elapsed < expected x 1.5): run the signals, log them to a debug stream, do nothing.
  • Soft band (expected x 1.5 <= elapsed < stuck): run the signals, and if two or more trip, push one notification to my phone -- once per run, not per check.
  • Stuck (elapsed >= stuck): existing heartbeat-stale path takes over, and the spiral signals become hints in the kill record.

The soft band is the new behavior. Before this, the orchestrator did literally nothing in this region. Under stuck-threshold, no signal. Over stuck-threshold, kill. The whole "the agent is busy but spinning" failure mode lived in the gap between those two policies.

No auto-kill

This was the hardest design decision and I want to be explicit about why I rejected auto-kill.

An agent confidently spiraling for 10 more minutes is less dangerous than the orchestrator confidently killing a run that was about to succeed in 5 more minutes. The asymmetry matters: a wasted 10 minutes is annoying, but a killed-just-before-success run forces a relaunch which costs the entire prior run's work plus another 10-60 minutes.

So the watchdog is human-in-the-loop. It pushes one notification -- "run X may be spiraling: 12 of 20 tool calls hit tmp_debug_mc.py, 3 tmp files in the clone, 1.4x expected wall-clock" -- and I decide. Kill, inject guidance via ticket edit, or let it cook.

The notification is idempotent per run. Once it fires, a spiral_warning_sent=true flag is written to the run's state file. Re-tripping the threshold doesn't re-page. If the agent recovers, great. If it doesn't, I already know.

What I left out (deliberately)

The signal count grew from four to five, but the move people always ask about first -- spawning a smaller LLM (Haiku, GPT-4o-mini) to read the last 50 log entries and verdict them as "progressing" or "spiraling" -- I still haven't shipped. Signal 5 is not that. Adding Signal 5 cost a few regexes and a counter; it recognizes a polling loop without asking a model anything. The LLM-as-judge is a different, more expensive thing, and it remains deliberately unbuilt.

Reasons:

  1. The cheap signals keep catching the cases I've actually seen. Each new spiral so far -- the test-rerun loop, then the polling loop -- turned out to be expressible as one more cheap pattern, not a case that needed a model to recognize. Adding LLM-as-judge before I've found a spiral the cheap signals genuinely cannot express is solving a hypothetical.
  2. LLM signals are non-deterministic, which means false positives become a budget question ("the watchdog cost me $0.03 to tell me my agent was fine") and a debugging nightmare ("why did the watchdog fire on this run and not that run?").
  3. The cheap signals are testable against a corpus of recorded runs. The corpus now has three spiraling runs (the gh.cmd case, the batch-parallel test loop, and the verify-polling loop above) and one healthy-run counter-case. The detector achieves correct verdicts on all of them. That's a regression suite. An LLM signal isn't.

If a spiral shows up that no cheap pattern can express, that's when the LLM judge earns its keep. The last two times I thought I'd reached that point -- the test-rerun loop, then the polling loop -- a cheap new signal turned out to be enough. Until then, every tool I add to my agent infrastructure has to justify its existence.

The layer the watchdog catches residual cases of

The signals and timing model are the detection layer. Before a spiral can be detected, it has to happen. A separate layer tries to prevent the most common spiral causes at the source.

Claude Code skills are behavioral constraints that load at agent startup. The ones in this project most relevant to spiral prevention:

  • retro-seed-verification -- requires agents to re-read current code before transcribing a retrospective claim into a ticket. Without it, agents write tickets based on bugs that no longer exist, then spiral trying to find and fix something that is not there.
  • stay-in-current-repo -- restricts file writes to the current working directory. Agents that confuse sibling clones create inconsistent file states and spiral trying to reconcile them.
  • no-jq -- bans jq from shell scripts and forces Python or DuckDB for JSON parsing. Without it, agents that hit a parsing problem tend to write tmp_parse_check.sh workarounds -- direct Signal 1 material.
  • tickle-vscode-scm -- after any git operation, touch .git/index so the IDE updates. Without it, agents run repeated git status / git log cycles trying to understand why the SCM panel is not reflecting their changes -- the repetition pattern Signal 2 catches.

CLAUDE.md load-bearing rules address structural causes:

  • CRITICAL: Never hunt the filesystem for executables -- prevents probe scripts (tmp_check_python.py, debug_find_pytest.sh) that pile up when a tool is not in the venv or PATH. Those probe scripts are exactly what Signal 1 detects.
  • CRITICAL: No /tmp/ writes from project scripts -- forces temporary files into repo-local scratch paths. A /tmp/debug_overlap.sh is invisible to Signal 1, which watches the clone directory. This rule keeps temporary files where the detector can see them.
  • CRITICAL: Minimum-diff UI bug fixes -- prevents compound fixes that touch more files than the reported bug requires. Extra file edits raise mutation entropy without matching commit progress -- the fingerprint Signal 3 looks for.

The control-plane layer lives at the orchestration control plane. Prompts like core.md and safety.md establish precautionary baselines: verify before implementing, check whether the work is already done, read the ticket scope before writing code. An agent that verifies its premise before starting is far less likely to spend 30 minutes debugging a fixture that was wrong from minute 5.

The watchdog's job is to catch whatever slips through all of this. In practice, most of the spirals in recent runs would have been blocked by one of the above constraints. The detector fires on the residual.

What this means if you're building agent infrastructure

If you're running long-lived AI agents, you probably have heartbeat detection. You probably don't have spiral detection. The gap between them is where almost all of my agent's wasted time has lived.

The detector is something like 200 lines of Python. Any team running agents in production can build the same thing in an afternoon. The hard part isn't the code -- it's noticing that "the agent is alive" isn't the same as "the agent is making progress" and deciding to instrument the difference.

If you build this, push notifications instead of auto-killing. The asymmetry of failure modes makes human-in-the-loop the right default.

In-repo only -- may not ship to public blog

Prior art (research after the fact). I built this watchdog and wrote the draft above before searching for prior art. Doing the web search afterward (May 2026), the underlying pattern turns out to be well-trodden -- the specific phrase "spiral detection" is rare and looks home-grown, but the failure mode and the cheap-signal approach are widely discussed. Captured here for in-repo reference; trim or drop in the public version.

Closest technique matches:

Closest framing matches (diagnostic-not-killing posture, soft-band partitioning, cost-driven motivation):

Production bug reports of the same failure mode in real frameworks:

What looks distinctive in this writeup vs. the public material I found:

  1. Three-region time model anchored on observed agent wall-clock x ticket-size buckets (small / medium / big with empirical thresholds from PRs #11-28) rather than iteration count or token budget. Most public articles use iteration count as the axis.
  2. Multi-signal AND-gate (>=2 of 3 cheap signals required before firing) rather than single-signal trip. Public writeups mostly trigger on one signal.
  3. "Spiral" naming and the explicit no-auto-kill stance with idempotent-per-run notification. The asymmetry-of-failure-modes argument doesn't appear cleanly in the public material; most production frameworks ship with hard iteration caps that auto-terminate.

Editor decision pending: drop this section entirely for public publication, OR convert to a brief "prior art" footnote linking to the strongest 2-3 sources (likely Modexa + Barun Saha + Pithy Cyborg) so the post doesn't read as if I'm claiming the loop-detection problem is novel.

Or: subscribe to the newsletter for more posts on building this stuff.