Turning an Agent Playbook Into a Script

Earlier this week we pointed a new pipeline at workout-tracker, one of our public repos, and let it choose its own next task. It passed over an authentication change that would have been destructive and had not been approved. It passed over an extra-large issue that nobody had planned. It chose the next eligible issue on the board, and then flagged something we had missed completely: the spec and plan for that issue existed only as untracked files in an old local worktree. Any agent starting from a clean checkout would have built the feature blind.

That run took five and a half minutes, and a script drove it, not a person steering a chat window. Getting there meant taking our old setup apart first.

About eight of our repos run on the same loop. We type /orchestrate, and an agent reads a handful of markdown files in docs/orchestration/, picks the next GitHub issue, does the work, opens a pull request, merges it on green CI and writes down what happened. We call one pass a "tick". The files come from our project template. In our largest repo, PLAYBOOK.md describes a tick in 569 lines of prose.

This week we moved that procedure into Claude Code dynamic workflows: JavaScript scripts that start subagents, decide what runs next and hold the results themselves. It took two days, and one of our estimates was off by almost a factor of three. If you run agents on a loop like this, most of what we learned will apply to you too.

Five versions of one command

Before changing anything, we had a separate agent review the whole setup across every repo. Its first finding: the /orchestrate command existed in seven directories as five different versions. Some were projects that had not pulled the latest template. Two were copies edited by hand. The rest were stale copies inside old git worktrees. One of the hand edits had a better rule than the template, "verify with real output", and it never made it back upstream.

The docs broke their own rules too. Each file has a line budget at the top, because every tick reads it and the cost compounds. One STATE.md had grown to 335 lines against a budget of about 150. A DECISIONS.md sat near 600 lines, read in full on every tick. And much of the playbook was procedure written as prose: fixed steps in a fixed order that the model re-derived from paragraphs on every run. It is the same drift we described in why agent config falls apart, one level up. If your agent instructions have line budgets, check them. Ours had quietly stopped meaning anything.

A memory note that was wrong

The review also turned up a problem outside the files. An old memory note, the kind Claude Code keeps between sessions, said agents never merge or deploy. The agent designing the new pipeline read it and proposed a system that stopped at an open pull request and handed the rest back to a human. That was never our policy. Our agents work on their own, merge on green CI and deploy. A task is complete when it is merged and deployed, not before.

One stale note had steered the whole design toward something weaker, and nothing in the output said so. When an agent's plan looks oddly timid, check its memory before you argue with the plan.

Scripts gather facts, agents make calls

The first workflow was deliberately small. status-sweep builds one table across all the repos: git state, open pull requests, where each orchestration cursor sits, which docs are over budget. Version one used four Haiku agents to run git and gh commands, plus a fifth to combine their answers. It worked. It also burned about 198,000 tokens and two minutes reporting facts a shell could read directly.

So we rewrote it. A bash script now collects every fact in about 12 seconds and hands them to one agent that writes the report. That run used about 54,000 tokens and took 55 seconds. We expected it to be ten times cheaper. It was about 3.7 times cheaper, because the remaining agent still has to read everything the script found, and that reading is most of the cost. The new version also caught something the old one missed: a local branch still tracking a remote branch that no longer existed.

That set the rule for everything after it.

If a command can produce the answer, a script does it. Agents get the parts that need judgment.

A tick with a fixed shape

The second workflow, orchestrate-tick, applies the same rule to the whole loop. The script owns the order of the stages, where a run may stop and which model each stage uses. The playbook still owns policy: what may be picked, what may be merged. The script decides the sequence, and the playbook decides the rules.

Every model is pinned in the script. Review runs on a different model from execute, for a reason that comes up further down. No stage inherits the model of the interactive session that started the tick, so switching to the most expensive model for something unrelated does not quietly make every tick expensive too.

The workout-tracker run at the top of this post was the first dry run of this pipeline. It was a dry run, so only part of the tick ran: two agents and about 146,000 tokens. The spec and plan it found went into the repo through a small docs pull request before any other work started.

Surviving a usage limit

A tick runs for minutes, and a subscription has usage limits. If the limit hits mid-tick, the final write-back never runs, and the next tick has no idea a branch was half-built.

So every stage now ends with a short script that stamps Checkpoint: <stage> @ <time> onto the tick's claim line in STATE.md. It works in a throwaway worktree on the orchestration home branch, pushes, retries on conflict and never force-pushes. The next tick reads the stamp and resumes from the first unfinished stage.

Claude Code has its own version of this. A workflow can pause at a usage limit and continue after the reset, but only in an interactive session, and resume only works within the same session. We use both. The native pause is switched on for interactive ticks, and the git checkpoint covers everything else: a new session, a crash, a scheduled or headless run.

Reviews that live on the pull request

Reviews used to happen inside an agent's context. If the tick died, the review died with it. Now the execute stage opens a draft pull request, and the review agents post their findings as comments on it. GitHub does not let you approve your own pull request, and the agents all work under one account, so anything that must block the merge is marked [blocking] in the text. A fix agent answers every thread, including the ones from Codex (OpenAI's coding agent) and from people. A re-review looks only at what changed since the last round, and reviews run once per push, not once per commit.

Reviews now survive a dead tick. A person can also steer work in progress by leaving a comment on the pull request, the same way they would with a colleague.

Where the tooling pushed back

Claude Code's auto mode runs a classifier over actions before they happen, and it blocked the agent from writing the workflow file. That was a fair call. The file was a script that merges and deploys by itself, which is exactly what a safety check should question. We fixed it with an explicit allow rule for the workflows folder, written down where anyone can see it, not a workaround. That difference matters, and it is the theme of our earlier post on agent guardrails.

One smaller surprise: a newly saved workflow did not show up by name until we started a new session. This post was written in a fresh session for that reason, picked up from a written handoff.

What other people are finding

Before publishing, we looked for teams doing the same thing, to check the approach and to find what to do better. The core split is not new. Anthropic separates workflows that follow predefined code paths from agents that direct themselves. Microsoft's Conductor makes the same bet in YAML: routing that costs no tokens, and room to put a cheap model on triage and a capable one on reasoning.

Two findings hit weak spots in our setup. A study from ETH Zurich found that repository context files did not generally improve task success, while raising inference cost by over 20% on average. The authors suggest that human-written files describe only minimal requirements. A 600-line decisions file read on every tick is hard to justify on those numbers. Research on models used as judges shows that they recognise and favour their own writing. Our execute and review stages ran on the same model, so the reviewer was grading its own homework.

The Claude Code docs had a cost detail we had missed. Agents in one run can share a prompt cache when their model, tools and output schema match, but a workflow agent's cache lasts five minutes by default. Cache reads cost a tenth of normal input or less. A tick whose stages sit minutes apart is probably paying full price for some of them. A longer cache costs more to write, so we are leaving it off until we have real numbers.

Where this leaves us

We handed the research to the agent that built the workflow, and every tick now runs with the result. Review is on Opus and sees only the issue and the diff. Only the pick stage reads the full decisions file. Stages hand over structured data, and a fix loop stops after three failed CI rounds. The longer cache stays off until the cost figures say it pays.

Very little of this needs Claude Code. The policy is markdown, the state is in git, the reviews are on GitHub and the fact-gathering is bash. The runtime is the one piece we want to make swappable, so the same tick can run on other coding agents.

Our agents keep improving the setup they run on. Next we will measure the longer cache on real ticks and share the numbers.