Agent operations playbook

Running an autonomous agent is not the same skill as building one. Once an agent is live and taking actions on a schedule or a trigger, someone owns five ongoing responsibilities:

• Deploy — get an agent into service with a defined purpose, a bounded scope, an owner, a risk-tier ceiling, and a kill switch that is tested before the first real run.
• Observe — know at any moment what every agent is doing, how much, how fast, how often it is asking for help, and how often its actions fail verification.
• Intervene — pause, drain, roll back, revoke credentials, or kill an agent quickly and without collateral damage when it misbehaves.
• Govern — keep the human approval gate staffed and meaningful, keep the audit log intact, and keep prompt/policy/model changes running through change management like any other production change.
• Improve — feed real run outcomes back into the agent's instructions and tier assignments, and widen autonomy only where the data earns it.

The rest of this page is those five, one section at a time.

Every autonomous agent runs the same four-beat loop, whether it wakes on a cron schedule, fires on a queue message, or is triggered by an alert:

1. Wake / trigger. The agent starts from a known, fixed context — its instructions file, its memory, and the current state of the work queue. It does not assume anything carried over in its head from last time; if it needs to know what happened before, it reads it back from a durable record.
2. Act. It picks the highest-value work within its scope, plans it, checks the plan against its risk tier, and either executes (low tier) or requests approval (higher tier). Irreversible, ambiguous, or out-of-scope work is written down and handed to a human, not attempted.
3. Report. Win or lose, it emits a truthful summary to its operator channel and appends a dated entry to its running log. "Nothing worth doing this cycle" is a valid, logged outcome.
4. Human review (asynchronous). The operator reads the reports on their own schedule, answers anything parked, and adjusts instructions. The agent does not block on this — it picks the answers up on its next wake.

You cannot operate what you cannot list. Every agent in service has a row in a fleet register, and an agent with no row does not run. The minimum columns:

Review the register on a fixed cadence (quarterly is enough for a small fleet). Agents accrete scope quietly; the review is where you claw it back.

A service has latency, traffic, errors, and saturation. An autonomous agent has a parallel set of five signals that tell you whether it is healthy without reading its logs line by line:

• Action rate — mutations per hour, split by tier. A step change up is the single most important alarm: it means the agent found a loop, a bad batch, or a new interpretation of its instructions.
• Approval-wait time — how long approval.request messages sit before a human answers. Rising wait time means the gate is understaffed and autonomy pressure is building.
• Verification-failure rate — fraction of actions that do not leave the target in the intended state on independent re-check. Above a few percent, something upstream is wrong — stale CMDB data, a changed API, drifted preconditions.
• Escalation rate — how often the agent stops and asks. Too high wastes the operator; suddenly too low can mean it has stopped recognising cases it should escalate.
• Heartbeat & queue depth — liveness plus backlog. A silent agent and a growing queue is an outage even if nothing errored.

Put those on one board with per-agent rows, alert on rate-of-change not just absolute thresholds, and keep the append-only audit log as the drill-down: every signal on the board should be a query away from the exact messages behind it.

Autonomous agents fail in recognisable ways. When something feels off, walk this list — it is ordered by how fast it does damage:

Reach for the gentlest control that solves the problem — but never hesitate to skip rungs when actions are actively causing harm. Each rung is something you should have practised before you need it.

• Just-in-time, per-action leases. An agent holds no standing credential. It redeems a scoped, short-TTL lease for the one plan it is executing, against the one target, and the lease expires on its own.
• Credentials never transit the work bus. A message references a lease by id; the agent redeems it out of band against the secrets broker. A captured message is not a captured credential.
• Rotation is scheduled, not event-driven. Signing keys and lease profiles rotate on a fixed interval whether or not anything looked wrong, so rotation is a boring routine and not a fire drill.
• On suspected compromise: revoke the lease profile (stops mutations without killing the process), rotate the agent's signing key, pull every audit-log row for its identity over the exposure window, and diff intended vs actual target state for each action in that window.
• Secrets stay out of logs and reports. The report channel and the running log are readable by more people than the vault; treat anything that lands there as public.

A prompt edit, a policy tweak, a tool addition, and a model upgrade are all production changes to the agent's behaviour — and the model upgrade is the one most likely to change behaviour silently, because you did not touch the code. Run them through the same lifecycle:

1. Shadow. New version runs against real inputs but its actions are logged, not applied. Compare its proposed actions to the live version's for a representative period.
2. Canary. Route a small, low-blast-radius slice of real work to the new version. Watch the five golden signals against the old version's baseline.
3. Dual-run for schema changes. If the change touches message shapes, producers emit both old and new, consumers upgrade, then the old shape retires.
4. Promote or roll back. Promotion is a deliberate step with the baseline comparison attached. Rollback is one command and is tested before the canary starts, not improvised after.
5. Record it. Effective instructions and model version are versioned artifacts. "Behaviour changed and we do not know which change did it" is the outage you are preventing.

• Approval answers "should this specific plan run?" Arbitration answers "two agents want incompatible things — which wins?" Same queue, two question shapes.
• A good approval request is self-contained: the plan, the dry-run diff, the tier rationale, the rollback, the blast radius, and any conflicting in-flight plan — all in one message. The approver should not have to go looking.
• Every request has a TTL. Unanswered in the window, it auto-escalates — it never sits silently and it never auto-approves.
• Staff the gate like on-call. A named rota, a paging path, and a response-time target. Approval-wait time on the board is how you know the staffing is right.
• Denials are data. A pattern of denials for one plan type means the agent's tier assignment or its instructions are wrong — fix the upstream, do not just keep denying.

1. Contain. Kill switch or credential revoke — stop new actions before anything else. A running agent during triage keeps changing the thing you are trying to understand.
2. Preserve the record. Snapshot the audit log range and the agent's effective instructions and model version before restarting or patching. This is the evidence.
3. Assess blast radius. Pull every action for the agent's identity over the incident window; diff intended vs actual state for each; list what needs manual correction.
4. Arbitrate the fix. Corrections above Tier 1 go through the gate like any other change — the incident does not suspend the approval model, it is exactly when you want it.
5. Post-incident review. The output is a change: a tighter scope, a new deny-list entry, a missing precondition check, a monitoring gap closed, or a kill-switch drill added. "The agent will be more careful" is not a fix.

The controls in the intervention ladder only work if they have been used. On a fixed cadence, in a controlled window, practise:

• Kill-switch drill — stop a live agent mid-action, confirm the in-flight action's rollback runs and the queue is intact on restart.
• Bad-approval drill — deliberately approve a plan that should have been denied; confirm verification catches the wrong end state and the rollback path works.
• Stuck-ticket drill — let an approval.request time out; confirm it escalates rather than stalling the ticket's other work.
• Credential-revoke drill — pull an agent's lease profile mid-run; confirm it degrades to plan-and-report and does not crash-loop.
• Model-rollback drill — roll a canary back to the previous model/prompt under time pressure; measure how long it actually takes.

Track: verification-pass rate, mean approval-wait time, escalations resolved per week, rollback count and cause, time-to-contain in drills, and scope-violation count (target: zero). These measure whether the fleet is trustworthy.

Do not optimise: raw action count, "percent of work done without a human", or tickets closed per hour. An agent that maximises those is an agent that has learned to stop asking — which is exactly the failure the whole model exists to prevent. Autonomy is a result you earn from a clean track record, not a KPI you push.

• Service-desk architecture — defines the fleet this page operates: the orchestrator, the domain agents, the Platform Ops agent, and the risk tiers referenced throughout.
• SOC & incident-response architecture — a second fleet with the same operating needs; "when an agent causes an incident" here is the mirror of eradication/recovery staying human-owned there.
• Agent-to-agent coordination protocol — the wire format the golden signals are computed from: heartbeat, result, escalation, and the synchronous audit tee.
• Operations model & SOP — the human-team procedures (rotas, change advisory, review cadence) this playbook's gate and drills plug into.
• Interactive ticket trace — one ticket through the lifecycle, including the stage-5 approval an operator on this playbook's gate rota would action.
• Field guide — the specific incidents from this project's own run that the failure modes above are generalised from.

This is an operating playbook, written to be adopted. It ships no control plane, no dashboard, and no agent runtime on this box — running a real fleet against real systems is a decision for whoever owns those systems, with their own console, their own on-call, and their own approval gate wired in.

The load-bearing choices — a stateless loop, a fleet register with a tier ceiling per agent, five golden signals, a rehearsed intervention ladder, change-managed prompts and models, and autonomy widened only on evidence — are the same principles the architecture pages are built on, expressed as day-to-day operations. Loosen them and you have a fleet you cannot safely leave unattended.

• Agent operations playbook — full edition — the expanded PDF: worked examples of each failure mode, a fleet-register template, a golden-signals alerting spec, drill runbooks, and a credential-incident checklist. Same reversible-first line as this page.
• Architecture review — send your own multi-agent or automation design and get a written review against these principles, with findings ranked by risk.
• Service-desk architecture — the fleet and the risk tiers this playbook operates.