Claude Code watchdog: supervising a scheduled agent from the outside

The cron wake loop guide covers making a single wake loud: capture $?, and on a non-zero exit fire an out-of-band alert with the tail of the log. That is the right first move, and it catches a bad run. It does nothing for a missing run, because the alert code is in the wrapper and the wrapper isn't executing:

• cron itself died or was never re-enabled after a package upgrade restarted it.
• the box rebooted — kernel auto-update, host migration, OOM — and the service that launches wakes didn't come back.
• the schedule has a silent gap. Wakes fire on the even hour; nginx fell over at 02:10; nobody finds out until a human loads the site at 09:00.
• the disk filled, so the wrapper can't write its log, its lock file, or its state — and often can't run at all.
• the wrapper has a bug and exits, cleanly, before it ever reaches the line that would have alerted you.

Every one of these is invisible to an in-process check. You need a second process, on its own schedule, that assumes the agent is already broken and goes looking for proof it isn't.

The watchdog is deliberately the opposite of the agent in every way that matters:

• Its own tight cron. This fleet's runs every 20 minutes (*/20 * * * *), decoupled from the wake schedule. If wakes are every two hours, the watchdog still notices a mid-cycle outage within twenty minutes.
• No LLM, no tokens, no spend cap. It is deterministic POSIX shell: curl, openssl, systemctl, df. It cannot be stalled by a full context window or halted by a budget ceiling, and it costs nothing to run.
• It checks the product, not the process. The question is not "did the last wake exit 0?" — it's "is the thing the agent is supposed to keep running still answering?" Externally-observable health: the site returns 200, the services are active, the certificate is valid, the disk has room.
• It is silent when healthy. A watchdog that pings every interval trains you to ignore it. This one writes a log line and says nothing.

The health probe, roughly as this fleet runs it:

set -euo pipefail
HOST="www.beaconwake.com"
anomalies=()   # short keys, for the change-signature
details=()     # human-readable lines for the alert body

# public HTTP: is nginx serving these paths?
for path in / /status.html /api/; do
  code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 \
         --resolve "$HOST:443:127.0.0.1" "https://$HOST$path" || echo 000)
  [ "$code" = 200 ] || { anomalies+=("http:$path"); details+=("HTTP $path -> $code"); }
done

That loop pins the connection to 127.0.0.1, so it is a pure "is the web server up and serving" check — fast, and immune to a DNS or routing problem masking a real local failure. Which is also its blind spot, and the reason for the next check.

The local --resolve loop confirms nginx is serving on the box. It says nothing about whether the outside world can reach it — DNS could be broken at the registrar, a firewall rule could have changed, public routing could be down. So the watchdog also makes exactly one request the normal way, with real name resolution, and retries it once to ride out a transient blip:

ext=000
for _ in 1 2; do
  ext=$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 "https://$HOST/" || echo 000)
  [ "$ext" = 200 ] && break
  sleep 5
done
[ "$ext" = 200 ] || { anomalies+=("http:external"); \
  details+=("external HTTPS -> $ext (real DNS+routing; local checks may still be green)"); }

Now the two results together tell you which thing is broken. Local green + external red is a path problem — DNS, firewall, upstream network — and restarting your app won't fix it. Both red is the app or the box. One request is enough; you are not load-testing your own site every twenty minutes.

A watchdog is only as good as its thresholds. Pick numbers where crossing the line is genuinely actionable, not "technically elevated". What this fleet checks, and why the number is where it is:

None of these needs a metrics stack. They are one curl, one openssl s_client, a short systemctl loop, a df, and a file-exists test.

This is the part that separates a useful watchdog from one you mute. An outage that lasts six hours should page you once, not eighteen times. The trick: reduce the current set of problems to a single sorted signature string, store it, and only alert when it differs from last time.

STATE_FILE=.watchdog_state
prev=$(cat "$STATE_FILE" 2>/dev/null || echo ok)

if [ "${#anomalies[@]}" -gt 0 ]; then
  sig=$(printf '%s\n' "${anomalies[@]}" | sort | tr '\n' ',')
  if [ "$sig" != "$prev" ]; then
    ./notify.sh "watchdog: ${#anomalies[@]} issue(s)
$(printf '%s\n' "${details[@]}")"
  fi
  echo "$sig" > "$STATE_FILE"
else
  [ "$prev" != ok ] && ./notify.sh "watchdog: all clear -- prior issue resolved ($prev)"
  echo ok > "$STATE_FILE"
fi

A persistent outage pings once. If a second thing then breaks, the signature changes and you get one more ping — because the situation genuinely changed. When everything recovers you get exactly one "all clear". This pattern is worth stealing for any cron job that can fail the same way repeatedly.

The watchdog is what catches the wake loop having silently stopped. So what catches the watchdog stopping? Same failure modes — its cron entry could be removed, the box could reboot, the script could grow a bug. You need one more layer, and it should be cheaper and simpler than everything above it:

• A dead-man's switch. The watchdog checks in on every healthy run — touches a URL, pings a hosted uptime service, updates a timestamp somewhere off the box. That external service alerts you if the check-in doesn't arrive on schedule. Absence of a signal is the signal.
• A "last successful wake" age check. Each wake writes a marker (a line in a log, a touched file). The watchdog — or the dead-man service — asserts that marker is newer than N intervals. This is the direct test for "the loop stopped" that an in-wrapper alert can never make.
• Keep the outermost layer off your infrastructure. A second cron on the same box that reboots doesn't help. A free external cron or uptime pinger does. The whole point is that it survives the failure that took out everything else.

You do not need infinite layers. Agent → watchdog → external dead-man is enough: each layer is simpler than the one it guards, and the last one lives somewhere the box's problems can't reach.

Once a watchdog can detect a problem, it is tempting to have it fix the problem. Sometimes that's right. Often it isn't. A rough line:

Safe to automate unattended:

• Restarting a dead stateless service. Better still, let the init system do it: Restart=on-failure with StartLimitIntervalSec and StartLimitBurst set, so a service that hard-fails on startup stops after a few tries instead of restart-looping and hammering whatever it depends on.
• Redeploying the last known-good build when a health check fails right after a deploy (next section).
• Clearing a transient resource — rotating a log that filled the disk, for instance — where the action is idempotent and can't lose data.

Page a human instead:

• Anything touching data or money, or anything not safely idempotent.
• Anything where you don't know why it broke — auto-remediation on an unknown cause turns one incident into two.
• Repeated failures. If restart #1 didn't hold, restart #4 won't either; escalate.

This fleet's watchdog is deliberately alert-only. The operator is reachable in minutes over Telegram, and an auto-remediation that misfires while nobody is watching is worse than a short wait. The agent's own crash handling leans the same way: on a crash loop it should degrade to plan-and-report rather than keep throwing itself at the same wall. Pick auto-heal where the blast radius is genuinely zero; alert everywhere else.

The "self-healing deploy" people mean when they say it: the agent ships a change, a smoke gate runs immediately after, and if the gate fails the previous release goes back automatically. This fleet's deploy.sh runs a local and a live smoke test on every deploy — every page and endpoint must return 200 — and a failure is meant to stop the line. The auto-revert loop around it is small:

# keep the current release before overwriting it
cp -r /var/www/html /var/www/releases/$(date +%s)

deploy_new_files
if ! ./smoke_test.py --live; then
  last_good=$(ls -1d /var/www/releases/* | tail -1)
  rsync -a --delete "$last_good/" /var/www/html/
  ./notify.sh "deploy rolled back: smoke gate failed, restored $last_good"
  exit 1
fi

Two things make this trustworthy: the smoke gate has to actually be strict (a gate that passes a broken site is worse than none), and the rollback target has to be a real artifact you kept, not "re-run the build and hope". Keep the last few releases; prune the rest.

watchdog.sh in this repo is about 125 lines of bash, set -euo pipefail, no dependencies beyond curl, openssl and systemctl. Cron runs it every 20 minutes, entirely separate from the two-hourly wake loop. In order it does:

1. the local --resolve-pinned 200 check on /, /status.html, /api/;
2. one real external https:// request, retried once, to separate "app down" from "path to app down";
3. TLS days-to-expiry via openssl s_client, warn under TLS_WARN_DAYS=15;
4. systemctl is-active over nginx beacon-api fail2ban cron;
5. root disk percent against DISK_WARN_PCT=90;
6. /var/run/reboot-required present and uptime past UPTIME_STUCK_HOURS=36;
7. collapse any anomalies to a sorted signature, compare to .watchdog_state, and call ./notify.sh exactly once when the signature changes — plus one "all clear" when it clears.

That is the whole thing. It has caught a stopped service and a stuck reboot flag in real operation, each time with a single message and a single recovery message. It does not restart anything, does not touch the repo, and does not know or care what the agent worked on — which is the next caveat.

Everything above checks that the product is healthy — site up, services up, disk fine, cert valid. It does not check that the agent is doing anything worthwhile. A wake that starts, reads its notes, decides there's nothing to do, and exits 0 — twelve times a day — passes every probe here. For "is it actually working," you need a different signal: a heartbeat plus queue-depth or output-rate monitoring, covered in the operations playbook. The watchdog tells you the lights are on. It can't tell you anyone's home.

This page leans on shell tools, not Claude Code flags, so it ages more slowly — but certbot's renewal window, systemd option names, and your own path list all drift. Two things to actually do rather than assume: run each probe by hand once and confirm it reports what you expect, then deliberately break something — stop a service, point a check at a bad path — and confirm the alert arrives and the signature dedupe works. An untested alert path is not an alert path. Found something out of date here? Tell us on the Agora.