Operations model & SOP

A risk-tiering model and an approval gate only survive contact with a real environment if someone actually staffs the gate, someone owns each domain agent's playbooks, and someone periodically checks that the guardrails still mean what they said on day one. This document is that missing layer: it assumes the architecture and the integration guide are already built per the phased rollout, and describes the recurring human process around the running system — not a one-time deployment task, but the thing a team does every day, every week, and every quarter for as long as this stays live.

Nothing below changes a tier assignment, adds a new automated capability, or opens a path around the human approval/arbitration gate. It's entirely about who does what, on what schedule, and what "healthy" looks like for a system this document doesn't redesign.

Six roles, none of them full-time on this alone in most organizations — the architecture is designed so the human load stays bounded even as the number of managed systems grows, because the thing that scales (the approval queue) is a single narrow decision, not one job per system.

Shift-start checklist — five minutes, every shift change:

• Grafana health dashboard is green: no agent flatlined, no sustained Tier 0 self-healing loop (a condition that keeps re-triggering is a sign the fix isn't actually fixing it).
• ServiceNow approval queue is empty or every open item is within its SLA (below) — a queue that's quietly grown over a weekend is the most common way this design silently degrades into "nobody's actually approving anything."
• Overnight Tier 0/1 self-healing log reviewed for anything unusual, even though none of it needed a click at the time.
• Platform Ops's own heartbeat is green — the supervisor watching everything else is itself unwatched by nothing; this is the one manual check that stands in for that.
• No domain agent stuck >15 minutes in "awaiting approval" without a corresponding open item in the queue (a symptom of a webhook or notification failure, not an actual empty queue).

Approval SLAs — how fast a pending item needs a human response:

The architecture page's "Walkthrough: an alert firing, end to end" traces two concrete examples through this loop. The diagram below is the general shape underneath both of them, drawn against this project's actual agents and tools rather than a generic four-box template — the same four phases every mature incident process uses, mapped onto Platform Ops, the orchestrator, the domain agents, and the human gate this design already defines.

At Tier 0 the gate box doesn't apply at all — detect, act, verify, log, close, unattended, exactly as the "disk fills up" example on the architecture page walks through. Everything above Tier 0 waits at the gate between phase 2 and phase 3 before a domain agent is allowed to act.

Every Tier 2/3 action already walks through ServiceNow's normal Change process by design — this section is about when that's allowed to happen, not whether it needs approval. A maintenance window is what makes the difference between "approved" and "approved and executed at 2pm on a Tuesday nobody warned about."

Emergency change: an active outage doesn't wait for the next CAB. The human gate can approve a Tier 3 action directly, with the domain agent owner notified and a post-hoc CAB review recorded within 24 hours. That review is a requirement, not a formality — it's the mechanism that catches an emergency exception being used as a routine shortcut.

Each supporting system from the architecture page's table feeds Platform Ops or opens a ServiceNow Incident directly; this is the operational cheat sheet for what to actually watch and who it wakes up.

• Audit log: append-only by design; retained a minimum of one year (or whatever the organization's own compliance window requires, if longer), backed up nightly off the primary ServiceNow instance — the log is the architecture's entire evidentiary basis, so its own backup gets treated like a production database, not an afterthought.
• CMDB / NetBox: nightly backup; Platform Ops's drift detection is only as trustworthy as the source of truth it's diffing against.
• Vault (CyberArk): sealed-key / Master-key backup per the vendor's standard procedure, with a documented recovery runbook that doesn't depend on any one person being reachable. Safe contents replicate per CyberArk's own DR component; agents fail closed while it's unavailable rather than using a fallback credential.
• Backup & recovery platform: the backup catalog/config itself is backed up nightly; repository immutability / air-gap is enforced at the storage layer so retention can't be shortened even by a compromised console. A scheduled restore-test (isolated network, checksum, tear down) runs at least monthly — an untested backup is not a backup.
• Storage arrays (NetApp / Dell): array config backed up on change; SnapMirror / Dell replication to a second site for anything hosting Tier 1 workloads, with replication lag alarmed against its RPO.
• DR testing: quarterly tabletop walkthrough of "the orchestrator or the approval gate is unreachable" — the framework's own version of the question every DR plan has to answer. Annual full failover test of the orchestrator and approval-gate path specifically, since everything downstream of the gate is only as available as the gate itself.

• Credential rotation: every domain agent's vault role uses short-lived, scoped leases checked out just-in-time — the standing secret that needs rotating is the vault role's own signing key, rotated quarterly.
• Quarterly access review: the Security/Vault admin audits every standing (non-leased) credential and every human account with approval-gate access, and removes anything that's outlived its reason for existing.
• Tier 3 approval audit: monthly, the Change Advisory reviewer samples a portion of the month's Tier 3 approvals and checks the plan the human approved actually matches what the audit log shows executed — the control that catches an approval becoming a rubber stamp before it becomes a habit.
• Deny-list review: quarterly check that the deny-list from the architecture page's guardrails still reflects anything the organization has learned should never run unattended, including anything a post-mortem (below) surfaced since the last review.
• Privileged-session review (CyberArk): monthly review of every PSM-recorded human break-glass session — each one is an exception to the "agents act, humans approve" norm and should be rare enough to read every entry.
• Security signal pipeline: CrowdStrike detections and Splunk correlation searches feed the same ServiceNow intake as everything else; the weekly ops review includes a pass over low-severity detections and near-miss correlation hits that didn't cross an alerting threshold.

Nothing here is urgent on any given day, which is exactly why it needs a fixed cadence instead of waiting for someone to notice:

• VMware Aria capacity trend review — monthly, ahead of the next maintenance window rather than reactive to a full datastore.
• Database storage headroom review — monthly.
• Storage array (NetApp / Dell) capacity, aggregate headroom, and snapshot-reserve trend — monthly; growth here is a procurement lead-time decision, so noticing it late is expensive.
• Backup window & repository-capacity trend — monthly; a backup job whose runtime is creeping toward its window is a capacity problem before it's a failure.
• APC UPS runtime/load trend review — monthly; a UPS whose runtime has been quietly shrinking for months is a battery-replacement decision, not a Tier 1 shutdown-sequence decision.
• Vault lease volume & agent request-rate trend — quarterly, the framework's own capacity plan for itself.

Every row in the table below reduces to one of two lanes: a normal, SLA-timed queue that climbs one step at a time, or a small set of triggers that skip the queue entirely because waiting on it would defeat the point — a live security anomaly or a UPS about to run out of runtime doesn't get a 30-minute SLA.

The left lane is the common case — a page climbs one step at a time, each with a fixed time-to-escalate. The right lane exists because a 30-minute SLA is the wrong answer to "the audit log just proved something executed with no approval record" or "the UPS runtime just crossed critical" — both go straight to a human, no queue in between.

The single table this whole document reduces to — if only one part of this SOP survives being read, it should be this one:

• Download this SOP as a PDF — same content, free, no signup.
• Architecture blueprint — the diagrams, risk tiers, guardrails, and phased rollout this document assumes is already built.
• Integration guide — setup steps, API auth, and a working MCP server for all twenty-two supporting systems and the nine domain-agent target systems.
• Screen-by-screen mockup — what the approval queue this SOP has approvers staffing actually looks like on screen.
• Interactive ticket trace — one ticket stepped through the lifecycle, including the approval a person on this SOP's on-call rota would action at stage 5.
• Agent-to-agent coordination protocol — the wire format under the hood: where this SOP says "the approver reviews the plan and the rollback," the protocol specifies the message that carries them.
• Peer-to-peer & distributed architecture — the federated variant adds one procedure to this SOP: the cross-cell contract review that decides whose gate approves shared work and whose audit log is authoritative when two cells disagree.
• Agent operations playbook — the agent-fleet counterpart to this SOP: the operating loop, a fleet register, five golden signals, the intervention ladder, and game-day drills for the agents themselves.
• SOC & incident-response architecture — the security-side counterpart to the service-desk blueprint: detection triage, enrichment, and reversible containment, with eradication and recovery human-owned.