Operations model & SOP

A risk-tiering model and an approval gate only survive contact with a real environment if someone actually staffs the gate, someone owns each domain agent's playbooks, and someone periodically checks that the guardrails still mean what they said on day one. This document is that missing layer: it assumes the architecture and the integration guide are already built per the phased rollout, and describes the recurring human process around the running system — not a one-time deployment task, but the thing a team does every day, every week, and every quarter for as long as this stays live.

Nothing below changes a tier assignment, adds a new automated capability, or opens a path around the human approval/arbitration gate. It's entirely about who does what, on what schedule, and what "healthy" looks like for a system this document doesn't redesign.

Six roles, none of them full-time on this alone in most organizations — the architecture is designed so the human load stays bounded even as the number of managed systems grows, because the thing that scales (the approval queue) is a single narrow decision, not one job per system.

Shift-start checklist — five minutes, every shift change:

• Grafana health dashboard is green: no agent flatlined, no sustained Tier 0 self-healing loop (a condition that keeps re-triggering is a sign the fix isn't actually fixing it).
• ServiceNow approval queue is empty or every open item is within its SLA (below) — a queue that's quietly grown over a weekend is the most common way this design silently degrades into "nobody's actually approving anything."
• Overnight Tier 0/1 self-healing log reviewed for anything unusual, even though none of it needed a click at the time.
• Platform Ops's own heartbeat is green — the supervisor watching everything else is itself unwatched by nothing; this is the one manual check that stands in for that.
• No domain agent stuck >15 minutes in "awaiting approval" without a corresponding open item in the queue (a symptom of a webhook or notification failure, not an actual empty queue).

Approval SLAs — how fast a pending item needs a human response:

The architecture page's "Walkthrough: an alert firing, end to end" traces two concrete examples through this loop. The diagram below is the general shape underneath both of them, drawn against this project's actual agents and tools rather than a generic four-box template — the same four phases every mature incident process uses, mapped onto Platform Ops, the orchestrator, the domain agents, and the human gate this design already defines.

At Tier 0 the gate box doesn't apply at all — detect, act, verify, log, close, unattended, exactly as the "disk fills up" example on the architecture page walks through. Everything above Tier 0 waits at the gate between phase 2 and phase 3 before a domain agent is allowed to act.

Every Tier 2/3 action already walks through ServiceNow's normal Change process by design — this section is about when that's allowed to happen, not whether it needs approval. A maintenance window is what makes the difference between "approved" and "approved and executed at 2pm on a Tuesday nobody warned about."

Emergency change: an active outage doesn't wait for the next CAB. The human gate can approve a Tier 3 action directly, with the domain agent owner notified and a post-hoc CAB review recorded within 24 hours. That review is a requirement, not a formality — it's the mechanism that catches an emergency exception being used as a routine shortcut.

Each supporting system from the architecture page's table feeds Platform Ops or opens a ServiceNow Incident directly; this is the operational cheat sheet for what to actually watch and who it wakes up.

• Audit log: append-only by design; retained a minimum of one year (or whatever the organization's own compliance window requires, if longer), backed up nightly off the primary ServiceNow instance — the log is the architecture's entire evidentiary basis, so its own backup gets treated like a production database, not an afterthought.
• CMDB / NetBox: nightly backup; Platform Ops's drift detection is only as trustworthy as the source of truth it's diffing against.
• Vault: sealed-key backup per the vault vendor's standard procedure, with a documented unseal runbook that doesn't depend on any one person being reachable.
• DR testing: quarterly tabletop walkthrough of "the orchestrator or the approval gate is unreachable" — the framework's own version of the question every DR plan has to answer. Annual full failover test of the orchestrator and approval-gate path specifically, since everything downstream of the gate is only as available as the gate itself.

• Credential rotation: every domain agent's vault role uses short-lived, scoped leases checked out just-in-time — the standing secret that needs rotating is the vault role's own signing key, rotated quarterly.
• Quarterly access review: the Security/Vault admin audits every standing (non-leased) credential and every human account with approval-gate access, and removes anything that's outlived its reason for existing.
• Tier 3 approval audit: monthly, the Change Advisory reviewer samples a portion of the month's Tier 3 approvals and checks the plan the human approved actually matches what the audit log shows executed — the control that catches an approval becoming a rubber stamp before it becomes a habit.
• Deny-list review: quarterly check that the deny-list from the architecture page's guardrails still reflects anything the organization has learned should never run unattended, including anything a post-mortem (below) surfaced since the last review.

Nothing here is urgent on any given day, which is exactly why it needs a fixed cadence instead of waiting for someone to notice:

• VMware Aria capacity trend review — monthly, ahead of the next maintenance window rather than reactive to a full datastore.
• Database storage headroom review — monthly.
• APC UPS runtime/load trend review — monthly; a UPS whose runtime has been quietly shrinking for months is a battery-replacement decision, not a Tier 1 shutdown-sequence decision.
• Vault lease volume & agent request-rate trend — quarterly, the framework's own capacity plan for itself.

Every row in the table below reduces to one of two lanes: a normal, SLA-timed queue that climbs one step at a time, or a small set of triggers that skip the queue entirely because waiting on it would defeat the point — a live security anomaly or a UPS about to run out of runtime doesn't get a 30-minute SLA.

The left lane is the common case — a page climbs one step at a time, each with a fixed time-to-escalate. The right lane exists because a 30-minute SLA is the wrong answer to "the audit log just proved something executed with no approval record" or "the UPS runtime just crossed critical" — both go straight to a human, no queue in between.

The single table this whole document reduces to — if only one part of this SOP survives being read, it should be this one:

• Download this SOP as a PDF — same content, free, no signup.
• Architecture blueprint — the diagrams, risk tiers, guardrails, and phased rollout this document assumes is already built.
• Integration guide — setup steps, API auth, and a working MCP server for all fifteen supporting systems and the nine domain-agent target systems.
• Screen-by-screen mockup — what the approval queue this SOP has approvers staffing actually looks like on screen.