Peer-to-peer & distributed multi-agent architecture

In the centralized design an orchestrator is the only actor that turns a request into a plan, routes each slice to a domain agent, and advances the ticket from one state to the next. Domain agents propose; they never decide and never talk to each other directly. Every message tees synchronously to one append-only audit log before delivery. One deny-list, enforced at one bus, covers the whole fleet. This is easy to reason about, easy to audit, and easy to gate: there is exactly one place where work is sequenced and exactly one place a human plugs in.

Its costs are the costs of any hub. The orchestrator is a single point of failure and a throughput ceiling. Every message pays a round-trip through it. It assumes one trust domain, one operator, one network that is usually up. Push a fleet across data centres, across organisations, or past the request rate one process can sequence, and the hub becomes the problem.

Remove the orchestrator. Now every agent is a peer: it can originate work, take work, and coordinate with any other peer it is entitled to talk to. There is no privileged router and no single writer of shared state. The same signed envelope still carries every message — trace_id, causation_id, idempotency_key, sig are unchanged — but to is now a peer or a topic, never "orchestrator", and no one actor owns ticket state. Five mechanisms replace what the hub used to do:

• Task assignment by contract-net. The originator broadcasts a plan.request to a capability topic (e.g. cap.identity). Peers that can do it and have spare capacity reply with a bid (cost, ETA, current queue depth). The originator awards to one bid with an execute carrying a fencing token — a monotonic number the target system uses to reject a later, stale winner. No central scheduler; the work advertises itself and the capable peers self-select.
• Shared state by replicated log or CRDT. Anything more than one agent must agree on — "is this target already being changed?", "what tier is this ticket?" — lives in a small replicated store. Two honest options: a quorum-replicated log (Raft / Multi-Paxos) when you need linearizable reads and can tolerate a partition stalling the minority side; or CRDTs (grow-only sets, LWW registers, OR-maps) when you need every partition to keep working and can reconcile afterwards. Ticket state becomes a fold over this shared history, not a row one process mutates.
• Membership & failure detection by gossip. Peers run a SWIM-style protocol: periodic direct pings, indirect pings through k random relays before declaring a peer dead, and a piggy-backed membership delta on every message. A phi-accrual detector turns "no heartbeat" into a suspicion level rather than a hard timeout, so a slow link degrades gracefully instead of flapping the whole roster.
• Audit by cross-signed local logs. Each agent keeps its own hash-chained log of every message it sent or received. Periodically each agent publishes its head hash to an audit topic; peers counter-sign what they witnessed. The union of the local logs, stitched on trace_id and cross-checked against the signed heads, reconstructs a whole ticket and is tamper-evident without a central tee — rewriting history requires colluding with every peer that counter-signed it.
• Policy enforced locally, everywhere. The deny-list is no longer one gate. It is compiled into every agent, version-pinned, and distributed like any other config; an agent refuses a forbidden action itself, before it acts, and emits an escalation. A peer running a stale policy version is fenced out by the others until it updates.

The approval gate does not disappear in a peer-to-peer fleet — it becomes a peer with a special capability. Any agent whose self-assessed plan is Tier ≥ 2 or that collides with another in-flight plan must obtain a signed approval.grant from a gate peer before it may send itself an execute. The grant is a capability token: scoped to one idempotency_key, one target, one time window, and useless for anything else. Because there is no orchestrator to trust the gate on everyone's behalf, the grant is verified independently by the acting agent and recorded in its local log and the shared state. Losing quorum on the shared store means new Tier ≥ 2 work cannot start — the fleet degrades to read-only and low-tier work rather than acting without a recorded human decision.

The same ticket the other pages follow — a password reset that turned into an account lockout — run with no orchestrator.

• Originate. The intake agent classifies the ticket, mints a trace_id, writes an intent to its local log, and broadcasts a plan.request on cap.identity.
• Bid & award. Two identity agents reply with bids; intake awards to the one with the shorter queue, sending execute-precursor plan.request with fencing token 42.
• Negotiate a dependency. The identity agent finds the unlock needs a group-membership check it cannot do. It does not route through a hub: it broadcasts its own plan.request on cap.directory, under the same trace_id, and awards it to a directory peer. Two peers now collaborate directly on one ticket.
• Claim the target. Before acting, the identity agent takes a lease on user:jdoe in the shared store. A second ticket touching the same account blocks on that lease instead of racing.
• Gate. The plan self-assesses as Tier 2 (privileged account). The identity agent requests an approval.request from a gate peer, which a human answers with a signed approval.grant scoped to idempotency_key = INC0012345:unlock:1. The grant is verified locally and appended to the shared history.
• Act, verify, release. The identity agent applies the unlock, emits a truthful result, runs an independent verify, releases the lease, and publishes its log head to the audit topic. Directory and intake peers counter-sign what they witnessed.
• Reconstruct. Anyone pulls every record with that trace_id from the union of local logs, checks it against the cross-signed heads, and gets the same ordered decision path the centralized audit log would have shown — assembled from the edges instead of read from the hub.

Pure peer-to-peer is rarely the right answer at every level of a system. The common shape is federated: centralized within a cell, peer-to-peer between cells. A cell is one trust domain — a site, a business unit, a customer tenant — and runs its own small orchestrator with its own gate and its own audit log. Cell orchestrators are then peers to each other: they discover capabilities, bid for cross-cell work, and cross-sign a shared inter-cell log, using exactly the peer-to-peer mechanisms above.

• Inside a cell you keep the centralized model's advantages: one sequencer, one deny-list, one obvious place for a human, simple end-to-end tracing.
• Between cells you get the distributed model's advantages: no global single point of failure, each cell survives the others being unreachable, organisations keep their own control point and their own audit boundary.
• The seam is explicit. A cross-cell request is a first-class thing with its own contract: which cell owns the outcome, whose gate approves it, which audit log is authoritative if the two disagree. You are designing one seam carefully instead of an N×N mesh loosely.

This is the same instinct as the service-desk design's per-domain agents, extended one level up: a domain agent orchestrates its own tools centrally; the fleet federates across domains.

Read each row as "if this is true of your situation, it points toward…". Most real systems match a mix; count the rows and note which ones are non-negotiable for you (compliance and trust usually are).

• Network partition. Decide per shared value whether it is CP (linearizable, minority stalls) or AP (available, reconcile later). Ticket state and lease ownership are usually CP — you do not want two peers each believing they hold user:jdoe. Capability discovery and heartbeats are fine as AP.
• Duplicate execution. Every execute carries a fencing token; the target system records the highest token it has seen and rejects anything lower. A stale winner that wakes up after a partition heals cannot re-apply.
• Orphaned work. A peer that wins a bid and then dies holds a lease with a TTL. On expiry the originator re-announces the plan.request; a new peer picks it up. Idempotency keys keep the retry from double-acting.
• Split-brain on the shared store. Consensus groups need an odd member count and a quorum to make progress; a minority partition goes read-only by design rather than forking state.
• Gossip storms. Bound fan-out (k relays, not broadcast), add jitter to gossip intervals, and cap membership-delta size per message so a flapping peer cannot amplify into fleet-wide traffic.
• Stale policy. Every agent advertises its deny-list version in its heartbeat. Peers refuse to award work to an agent behind the pinned minimum, so a missed policy update takes that agent out of rotation instead of letting it act on old rules.

• Every agent enforces the deny-list itself. With no bus in the middle, "refuse forbidden actions" is a property of each agent, tested in each agent, and version-gated by peers. The list is config, distributed and pinned like any other critical config.
• Membership is by issued certificate. A peer joins only with a signed capability cert from the fleet CA, naming what it may originate and accept. Unsigned or expired peers are ignored. This is the sybil defence — you cannot flood the mesh with fake bidders without the CA.
• Capabilities are least-privilege and short-lived. A cert grants "bid on cap.identity, accept execute.identity" and nothing else, with a TTL measured in hours. Revocation is a short TTL plus a gossiped revocation list, not a CRL nobody checks.
• Larger blast radius, so contain it structurally. A compromised peer can talk directly to every peer its cert allows. Keep certs narrow, keep cells small, and make the audit cross-signing mandatory so a rogue peer's actions still land in logs its victims counter-signed.
• Credentials still never ride the wire. Unchanged from the centralized protocol: an execute references a just-in-time credential lease by id, redeemed out of band against the secrets broker, scoped to one plan and one target.

Going peer-to-peer costs you:

• A single place to read "what is the whole system doing right now".
• Trivial global ordering — you now choose a consistency model and live with its edges.
• One deny-list at one gate — policy is now a distributed-config problem.
• One-shot rollback of a whole ticket — compensation is per-peer and coordinated.
• A short list of failure modes — you inherit split-brain, stale membership, gossip amplification, and fencing bugs.

And buys you: no fleet-wide single point of failure; horizontal scale by adding peers; operation across organisations and regions that will never share one hub; latency locality; and continued partial operation when part of the network is unreachable.

• Agent-to-agent coordination protocol — the centralized design this page is the counterpart to. The envelope, the closed message-type set, idempotency keys, and the credential-lease rule carry over unchanged; what changes is who routes and who advances state.
• Service-desk architecture — a fully centralized instance: one orchestrator, nine domain agents, one gate. Read this page for what it would take to federate that design across sites or tenants.
• SOC & incident-response architecture — another centralized instance (SIEM/SOAR as the hub). Detection sources distributed across regions are a natural candidate for the hybrid model.
• Agent operations playbook — the five golden signals and the intervention ladder still apply, but in a peer-to-peer fleet you watch them per peer and per cell, and "quorum health" and "gossip convergence time" join the list.
• Operations model & SOP — the human procedures. Federation adds one: the cross-cell contract review, deciding whose gate and whose audit log is authoritative for shared work.
• Interactive ticket trace — the same lockout ticket stepped through the centralized lifecycle; the worked example above is its peer-to-peer twin.

This is an architecture concept, written to be compared against the centralized model and implemented where it fits. It ships no mesh, no gossip daemon, and no running peer on this box. Standing up a real distributed agent fleet — picking a consensus implementation, running a CA, operating gossip membership at scale — is a decision for whoever owns the systems being changed, made with their own infrastructure and their own approval gates wired in.

The load-bearing claim of the page: decentralization is not a default, it is a response to a specific pressure — scale past one sequencer, a footprint past one trust domain, or an availability requirement past one hub. Absent one of those, the centralized model's simplicity is worth keeping.