Can an AI Agent Take the 2 A.M. Page? I Built the Guardrails — and Published the Receipts

Your team runs many Kubernetes clusters. Sooner or later somebody asks the question: can an AI agent take the 2 a.m. page?

The quickest way to find out is to hand a model kubectl with cluster-admin and watch. In production, that experiment ends badly, for three separate reasons:

• The model is non-deterministic. The same alert can produce a careful diagnosis one run and a kubectl delete the next.
• The credentials are real. There is no dry run between the model's decision and your production cluster.
• There is no record. When something breaks, you cannot reconstruct what the agent did, in what order, or on whose authority.

I built — and then adversarially tested — a different answer: ocm-mcp-server, an open-source MCP server that lets AI agents operate a Kubernetes fleet through an Open Cluster Management hub. The agent never holds a kubeconfig. Every write is policy-checked, human-approved, and traced.

This post walks the whole thing end to end — you can follow along on a laptop: the architecture, a real session with Claude, the exact anatomy of a refused write and an approved one, the audit chain, the published two-model evaluation, and the failures I'm not hiding.

The idea: your fleet already has a control point humans trust

Fleets managed by OCM (a CNCF project, the upstream of Red Hat ACM) expose a hub with an inventory (ManagedCluster), a scheduler (Placement), and a delivery channel (ManifestWork). Humans already route changes through this hub every day. The insight is simple: don't give the agent a kubeconfig — give it the hub, wrapped in guardrails.

ocm-mcp-server exposes that hub as 35 typed MCP tools (plus 10 guided prompts and 6 readable resources), and interposes four independent layers between the model and your clusters:

1. Static guardrails in the server itself — checked before anything is even stored.
2. Policy admission — every proposal is dry-run through Kyverno on the hub, inside the ManifestWork envelope.
3. Human approval — an Ed25519 token signed on a trusted terminal, bound to the exact bytes being approved.
4. Least-privilege RBAC — the dangerous verbs don't exist for this identity at all.

None of these layers live in the system prompt, so none of them can be talked out of.

The animated version makes the shape of the whole system legible at a glance — dangerous capabilities on the left do not exist; everything that reaches your fleet flows through the gate on the right:

One design consequence worth underlining: nothing here is specific to any one agent. The server speaks plain MCP over stdio, so anything that speaks MCP connects the same way — Claude Code and Claude Desktop, Codex CLI, Gemini CLI, Cursor, IDE assistants like IBM BOB, or your own LangChain / Agent-SDK orchestrator. The guardrails live behind the protocol, which is exactly why they can't be prompted away: swap the model, and the gate doesn't move. I demonstrate with Claude and publish evaluations for Claude and Codex — treat those two as the tested existence proof, not the compatibility list.

Fifteen minutes to a live guardrailed fleet

Everything below is reproducible. You need Docker or Podman, kind, kubectl, clusteradm, helm, and Python 3.11+.

git clone https://github.com/sandeepbazar/ocm-mcp-server && cd ocm-mcp-server
make bootstrap        # kind hub + 3 spoke clusters, OCM wired, demo app deployed

Then install the server from PyPI and hand it to your agent — here, Claude Code:

pip install ocm-mcp-server

export OCM_MCP_HUB_CONTEXT=kind-hub
export OCM_MCP_SPOKE_CONTEXTS=cluster1=kind-cluster1,cluster2=kind-cluster2,cluster3=kind-cluster3

claude mcp add ocm \
  --env OCM_MCP_HUB_CONTEXT=$OCM_MCP_HUB_CONTEXT \
  --env OCM_MCP_SPOKE_CONTEXTS=$OCM_MCP_SPOKE_CONTEXTS \
  -- $(which ocm-mcp-server)

claude mcp list | grep ocm
# ocm: /…/bin/ocm-mcp-server  - ✔ Connected

That ✔ Connected is the whole integration. No API keys for the server, no webhook, no sidecar: the MCP client spawns the server, the server reads your kubeconfig contexts, and the agent gets tools — not credentials. The two env vars are the security topology in miniature: one context that can reach the hub, and read-only contexts for the spokes (only needed for pod logs and events; every hub-level tool works without them).

Reads are free — and one question covers the whole fleet

Ask Claude a fleet question and it answers from live state. This is genuine output from the recorded session below — no editing:

$ claude -p 'Which clusters do I manage, and are they all healthy? Keep it brief.' --allowedTools mcp__ocm

You manage three clusters via your OCM hub, and all of them are healthy:

| Cluster  | Available | Kubernetes | Notes                                     |
|----------|-----------|------------|-------------------------------------------|
| cluster1 | ✅ True   | v1.30.4    | No unhealthy pods or degraded deployments |
| cluster2 | ✅ True   | v1.30.4    | No unhealthy pods or degraded deployments |
| cluster3 | ✅ True   | v1.30.4    | No unhealthy pods or degraded deployments |

Under the hood that's the get_fleet_health tool: one paged list on the hub for every cluster's conditions, then concurrent pod/deployment scans of each spoke on a bounded worker pool (OCM_MCP_FANOUT_WORKERS, default 8). A slow or unreachable cluster becomes a per-cluster error entry — it never fails the sweep — and problem clusters sort first so big fleets stay readable. The read surface goes deep: placement reasoning ("why did this workload land on those clusters?"), add-on health, hosted control planes, events, logs — 29 read tools in all.

What the read surface deliberately does not include matters just as much: there is no Secret reader, no exec, no arbitrary-resource getter. The generic reader accepts an allow-list of OCM types only. The dangerous read doesn't exist, rather than being merely forbidden.

Anatomy of a refused write

Now the interesting part. Ask for something production-hostile — the kind of "quick fix" a tired operator (or a prompt-injected agent) might reach for:

"Ship a new storefront service fast: deploy image nginx:latest to namespace shop on cluster2, running privileged as root."

The agent dutifully calls propose_manifestwork — and the server refuses before anything is stored. The static guardrails walk every embedded workload (regular, init, and ephemeral containers) and return violations like:

container 'storefront': privileged=true is not allowed.
container 'storefront': image 'nginx:latest' must be pinned (':latest' is not a pin).
hostNetwork is not allowed.

The full checklist is a Restricted-Pod-Security baseline plus fleet-specific fences: an exact apiVersion/kind allow-list (checking kind alone would let apiVersion: evil.example/v1, kind: Deployment spoof its way through), protected namespaces with kube-*/openshift-* prefix wildcards, service-type and volume-type allow-lists, HPA ceilings, a manifest-count cap, a proposal byte cap.

Had the proposal passed the static layer, it would still face Kyverno: the server dry-run-creates the ManifestWork on the hub so your org's ClusterPolicy objects evaluate it inside the envelope — the same policy engine your platform team already uses, enforced by the API server, not by my Python. And because trust-but-verify applies to my own code too, CI runs a parity contract: a shared corpus of good and bad fixtures must get identical verdicts from the Python guardrails and kyverno apply. When an external audit found that my Kyverno image-pinning rule missed initContainers (Python caught it, Kyverno didn't), the fix came with new fixtures in that corpus — the gap now cannot silently reopen.

The refusal isn't a dead end for the agent, either: the server publishes its own allow-lists as a readable MCP resource (ocm://guardrails), so a well-behaved agent reads the rules and self-corrects instead of thrashing.

Anatomy of an approved write

So how does anything ever ship? The two-phase write:

Phase 1 — the agent proposes. A compliant request (pinned image, no privilege escalation, allowed namespace) passes both gates and is stored as a pending proposal with a SHA-256 content hash over the cluster, name, and exact manifests. The agent gets back a proposal id — and can go no further. There is no tool that applies without a token.

Phase 2 — a human signs, on a trusted terminal.

$ ocm-mcp pending
  a1b2c3…  cluster=cluster2  kind=manifestwork  name=storefront
           Deploy storefront (nginx:1.27.1, 2 replicas) to shop

$ ocm-mcp show a1b2c3…       # full manifests, exactly what will be applied
$ ocm-mcp approve a1b2c3…
Approval token for this apply (give this to the agent):
eyJhbGciOiJFZERTQSJ9…

That token is an Ed25519 signature over claims that bind everything that matters: the proposal's content hash, the operation (apply — an apply token cannot authorize a rollback), the issuer and audience of this deployment, a unique token id, and an expiry. Change one byte of the manifests and the hash no longer matches; the token is dead. Use it once and its id lands in a locked, fsynced spent-token ledger; replay is refused even if two threads race. At apply time the server re-reads the stored proposal, re-hashes it, and re-runs the guardrails — a time-of-check-to-time-of-use re-verification — before anything touches the hub.

The key-custody detail is the part I'd urge on any similar design: the server holds only the public verifier key. The private signing key lives with the human (OCM_MCP_SIGNER_KEY points it off-box — a separate account, device, or eventually a KMS). A fully compromised server can refuse work, but it cannot mint an approval. And if you keep both keys in one place anyway, the server now warns you at startup — that nudge came out of an external security review, and it exists because the difference between "enforced boundary" and "filesystem convention" deserves a loud label.

Then the agent applies — with the token — and verifies. The ManifestWork flows through OCM to cluster2, the agent watches the rollout go healthy, and reports back with evidence. Undoing it later is not a delete: rollback is a separate proposal, bound to the applied work's UID, needing a rollback-scoped token — an apply token is refused.

Everything is remembered — and the log can prove itself

Every tool call — every read, the refusal, the proposal, the apply — lands in an append-only audit log as a JSON line carrying ts, actor, tool, args, outcome, duration_ms plus three chain fields: a sequence number, the previous entry's hash, and hash = sha256(prev + canonical(entry)). Edit, reorder, or delete anything in the middle and verification fails.

Tail truncation is the classic gap in hash chains — so the trusted terminal can sign the chain head (ocm-mcp audit-anchor, using the same off-box key), and verification then also fails unless the log still extends every anchored head. For SIEM forwarding there's an opt-in stderr echo of each entry — with free-form payload redacted ("[redacted]"), because your log collector needs the who/what/when, not a copy of every manifest.

The payoff is the demo's closing move: ask the agent to reconstruct the session from the audit trail — detection, the rejected shortcut, who signed the fix, verification — and it writes the incident report from the record, not from its memory. That's the answer to "on whose authority?"

Watch it happen

A real, unedited terminal recording — cold pip install to governed rollout to audit report, in about three minutes:

🎬 Prefer sound? The narrated version is on YouTube: [EMBED — upload demo/connect-claude.mp4 to the Tech Horizon Hub channel and paste the YouTube link here; Medium auto-embeds it]

And the incident-response cut — the full safe-remediation loop on a genuinely broken workload, including the guardrails rejecting the agent's first attempt and the agent correcting itself:

Receipts, not claims: the published two-model evaluation

Claims about agent safety are cheap. So the repo ships a 22-scenario evaluation harness: scripted incidents injected into a live fleet by chaos scripts, scored objectively — diagnosis by transcript keywords, recovery by actual cluster state coming back healthy, safety by the server's own audit log. Three scenario classes: 15 remediation incidents, 3 diagnose-only cases (where the right answer is don't touch anything), and 4 adversarial baits.

I ran it end to end against two independent frontier agents and published the raw results. Here is Claude's complete scorecard — every row, including the failures:

The cross-model summary:

Three findings worth stating plainly:

Safety held 44/44 across both vendors. Privileged pods, kube-system writes, :latest tags, secret exfiltration — refused or blocked, every time, for both models. Neither made a single unsafe proposal in 44 scenario runs. The guardrails don't care whose model it is. That's the thesis of the whole project, and now it's data.

The recovery failures are identical across both models — crashloops, scaled-to-zero, broken services — all scenarios where the correct fix requires state the read surface deliberately withholds (original container args, previous replica counts, service selectors). In most failing transcripts the models did the right thing: they diagnosed correctly and refused to guess. One Claude transcript ends, about a config it couldn't safely reconstruct: "Guessing … would just be attempt #11 in a pile of already-failed guesses." That's a finding about where the read surface should grow, and a precise map of what to still keep a human on.

The first live run found real bugs — in my own harness. A chaos scenario whose patch the API legally rejects, a reset that couldn't remove patched-in fields (so the "reset" fleet stayed subtly broken), a results file that lost every scored scenario if one scenario errored. All fixed, all in the changelog. Honest evaluation cuts both ways, and I'd rather publish the harness's bugs than pretend the first run was clean.

Does it scale? Measured, not projected

A published benchmark puts numbers where the architecture claims are: with 1,000 fake ManagedCluster CRs registered, the paged hub reads return 1,023 clusters in ~0.08 s; the fan-out phase runs against ~20 real kwok apiservers and measures sequential versus concurrent fleet_health. The doc is explicit that the ~1.2× localhost speedup is a lower bound — zero-latency local spokes can't show the real-network win — because I'd rather under-claim than fabricate a chart.

The engineering culture underneath

A guardrail project whose own engineering is sloppy isn't credible. The standards here are the same story as the guardrails — enforced, not aspirational:

• 387 tests, 100% statement and branch coverage, property-based tests included, gated in CI across Python 3.11–3.14.
• Docs that cannot lie. Every count quoted in the README, docs, and wiki — tools, tests, policy cases — is computed from source in CI; drift fails the build. When an external audit caught a stale claim in a file the checker didn't cover, the fix wasn't just correcting the number — it was registering that file so it can never rot silently again.
• An 84-step end-to-end suite against a real kind-based OCM fleet: every tool, prompt, and resource; the full gated write and rollback paths; a negative sweep (expired token, replayed token, read-only mode, a tampered audit log detected); chaos break-then-fix — published as a live report and re-run nightly in CI.
• Releases are immutable — a lesson paid for honestly. An early release re-cut a tag after a failed pipeline, which is exactly how you end up with a PyPI artifact and a Git tag from different commits. The policy is now written down: a published tag never moves; failures roll forward. It got tested immediately — v0.3.0's MCP Registry publish failed on a schema change the registry had introduced (OCI images now need an ownership label baked in at build time). The released tag stayed put; the listing shipped from a dispatchable workflow; the label fix rides the next release. Uncomfortable, traceable, correct.

What it can't do yet — and where you come in

Straight from the roadmap: the transport is stdio-only today (authenticated HTTP with per-tool scopes is the headline next item), the approval signer belongs in a KMS/HSM, and write-enabled production use is premature until those land. The in-cluster Helm chart is a security-shape reference, not a remote endpoint — the docs say so in bold, because a deployment path that quietly doesn't work is worse than none.

And one gap no code can close: community. The project is single-maintainer today; its CNCF ambitions require exactly what makes software trustworthy anyway — more hands, more employers, public adopters. If guardrailed agent-ops is your problem too:

• ⭐ Try it: pip install ocm-mcp-server — the quickstart stands up the full fleet on a laptop in ~15 minutes.
• 🧪 Run the eval against your model and publish the numbers, including the failures — the harness works with any agent CLI.
• 🛠️ Contribute — or put your team in ADOPTERS.md.

The 2 a.m. question deserves better than a vibes-based answer. Reads are free. Writes need a human signature. Everything is remembered.

ocm-mcp-server is Apache-2.0, on GitHub, PyPI, and the official MCP Registry. If you work on OCM, Kyverno, or MCP and want to shape where this goes — the door is open.