Architecture

The choke-point idea #

Fleet operations already flow through a hub: Open Cluster Management gives us cluster inventory (ManagedCluster), scheduling (Placement), and delivery (ManifestWork). Instead of handing an agent N kubeconfigs, we hand it a narrow, typed view of that hub - one place to observe, one place to constrain.

agent (any MCP client)
   │  typed MCP tools
   ▼
ocm-mcp-server ──── audit.jsonl (every call)
   │                 └─ OTel spans → Jaeger
   │ static guardrails (layer 1)
   ▼
OCM hub API
   │ Kyverno admission, incl. dry-run   (layer 2)
   │ human approval token               (layer 3)
   │ least-privilege RBAC               (layer 4)
   ▼
ManifestWork → work agent on each managed cluster

Components #

Low-level design - the full internals, vertically #

Everything below is the same system, cut five ways: the full component stack, the anatomy of one read call, the complete gated write sequence, the integrity machinery (state, keys, audit), and the observability signals. File references are to src/ocm_mcp_server/.

1. The full vertical stack #

Every box is a real component; every arrow is a real call path or protocol.

2. Anatomy of one read call - list_clusters #

What actually happens, function by function, for the simplest tool:

Spoke-touching reads (get_cluster_health, query_events, get_pod_logs) differ in one step: k8s.spoke_core(cluster) resolves the per-cluster read context and every list is bounded (limit=OCM_MCP_HEALTH_LIMIT, request timeout (5, 30)s), with an explicit note when the cluster has more than the limit.

get_fleet_health is get_cluster_health widened to the whole fleet in one call: one paged hub list for every cluster's conditions, then a bounded thread pool fans out the same per-cluster spoke scan concurrently instead of the agent looping the single-cluster tool once per member.

3. The gated write path - every check, in order #

Nothing in this sequence is advisory; each numbered gate refuses on its own.

4. Rollback and lifecycle actions - the same gate, different verbs #

5. Integrity machinery - audit chain, anchors, keys, state #

6. Observability - three independent signals per tool call #

Every tool call passes through the traced_tool decorator (tracing.py), which emits up to three records with different jobs and different trust properties:

Why three signals instead of one: the audit log is a safety artifact - append-only, hash-chained, anchor-signed, and always on; incident reports and the eval harness are built from it. The OTel span is a debugging artifact - it shows call structure and latency in Jaeger but is deliberately not trusted for anything safety-related, so it can be disabled (and is fail-soft: without the [tracing] extra installed and OTEL_EXPORTER_OTLP_ENDPOINT set, the span layer is a no-op and tools behave identically). Metrics are an alerting artifact - aggregate counters with no per-call payload at all.

Enable tracing with two switches (full guide: deployment.md):

pip install "ocm-mcp-server[tracing]"
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318   # e.g. Jaeger all-in-one

Tested at three levels: unit tests cover span creation, redaction, and the no-op paths; the e2e suite's tracing-export step points a fresh server process at a local OTLP sink and asserts a real trace batch arrives naming the tool.* span and the ocm-mcp-server service; and make bootstrap starts a real Jaeger container (--no-jaeger to skip) so every local-fleet tool call is browsable at http://localhost:16686.

7. Where each guarantee is enforced (quick index) #

The parity between the left and middle columns is not aspirational: CI runs the same fixture corpus through both layers and fails on any verdict mismatch.

Design decisions worth arguing about #

Why validate ManifestWorks, not Pods? Policies on the managed clusters see resources only after delivery. Validating the envelope on the hub rejects bad content before it ever leaves - at proposal time, via server-side dry-run, so the agent gets the policy message as feedback and can self-correct.

Why an Ed25519-signed token instead of a "yes" in chat? A chat approval approves a conversation. The token approves a content hash and an operation: if the agent mutates the proposal after approval, the signature no longer verifies, and an apply token cannot authorize a rollback. Approval is asymmetric - the CLI signs with a private key the server never holds, so a compromised server cannot mint one - provided the private signing key is kept off the server (OCM_MCP_SIGNER_KEY on a separate account/device); co-located, that is a filesystem convention, not a boundary. Tokens are single-use and expire (default 1 h), minted only by the CLI on a trusted terminal.

Why per-spoke read ServiceAccounts in the quickstart? Simplicity. The production-correct path is the OCM cluster-proxy add-on (hub-mediated access, no direct spoke credentials on the server host); the tool surface is identical, so swapping the transport does not change the agent's world.

Why no Secrets/exec tools at all? Any tool that exists will eventually be called. Capabilities that are absent cannot be prompt-injected into use.

Scaling the pattern #

• More clusters: nothing changes - the hub is the fan-out point.
• More agents: one server per agent identity, each with its own RBAC and audit.
• Other hubs: the guardrail pattern (static → policy dry-run → human token → RBAC) ports to any declarative delivery API, not just OCM.

Source: docs/architecture.md — this page is generated from it, so edit the markdown, not the site.