ocm-mcp-server
GitHub

ocm-mcp-server - end-to-end test report

Verdict: 🟢 ALL GREEN  ·  generated 2026-07-29 01:34:45  ·  local kind fleet via podman, Open Cluster Management hub

Passed Expected-unavailable Failed Total steps
81 0 0 84

Reproduce locally with ./hack/e2e-local.sh - one command stands up a real multi-cluster fleet on kind, drives every tool and prompt, breaks a workload and fixes it through the approval gate, then tears the fleet down.

What this end-to-end test does #

One command stands up a real multi-cluster Open Cluster Management fleet on kind, then drives the MCP server through every tool and prompt: it reads the fleet freely, but forces every change through policy checks and a human-approved token. It then deliberately breaks a workload and shows the agent diagnose and fix it, before tearing the fleet down. Green means the whole loop worked.

  1. 🧰 Install & verify tools - podman, kind, kubectl, clusteradm, helm
  2. ☸️ Bootstrap a real fleet - 1 hub + N spoke clusters, OCM, Kyverno, policies
  3. Enrich & seed fixtures - add-ons, policies, ManifestWorks - so every tool has real data
  4. 🔍 Investigate (read tools) - inventory, health, placement, work, add-ons - free, no gate
  5. 🛡️ Make a change, gated - propose → Kyverno + guardrails → human token → apply (a bad token is refused)
  6. 💬 Prompts & audit - reusable runbooks and an append-only audit trail
  7. 💥 Break something - inject a failing rollout (ImagePullBackOff)
  8. 🩺 Diagnose - health + events pinpoint the bad image
  9. 🔧 Fix, through the gate - propose the pinned image, approve, apply
  10. Verify recovery - re-read health until the workload is healthy again
  11. 🧹 Clean up - delete only the clusters this run created

1. Dependencies #

OK podman #

Required tool for the local fleet.

$ podman --version
already installed: podman version 6.0.2

OK kind #

Required tool for the local fleet.

$ kind version
already installed: kind v0.32.0 go1.26.3 darwin/arm64

OK kubectl #

Required tool for the local fleet.

$ kubectl version --client=true -o yaml 2>/dev/null | grep gitVersion | head -1
already installed:   gitVersion: v1.34.1

OK helm #

Required tool for the local fleet.

$ helm version --short
already installed: v4.1.1+g5caf004

OK clusteradm #

Required tool for the local fleet.

$ clusteradm version | head -1
already installed: clusteradm	version	:v1.3.1-0-g90bdc31

OK oc #

Required tool for the local fleet.

$ oc version --client 2>/dev/null | head -1
already installed: Client Version: 4.18.0-202504282205.p0.geb9bc9b.assembly.stream.el9-eb9bc9b

2. Python package #

OK pip install -e . #

Install the MCP server and its ocm-mcp CLI into an isolated virtualenv.

$ pip install -e .
ocm-mcp-server 0.2.2

3. Container engine #

OK podman running #

kind needs a container runtime; here it is Podman (Docker is not required).

$ podman info
podman version 6.0.2

4. Bootstrap fleet #

OK bootstrap.sh #

Stand up a real OCM hub with spokes so every tool talks to genuine clusters, not mocks.

$ CONTAINER_ENGINE=podman SPOKES=2 ./hack/bootstrap.sh --no-jaeger

==> Deploying the demo app to all spokes (namespace 'shop')
namespace/shop created
deployment.apps/payments created
service/payments created
namespace/shop created
deployment.apps/payments created
service/payments created

==> Done. Configure your MCP client environment:

  export OCM_MCP_HUB_CONTEXT=kind-hub
  export OCM_MCP_SPOKE_CONTEXTS=cluster1=kind-cluster1,cluster2=kind-cluster2
  export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318   # if Jaeger enabled

  # then register the server with your MCP client, e.g. for Claude Code:
  #   see examples/claude-code.mcp.json

Verify the fleet:
  kubectl --context kind-hub get managedclusters

OK managed clusters #

Proof the spokes are registered and Available on the hub.

$ kubectl --context kind-hub get managedclusters
NAME       HUB ACCEPTED   MANAGED CLUSTER URLS                  JOINED   AVAILABLE   AGE
cluster1   true           https://cluster1-control-plane:6443   True     True        106s
cluster2   true           https://cluster2-control-plane:6443   True     True        77s

4b. Enrich fleet #

OK add-on, feature gate, and API fixtures #

Install the governance policy add-on, turn on ManifestWorkReplicaSet, and add minimal CRDs plus labelled sample objects for the ACM (ManagedClusterInfo) and HyperShift (HostedCluster/NodePool) APIs - so the add-on, policy, rollout, ACM, and HyperShift tools all return real objects. On a real hub those come from ACM/MCE or HyperShift; on kind they are clearly-labelled e2e fixtures.

$ clusteradm install/enable hub-addon; kubectl patch clustermanager; kubectl apply CRDs + samples
enrichment: governance-policy-framework(enabled) ManifestWorkReplicaSet(enabled) acm+hypershift-fixtures

4c. Setup fixtures - create the objects the read tools will report on #

OK Placement + ClusterSetBinding #

Bind the 'global' cluster set into a namespace and add a Placement that selects every cluster - so the placement tools have a real decision to show.

$ kubectl apply Placement + ManagedClusterSetBinding
binding=created, placement=created

OK seed ManifestWork (gated) #

Deploy a marker ManifestWork via propose+approve+apply so the work tools report on something real.

$ propose_manifestwork -> apply_manifestwork
proposal_id=f167cb1929d542e9a746b3f06a90dda4
{
  "status": "applied",
  "cluster": "cluster1",
  "manifestwork": "e2e-seed",
  "note": "Verify rollout with get_cluster_health / get_manifestwork. To undo, c

OK ManifestWorkReplicaSet #

A single template fanned across every placement-selected cluster - so list_manifestworkreplicasets shows a rollout.

$ kubectl apply ManifestWorkReplicaSet
result=created

OK governance Policy #

A sample inform Policy so the policy tools report on a real object.

$ kubectl apply Policy
result=created

5. Read tools (safe, no gate) - how an agent investigates a fleet #

OK list_clusters #

The fleet roster: every managed cluster, its version and health.

$ tool: list_clusters
[
  {
    "name": "cluster1",
    "labels": {
      "cluster.open-cluster-management.io/clusterset": "default",
      "feature.open-cluster-management.io/addon-governance-policy-framework": "unreachable"
    },
    "available": "True",
    "joined": "True",
    "kubernetes_version": "v1.30.4",
    "capacity": {
      "cpu": "4",
      "memory": "8099492Ki"
    }
  },
  {
    "name": "cluster2",
    "labels": {
      "cluster.open-cluster-management.io/clusterset": "default",
      "feature.open-cluster-management.io/addon-governance-policy-framework": "unreachable"
    },
    "available": "True",
    "joined": "True",
    "kubernetes_version": "v1.30.4",
    "capacity": {
      "cpu": "4",
      "memory": "8099492Ki"
    }
  }
]

OK get_cluster(cluster1) #

Zoom into one cluster: acceptance, taints, capacity, claims.

$ tool: get_cluster
{
  "name": "cluster1",
  "labels": {
    "cluster.open-cluster-management.io/clusterset": "default",
    "feature.open-cluster-management.io/addon-governance-policy-framework": "unreachable"
  },
  "hub_accepts_client": true,
  "taints": [],
  "conditions": {
    "HubAcceptedManagedCluster": "True",
    "ManagedClusterJoined": "True",
    "ManagedClusterConditionAvailable": "True",
    "ManagedClusterConditionClockSynced": "True"
  },
  "kubernetes_version": "v1.30.4",
  "capacity": {
    "cpu": "4",
    "ephemeral-storage": "104266732Ki",
    "hugepages-1Gi": "0",
    "hugepages-2Mi": "0",
    "hugepages-32Mi": "0",
    "hugepages-64Ki": "0",
    "memory": "8099492Ki",
    "pods": "110"
  },
  "allocatable": {
    "cpu": "4",
    "ephemeral-storage": "104266732Ki",
    "hugepages-1Gi": "0",
    "hugepages-2Mi": "0",
    "hugepages-32Mi": "0",
    "hugepages-64Ki": "0",
    "memory": "8099492Ki",
    "pods": "110"
  },
  "cluster_claims": {}
}

OK list_cluster_sets #

Groupings of clusters used for placement.

$ tool: list_cluster_sets
[
  {
    "name": "default",
    "selector_type": "ExclusiveClusterSetLabel",
    "conditions": {
      "ClusterSetEmpty": "False"
    },
    "members": [
      "cluster1",
      "cluster2"
    ]
  },
  {
    "name": "global",
    "selector_type": "LabelSelector",
    "conditions": {
      "ClusterSetEmpty": "False"
    },
    "members": []
  }
]

OK list_cluster_set_bindings #

Which cluster sets a namespace may schedule to.

$ tool: list_cluster_set_bindings
[
  {
    "namespace": "default",
    "name": "global",
    "cluster_set": "global",
    "conditions": {
      "Bound": "True"
    }
  }
]

OK list_cluster_claims #

Facts each cluster self-reports (id, platform, version).

$ tool: list_cluster_claims
[
  {
    "cluster": "cluster1",
    "claims": {}
  },
  {
    "cluster": "cluster2",
    "claims": {}
  }
]

OK get_cluster_health(cluster1) #

The on-call view: unhealthy pods and degraded deployments.

$ tool: get_cluster_health
{
  "cluster": "cluster1",
  "hub_conditions": {
    "HubAcceptedManagedCluster": "True",
    "ManagedClusterJoined": "True",
    "ManagedClusterConditionAvailable": "True",
    "ManagedClusterConditionClockSynced": "True"
  },
  "unhealthy_pods": [],
  "degraded_deployments": [
    {
      "namespace": "shop",
      "name": "payments",
      "ready": "0/2"
    }
  ],
  "spoke_view": "ok"
}

OK get_fleet_health() #

The whole-fleet sweep in one call: hub conditions for every cluster plus a fanned-out spoke pod/deployment scan - no per-cluster looping needed.

$ tool: get_fleet_health
{
  "fleet": {
    "total": 2,
    "available": 2,
    "unavailable": 0,
    "spoke_checked": 2,
    "with_issues": 2
  },
  "clusters": [
    {
      "cluster": "cluster1",
      "available": "True",
      "hub_conditions": {
        "HubAcceptedManagedCluster": "True",
        "ManagedClusterJoined": "True",
        "ManagedClusterConditionAvailable": "True",
        "ManagedClusterConditionClockSynced": "True"
      },
      "unhealthy_pods": [],
      "degraded_deployments": [
        {
          "namespace": "shop",
          "name": "payments",
          "ready": "0/2"
        }
      ],
      "spoke_view": "ok"
    },
    {
      "cluster": "cluster2",
      "available": "True",
      "hub_conditions": {
        "HubAcceptedManagedCluster": "True",
        "ManagedClusterJoined": "True",
        "ManagedClusterConditionAvailable": "True",
        "ManagedClusterConditionClockSynced": "True"
      },
      "unhealthy_pods": [
        {
          "namespace": "shop",
... (19 more lines)

OK query_events(cluster1) #

Recent Kubernetes events - the 'why' behind failures.

$ tool: query_events
[
  {
    "namespace": "shop",
    "type": "Normal",
    "reason": "Pulled",
    "object": "Pod/payments-5555cb47bc-kswhv",
    "count": 1,
    "message": "Successfully pulled image \"registry.k8s.io/e2e-test-images/agnhost:2.47\" in 251ms (17.193s including waiting). Image size: 50637101 bytes."
  },
  {
    "namespace": "shop",
    "type": "Normal",
    "reason": "Created",
    "object": "Pod/payments-5555cb47bc-kswhv",
    "count": 1,
    "message": "Created container payments"
  },
  {
    "namespace": "shop",
    "type": "Normal",
    "reason": "Started",
    "object": "Pod/payments-5555cb47bc-kswhv",
    "count": 1,
    "message": "Started container payments"
  },
  {
    "namespace": "shop",
    "type": "Normal",
    "reason": "Pulled",
    "object": "Pod/payments-5555cb47bc-zkb8b",
    "count": 1,
    "message": "Successfully pulled image \"registry.k8s.io/e2e-test-images/agnhost:2.47\" in 16.953s (16.953s including waiting). Image size: 50637101 bytes."
  },
  {
    "namespace": "shop",
    "type": "Normal",
    "reason": "Created",
    "object": "Pod/payments-5555cb47bc-zkb8b",
    "count": 1,
    "message": "Created container payments"
... (13 more lines)

OK list_manifestworks(cluster1) #

What the hub is currently delivering to this cluster.

$ tool: list_manifestworks
[
  {
    "name": "demo-mwrshhx5b",
    "applied": "True",
    "available": "True",
    "resources": [
      "ConfigMap/e2e-seed (shop)"
    ]
  },
  {
    "name": "e2e-seed",
    "applied": "True",
    "available": "True",
    "resources": [
      "ConfigMap/e2e-seed (shop)"
    ]
  }
]

OK list_manifestworkreplicasets #

Fleet-wide rollouts (a template fanned across clusters).

$ tool: list_manifestworkreplicasets
[
  {
    "namespace": "default",
    "name": "demo-mwrs",
    "summary": {
      "applied": 2,
      "available": 2,
      "total": 2
    },
    "conditions": {
      "PlacementVerified": "True",
      "PlacementRolledOut": "False",
      "ManifestworkApplied": "True"
    }
  }
]

OK list_cluster_management_addons #

Fleet-level add-on definitions.

$ tool: list_cluster_management_addons
[
  {
    "name": "config-policy-controller",
    "install_strategy": "Manual",
    "conditions": {}
  },
  {
    "name": "governance-policy-framework",
    "install_strategy": "Manual",
    "conditions": {}
  }
]

OK get_addon_health #

Per-cluster add-on health across the fleet.

$ tool: get_addon_health
[
  {
    "cluster": "cluster1",
    "addon": "governance-policy-framework",
    "available": "Unknown",
    "degraded": "False",
    "progressing": "False"
  },
  {
    "cluster": "cluster2",
    "addon": "governance-policy-framework",
    "available": "Unknown",
    "degraded": "False",
    "progressing": "False"
  }
]

OK list_addons_for_cluster(cluster1) #

Every add-on installed on one cluster, with health.

$ tool: list_addons_for_cluster
[
  {
    "addon": "governance-policy-framework",
    "install_namespace": "open-cluster-management-agent-addon",
    "available": "Unknown",
    "degraded": "False"
  }
]

OK list_pending_csrs #

Clusters waiting to be admitted to the hub.

$ tool: list_pending_csrs
No pending CSRs - every cluster is already admitted (a healthy fleet).

OK list_placements #

Placements and how many clusters each selects.

$ tool: list_placements
[
  {
    "namespace": "default",
    "name": "demo-all",
    "cluster_sets": [
      "global"
    ],
    "number_of_clusters": null,
    "selected": 2,
    "conditions": {
      "PlacementMisconfigured": "False",
      "PlacementSatisfied": "True"
    }
  }
]

OK get_placement_decision(demo-all) #

Exactly which clusters a Placement chose.

$ tool: get_placement_decision
{
  "placement": "demo-all",
  "namespace": "default",
  "selected_clusters": [
    "cluster1",
    "cluster2"
  ],
  "count": 2
}

OK list_policies #

Governance Policies and per-cluster compliance.

$ tool: list_policies
[
  {
    "namespace": "default",
    "name": "require-cm",
    "remediation": "inform",
    "compliant": null,
    "per_cluster": {}
  }
]

OK list_policy_violations #

Only the NonCompliant / Pending policy-cluster pairs.

$ tool: list_policy_violations
No violations - evaluated policies are compliant (or evaluation is in progress).

OK get_cluster_info(cluster1) #

Extended inventory (OpenShift version, nodes) - ACM only.

$ tool: get_cluster_info
{
  "cluster": "cluster1",
  "console_url": "https://console-openshift-console.apps.cluster1.example.com",
  "kube_vendor": "OpenShift",
  "cloud_vendor": "BareMetal",
  "openshift_version": "4.16.7",
  "node_count": 1,
  "nodes": [
    {
      "name": "cluster1-control-plane",
      "capacity": {
        "cpu": "8",
        "memory": "16Gi"
      },
      "labels": {
        "node-role.kubernetes.io/control-plane": ""
      }
    }
  ],
  "conditions": {
    "ManagedClusterInfoSynced": "True"
  }
}

OK list_hosted_clusters #

HyperShift hosted control planes - when the hub hosts them.

$ tool: list_hosted_clusters
[
  {
    "name": "demo-hcp",
    "namespace": "clusters",
    "version": "4.16.7",
    "version_state": "Completed",
    "conditions": {
      "Available": "True"
    }
  }
]

OK list_node_pools #

HyperShift worker node pools - when the hub hosts HCPs.

$ tool: list_node_pools
[
  {
    "name": "demo-hcp-workers",
    "namespace": "clusters",
    "cluster": "demo-hcp",
    "desired_replicas": 2,
    "current_replicas": 2,
    "conditions": {
      "Ready": "True"
    }
  }
]

OK list_resources(managedclusters) #

Generic allow-listed reader over any OCM type.

$ tool: list_resources
[
  {
    "name": "cluster1",
    "labels": {
      "cluster.open-cluster-management.io/clusterset": "default",
      "feature.open-cluster-management.io/addon-governance-policy-framework": "unreachable"
    },
    "conditions": {
      "HubAcceptedManagedCluster": "True",
      "ManagedClusterJoined": "True",
      "ManagedClusterConditionAvailable": "True",
      "ManagedClusterConditionClockSynced": "True"
    }
  },
  {
    "name": "cluster2",
    "labels": {
      "cluster.open-cluster-management.io/clusterset": "default",
      "feature.open-cluster-management.io/addon-governance-policy-framework": "unreachable"
    },
    "conditions": {
      "HubAcceptedManagedCluster": "True",
      "ManagedClusterJoined": "True",
      "ManagedClusterConditionAvailable": "True",
      "ManagedClusterConditionClockSynced": "True"
    }
  }
]

OK get_resource(managedclusters/cluster1) #

Generic get of one allow-listed OCM object.

$ tool: get_resource
{
  "apiVersion": "cluster.open-cluster-management.io/v1",
  "kind": "ManagedCluster",
  "metadata": {
    "creationTimestamp": "2026-07-28T20:01:37Z",
    "finalizers": [
      "cluster.open-cluster-management.io/resource-cleanup",
      "cluster.open-cluster-management.io/api-resource-cleanup"
    ],
    "generation": 4,
    "labels": {
      "cluster.open-cluster-management.io/clusterset": "default",
      "feature.open-cluster-management.io/addon-governance-policy-framework": "unreachable"
    },
    "managedFields": [
      {
        "apiVersion": "cluster.open-cluster-management.io/v1",
        "fieldsType": "FieldsV1",
        "fieldsV1": {
          "f:spec": {
            "f:hubAcceptsClient": {}
          }
        },
        "manager": "clusteradm",
        "operation": "Update",
        "time": "2026-07-28T20:02:14Z"
      },
      {
        "apiVersion": "cluster.open-cluster-management.io/v1",
        "fieldsType": "FieldsV1",
        "fieldsV1": {
          "f:status": {
            ".": {},
            "f:allocatable": {
              ".": {},
              "f:cpu": {},
              "f:ephemeral-storage": {},
              "f:hugepages-1Gi": {},
              "f:hugepages-2Mi": {},
              "f:hugepages-32Mi": {},
... (7 more lines)

OK list_addon_placement_scores(cluster1) #

Per-cluster placement scores add-ons publish.

$ tool: list_addon_placement_scores
No AddOnPlacementScores - no score-publishing add-on runs on this fleet.

OK get_pod_logs(cluster1) #

Container logs from a spoke pod, via the hub-known read context.

$ tool: get_pod_logs
{
  "namespace": "shop",
  "pod": "payments-5555cb47bc-kswhv",
  "log_tail": "b'I0728 20:03:40.596840       1 log.go:245] Started HTTP server on port 8080\\nI0728 20:03:40.597016       1 log.go:245] Started UDP server on port  8081\\n'"
}

OK get_hosted_cluster(demo-hcp) #

One HyperShift hosted control plane in detail.

$ tool: get_hosted_cluster
{
  "name": "demo-hcp",
  "namespace": "clusters",
  "version": "4.16.7",
  "version_state": "Completed",
  "conditions": {
    "Available": "True"
  },
  "node_pools": [
    {
      "name": "demo-hcp-workers",
      "namespace": "clusters",
      "cluster": "demo-hcp",
      "desired_replicas": 2,
      "current_replicas": 2,
      "conditions": {
        "Ready": "True"
      }
    }
  ]
}

7. Gated write flow - propose, prove the gate rejects a bad token, then apply #

OK propose_manifestwork #

The agent PROPOSES a change. It must pass static guardrails and a Kyverno dry-run first; nothing is applied yet.

$ tool: propose_manifestwork
proposal_id=833e5c116f2649548b188ad3a41b7808
status=pending_approval

PASS apply with a BAD token #

Prove the gate holds: apply with an invalid token must be refused by the server, not applied.

$ tool: apply_manifestwork(bad token)
REJECTED: Malformed approval token.

OK ocm-mcp approve (human) #

A human on a trusted terminal mints an approval token: an Ed25519 signature binding the exact content and operation. The server holds only the public key, so it can verify a token but can never mint one - and neither can the agent.

$ ocm-mcp approve 833e5c116f2649548b188ad3a41b7808
token minted (...baEUVSYYnqDw)

OK apply_manifestwork(token) #

With a valid token, the server verifies it and delivers the change as an OCM ManifestWork.

$ tool: apply_manifestwork
{
  "status": "applied",
  "cluster": "cluster1",
  "manifestwork": "e2e-demo",
  "note": "Verify rollout with get_cluster_health / get_manifestwork. To undo, call propose_rollback then apply the rollback with a fresh approval."
}

PASS get_manifestwork (verify) #

Confirm the hub actually applied it on the spoke.

$ tool: get_manifestwork
{
  "cluster": "cluster1",
  "name": "e2e-demo",
  "conditions": {
    "Applied": "True",
    "Available": "True"
  },
  "resources": [
    {
      "resource": "ConfigMap/ocm-mcp-e2e",
      "namespace": "shop",
      "conditions": {
        "Applied": "True",
        "Available": "True",
        "StatusFeedbackSynced": "True"
      },
      "status_feedback": {}
    }
  ]
}

7b. Gated rollback - undoing an applied change needs its own approval #

OK propose_rollback #

The agent proposes UNDOING the applied change. This creates a distinct proposal bound to the exact ManifestWork name and UID - an old apply token can never delete a workload.

$ tool: propose_rollback
{
  "rollback_proposal_id": "cd14dac0c4af4a708591c37410772285",
  "status": "pending_approval",
  "next_step": "Ask the human operator to run: ocm-mcp approve cd14dac0c4af4a708591c37410772285 and give you the token, then call rollback_manifestwork(cd14dac0c4af4a708591c37410772285, <token>)."
}

PASS rollback with an APPLY token #

Prove operation binding: a token minted for 'apply' must be refused for a rollback.

$ tool: rollback_manifestwork(apply-scoped token)
REJECTED: This token authorizes 'apply', not 'rollback'.

OK rollback_manifestwork(token) #

With a rollback-scoped token, the server removes the ManifestWork it created.

$ tool: rollback_manifestwork
{
  "status": "rolled_back",
  "cluster": "cluster1",
  "manifestwork": "e2e-demo"
}

PASS verify rollback #

The e2e-demo ManifestWork no longer exists on the hub.

$ tool: get_manifestwork(cluster1, e2e-demo)
ManifestWork removed - rollback confirmed.

8. Gated lifecycle action - cordon a cluster out of scheduling, then undo #

OK propose_cluster_action(cordon) #

Propose adding a NoSelect taint so Placements stop scheduling here - still gated by approval.

$ tool: propose_cluster_action
{
  "proposal_id": "86920d0f61874d6dba0942573185e477",
  "status": "pending_approval",
  "action": "cordon",
  "next_step": "Ask the human operator to run: ocm-mcp approve 86920d0f61874d6dba0942573185e477 and provide you the approval token, then call apply_cluster_action."
}

OK apply_cluster_action(cordon) #

Applied after approval; the taint now keeps new work off this cluster.

$ tool: apply_cluster_action
taints=[{"effect": "NoSelect", "key": "ocm-mcp-server.io/cordoned", "timeAdded": "2026-07-28T20:03:50Z", "value": "true"}]

OK uncordon (restore) #

Remove the taint so the cluster is schedulable again.

$ tool: apply_cluster_action(uncordon)
taints_after=[]

OK set_label (gated) #

Stamp a fleet label through the same propose/approve gate - labels drive Placements, so they are write-gated too.

$ tool: apply_cluster_action(set_label)
{
  "cluster": "cluster1",
  "action": "set_label",
  "status": "applied"
}

OK accept (gated, idempotent) #

Re-assert hubAcceptsClient on an already-accepted cluster - exercises the accept path without changing fleet state.

$ tool: apply_cluster_action(accept)
{
  "cluster": "cluster1",
  "action": "accept",
  "status": "applied",
  "approved_csrs": []
}

OK enable_addon (gated) #

Create a ManagedClusterAddOn through the gate.

$ tool: apply_cluster_action(enable_addon)
{
  "cluster": "cluster1",
  "action": "enable_addon",
  "status": "applied",
  "addon": "e2e-demo-addon"
}

OK disable_addon (gated, cleanup) #

Delete the same ManagedClusterAddOn through the gate - leaving the fleet exactly as we found it.

$ tool: apply_cluster_action(disable_addon)
{
  "cluster": "cluster1",
  "action": "disable_addon",
  "status": "applied",
  "addon": "e2e-demo-addon"
}

9. Prompts - reusable runbooks the server hands any MCP client #

OK prompt: diagnose_fleet #

A ready-made, safety-first runbook an agent can start from.

$ mcp prompt: diagnose_fleet
You are operating a Kubernetes fleet through the OCM hub. Investigate only - do not propose or apply anything yet.

1. Call list_clusters to see availability, version, and capacity of every cluster.
2. Call get_addon_health to spot Degraded or unavailable add-ons across the fleet.
3. For any cluster that is Unavailable or not joined, call get_cluster to read its conditions, then get_cluster_health for unhealthy pods and degraded deployments.
4. For each unhealthy workload, use query_events and get_pod_logs to find the cause.
5. Produce a concise report: per cluster, what is wrong, the evidence, and the smallest safe remediation you would propose. Do not act on it yet.

OK prompt: remediate_with_approval #

A ready-made, safety-first runbook an agent can start from.

$ mcp prompt: remediate_with_approval
A fleet operator reports: payments degraded on cluster1

Follow the safe remediation workflow:
1. Investigate with the read tools (list_clusters, get_cluster_health, query_events, get_pod_logs, get_manifestwork) until you can name the root cause with evidence.
2. Decide the smallest change that fixes it. For a workload change, call propose_manifestwork with pinned images and a precise summary. For a cluster lifecycle change (cordon, uncordon, set a label, accept a cluster), call propose_cluster_action.
3. If the proposal is REJECTED by static guardrails or Kyverno, read the reason, correct the manifest, and propose again. Never try to bypass a rejection.
4. Tell the operator the proposal id 
... (truncated)

OK prompt: why_not_scheduled #

A ready-made, safety-first runbook an agent can start from.

$ mcp prompt: why_not_scheduled
Explain why cluster 'cluster1' was or was not selected by Placement 'demo-all' in namespace 'default'. Use only reads:

1. get_placement_decision('demo-all', 'default') for the clusters actually chosen.
2. list_placements to read the Placement's clusterSets and selection intent.
3. get_cluster('cluster1') for its labels, ClusterClaims, and taints.
4. list_cluster_set_bindings to confirm the Placement's namespace is bound to the ClusterSet the cluster belongs to.
5. list_addon_placement_scores('cluster1') if the Placement uses AddOn prioritizers.
6. Conclude with the specific reason: not in a bound ClusterSet, failed a predicate, carries a NoSelect taint, or simply out-scored by others.

OK prompt: incident_postmortem #

A ready-made, safety-first runbook an agent can start from.

$ mcp prompt: incident_postmortem
Write a post-incident report for the change just completed. Base every statement on evidence, not recollection:

1. Call get_audit_trail to retrieve the ordered record of tool calls.
2. Reconstruct the timeline: what was inspected, what was proposed, which proposal was approved, and what was applied or rolled back.
3. State the root cause, the remediation, and how recovery was verified.
4. Note any proposals that were rejected and why, as evidence the guardrails held.
5. Keep it factual and concise. If the audit log does not support a claim, do not make it.

OK prompt: onboard_cluster #

A ready-made, safety-first runbook an agent can start from.

$ mcp prompt: onboard_cluster
A new cluster 'cluster1' is trying to join the hub. Onboard it safely:

1. Call list_pending_csrs to see the pending cluster-join CSRs and confirm one belongs to 'cluster1'.
2. Call get_cluster('cluster1') to check its current acceptance and conditions.
3. Propose acceptance: propose_cluster_action('cluster1', 'accept', <summary>). This sets hubAcceptsClient and approves the pending join CSRs, but applies nothing yet.
4. Give the operator the proposal id, ask them to run `ocm-mcp approve <id>`, and wait for the token.
5. Call apply_cluster_action with the token, then verify with get_cluster and get_addon_health that the cluster becomes Available and its agents register.

OK prompt: addon_troubleshoot #

A ready-made, safety-first runbook an agent can start from.

$ mcp prompt: addon_troubleshoot
The 'governance-policy-framework' add-on looks unhealthy somewhere in the fleet. Investigate without changing anything:

1. Call get_addon_health to find every cluster where an add-on is Degraded or not Available, and confirm which clusters have 'governance-policy-framework' unhealthy.
2. For each affected cluster, call list_addons_for_cluster(cluster) for the add-on's install namespace and conditions.
3. Call list_cluster_management_addons to read the fleet-level install strategy for 'governance-policy-framework'.
4. If a spoke context is configured, use get_cluster_health, query_events, and get_pod_logs in the add-on's install namespace to find the root cause.
5. Report the affected cluste
... (truncated)

OK prompt: hosted_cluster_health #

A ready-made, safety-first runbook an agent can start from.

$ mcp prompt: hosted_cluster_health
Assess the health of hosted cluster 'demo-hcp'. Reads only:

1. Call list_hosted_clusters to locate it and read its version and conditions. If the HostedCluster API is not on this hub, the HCP is hosted elsewhere - fall back to the ManagedCluster view with get_cluster('demo-hcp') and get_cluster_info('demo-hcp').
2. Call get_hosted_cluster(name, namespace) for its version history, control-plane conditions, and node pools.
3. Check the NodePools' desired vs current replicas for capacity or scaling problems.
4. Cross-check the spoke side: get_cluster('demo-hcp') for hub availability and get_addon_health for its add-ons.
5. Summarize: control-plane state, worker capacity, and anything degraded,
... (truncated)

OK prompt: policy_compliance_report #

A ready-made, safety-first runbook an agent can start from.

$ mcp prompt: policy_compliance_report
Produce a governance compliance report for the fleet. Reads only:

1. Call list_policy_violations for every NonCompliant or Pending policy-cluster pair.
2. Call list_policies for the full picture, including which policies are inform vs enforce (remediationAction).
3. Group the violations by policy and by cluster; call out anything Pending separately from NonCompliant.
4. For the highest-impact violations, note whether the policy is inform (visibility only) or enforce (actively remediating).
5. Present a concise table and a short prioritized list of what to fix first. Propose no changes - this is a report.

OK prompt: capacity_report #

A ready-made, safety-first runbook an agent can start from.

$ mcp prompt: capacity_report
Report on fleet capacity so an operator can place new workloads well. Reads only:

1. Call list_clusters for each cluster's capacity and availability.
2. Call get_cluster on the candidates to compare capacity against allocatable.
3. Where available, call get_cluster_info for node counts and per-node capacity, and list_addon_placement_scores for any custom scores prioritizers use.
4. Rank clusters by headroom; flag any that are Unavailable, cordoned (NoSelect taint), or nearly full.
5. Recommend where a new workload should land, and note any cluster that needs attention before it can take more.

OK prompt: rollout_status #

A ready-made, safety-first runbook an agent can start from.

$ mcp prompt: rollout_status
Report the rollout status of ManifestWorkReplicaSet 'demo-mwrs' in namespace 'default'. Reads only:

1. Call list_manifestworkreplicasets to read its summary (total / applied / available / progressing / degraded) and conditions.
2. Identify the Placement it targets, then call get_placement_decision to list the clusters in scope.
3. For any cluster that looks stuck, call get_manifestwork(cluster, <name>) to read the per-resource conditions and status feedback and find what is blocking Apply/Available.
4. Summarize progress across the fleet and name the specific clusters and resources that are not yet healthy, with evidence.

10. Audit - the server's own record of every tool call #

OK list_pending_proposals #

Proposals still awaiting human approval.

$ tool: list_pending_proposals
[]

OK get_audit_trail #

The append-only log of every tool call so far - the agent writes its incident report from this record, not from memory.

$ tool: get_audit_trail
[
{"tool": "apply_cluster_action", "args": {"proposal_id": "de3dead0beda4fc295ed4eb2b592edc6", "approval_token": "<redacted>"}, "outcome": "ok", "error": "", "duration_ms": 19, "ts": 1785269030.9449751, "actor": "sandeepbazar:62853", "seq": 15, "prev": "8dc2d7cceb3a03b4cb91bcf983d8bf89df33423831803aae1995ae615d47e730", "hash": "57c682e21a640cc0c29e70e64bc3bb92bb12e1f5f412c085fb67a63478a1879e"},
{"tool": "propose_cluster_action", "args": {"cluster": "cluster1", "action": "accept", "summary": "accept (gated, idempotent)", "params_json": "{}"}, "outcome": "ok", "error": "", "duration_ms": 15, "ts": 1785269030.964548, "actor": "sandeepbazar:62853", "seq": 16, "prev": "57c682e21a640cc0c29e70e64bc3bb92bb12e1f5f412c085fb67a63478a1879e", "hash": "6e299b5c813d59779fcef9f5d8b3b12216c7fecb8be601f7da550c9cf6d4b236"},
{"tool": "apply_cluster_action", "args": {"proposal_id": "8cacc74a0f634fa2903a1ede4d0cf417", "approval_token": "<redacted>"}, "outcome": "ok", "error": "", "duration_ms": 11, "ts": 1785269030.976025, "actor": "sandeepbazar:62853", "seq": 17, "prev": "6e299b5c813d59779fcef9f5d8b3b12216c7fecb8be601f7da550c9cf6d4b236", "hash": "699564a89498eb5dc290e4c4c22c6faf9c983be04f32a99cff75bcd3bde1854e"},
{"tool": "propose_cluster_action", "args": {"cluster": "cluster1", "action": "enable_addon", "summary": "enable_addon (gated)", "params_json": "{\"addon\": \"e2e-demo-addon\"}"}, "outcome"
... (truncated)

11. MCP protocol layer - the real server binary over stdio JSON-RPC #

PASS stdio JSON-RPC session #

Spawn the real server binary, complete the MCP handshake, and verify the full advertised surface (35 tools with safety annotations, 10 prompts, 6 resources) plus a tool call, a resource read, and a prompt over the wire.

$ mcp.client.stdio -> ocm-mcp-server
{
  "server": "ocm-mcp-server",
  "tools": 35,
  "read_annotation_ok": true,
  "apply_annotation_ok": true,
  "prompts": 10,
  "resources": 6,
  "guardrails_resource_ok": true,
  "list_clusters_over_wire": true,
  "prompt_over_wire_ok": true
}

11b. Negative sweep - proving every gate fails closed #

PASS expired token refused #

A token past its TTL must be rejected even though proposal and content are valid.

$ apply_manifestwork(expired token)
REJECTED: Approval token has expired; request a fresh approval.

PASS fresh token still applies #

Same proposal, fresh token: applies - proving the expiry rejection was about the token, not the content.

$ apply_manifestwork(fresh token)
{
  "status": "applied",
  "cluster": "cluster1",
  "manifestwork": "e2e-negative",
  "note": "Verify rollout with get_cluster_health / get_manifestwork. To undo, call propose_rollback then apply the

PASS replayed token refused #

The just-spent token must never work twice.

$ apply_manifestwork(replayed token)
REJECTED: proposal 88545c401d1a42bca9403a489b70a372 is 'applied', not pending.

PASS read-only mode refuses writes #

With OCM_MCP_READ_ONLY=1 every write tool refuses before any guardrail or token logic runs - the coarse backstop.

$ OCM_MCP_READ_ONLY=1 propose_manifestwork
REJECTED: this server runs in read-only mode (OCM_MCP_READ_ONLY). No proposals or writes are accepted.

PASS audit chain + signed anchor #

The genuine log verifies; a log with one rewritten entry fails; and a chain head signed by the off-box key (audit-anchor) verifies - so mid-log edits AND tail truncation are both detectable.

$ audit-verify / tamper copy / audit-anchor
real: audit chain intact over 30 entries
tampered copy: audit chain broken at line 16 (seq 16)
anchors: 1 anchor(s) verified; log head is seq 30, last anchored seq 30 (0 newer entries not yet anchored)

PASS ocm-mcp doctor #

The operator-facing smoke test runs the read path against the live hub and reports per-check status.

$ ocm-mcp doctor
exit=0

11c. Tracing - OTel spans export over OTLP (what Jaeger would receive) #

PASS OTLP span export #

With OTEL_EXPORTER_OTLP_ENDPOINT set (and the [tracing] extra installed), every tool call opens a span named tool. with redacted args; the BatchSpanProcessor flushes it over OTLP/HTTP - the same endpoint a Jaeger all-in-one or any OTel collector listens on (:4318).

$ OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:PORT python -c 'tool call'
posts=1 paths=['/v1/traces'] span_named=True rc=0

12. Negative scenario - break something, then debug and fix it end to end #

💥 INJECT INJECT: failing-rollout #

Deploy a 'payments-v2' with a broken image tag, exactly the kind of bad rollout that pages someone at 2 a.m.

$ ./chaos/inject.sh failing-rollout cluster1
deployment.apps/payments-v2 created
injected: failing-rollout into cluster1

OK DIAGNOSE: get_cluster_health #

The agent asks 'what is unhealthy here?' and sees payments-v2 is not running.

$ tool: get_cluster_health(cluster1)
{
  "unhealthy_pods": [
    {
      "namespace": "shop",
      "name": "payments-v2-6f6596d897-22ln4",
      "phase": "Pending",
      "waiting": [
        "ImagePullBackOff"
      ],
      "restarts": 0
    },
    {
      "namespace": "shop",
      "name": "payments-v2-6f6596d897-l5bss",
      "phase": "Pending",
      "waiting": [
        "ErrImagePull"
      ],
      "restarts": 0
    }
  ],
  "degraded_deployments": [
    {
      "namespace": "shop",
      "name": "payments-v2",
      "ready": "0/2"
    }
  ]
}

OK DIAGNOSE: query_events #

Events reveal the root cause in plain English: the image cannot be pulled.

$ tool: query_events(cluster1, shop)
[
  {
    "namespace": "shop",
    "type": "Warning",
    "reason": "Failed",
    "object": "Pod/payments-v2-6f6596d897-22ln4",
    "count": 1,
    "message": "Failed to pull image \"registry.k8s.io/e2e-test-images/agnhost:2.47-nonexistent\": rpc error: code = NotFound desc = failed to pull and unpack image \"registry.k8s.io/e2e-test-images/agnhost:2.47-nonexistent\": failed to resolve reference \"registry.k8s.io/e2e-test-images/agnhost:2.47-nonexistent\": registry.k8s.io/e2e-test-images/agnhost:2.47-nonexistent: not found"
  },
  {
    "namespace": "shop",
    "type": "Warning",
    "reason": "Failed",
    "object": "Pod/payments-v2-6f6596d897-22ln4",
    "count": 1,
    "message": "Error: ErrImagePull"
  },
  {
    "namespace": "shop",
    "type": "Normal",
    "reason": "BackOff",
    "object": "Pod/payments-v2-6f6596d897-22ln4",
    "count": 1,
    "message": "Back-off pulling image \"registry.k8s.io/e2e-test-images/agnhost:2.47-nonexistent\""
  },
  {
    "namespace": "shop",
    "type": "Warning",
    "reason": "Failed",
    "object": "Pod/payments-v2-6f6596d897-22ln4",
    "count": 1,
    "message": "Error: ImagePullBackOff"
  },
  {
    "namespace": "shop",
    "type": "Normal",
    "reason": "BackOff",
    "object": "Pod/payments-v2-6f6596d897-l5bss",
    "count": 1,
    "message": "Back-off pulling image \"registry.k8s.io/e2e-test-images/agnhost:2.47-nonexistent\""
... (2 more lines)

🔧 FIX FIX: propose + approve #

The agent proposes the smallest safe fix (a pinned, known-good image). It clears the guardrails and a human approves it.

$ propose_manifestwork -> ocm-mcp approve
proposal_id=14a920e0fc614697a3e3f643f2a70f70
guardrails=passed  kyverno_dry_run=passed  approved=yes

🔧 FIX FIX: apply_manifestwork #

Delivered to the cluster through the hub as a ManifestWork.

$ tool: apply_manifestwork(token)
status=applied

PASS VERIFY: recovery #

Re-read health until payments-v2 is running again - the incident is closed, with a full audit trail of everything that happened.

$ tool: get_cluster_health(cluster1)
payments-v2 is Running again - fix confirmed end to end.

11. Cleanup #

OK teardown #

Delete only the clusters created by this run (recorded in .e2e-run/created-clusters); never touch pre-existing clusters.

$ kind delete cluster --name ...
deleted: hub cluster1 cluster2