Cilium Explain turns raw Kubernetes and Cilium state into a deterministic diagnosis: it tells you why Pod A can't reach Pod B, with the evidence to prove it.
When connectivity breaks in a Cilium cluster, the data you need is scattered across kubectl, Hubble flows, EndpointSlices, and CiliumNetworkPolicies — and correlating it by hand, under incident pressure, is the slowest part of every outage. Cilium Explain is a read-only troubleshooting companion for Cilium and Hubble that does the correlation for you and answers the questions directly:
- Why can't Pod A communicate with Pod B?
- Which policy blocked the traffic?
- Is DNS working? Is the Service configured correctly? Are the Endpoints healthy?
- What evidence proves this conclusion, and what should I check next?
Every conclusion is produced by a deterministic rule engine over explicitly collected facts. No AI, no heuristics, no guessing: every finding carries the evidence it is based on and a read-only next step. When the required evidence is absent (e.g. Hubble is unreachable, or a policy uses constructs outside the analyzed subset), the tool says "No Evidence" — it never assumes traffic was allowed.
⚠️ Early-stage project. Not affiliated with the Cilium project — built as a companion tool; feedback welcome.
go run ./cmd/cilium-explain explain \
--source frontend/frontend-abc \
--destination payments/payment-service \
--port 443Checking connectivity: frontend/frontend-abc -> payments/payment-service:443
✓ Source Pod
✓ DNS
✓ Service
✖ Endpoints
Reason
Service "payments/payment-service" has no Ready endpoints (2 not ready).
Evidence
EndpointSlice
Ready endpoints
0
Recommendation
Inspect the endpoints of Service payment-service
kubectl get endpointslices -n payments -l kubernetes.io/service-name=payment-service
--------------------
✖ Policy
Reason
Traffic denied: CiliumNetworkPolicy "payments-policy" selects the backends of
"payments/payment-service" and no ingress rule matches the source on port 443
(ingress default-deny).
Recommendation
kubectl describe ciliumnetworkpolicy -n payments payments-policy
--------------------
====================
Primary Root Cause
Service "payments/payment-service" has no Ready endpoints (2 not ready).
Secondary Finding
Traffic denied: CiliumNetworkPolicy "payments-policy" selects the backends of "payments/payment-service" and no ingress rule matches the source on port 443 (ingress default-deny).
Two independent problems, found in one pass, ranked by which to fix first — each backed by evidence and a copy-pasteable read-only next step. When every check passes:
✔ Connectivity analysis completed.
No problems detected.
Traffic should be permitted.
Want more? docs/scenarios/ contains three realistic incident walkthroughs, each reproducible locally with kind + Cilium (or without any cluster, via the test suite).
During a connectivity incident, the truth is spread across four different surfaces: workload state (kubectl get pods), service plumbing (Services, EndpointSlices), policy intent (CiliumNetworkPolicies), and observed traffic (Hubble flows). Each existing tool shows one of these surfaces raw — and leaves the correlation to you. You must know which layer to suspect, in which order to check, and how to join a Hubble drop to the policy that caused it. Worse, the first problem you find can mask a second one, costing a full extra incident cycle.
Cilium Explain performs that correlation deterministically. Collectors snapshot all four surfaces into typed facts; a rule engine evaluates every check (it never stops at the first failure); a correlator ranks the failures into a Primary Root Cause and Secondary Findings. Same cluster state, same output — every time, with the supporting evidence attached to every conclusion.
| Tool | Shows Data | Explains Why |
|---|---|---|
| kubectl | ✅ | ❌ |
| Hubble | ✅ | ❌ |
| Cilium CLI | ✅ | ❌ |
| Cilium Explain | ✅ | ✅ |
One-line positioning: Hubble shows the flows. explain tells you why one failed. explain state shows the datapath behind both.
- Deterministic explainability — every conclusion follows from explicit facts via pure rules; identical state produces identical output. No AI, no heuristics.
- Strictly read-only — only
get/listAPI verbs and read-onlyciliumlist/get commands. The tool never applies, patches, or deletes anything. - Evidence-based diagnosis — every finding carries what happened, why, the observations proving it, and a read-only recommended next step. Missing evidence is reported as No Evidence, never assumed away.
- Multiple root causes — the engine always runs every rule and reports all detected problems, ranked (Primary Root Cause / Secondary Findings), so one failure can't hide another.
- Datapath visualization —
explain staterenders the live eBPF datapath (load-balancer, connection tracking, ipcache, compiled policy) with all raw IPs and numeric identities resolved to names. - Web dashboard —
explain state --webserves an embedded single-binary dashboard with a graph view and searchable tables. - JSON output —
--jsonproduces a machine-readable snapshot of any state view.
┌────────────┐ ┌────────────┐ ┌────────────┐
│ Kubernetes │ │ Cilium │ │ Hubble │
│ API │ │ CRDs │ │ Observer │
└─────┬──────┘ └─────┬──────┘ └─────┬──────┘
└───────────┬────┴───────────────┘
▼
CLI ──▶ Collectors ──▶ Facts ──▶ Rule Engine ──▶ Findings
│
Renderer ◀── Explanation ◀── Correlator ◀──┘
- Collectors read cluster state (and nothing else): the Kubernetes API (Pod, Service, EndpointSlice), Cilium CRDs (CiliumNetworkPolicy, CiliumEndpoint), and the Hubble Observer gRPC API (recent flows and verdicts). Strictly read-only — the tool never applies, patches or deletes anything.
- Rules are pure functions
Facts → Finding, one per file: Source Pod, DNS, Service, Endpoints, Policy. They never touch an API. - The engine never stops at the first failure — every rule always runs, so a single invocation shows every detected problem.
- The correlator ranks failed findings into a Primary Root Cause and Secondary Findings (severity first, then pipeline order). Inconclusive "No Evidence" findings are reported but never promoted to root cause.
- Renderers only format. Phase 1 ships a terminal renderer; JSON/YAML/Markdown implement the same interface later.
The full system design lives in docs/architecture.md.
Requires Go 1.25+.
go build ./...
go run ./cmd/cilium-explain explain \
--source frontend/frontend-abc \
--destination payments/payment-service \
--port 443Connectivity to the cluster:
- Kubernetes & Cilium APIs — via
--kubeconfig(default:$KUBECONFIG, then~/.kube/config, then in-cluster config). - Hubble Observer API — via
--hubble-server(defaultlocalhost:4245; e.g.kubectl port-forward -n kube-system svc/hubble-relay 4245:80). If Hubble is unreachable the analysis continues and reports the missing flow evidence as a warning.
Required RBAC is read-only: get/list on Pods, Services, EndpointSlices, CiliumNetworkPolicies and CiliumEndpoints.
The policy rule statically evaluates CiliumNetworkPolicies with Cilium semantics for a well-defined subset: endpointSelector matching against the Service's backend labels, ingress default-deny once a policy selects the destination, fromEndpoints (matchLabels and matchExpressions, including the k8s:io.kubernetes.pod.namespace label), fromEntities (all/cluster/world), numeric toPorts including port ranges, and ingressDeny precedence. Constructs outside this subset — CIDR peers, named ports, L7 rules — yield an explicit indeterminate finding rather than a guessed verdict. Observed Hubble POLICY_DENIED drops for the exact query are treated as authoritative proof of denial.
Phase 1 does not issue live DNS queries (the tool runs outside the cluster and stays read-only). CoreDNS answers <service>.<ns>.svc.cluster.local straight from the Service object, so the DNS rule evaluates exactly those facts: the Service exists and has a ClusterIP (or is intentionally headless).
Hubble shows the flows.
explaintells you why one failed.explain stateshows the datapath behind both.
Where explain reasons about traffic, explain state visualizes the live eBPF datapath state — not flows, but what the kernel is configured to do right now, read from the Cilium agent's BPF maps. No flow logs, no custom eBPF programs: every view comes from read-only cilium list/get commands executed in the agent pod (via kubectl exec).
cilium-explain explain state lb # service load-balancer map (frontend → backends)
cilium-explain explain state ct # connection tracking table (+ count-by-state summary)
cilium-explain explain state ipcache # IP → identity → labels explorer
cilium-explain explain state policy # compiled ALLOW/DENY policy matrix per endpoint
cilium-explain explain state --web # web dashboard with all four views on :8080
cilium-explain explain state lb --json # machine-readable output for any viewRaw BPF maps contain IPs and numeric security identities; the identity-join resolver (internal/join) composes the ipcache, the identity list and the endpoint list so every view renders payments/payment-7f9 (app=payments) instead of 10.0.1.5 / id 12345:
Load-balancer map (2 services)
FRONTEND TYPE BACKEND POD
10.96.0.10:443 ClusterIP 10.0.1.5:80 payments/payment-7f9
172.18.0.2:31443 NodePort 10.0.1.5:80 payments/payment-7f9
payments/payment-7f9 (endpoint 217, identity 12345)
DIR PEER PORT/PROTO ACTION
egress any any/ANY ALLOW
ingress world any/ANY DENY
ingress app=frontend (frontend) 80/TCP ALLOW
The policy matrix is the standout view: it shows the compiled, enforced policy rows loaded in the kernel — the state kubectl get cnp cannot show you.
--web serves an embedded single-binary dashboard (web/, via embed.FS) with a Cytoscape graph for the LB view and searchable tables for the rest, backed by GET /api/state/{lb,ct,ipcache,policy,snapshot}. Snapshots are cached for 10 s; /api/state/snapshot forces a refresh.
Useful flags: --cilium-namespace (default kube-system), --cilium-pod (pin an agent; default: first running agent — the snapshot reflects that node's view), --kubectl, --kubeconfig.
Everything stays read-only and deterministic: sources that cannot be read degrade to warnings, unresolvable numbers fall back to identity <n> — never a guess.
Beyond the fixture-driven test suite, the tool has been validated end-to-end against a real two-node kubeadm cluster (Kubernetes v1.36, Cilium CNI, containerd):
explain state lbresolved all 22 services in the BPF load-balancer map to names — local pods by pod name, remote-node coredns backends via ipcache identity labels, and the API server as the reservedkube-apiserveridentity.explain state ctdecoded 4,000+ live connection-tracking entries (raw byte-array addresses, network-byte-order ports, address-swapped tuples per Cilium issue #5848) — self-validating against known traffic such as kubelet readiness probes and Cilium health checks.explain state ipcachejoined every/32prefix to its pod, and wide prefixes to reserved identities (world,remote-node,host).explain state policydumped the compiled per-endpoint policy maps, andexplain state --webserved all four dashboard views.
The parser test fixtures in internal/state/testdata/ are recorded outputs from that live cluster, so the unit tests exercise the exact JSON shapes a real Cilium agent emits. Flag bit semantics (policy deny flag, CT tuple/state flags) were verified against the Cilium source.
go test ./...No live cluster is required: rules are covered by table-driven tests, collectors by fake clients (k8s.io/client-go/kubernetes/fake and in-package conversion tests), the engine by integration tests over the fake collectors in internal/collectors/fake, and the datapath state parsers and identity-join resolver by recorded JSON fixtures.
cmd/cilium-explain/ CLI entrypoint
docs/ architecture + reproducible incident scenarios
internal/cli/ Cobra commands; orchestration only
internal/models/ typed Facts / Finding / Explanation / state models (stdlib-only)
internal/collectors/ collector interfaces + kubernetes, cilium, hubble, fake
internal/engine/rules/ deterministic rules, one per file
internal/engine/correlate/ finding ranking (primary cause / secondary findings)
internal/engine/explain/ pipeline orchestration
internal/join/ identity-join resolver (IP / identity → names)
internal/state/ datapath state snapshots + BPF map output parsers
internal/server/ /api/state/* routes + embedded dashboard serving
internal/renderer/ terminal renderer (Renderer interface for future formats)
web/ embedded web dashboard (embed.FS)
pkg/ reserved for future public APIs
Layering rules enforced by the package structure: collectors never print, rules never reach an API, the renderer never touches collectors, and the CLI only wires things together.
- Phase 2 — resource change history and timelines,
doctorcommand, REST API, egress policy evaluation on the source, named-port and CIDR-peer resolution, JSON/YAML/Markdown renderers, TLS to Hubble Relay. Extension points are marked// TODO(phase2):in the code. - Phase 3 — optional AI summarizer (rephrasing findings only — never producing them), plugin system, multi-cluster.
- Datapath state (later) — NAT table view, tunnel/overlay topology, multi-node aggregation, direct map reads. Marked
// TODO(later):in the code.
- Ingress evaluation only, on the destination side; egress policies on the source pod are collected as future work.
- CiliumClusterwideNetworkPolicies and Kubernetes NetworkPolicies are not yet evaluated.
- TCP service traffic only; named policy ports and CIDR peers produce an explicit indeterminate finding instead of a verdict.
- Backends are identified via the Service selector; selectorless Services cannot be analyzed further.
explain statesnapshots reflect a single agent's node-local datapath view (pin a node with--cilium-pod).
Contributions are welcome. The design goal is that adding a diagnosis means adding one Rule implementation plus its table-driven test — and adding a datapath view means one parser file plus a snapshot field — if a change requires touching more layers than that, it probably belongs in a design discussion first. Please keep go test ./..., go vet ./... and gofmt clean.
Apache License 2.0. See the LICENSE file for details.