Introduction to Cilium

Starting from a Kubernetes cluster without reliable datapath visibility, participants position the CNI, Cilium Agent, Operator, Hubble, and the eBPF-based datapath in one operating model. Using a prepared application, they trace responsibilities from pod startup to communication and identify which component distributes configuration, processes traffic, or supplies observability data.

A Cilium installation is only as sound as its prerequisites and selected modes. Participants check kernel capabilities, CNI state, routing, and IPAM constraints, derive suitable installation values, and install Cilium with the CLI or Helm in the lab environment. A successful cilium status --wait provides the first reproducible acceptance checkpoint.

After installation, a green status indicator is not yet a network model. Participants trace a pod-to-pod request and a Service request through endpoints, security identities, routing, and load balancing. They record source identity, destination, path, and expected response, then compare their hypothesis with the observed Cilium and Kubernetes state.

When dependencies appear only as IP addresses, faults and responsibilities remain difficult to assign. The group enables Hubble Relay, uses the Hubble CLI as its primary tool, and optionally uses the open-source UI as a Service map. It generates defined test traffic and filters current flows by namespace, pod, direction, and verdict. At the end, a documented CLI filter demonstrates the expected Service path and one deliberately failed request.

An installed CNI needs reproducible acceptance criteria, not isolated ping tests. Participants run the Cilium Connectivity Test, distinguish test failures from infrastructure problems, and verify selected paths with Hubble. The block ends with a baseline of status, successful checks, and one explained deviation that will serve as a later comparison.

Open pod communication does not become secure by immediately adding a long list of rules. Participants manually create a minimal communication matrix for the test application from defined test traffic and current Hubble flows, compare Kubernetes NetworkPolicy with CiliumNetworkPolicy, and introduce default deny in a controlled way. Before-and-after tests prove which connection is intentionally blocked and which required application path remains available.

Changing pod IP addresses must not make access control unusable. Participants write L3/L4 rules with labels, namespaces, ports, and Cilium identities in the cluster, deliberately apply an incorrect selector, and correct it using realized endpoint policy. The result is a tested rule with clearly documented allowed and denied paths.

Outbound access to external services can rarely be described permanently with individual IP addresses. Using CiliumNetworkPolicy, the group builds a DNS- and FQDN-based egress rule, allows the required name resolution, and checks responses and TTL-dependent state in the test application. Hubble and Cilium policy data then show that only the intended destination remains reachable.

An allowed TCP port does not define which HTTP action should be permitted. Participants add HTTP L7 rules for method and path to a CiliumNetworkPolicy, position the role of the Envoy proxy, and test allowed and denied requests in the lab. Visible HTTP verdicts confirm the effect and expose the additional operational dependency.

A plausible policy can still cause unexpected blocks because of selectors, direction, or DNS state. In a prepared failure scenario, participants work from a Hubble drop through the endpoint and realized policy to the responsible rule. They correct only the false assumption and prove with positive and negative tests that the boundary remains intact.

Service reachability cannot remain a black box. In a cluster with kube-proxy replacement enabled, participants deploy ClusterIP and NodePort Services, inspect their paths, compare EndpointSlices with Cilium Service state, and examine the associated BPF load-balancing map. The test succeeds when all expected backends are programmed and reachable through the intended paths.

Encryption is credible only when its coverage and exclusions are verified. Participants enable transparent WireGuard encryption in the lab, check node prerequisites, and generate traffic across multiple nodes. cilium-dbg status, WireGuard peers, and traffic on cilium_wg0 demonstrate encrypted cross-node paths; Hubble correlates the workload flows that must be assessed separately.

During an incident, the order of signals determines how quickly false hypotheses are eliminated. The group creates a diagnostic path from cilium status, endpoint state, logs, health, and Connectivity Tests, then collects a transferable package with cilium sysdump. Before sharing it, participants identify sensitive content and document the smallest reliable finding.

When a datapath or policy problem remains below Hubble visibility, diagnostics proceed deliberately to eBPF. Participants correlate endpoint and node, inspect relevant programs and Cilium maps for IP cache, policy, or load balancing, and use bpftool when necessary. They do not program eBPF; instead, they prove a concrete difference between intended and realized state.

A prepared final incident connects installation, Service path, policy, and observability. Participants prioritize hypotheses, restore the faulty configuration, and provide an evidence set from status, flow, policy, and, when necessary, eBPF state. They then transfer the process into a rollout checklist with responsibilities, checkpoints, and clear escalation boundaries.