/ 
Rex-Rancher-0.003
/ Rex::Rancher::Cilium

NAME

Rex::Rancher::Cilium - Cilium CNI installation for Rancher Kubernetes distributions

VERSION

version 0.003

SYNOPSIS

use Rex::Rancher::Cilium;
use JSON::MaybeXS;    # JSON()->true below

# Install Cilium on an RKE2 cluster (defaults to version 1.20.0)
install_cilium(
  distribution => 'rke2',
);

# Install, upgrade or leave alone -- decided from the Helm release,
# read through the local kubeconfig; with Gateway API and extra values
install_cilium(
  distribution        => 'rke2',
  kubeconfig          => "$ENV{HOME}/.kube/mycluster.yaml",
  version             => '1.20.0',
  gateway_api         => 1,
  gateway_api_version => 'v1.6.1',
  helm_values         => { hubble => { relay => { enabled => JSON()->true } } },
);

# Install Cilium on a K3s cluster with explicit version
install_cilium(
  distribution     => 'k3s',
  k8s_service_host => '10.0.0.1',    # the control plane, not localhost
  version          => '1.20.0',
  cli_version      => 'v0.19.7',
);

# Upgrade an existing Cilium installation, keeping what it runs,
# and wait until it is ready again
upgrade_cilium(
  distribution => 'rke2',
  kubeconfig   => "$ENV{HOME}/.kube/mycluster.yaml",
  version      => '1.20.0',
  wait         => 1,
);

# Only move the Gateway API CRDs (restarts cilium-operator if applied)
ensure_gateway_api_crds(
  kubeconfig => "$ENV{HOME}/.kube/mycluster.yaml",
  version    => 'v1.6.1',
);

DESCRIPTION

Rex::Rancher::Cilium provides Cilium CNI installation and upgrade for Rancher Kubernetes distributions (RKE2 and K3s). The Cilium CLI runs on the remote server host via SSH; the Helm release state and the Gateway API CRDs are read and written from the local machine via Kubernetes::REST when a local kubeconfig is given.

Prerequisites

The server must already be running with its own CNI switched off in config.yaml, so that Cilium is the only one: on RKE2 cni: none and disable-kube-proxy: true, on K3s flannel-backend: none, disable-network-policy: true, disable-kube-proxy: true and cluster-cidr: 10.42.0.0/16; on both Cilium takes over kube-proxy's role. "install_server" in Rex::Rancher::Server sets these options by default when cilium => 1.

Helm values

Distribution-specific Helm values are written to /tmp/cilium-values-<dist>.yaml:

RKE2

kubeProxyReplacement: true, k8sServiceHost: 127.0.0.1, k8sServicePort: "6443", cni.exclusive: false, operator.replicas: 1, ipam.mode: kubernetes (or ipam_mode); with cluster_cidr also ipam.operator.clusterPoolIPv4PodCIDRList: [cluster_cidr], which Cilium uses only in cluster-pool mode (ipam_mode or helm_values; under kubernetes the pods follow the node podCIDRs).

K3s

kubeProxyReplacement: true, k8sServiceHost: k8s_service_host, k8sServicePort: "6443", cni.exclusive: true, operator.replicas: 1, ipam.mode: cluster-pool (or ipam_mode) with ipam.operator.clusterPoolIPv4PodCIDRList: [10.42.0.0/16] (K3s's cluster-cidr; cluster_cidr replaces it).

Both distributions share the same CNI binary/config paths (/opt/cni/bin, /etc/cni/net.d). gateway_api adds gatewayAPI.enabled: true, and helm_values is merged over all of it. --set kubeProxyReplacement=true is also passed on the command line and wins over any value file. With kubeconfig and a running Cilium, its IPAM mode, pool, K3s k8sServiceHost and operator.replicas replace these defaults (see "install_cilium").

Default versions

The module ships with pinned defaults for reproducibility: Cilium 1.20.0 and Cilium CLI v0.19.7, the pair kubernetes-ocp runs. Override with the version and cli_version options. The default version is that of a fresh install: a running Cilium of another minor version keeps its own (see version in "install_cilium"). The Gateway API version has no default; see gateway_api_version (v1.6.1 or newer for Cilium 1.20).

FUNCTIONS

install_cilium(%opts)

Install Cilium CNI on a Rancher Kubernetes cluster (RKE2 or K3s), or bring an existing installation to the requested version and values.

The Cilium CLI binary is downloaded from GitHub to /usr/local/bin/cilium on the remote host (skipped if the correct version is already present). Distribution-appropriate Helm values, merged with helm_values, are written to /tmp/cilium-values-<dist>.yaml and handed to the CLI.

With kubeconfig (a local kubeconfig path), the existing Helm release is read from the cluster via Kubernetes::REST before anything is installed, and the outcome depends on its state:

  • no release: cilium install.

  • deployed at the requested version with every requested value already in effect: nothing is done.

  • deployed at another version, or with a requested value differing: cilium upgrade.

  • failed with an earlier revision still deployed (a failed upgrade): cilium upgrade again.

  • failed or pending-install with no deployed revision (a failed or interrupted first install), uninstalling or uninstalled: the stale release is removed (cilium uninstall, then its Helm release Secrets) and Cilium is installed fresh. No working Cilium exists in these states, so nothing running is taken down. A Secret already gone (404) is fine; any other API error deleting one dies before the install.

  • an upgrade that would change ipam.mode (the release sets one explicitly and the requested values differ): dies before cilium upgrade, naming both modes. Cilium cannot switch IPAM mode under running pods; redeploy the cluster, or pass the deployed mode in helm_values. (With the running configuration below, this only fires when release and ConfigMap disagree.)

  • pending-upgrade or pending-rollback, or pending-install over a deployed revision: dies before the host is touched, with the message of "upgrade_cilium". An earlier run was interrupted or another is still running, and the deployed revision still carries the pod network; the message names the Secret to delete once no other deploy is running.

What a Cilium on the cluster runs is read first and kept wherever the caller did not ask for something else, before anything touches the host. This does not depend on the release state: a failed or pending-install release, or none at all, can still have left cilium-config and pods behind, and the ConfigMap is what those pods run with, so a reinstall uses it too:

  • ipam.mode and the pool (ipam.operator.clusterPoolIPv4PodCIDRList) come from the ConfigMap kube-system/cilium-config (ipam, cluster-pool-ipv4-cidr), not from the release values: a Cilium installed without an explicit mode runs the chart default cluster-pool, and the kubernetes default of the RKE2 values must not switch it (its pods would lose their addresses). A ConfigMap without ipam counts as cluster-pool, the agent's default. The default pool gives way to the running one; in any other mode it is dropped.

  • a mode or pool the caller set in helm_values that differs from the running one dies, naming both: Cilium cannot change either under running pods. The pool is compared only in cluster-pool mode, as a set.

  • cluster_cidr is the pool of a fresh install, not a requested one: on a running cluster-pool Cilium with another pool the running pool is kept, with a warning naming both.

  • ipam_mode is likewise the mode of a fresh install: on a running Cilium in another mode the running mode (and, in cluster-pool, its pool) is kept, with a warning naming both. Only ipam.mode in helm_values counts as requested and dies on a difference.

  • on K3s without k8s_service_host (or helm_values->{k8sServiceHost}), k8sServiceHost is the KUBERNETES_SERVICE_HOST of the running cilium DaemonSet.

  • operator.replicas keeps the spec.replicas of the running cilium-operator Deployment (without one, the release's value) instead of the default 1, unless helm_values sets it.

Any API error other than a 404 while reading these dies rather than fall back to the defaults.

After a fresh install the cilium DaemonSet must exist, or it dies: the CLI has been seen to exit 0 without creating anything. An API error other than a 404 reading it dies naming that error. With wait, the function returns only once Cilium is ready (see there).

Without kubeconfig the release state cannot be read, and the previous behaviour applies: cilium install runs, and its "cannot re-use a name" error (the release already exists) counts as success, with a warning that version and values were not reconciled. Helm refuses to install over the existing release, so nothing about the running Cilium changes on this path; changing it needs kubeconfig, here or with "upgrade_cilium".

On both distributions kubeProxyReplacement=true is passed to enable Cilium's eBPF-based kube-proxy replacement, so the server config must have switched off the distribution's CNI and kube-proxy: on RKE2 cni: none and disable-kube-proxy: true, on K3s flannel-backend: none, disable-network-policy: true, disable-kube-proxy: true and cluster-cidr: 10.42.0.0/16 ("install_server" in Rex::Rancher::Server writes these). Cilium then needs an API server address that works before any Service does, on every node: on RKE2 that is 127.0.0.1:6443, where servers and agents alike serve it. K3s agents serve it on 127.0.0.1:6444 instead, so on K3s Cilium is given the control plane's own address, k8s_service_host, and dies without one.

Both distributions have been run live through Rex::Rancher. The k3s values are those kubernetes-ocp verified live (k3s v1.36.4+k3s1, Cilium 1.20.0, Gateway API v1.6.1 standard), with the Cilium and CLI versions this module defaults to; Rex::Rancher's k3s path has since brought Cilium up live with k8s_service_host (see Rex::Rancher::Distribution::K3s).

Options:

distribution

rke2 (default) or k3s.

version

Cilium version to install, e.g. 1.20.0. Default: 1.20.0.

Cilium upgrades and rolls back one minor version at a time. With kubeconfig, the version a Cilium on the cluster runs (the cilium-agent image tag of the cilium DaemonSet, else the chart of a deployed release) is read before the host is touched:

  • a version more than one minor version from it, up or down, dies naming the minor to go to first;

  • without version, the default applies only to a fresh install and to a newer patch of the running minor. A running Cilium of an older minor keeps its version, with a warning (pass version to upgrade, one minor at a time); one newer than the default keeps its version too. So a re-run after a Rex::Rancher upgrade never moves a cluster by more than a patch release.

Without kubeconfig nothing is compared, and nothing running changes (see below). Cilium 1.18 and newer need Linux 5.10 or newer on every node (4.18 on RHEL 8.10); Cilium 1.20 is tested on Kubernetes 1.33 to 1.36.

cli_version

Cilium CLI version to download, e.g. v0.19.7. Default: v0.19.7, which supports Cilium 1.16 and newer; an older Cilium needs an older CLI.

k8s_service_host

K3s only, and required there: the control plane address Cilium reaches the API server at on port 6443 from every node (k8sServiceHost), e.g. the server's IP or a name in its certificate. A loopback address dies, because K3s agents serve the API on 127.0.0.1:6444, not 6443. helm_values->{k8sServiceHost} may take its place, and with kubeconfig a running Cilium's own address does (see above); without any of them it dies before the host is touched. "rancher_deploy_server" in Rex::Rancher passes the first tls_san. Passing it on RKE2 dies: RKE2 uses 127.0.0.1, which works on every node there.

cluster_cidr

The cluster's pod network, the server's cluster_cidr (see "install_server" in Rex::Rancher::Server): one IPv4 CIDR, anything else dies before the host is touched. Written as Cilium's pool, ipam.operator.clusterPoolIPv4PodCIDRList, unless helm_values sets one. It takes effect only in cluster-pool mode: on K3s, in place of 10.42.0.0/16, and on RKE2 with ipam_mode => 'cluster-pool' (or that mode in helm_values). RKE2's default stays ipam.mode: kubernetes, where Cilium ignores the pool value and pods get addresses from the node podCIDRs the cluster cuts from its cluster-cidr -- the same range, reached through the server's config.yaml. It is the pool of a fresh install: a running cluster-pool Cilium keeps its own pool, with a warning when it differs (see above). "rancher_deploy_server" in Rex::Rancher passes its own cluster_cidr.

ipam_mode

Cilium's IPAM mode (ipam.mode) on a fresh install: kubernetes or cluster-pool, anything else dies before the host is touched (other modes need more than a mode and stay with helm_values). Default: "default_ipam_mode" in Rex::Rancher::Distribution, kubernetes on RKE2 and cluster-pool on K3s. With cluster-pool, cluster_cidr is the pool on RKE2 too. A Cilium already running (kube-system/cilium-config readable through kubeconfig) keeps its own mode, with a warning when it differs (see above); without kubeconfig nothing running is changed anyway. An ipam.mode in helm_values that differs from it dies as contradictory. "rancher_deploy_server" in Rex::Rancher passes its own ipam_mode.

api_server

Kubernetes API server URL, passed as --api-server to the CLI. Optional; the CLI uses the kubeconfig's server address if omitted.

kubeconfig

Local path to the cluster kubeconfig (as saved by "rancher_deploy_server" in Rex::Rancher). Enables the release-state handling and the running-configuration reads above and is required for gateway_api and wait. Optional.

wait

If true, wait after install, upgrade or no-op until Cilium is ready: the cilium DaemonSet has rolled out its current generation to every node it schedules on and all those pods are ready, and every cilium-operator replica is updated and ready. Requires kubeconfig; read from the local machine. Dies after wait_duration naming the last state (pods ready and updated of each); a missing DaemonSet or Deployment is waited out, any other API error (no access, no connection) dies at once. Default: off, which only checks that the DaemonSet exists after a fresh install.

wait_duration

Seconds wait waits, a whole number above 0. Default: 600. Only used with wait.

helm_values

Hashref of additional Helm values, deep-merged over the defaults (hashes merge key by key, anything else replaces the default). Optional.

gateway_api

If true, apply the Gateway API CRDs before Cilium and set gatewayAPI.enabled: true. Default: off. Requires kubeconfig and gateway_api_version. The CRDs are fetched from the kubernetes-sigs/gateway-api GitHub release on the machine running Rex and applied through Kubernetes::REST (no kubectl). They are skipped when the cluster already carries that bundle version and channel; when they are applied to a cluster with a running cilium-operator, the operator is restarted so it picks up the new CRDs. Only a 404 counts as missing: an API error reading the probe CRD (gateways.gateway.networking.k8s.io), a CRD waited on to become Established, or the cilium-operator Deployment dies at once instead of applying, waiting out 30s, or skipping the restart.

RKE2 v1.37+ ships the same CRDs as its own chart, rke2-gateway-api-crd, which would overwrite them. "rancher_deploy_server" in Rex::Rancher disables it for you; calling this directly, put it in install_server's disable. While that chart's Helm release exists this dies before applying anything, naming the way out: disable the chart and restart rke2-server (Helm uninstalls it but keeps its gateway.networking.k8s.io CRDs, so Gateways and routes survive), or use RKE2's CRDs via helm_values without gateway_api.

gateway_api_version

Gateway API release to apply, e.g. v1.6.1. It must match what the Cilium version supports; there is no default because the two are version-locked. Cilium 1.20 and newer need v1.6.1 or newer (TLSRoute and BackendTLSPolicy at v1); an older one with such a Cilium dies before the host is touched -- for a pinned version already in option checks, for the default once it is known whether a running Cilium keeps an older version.

gateway_api_channel

experimental (default) or standard. What Cilium needs depends on both versions. Cilium up to 1.16 requires TLSRoute v1alpha2, which only the experimental channel carries. Cilium 1.17 to 1.19 require standard-channel CRDs only and handle TLSRoute v1alpha2 when it is there; before Gateway API v1.5 the standard channel has no TLSRoute, so standard costs TLS passthrough. Gateway API v1.5 moved TLSRoute (as v1) into the standard channel, v1.6 also TCPRoute and UDPRoute; Cilium 1.20 requires TLSRoute v1 and BackendTLSPolicy v1, which the v1.5+ standard channel carries. The default stays experimental: it is the channel every Cilium version works with, and it keeps TLSRoute v1alpha2 objects of an older Cilium readable (the v1.6 standard TLSRoute CRD drops that version, and such objects disappear).

Gateway API v1.5+ ships an admission policy that refuses experimental CRDs on top of standard ones: a cluster that started on standard cannot move to experimental.

upgrade_cilium(%opts)

Upgrade an existing Cilium installation to a new version using cilium upgrade, unconditionally. The Cilium CLI is updated first if needed. The same Helm values generation logic as "install_cilium" is used, and gateway_api applies the CRDs the same way (restarting a running cilium-operator when they were applied).

kubeconfig is required: without it the function dies before anything touches the host. It does the same, pointing to "install_cilium", when the Helm release cilium in kube-system has no deployed revision: cilium upgrade is a Helm upgrade and cannot install, so a ConfigMap kube-system/cilium-config or a cilium DaemonSet without a deployed release is not enough. A release left failed, pending-install, uninstalling or uninstalled without one is what "install_cilium" removes and installs again. A release stuck in pending-upgrade or pending-rollback (or pending-install over a deployed revision) dies before the host as well, with the message of "install_cilium" naming the Secret to delete. A failed upgrade over a deployed revision is upgraded again. The running configuration is read and kept exactly as in "install_cilium" (IPAM mode and pool from kube-system/cilium-config, K3s k8sServiceHost from the DaemonSet, operator.replicas from the cilium-operator Deployment), so an upgrade needs no values the caller has to look up first, and a change of IPAM mode or pool asked for in helm_values dies before the host is touched. The version follows the same one-minor-at-a-time rule as there: pass version to upgrade to a new minor. Without it, a running Cilium of an older minor keeps its version (with a warning) and cilium upgrade only re-applies the values. Without the API none of that can be checked, and the generated values would be applied as they stand -- on RKE2 the default ipam.mode: kubernetes switches a cluster-pool cluster and its pods lose their addresses.

Options are the same as "install_cilium", wait and wait_duration included, except that kubeconfig is not optional.

upgrade_cilium(
  distribution => 'rke2',
  kubeconfig   => "$ENV{HOME}/.kube/mycluster.yaml",
  version      => '1.20.0',
);

ensure_gateway_api_crds(%opts)

Apply the Gateway API CRDs of one bundle version and channel, the same way gateway_api does in "install_cilium", without touching Cilium's Helm release: for a Gateway API pin that moved while Cilium did not. Everything runs from the local machine through Kubernetes::REST; the remote host is not used.

The CRDs are skipped when the cluster already carries that bundle version and channel. When they are applied and a cilium-operator Deployment exists, it is restarted so it picks up the new CRDs; API errors other than a 404 die, as with gateway_api. While RKE2's rke2-gateway-api-crd Helm release exists this dies before applying anything, as gateway_api does. Returns 1 when the CRDs were applied, 0 when they were already current.

Options:

kubeconfig

Local path to the cluster kubeconfig. Required.

version

Gateway API release, e.g. v1.2.0. Required; it must match what the running Cilium supports.

channel

experimental (default) or standard; see gateway_api_channel in "install_cilium".

ensure_gateway_api_crds(
  kubeconfig => "$ENV{HOME}/.kube/mycluster.yaml",
  version    => 'v1.2.0',
  channel    => 'standard',
);

validate_cilium_opts(%opts)

Check the options of "install_cilium" without touching anything: dies with the same message install_cilium would for an unknown distribution, bad helm_values, cluster_cidr, ipam_mode or gateway_api settings, and -- also with kubeconfig, where install_cilium could still read it from a running Cilium -- a K3s cluster without a usable k8s_service_host. Returns 1. Not exported; "rancher_deploy_server" in Rex::Rancher calls it as Rex::Rancher::Cilium::validate_cilium_opts before the node is prepared.

SEE ALSO

Rex::Rancher, Rex::Rancher::Server, Rex, https://docs.cilium.io/

SUPPORT

Issues

Please report bugs and feature requests on GitHub at https://github.com/Getty/rex-rancher/issues.

CONTRIBUTING

Contributions are welcome! Please fork the repository and submit a pull request.

AUTHOR

Torsten Raudssus <getty@cpan.org>

COPYRIGHT AND LICENSE

This software is copyright (c) 2026 by Torsten Raudssus <torsten@raudssus.de> https://raudssus.de/.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.