NAME

Rex::Rancher::Server - Rancher Kubernetes server (control plane) installation

VERSION

version 0.003

SYNOPSIS

use Rex::Rancher::Server;

# Install RKE2 server (default)
install_server(
  token   => 'my-cluster-secret',
  tls_san => ['lb.example.com'],
);

# Install K3s server
install_server(
  distribution => 'k3s',
  token        => 'my-cluster-secret',
  tls_san      => ['lb.example.com'],
);

# Join additional control plane node (HA setup)
install_server(
  distribution => 'rke2',
  token        => 'my-cluster-secret',
  server       => 'https://first-server:9345',
);

# Retrieve kubeconfig and join token from a running server
my $kubeconfig = get_kubeconfig('rke2');
my $token      = get_token('rke2');

# The kubeconfig for this machine: pointed at the server's address, saved 0600
fetch_kubeconfig(
  distribution => 'rke2',
  server       => 'lb.example.com',
  file         => "$ENV{HOME}/.kube/cluster.yaml",
);

# Update registry mirrors on an already-running node
update_registries(
  distribution => 'rke2',
  registries   => {
    mirrors => { 'docker.io' => { endpoint => ['http://cache:5000'] } },
  },
);

DESCRIPTION

Rex::Rancher::Server handles control plane installation for both RKE2 and K3s Kubernetes distributions. It provides a unified interface for installing, configuring, and managing server nodes.

RKE2 installation

By default the official install script at https://get.rke2.io is fetched and run via curl -sfL … | sh -; with install_method => 'artifact' the checksum-verified release tarball is installed instead (see "install_server"). The service is started with --no-block to avoid systemd's 90-second activation timeout (RKE2's first start pulls many container images), then polled with systemctl is-active for up to 10 minutes; a failed or never-active service dies with its journal tail. A running rke2-server is left running on a re-run (systemctl start) unless it has to take something new (see "Re-runs"). Before the service is (re)started, an agent/etc/containerd/config.toml.tmpl that holds only imports and version = 2, as Rex::GPU 0.001 wrote it, is removed with a warning: rke2 renders it instead of its own containerd config, so no SystemdCgroup, sandbox image or registry mirrors. The content decides, never the path: any other template stays, with a log line. If the removal fails, install_server dies before the start. While agent/etc/containerd/config.toml is still that template's output, a running service is then restarted once, with a warning, so rke2 renders its own containerd config (see "remove_bare_containerd_template" in Rex::Rancher::Distribution). K3s is restarted on every run anyway (below), so it renders its config either way. After that the function waits until the kubeconfig file appears at /etc/rancher/rke2/rke2.yaml; API readiness is confirmed separately by the caller using "wait_for_api" in Rex::Rancher::K8s.

Re-runs

A running RKE2 reads its configuration only when it starts, so install_server (and "install_agent" in Rex::Rancher::Agent for rke2-agent) restarts a running service with --no-block, logging why, when the host has changed under it since its main process started:

config.yaml, config.yaml.d/, registries.yaml or /etc/default/rke2-server (-agent) modified, or a containerd config.toml.tmpl, config-v3.toml.tmpl or config-v3.toml.d/ drop-in (as Rex::GPU writes) — by modification time, and Rex rewrites a file only when its content differs, so the same options twice change nothing. An upgraded nvidia-container-runtime alone is no reason, containerd runs it anew per container;
an installed rke2 binary of another version than the running one, where the version skew policy lets it be restarted: a patch release, or the next minor with version pinned.

Before any of that, the installed binary is compared with the running one (RKE2 and K3s): the next minor without a pinned version is not restarted onto, with a warning, and a jump of more than one minor or a downgrade (possible only when the stable channel could not be resolved before the install, or moved in between) dies without a restart. See version under "install_server".

Nothing changed: systemctl start, the service keeps running. Changes made by hand or left by an interrupted earlier run count as well. What cannot be determined (no ps) counts as unchanged, with a warning. Details in "restart_reasons" in Rex::Rancher::Distribution.

A restart takes this server's API and etcd member down for its duration and touches no other node. Several servers of one HA cluster restarting at the same time can lose etcd quorum: deploy them one after another (Rex's default), not in parallel. K3s is restarted on every run, as its install script rewrites the unit each time, except onto an unpinned new minor, and except with hold_running: then it is restarted as RKE2 is, and also when the install script changed the content of its unit or env file (see hold_running under "install_server").

K3s installation

The official install script at https://get.k3s.io is used (piped, or run against the checksum-verified binary with install_method => 'artifact'), with K3S_URL set when joining an existing server. The script runs with INSTALL_K3S_SKIP_START: instead of its own blocking restart, k3s.service is restarted with --no-block, then the same systemctl is-active wait (at most 10 minutes, journal tail on failure) as for RKE2 follows, so a joining server that cannot reach the first one dies instead of hanging the deploy. A bare config.toml.tmpl in /var/lib/rancher/k3s/agent/etc/containerd is removed before that restart, as on RKE2. Then the kubeconfig wait. The token is read from config.yaml and never passed on the command line. Traefik and ServiceLB are disabled by default (disable in config.yaml, see "install_server") to leave room for Cilium and external load balancers.

Config layout

Both distributions use /etc/rancher/<dist>/config.yaml with the same key names (token, tls-san, node-name, node-label, disable, cni, etc.). When cilium => 1 (the default), the distribution's own CNI is switched off so that Cilium is the only one: on RKE2 cni: none and disable-kube-proxy: true, on K3s flannel-backend: none, disable-network-policy: true, disable-kube-proxy: true and cluster-cidr: 10.42.0.0/16; on both, Cilium's kube-proxy replacement takes over. See "install_server"'s cilium.

Registry mirrors are written to registries.yaml in the same directory. Both files are 0600 root:root: config.yaml holds the join token, registries.yaml may hold registry credentials.

FUNCTIONS

install_server(%opts)

Write the cluster configuration file, optionally write registries.yaml, install the distribution, start the service, wait until systemctl is-active reports it active, and then wait until the kubeconfig file is written to disk by the server process.

Returns 1 on success. Dies if installation fails, the distribution is unknown, the host carries Cilium datapath state from an earlier cluster but no RKE2 or K3s (see below), the installed version differs from a pinned version, a server already set up on the host would get another cluster-cidr (see "cluster_cidr"), or the service does not become active within 10 minutes. A service that ends up failed or never gets active makes the die message carry the last 50 lines of its journal (journalctl -u SERVICE -n 50 --no-pager).

The first thing read from the host, before anything is written or installed, is whether an earlier cluster left Cilium's datapath behind: the pins in /sys/fs/bpf/cilium, the /run/cilium/cgroupv2 mount or the cilium_host device, on a host with neither RKE2 nor K3s (no rke2 or k3s on PATH, none of their server and agent units active). The vendor uninstall scripts leave that state until a reboot, and its socket load balancer makes connections to the old cluster's service addresses hang, so every image pull of the new install stalls on a registry mirror among them. Such a host dies naming what was found and asking for a reboot; see "check_cilium_residue" in Rex::Rancher::Uninstall. With RKE2 or K3s on the host the state is its running cluster's (a re-run, an upgrade) and is not checked.

Options:

distribution

rke2 (default) or k3s. Both have been run live through Rex::Rancher; what the k3s runs covered is listed in Rex::Rancher::Distribution::K3s.

token

Shared secret used for node joining. If omitted, the token the server is already sealed with (/var/lib/rancher/rke2/server/token, K3s: /var/lib/rancher/k3s/server/token) is reused, so re-running install_server on a live control plane never rotates its token. Only on a fresh server (no such file) is a new one generated (up to 48 random alphanumeric characters, never fewer than 32). A passed token always wins.

The token is written to config.yaml only; it is never put on the installer command line or into its environment, where ps would show it. config.yaml is written 0600 root:root, including when it already exists.

server

URL of an existing server node to join. Used for multi-server HA setups (omit for the first/only server). For RKE2 the port is 9345; for K3s it is 6443.

tls_san

Additional TLS Subject Alternative Names for the API server certificate, as an arrayref or a comma-separated string. Include the load balancer address, public IP, or DNS name so that kubeconfig clients can connect.

version

Pinned version string, e.g. v1.30.4+rke2r1 for RKE2 or v1.30.4+k3s1 for K3s, handed to the installer as INSTALL_RKE2_VERSION / INSTALL_K3S_VERSION. If omitted, the latest stable release is installed.

When given, the version the installed binary reports (rke2 --version / k3s --version) is compared with it after the installer ran, and a mismatch dies before the service is started, on RKE2 and K3s alike (the K3s install script runs with INSTALL_K3S_SKIP_START). This catches a pinned install or upgrade that failed while an older binary is still on the host.

Against a running server the version skew policy applies, for RKE2 and K3s alike, before anything is written or installed: a version more than one minor ahead of the running one, or older than it, dies with both versions and the host unchanged. Without version the version checked is the one the install script would take from the stable channel (https://update.rke2.io/v1-release/channels/stable, K3s: update.k3s.io), resolved on the host with curl; if that fails, a warning, and the check happens after the install instead, before any restart (see "Re-runs"). A patch release of the running minor is installed and the service restarted, pinned or not. The next minor restarts it only when version is pinned; unpinned, it is installed but the running server is not restarted and keeps its old version until its next start (a reboot), with a warning that says so and names systemctl restart SERVICE. Upgrade a cluster server by server, one minor at a time, then the agents.

A server that is not running (stopped, crashed, never started) is held to the same rules against the installed binary (rke2 --version / k3s --version), the version it would start on: a jump or a downgrade dies before anything is installed, and the next minor without version is installed with a warning, since nothing holds a stopped service back from starting on it. No binary on the host is a fresh install and is not checked. If the stable channel cannot be resolved there, only a warning says the skew is not checked.

hold_running

If true, the host stays on the version it runs: the version of the running server (its main process, /proc/PID/exe --version) is this run's version, for the version skew check, the installer, the check of the installed version and the restart decision alike. A server that is not running holds the installed binary's version (rke2 --version / k3s --version); with neither, version applies, and without it the stable channel, as without hold_running. A running server whose version cannot be read holds the installed binary instead, and without one version, each with a warning. It is read before anything is written or installed, RKE2 and K3s alike (see "held_version" in Rex::Rancher::Distribution).

A version given as well is installed only where there is nothing to hold; otherwise the held version wins, and a warning names both. install_method => 'artifact' still requires version, for a host with nothing to hold.

Held, the server is restarted only when its configuration changed (see "Re-runs"; the binary stays on the version it runs): RKE2 as without hold_running, and K3s, which is otherwise restarted on every run, the same way, and also when the install script wrote /etc/systemd/system/k3s.service or k3s.service.env with other content than before, the check the K3s install script itself makes before it restarts K3s. Default: 0.

install_method

How the distribution gets onto the host. script (default) pipes the official install script into sh (curl -sfL https://get.rke2.io | sh -, K3s: https://get.k3s.io), exactly as without this option.

artifact pre-downloads the release artifact for the node's own architecture (uname -m on the host: amd64 or arm64, anything else dies) from the GitHub release, verifies it against the release's official sha256sum-ARCH.txt and dies loudly on a mismatch, then runs the install script against the local file: RKE2 via INSTALL_RKE2_ARTIFACT_PATH (tarball rke2.linux-ARCH.tar.gz), K3s by installing the binary to /usr/local/bin/k3s and running the script with INSTALL_K3S_SKIP_DOWNLOAD=binary. Downloads run on the host with curl (no SFTP, nothing is uploaded) into /tmp/rke2-artifacts / /tmp/k3s-artifacts, which are emptied first. Requires version; dies without one.

On RPM-based hosts (Rocky, RHEL) RKE2's install script uses the tarball instead of its RPM method when given an artifact path, so no rke2-selinux package is installed, and a host that already carries RKE2 from RPMs is refused by the script ("existing RKE2 RPMs").

node_name

Kubernetes node name, written as node-name to config.yaml. If omitted, the system hostname is used.

disable

Packaged components to switch off, as an arrayref or a comma-separated string, written as disable to config.yaml. The names are distribution-specific. Default: ['rke2-ingress-nginx', 'rke2-traefik', 'rke2-traefik-crd'] on rke2 (no bundled ingress controller: RKE2 ships the Traefik charts since v1.30.3, opt-in, and deploys Traefik by default on new clusters since v1.36; a name the installed RKE2 does not ship is ignored), ['traefik', 'servicelb'] on k3s. A given list replaces the default rather than extending it; [] disables nothing. Independent of cilium.

# keep the default and also drop metrics-server
disable => [qw( rke2-ingress-nginx rke2-traefik rke2-traefik-crd
                rke2-metrics-server )],

cluster_cidr

The pod network, one IPv4 CIDR such as 10.42.0.0/16, written as cluster-cidr to config.yaml on RKE2 and K3s alike, with or without cilium. Anything else (including a dual-stack list) dies before the host is touched. Every server of a cluster needs the same value (RKE2 refuses a joining server that differs), and it cannot be changed on a running cluster. "install_cilium" in Rex::Rancher::Cilium takes the same value as Cilium's pool, used in cluster-pool mode (see there). Default: nothing written on RKE2 (RKE2's own default, 10.42.0.0/16, applies); on K3s with cilium 10.42.0.0/16 is written, without it nothing.

On a server already set up on the host (its service active, or server/token there), RKE2 and K3s alike, the value it runs with is read first: cluster-cidr from config.yaml and its config.yaml.d/ drop-ins, merged as the distribution merges them (the last one wins), or without one the built-in 10.42.0.0/16. When this run would give it another one (cluster_cidr, or without it the default above), install_server dies with both values before anything is written or installed; that includes a re-run that leaves cluster_cidr out against a server set up with another. Pass the value it runs with. A file there that cannot be read or parsed dies too, with its name. A server not set up yet is not checked.

node_labels

Node labels applied at join time, as an arrayref of key=value strings.

registries

Private registry mirror configuration. Written to registries.yaml in the distribution config directory, 0600 root:root (it may hold registry passwords). Structure:

{
  mirrors => {
    'docker.io' => { endpoint => ['http://registry.internal:5000'] },
  },
  configs => {
    'registry.internal:5000' => {
      auth => { username => 'user', password => 'pass' },
    },
  },
}

cilium

If true (default: 1), switch off the distribution's own CNI in the server config so that Cilium is the only one. Set to 0 to keep the distribution's default CNI (Canal on RKE2, Flannel on K3s).

On rke2, cni: none and disable-kube-proxy: true are written, preparing the node for Cilium with full kube-proxy replacement.

On k3s, flannel-backend: none, disable-network-policy: true, disable-kube-proxy: true and cluster-cidr: 10.42.0.0/16 (or cluster_cidr) are written: Flannel, k3s's embedded network policy controller and kube-proxy are switched off, and Cilium takes over all three with kube-proxy replacement. cluster-cidr is k3s's own default, written out because Cilium's cluster-pool IPAM is given the same range (see Rex::Rancher::Cilium). These are server settings that k3s agents take from the server; an additional server joining with server gets the same keys, as k3s requires them to match across servers. The same keys and Cilium values were verified live in kubernetes-ocp (k3s v1.36.4+k3s1, Cilium 1.20.0, Gateway API v1.6.1), the Cilium version Rex::Rancher::Cilium defaults to, and have since been run live through Rex::Rancher's k3s path (a control plane with one worker; see Rex::Rancher::Distribution::K3s).

nvidia_runtime_path

If true, and nvidia-container-runtime is on the host's PATH, write PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin to /etc/default/rke2-server before the installer runs. The rke2 unit sets no PATH, and rke2 looks for the NVIDIA runtime only when the service starts; without it a host- or vendor-installed toolkit (/usr/bin, e.g. DGX OS) is not wired into containerd. Other lines of the file are kept, an existing PATH= line is replaced. If the file changed while the service is already running, the service is restarted (see "Re-runs"). The GPU Operator's toolkit (/usr/local/nvidia/toolkit) is found by rke2 without this. No effect on k3s. Default: 0; "rancher_deploy_server" in Rex::Rancher turns it on for gpu => 1, gpu_setup => 0.

install_server(
  distribution => 'rke2',
  token        => 'my-cluster-secret',
  tls_san      => ['loadbalancer.example.com'],
  node_labels  => ['role=control-plane'],
  version      => 'v1.30.4+rke2r1',
  node_name    => 'cp-01',
);

# Checksum-verified release artifact instead of curl | sh
install_server(
  version        => 'v1.30.4+rke2r1',
  install_method => 'artifact',
);

preflight_server(%opts)

my $checked = Rex::Rancher::Server::preflight_server(%opts);
# { version => ..., install_method => 'script', cluster_cidr => ... }

Everything "install_server" checks before it writes or installs, with the same options and in the same order, and nothing else: it only reads the host. "install_server" runs it first thing; "rancher_deploy_server" in Rex::Rancher runs it before it prepares the node, and a caller that prepares the node on its own can do the same, so a refused host is left as it was. Not exported.

1. the pure option checks: distribution, install_method (with version), cluster_cidr;
2. Cilium datapath state an earlier cluster left on a host without RKE2 or K3s ("check_cilium_residue" in Rex::Rancher::Uninstall), the first read of the host;
3. with hold_running, the version to hold ("held_version" in Rex::Rancher::Distribution);
4. the version skew against the running server or installed binary ("check_version_skew" in Rex::Rancher::Distribution), with the held version under hold_running, else version;
5. the cluster-cidr of a server already set up on the host ("check_established_cluster_cidr" in Rex::Rancher::Distribution).

Dies as "install_server" would. Returns a hashref: the version this run installs (held, pinned, or undef for the stable channel), the install_method and the cluster_cidr (undef when not given).

update_registries(%opts)

Update registries.yaml on an already-running node and restart the distribution service to pick up the new registry mirror configuration.

Use this to add or change registry mirrors after the cluster is up — for example, after deploying an in-cluster registry that you want every node to use as a pull-through cache.

Before the restart, an agent/etc/containerd/config.toml.tmpl that holds only imports and version = 2, as Rex::GPU 0.001 wrote it, is removed with a warning, for RKE2 and K3s alike: the restart would render it instead of the distribution's own containerd config, which carries the registry mirrors, so the new ones would not take effect. Any other template stays, with a log line; no template changes nothing (see "remove_bare_containerd_template" in Rex::Rancher::Distribution). If the removal fails, update_registries dies after writing registries.yaml and before the restart, so nothing is restarted.

Required options:

registries: Registry mirror hashref (same structure as install_server's registries option).

Optional options:

distribution: rke2 (default) or k3s. Controls which service is restarted.

update_registries(
  distribution => 'rke2',
  registries   => {
    mirrors => {
      'docker.io'         => { endpoint => ['http://registry.internal:5000'] },
      'registry.internal' => { endpoint => ['http://registry.internal:5000'] },
    },
  },
);

get_kubeconfig($distribution)

Read the kubeconfig file from the remote server and return its content as a string. The file is read directly via cat over SSH; no SFTP is used.

$distribution defaults to rke2.

Note: RKE2 and K3s both write a loopback server address (https://127.0.0.1, see "patch_kubeconfig_server"). The content comes back as the node has it; "fetch_kubeconfig" points it at an address that works from elsewhere and saves it, as "rancher_deploy_server" in Rex::Rancher does.

Dies if the file cannot be read.

fetch_kubeconfig(%opts)

my $kubeconfig = fetch_kubeconfig(
  distribution => 'rke2',
  server       => 'cp.example.com',
  file         => "$ENV{HOME}/.kube/cluster.yaml",
);

The server's kubeconfig for use from this machine: read from the host Rex is connected to ("get_kubeconfig", over the exec channel, no SFTP), pointed at server ("patch_kubeconfig_server"), passed through filter if given, written to file if given, and returned. The CA and the admin client certificate and key are kept.

Options:

distribution

rke2 (default) or k3s. Anything else dies before the host is read.

server

The address the kubeconfig will reach the Kubernetes API at: a host name or an IPv4/IPv6 address, without scheme or port. It must be a name the API server certificate is made for (the node's own IPs and host name, or a tls_san), since the CA stays. Omitted, the kubeconfig keeps the loopback address the node wrote, which works only on the node itself (a Local connection) or through a tunnel to its port 6443.

There is no fallback to the host Rex is connected to: the SSH address need not be one the API is reached at or certified for (an ssh_config alias, a jump host, a NAT address). Where it is, pass it: server => connection->server.

filter

A code reference. It gets the patched kubeconfig and returns the one to write and return: the place for a caller's own policy, which this module does not bring (see the example below). If it returns nothing (undef or an empty string), fetch_kubeconfig dies and writes nothing. A filter that is not a code reference dies before the host is read.

file

Local path the result is written to, mode 0600 from the moment it exists: the kubeconfig carries the cluster admin's client certificate and key. An existing file is set to 0600 before its content is replaced, in place, so a symlink keeps pointing where it did. The directory must exist. Without file nothing is written.

Dies naming the cause if the kubeconfig cannot be read from the host, is empty, or cannot be written.

A caller that wants the CA dropped and certificate verification off, for instance because it reaches the API at a name the certificate does not carry, says so in filter; this module keeps the CA itself:

my $kubeconfig = fetch_kubeconfig(
  distribution => 'k3s',
  server       => connection->server,
  file         => "$ENV{HOME}/.kube/k3s.yaml",
  filter       => sub {
    my ( $kc ) = @_;
    $kc =~ s/^[ \t]*certificate-authority-data:.*\n//mg;
    $kc =~ s/^([ \t]*)(server: https:\/\/\S+)\n/$1$2\n$1insecure-skip-tls-verify: true\n/mg;
    return $kc;
  },
);

patch_kubeconfig_server($content, $server)

my $kubeconfig = patch_kubeconfig_server($content, '2001:db8::10');
# server: https://[2001:db8::10]:6443

Return $content, a kubeconfig as RKE2 or K3s wrote it, with its server URL pointed at $server. Pure: no host, no connection, so it serves a kubeconfig read over any channel, not only through "get_kubeconfig".

RKE2 and K3s write https://127.0.0.1:PORT, or https://[::1]:PORT when the cluster's (first) service CIDR is IPv6; both become https://SERVER:PORT, the port kept. Nothing else changes: the CA (certificate-authority-data) and the client certificate and key stay, so $server must be a name the API server certificate is made for. A server URL that is not loopback (a configured bind-address) is left alone.

$server is a host name or an IPv4/IPv6 address, without scheme or port; an IPv6 address is put in brackets (https://[2001:db8::10]:6443), one already in brackets is taken as it is. Dies without $content or $server.

get_token($distribution)

Read the node join token from the server and return it as a string (trailing newline stripped).

$distribution defaults to rke2.

The token is stored at:

RKE2: /var/lib/rancher/rke2/server/node-token
K3s: /var/lib/rancher/k3s/server/node-token

Dies if the file cannot be read (e.g. server not yet started).

SUPPORT

Issues

Please report bugs and feature requests on GitHub at https://github.com/Getty/rex-rancher/issues.

CONTRIBUTING

Contributions are welcome! Please fork the repository and submit a pull request.

AUTHOR

Torsten Raudssus <getty@cpan.org>

COPYRIGHT AND LICENSE

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.

To install Rex::Rancher, copy and paste the appropriate command in to your terminal.

cpanm

cpanm Rex::Rancher

CPAN shell

perl -MCPAN -e shell
install Rex::Rancher

For more information on module installation, please visit the detailed CPAN module installation guide.

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	Go to GitHub issues (only if GitHub is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)