rke: Can't install Kubernetes >=1.22 on Flatcar Linux due to missing SELinux custom policies
RKE version:
1.3.3 (using terraform RKE provider v1.3.0)
Docker version: (docker version,docker info preferred)
Client:
Context: default
Debug Mode: false
Server:
Containers: 17
Running: 9
Paused: 0
Stopped: 8
Images: 13
Server Version: 20.10.11
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cde01e96ed658bc5050abe1bb601b4b4510ba7a2
runc version: e4bccdbd64361ac5ea8ba90bb8845add78f957a6
init version:
Security Options:
seccomp
Profile: default
selinux
cgroupns
Kernel Version: 5.10.84-flatcar
Operating System: Flatcar Container Linux by Kinvolk 3033.2.0 (Oklo)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.807GiB
Name: worker-01
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Operating system and kernel: (cat /etc/os-release, uname -r preferred)
NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=3033.2.0
VERSION_ID=3033.2.0
BUILD_ID=2021-12-10-1820
PRETTY_NAME="Flatcar Container Linux by Kinvolk 3033.2.0 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar-linux.org/"
BUG_REPORT_URL="https://issues.flatcar-linux.org"
FLATCAR_BOARD="amd64-usr"
5.10.84-flatcar
Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
Master/Worker nodes provisionned by terraform using the RKE provider v1.3.0. Nodes are vSphere virtual machines based on the Flatcar OVA.
cluster.yml file:
As I’m using the terraform provider, here’s the tf rke_cluster declaration
resource "rke_cluster" "main" {
kubernetes_version = "v1.22.4-rancher1-1"
cluster_name = "test-cluster"
authentication {
strategy = "x509"
sans = "<...redacted...>"
}
dynamic "nodes" {
for_each = flatten([local.rke_cluster_master_nodes, local.rke_cluster_worker_nodes])
content {
address = nodes.value["address"]
ssh_key = nodes.value["id_rsa"]
labels = nodes.value["labels"]
role = nodes.value["roles"]
hostname_override = nodes.value["name"]
user = nodes.value["user"]
}
}
dns {
provider = "coredns"
}
ingress {
provider = "none"
}
network {
plugin = "calico"
options = {
"calico_cloud_provider" : "none",
"calico_flex_volume_plugin_dir" : "/var/lib/kubelet/volumeplugins"
}
}
services {
kube_api {
audit_log {
enabled = true
}
secrets_encryption_config {
enabled = true
}
}
}
upgrade_strategy {
drain = false
max_unavailable_worker = 1
max_unavailable_controlplane = 1
}
}
Steps to Reproduce:
Try to update a kubernetes cluster from 1.21 (or possibly earlier versions) to 1.22 when using Flatcar OS 3033.2.0. I imagine that a fresh 1.22 installation would lead to the same result.
Results:
The following error occurs:
Failed running cluster err:[[selinux] Host [10.130.0.241] does not recognize SELinux label [label=type:rke_container_t]. This is required for Kubernetes version [>=1.22.0-rancher0]. Please install rancher-selinux RPM package and try again]
As shown in docker info above, SELinux is enabled on dockerd, triggering this specific step from RKE. Starting from 1.22, a dedicated custom SELinux policy must be installed on SELinux-enabled nodes. As I’m using Flatcar Linux, it’s not possible to deploy this RPM as-is.
I’m quite a newbie when it comes to SELinux and I don’t see how I can easily work around this as disabling SELinux on the docker daemon is not an option for me. Is there any plan on RKE side to better integrate this with Flatcar Linux ? I may be missing a simple way to circumvent this so don’t hesitate to tell me ^^
Thanks
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 7
- Comments: 31 (6 by maintainers)
disable SELinux on docker …
On Mon, 4 Dec 2023, 15:54 tailtwo, @.***> wrote:
unstale
This is still an issue, so unstale please!
A response from anyone at Rancher would be highly appreciated!
Anyone managed updating k8s to >= 1.22 without disabling SELinux?
Any guiadance from Rancher regarding this issue?
Any Updates?
Is there any other option than disabling SELinux?
Any updates? Disabling SELinux in production clusters isn’t really an option!
@bitfisher I opened an issue on Flatcar side too (see https://github.com/flatcar-linux/Flatcar/issues/598). The current workaround is to have selinux disabled for the docker service. This is of course not ideal but work seems currently on-going on Flatcar side to smooth out SELinux related stuff https://github.com/flatcar-linux/Flatcar/issues/673