rke: Can't install Kubernetes >=1.22 on Flatcar Linux due to missing SELinux custom policies

RKE version:

1.3.3 (using terraform RKE provider v1.3.0)

Docker version: (docker version,docker info preferred)

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 17
  Running: 9
  Paused: 0
  Stopped: 8
 Images: 13
 Server Version: 20.10.11
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: cde01e96ed658bc5050abe1bb601b4b4510ba7a2
 runc version: e4bccdbd64361ac5ea8ba90bb8845add78f957a6
 init version: 
 Security Options:
  seccomp
   Profile: default
  selinux
  cgroupns
 Kernel Version: 5.10.84-flatcar
 Operating System: Flatcar Container Linux by Kinvolk 3033.2.0 (Oklo)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.807GiB
 Name: worker-01
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Operating system and kernel: (cat /etc/os-release, uname -r preferred)

NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=3033.2.0
VERSION_ID=3033.2.0
BUILD_ID=2021-12-10-1820
PRETTY_NAME="Flatcar Container Linux by Kinvolk 3033.2.0 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar-linux.org/"
BUG_REPORT_URL="https://issues.flatcar-linux.org"
FLATCAR_BOARD="amd64-usr"
5.10.84-flatcar

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)

Master/Worker nodes provisionned by terraform using the RKE provider v1.3.0. Nodes are vSphere virtual machines based on the Flatcar OVA.

cluster.yml file: As I’m using the terraform provider, here’s the tf rke_cluster declaration

resource "rke_cluster" "main" {
  kubernetes_version = "v1.22.4-rancher1-1"
  cluster_name       = "test-cluster"
  authentication {
    strategy = "x509"
    sans     = "<...redacted...>"
  }
  dynamic "nodes" {
    for_each = flatten([local.rke_cluster_master_nodes, local.rke_cluster_worker_nodes])
    content {
      address           = nodes.value["address"]
      ssh_key           = nodes.value["id_rsa"]
      labels            = nodes.value["labels"]
      role              = nodes.value["roles"]
      hostname_override = nodes.value["name"]
      user              = nodes.value["user"]
    }
  }
  dns {
    provider = "coredns"
  }
  ingress {
    provider     = "none"
  }
  network {
    plugin  = "calico"
    options = {
        "calico_cloud_provider" : "none",
        "calico_flex_volume_plugin_dir" : "/var/lib/kubelet/volumeplugins"
    }
  }
  services {
    kube_api {
      audit_log {
        enabled = true
      }
      secrets_encryption_config {
        enabled = true
      }
    }
  }
  upgrade_strategy {
    drain                        = false
    max_unavailable_worker       = 1
    max_unavailable_controlplane = 1
  }
}

Steps to Reproduce:

Try to update a kubernetes cluster from 1.21 (or possibly earlier versions) to 1.22 when using Flatcar OS 3033.2.0. I imagine that a fresh 1.22 installation would lead to the same result.

Results:

The following error occurs:

Failed running cluster err:[[selinux] Host [10.130.0.241] does not recognize SELinux label [label=type:rke_container_t]. This is required for Kubernetes version [>=1.22.0-rancher0]. Please install rancher-selinux RPM package and try again]

As shown in docker info above, SELinux is enabled on dockerd, triggering this specific step from RKE. Starting from 1.22, a dedicated custom SELinux policy must be installed on SELinux-enabled nodes. As I’m using Flatcar Linux, it’s not possible to deploy this RPM as-is.

I’m quite a newbie when it comes to SELinux and I don’t see how I can easily work around this as disabling SELinux on the docker daemon is not an option for me. Is there any plan on RKE side to better integrate this with Flatcar Linux ? I may be missing a simple way to circumvent this so don’t hesitate to tell me ^^

Thanks

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 7
  • Comments: 31 (6 by maintainers)

Most upvoted comments

disable SELinux on docker …

On Mon, 4 Dec 2023, 15:54 tailtwo, @.***> wrote:

Unstale again please. There still isn’t a way to upgrade past 1.21 on CoreOS.

— Reply to this email directly, view it on GitHub https://github.com/rancher/rke/issues/2788#issuecomment-1838939294, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEOPPA427OUWQ7NAPUSZW63YHXW4PAVCNFSM5K2UKAB2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBTHA4TGOJSHE2A . You are receiving this because you were mentioned.Message ID: @.***>

unstale

This is still an issue, so unstale please!

A response from anyone at Rancher would be highly appreciated!

Anyone managed updating k8s to >= 1.22 without disabling SELinux?

Any guiadance from Rancher regarding this issue?

Any Updates?

Is there any other option than disabling SELinux?

Any updates? Disabling SELinux in production clusters isn’t really an option!

@bitfisher I opened an issue on Flatcar side too (see https://github.com/flatcar-linux/Flatcar/issues/598). The current workaround is to have selinux disabled for the docker service. This is of course not ideal but work seems currently on-going on Flatcar side to smooth out SELinux related stuff https://github.com/flatcar-linux/Flatcar/issues/673