terraform-provider-google: Adding cpu_cfs_quota breaks node pool

Terraform v0.15.2 on darwin_amd64

Google 1.20.5-gke.2000

google-beta 3.66.1

Since we upgraded GKE to 1.20.5-gke.2000

our terraform plan failed for various node pools with the error:

ERR │ Error: googleapi: Error 400: At least one of ['node_version', 'image_type', 'updated_node_pool', 'locations', 'workload_metadata_config', 'upgrade_settings', 'kubelet_config', 'linux_node_config', 'tags', 'taints', 'labels', 'node_network_config', 'gcfs_config'] must be specified., badRequest
  ERR │ 
  ERR │   on ../../../modules/10_k8s/k8s-nodepool-actions.tf line 9, in resource "google_container_node_pool" "actions":
  ERR │    9: resource "google_container_node_pool" "actions" {

After adding kubelet_config (was’t set beore) with the default values

kubelet_config {
      cpu_manager_policy   = "static"
      cpu_cfs_quota        = true
      cpu_cfs_quota_period = "100us"
    }

the node pools get completely broken and had to be deleted, as no pods couldn’t run on it anymore:

Error: failed to create containerd task: OCI runtim
 e create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: se
 tting cgroup config for procHooks process caused: failed to write "100" to "/sys/fs/cgroup/cpu,cpuacct/kubepods/besteffort/pod8230a76c-fc9e-4ad0-
 be3d-c828025fdb5b/download-dashboards/cpu.cfs_period_us": write /sys/fs/cgroup/cpu,cpuacct/kubepods/besteffort/pod8230a76c-fc9e-4ad0-be3d-c828025
 fdb5b/download-dashboards/cpu.cfs_period_us: invalid argument: unknown

Re-creating the node pools fixed the problem, but running our terraform plan again, reveals again the first problem and also with complete random nodepools on dev and prod.

This is the nodepool config:

resource "google_container_node_pool" "iris" {
  for_each           = var.node_pool_config_iris
  project            = var.project_k8s
  provider           = google-beta
  name               = each.value.name
  location           = var.region
  cluster            = google_container_cluster.ee_k8s.name
  initial_node_count = 1
  version            = var.k8s_version

  node_config {
    image_type   = "COS_CONTAINERD"
    disk_size_gb = "50"
    machine_type = each.value.machine_type
    preemptible  = each.value.preemptible

    oauth_scopes = [
      "https://www.googleapis.com/auth/compute",
      "https://www.googleapis.com/auth/devstorage.read_write",
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring",
    ]

    metadata = {
      disable-legacy-endpoints = "true"
    }

  }

  management {
    auto_repair  = true
    auto_upgrade = true
  }

  autoscaling {
    min_node_count = 0
    max_node_count = 40
  }

  timeouts {
    create = "90m"
    delete = "90m"
    update = "90m"
  }

  upgrade_settings {
    max_surge       = 10
    max_unavailable = 1
  }
}

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 5
Comments: 27

Most upvoted comments

@jarrod-manwaring @josandstrom , The api team is communicating that they have a fix in motion on the bug. I’ll ask for a timeline once the fix it checked in. I am thinking < 2 weeks max

ScottSuarez on Jun 16, 2021

I’m also running into this problem and I don’t think I can easily apply the lifecycle workaround @hprotzek offered, because I’m using the GKE Google module (I initially opened an issue against it)

Before I saw this thread, I tried to manually reset the node pool’s kubelet_config using gcloud beta container node-pools update pool-xxxx --system-config-from-file=config.txt with the default empty config described here, in the hope that it would help restore a known good state the provider could work with.

I’ve also tried the cpu_cfs_quota = false cpu_manager_policy = "" workaround @ScottSuarez offered, but the GKE TF module only supports cpu_manager_policy for some reason (I’ve opened a PR to add the other attributes) so that’s unlikely to work either, unless I start using my patched version (which would cause us other problems)

In addition to that, since I did the manual system config reset, I seem to have a similar issue with linux_node_config :

          - kubelet_config {
              - cpu_cfs_quota = false -> null
            }

          - linux_node_config {
              - sysctls = {} -> null
            }

Looking forward to the upstream fix, because I can’t see a good workaround in my case at the moment !

renaudguerin on Jun 29, 2021

This looks like an upstream issue since the api is returning an empty "kubeletConfig": {} in the rest response. The underlying sdk supporting resource deserializes the rest response into the NodePool object. Because the empty kubeletConfig is present NodeKubeletConfig is populated on the NodePool.NodeConfig. Since this field is populated the default values are specified on the field including the cpu_cfs_quota which golang defaults to false. This is inherently incorrect since if cpu_cfs_quota is defined it should be enabled by default.

This is causing issues for terraform as we try to determine the state of the resource from the NodePool object but the state is corrupted due to this behavior.

I can probably patch a workaround for this but maintaining numerous workaround can be a bit of maintenance burden. Two questions : how impactful is this for you? Could you wait for this be fixed upstream or is it pressing to be resolved now? I’ve filed an internal bug on this issue against the api.

I’m not sure why you can’t speicify the default values for the config. The original response you got from the api when you tried to do so seems to infer that there was something wrong with the formatting of quota_period. failed to write "100"

ScottSuarez on Jun 8, 2021

The fix should have been promoted to prod recently. Hopefully a full role-out by the end of the week. I’ll check in next week to ensure that it’s fixed

ScottSuarez on Jun 29, 2021

I deleted the affected node pools and let Terraform re-create them with the config from above. When I re-run terraform I get this:

~ node_config {
            tags              = [
                "internal",
            ]
            # (12 unchanged attributes hidden)

          - kubelet_config {
              - cpu_cfs_quota = false -> null
            }

When I look into the node pool with gcloud:

config:
  diskSizeGb: 50
  diskType: pd-standard
  imageType: COS_CONTAINERD
  kubeletConfig: {}

So it seems the terraform provider thinks the default of cpu_cfs_quota is false, which is wrong according to https://cloud.google.com/sdk/gcloud/reference/beta/container/node-pools/create#--system-config-from-file

hprotzek on Jun 1, 2021