gcs-fuse-csi-driver: The sidecar container does not respect terminationGracePeriodSeconds when the Pod restartPolicy is OnFailure or Always

Symptom

When the Pod restartPolicy is OnFailure or Always, and the Pod does not belong to a Job workload, the sidecar container gke-gcsfuse-sidecar does not respect the Pod terminationGracePeriodSeconds. The sidecar container will keep running until the terminationGracePeriodSeconds passed.

For example, the Pod terminationGracePeriodSeconds is set to be 60s. When you run kubectl delete pod xxx, the main workload container may have a logic to catch the SIGTERM signal, clean up, and gracefully exit in a few seconds. At this moment, the sidecar container is supposed to terminate gracefully as well because there will be no I/O that relies on the sidecar container. However, the sidecar container will wait the full terminationGracePeriodSeconds 60 seconds until Kubernetes sends the SIGKILL signal to forcibly kill it.

Note that, if the Pod restartPolicy is Never, or if the Pod belongs to a Job workload, after all the main workload containers exit, the sidecar container will automatically terminate immediately.

Root cause

The root cause of this behavior is that, when the sidecar container catches the SIGTERM signal, it does not propagate the signal to gcsfuse processes. Instead, the sidecar container main process will sleep and wait for the SIGKILL signal to forcibly kill all the gcsfuse processes.

Currently, gcsfuse cannot gracefully handle the SIGTERM signal – when it receives the SIGTERM signal , it will terminate immediately, even though there may be in-flight I/O happening. As a result, the sidecar container main process does not propagate the signal to gcsfuse processes in order to avoid any data loss.

Solution

When gcsfuse supports SIGTERM signal graceful handling, the sidecar container will propagate the signal to gcsfuse processes.

Workaround

There are two workarounds to avoid this issue.

  1. Consider setting the Pod restartPolicy to Never if possible.
  2. In your application, add a piece of logic to notify the sidecar container to exit. Below is an example:
apiVersion: v1
kind: Pod
metadata:
  name: gcsfuse-csi-grace-test
  annotations:
    gke-gcsfuse/volumes: "true"
spec:
  terminationGracePeriodSeconds: 60
  containers:
  - name: busybox
    image: busybox
    command:
      - "/bin/sh"
      - "-c"
      - handle_term() { echo "Caught SIGTERM signal! Finishig up the application!"; sleep 30; touch /gcsfuse-tmp/.volumes/exit; };
        trap handle_term SIGTERM;
        echo "I am sleeping!";
        sleep infinity & wait $!;
    volumeMounts:
    - name: gcs-fuse-csi-ephemeral
      mountPath: /data
    - name: gke-gcsfuse-tmp
      mountPath: /gcsfuse-tmp
  volumes:
  - name: gcs-fuse-csi-ephemeral
    csi:
      driver: gcsfuse.csi.storage.gke.io
      volumeAttributes:
        bucketName: your-bucket-name

Here is some explanation:

  • The busybox is your main workload container.
  • A volume gke-gcsfuse-tmp is mounted to the mount path /gcsfuse-tmp. You don’t need to define the volume gke-gcsfuse-tmp because it is automatically injected by GKE.
  • In your application, there is a process running in the background to catch the SIGTERM signal.
  • When the SIGTERM signal is seen, your application does some clean up work.
  • After the cleanup is done, your application creates a new file /gcsfuse-tmp/.volumes/exit.
  • The sidecar container also has the volume gke-gcsfuse-tmp mounted, and will detect the file /gcsfuse-tmp/.volumes/exit and exit immediately.
  • In the above example, after the main application caught the SIGTERM signal, it took 30 seconds to do the cleanup, and then notifies the sidecar container to exit. The Pod is able to terminate gracefully in ~ 30 seconds after the Pod delete operation.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 19 (9 by maintainers)

Most upvoted comments

The new GKE version rollout completed. Starting from GKE 1.29.3-gke.1093000, the CSI driver injects the GCSFuse sidecar container as an init container that respects the terminationGracePeriodSeconds.

To try out the new feature, please upgrade your GKE cluster to 1.29.3-gke.1093000 or later, and make sure ALL your nodes are also upgraded to GKE version 1.29 or later, then re-deploy your workloads.

Closing this issue for now.

Hi @timscottbell we released a new CSI version and started the rollout. The new ETA is 4/4 to complete the rollout. I will update the doc to mention the GKE version will you need to upgrade to.

@timscottbell , thanks for the information, this is very helpful for me to understand your use case. I understand your frustration, and I am prioritizing this bug fix. I am thinking about another way to fix this issue. Will keep you posted.

Great, thanks @songjiaxun . Which doc are you referring to?

We will update the GKE GCS FUSE CSI Driver doc to guid you which GKE version is available to include the new feature or fix.

Hi @timscottbell , I pushed a fix to properly handle the sidecar auto-termination: https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver/commit/3c9cc36c392962ef6770ae8b363938464834e6a6

Basically, in your use case, when you scale down your Deployment workload, with the new fix, the following steps will happen:

  1. Kubernetes sends SIGTERM to the Pod. The terminationGracePeriodSeconds (300s) count down starts.
  2. The workload container and the gcsfuse sidecar container catch the SIGTERM signal.
  3. The Pod goes to Terminating state.
  4. The workload container finishes up the application and exits. (e.g., the cleanup takes 30s) The workload container must exit with whatever exit code.
  5. The CSI driver monitors the Pod. When the Pod is in a Terminating state, AND all the workload containers exited, the CSI driver sends a exit file to the gcsfuse sidecar container. (This will take 10s.)
  6. The gcsfuse sidecar container exits when it receives the exit file.
  7. So in this example, after about 40s, the gcsfuse sidecar container will terminate automatically.

I’ve added a test case in our e2e test: https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver/blob/3c9cc36c392962ef6770ae8b363938464834e6a6/test/e2e/testsuites/workloads.go#L110-L157

@timscottbell please let me your thoughts. If this looks good to you, I will start the rollout this week. Thank you!