gcs-fuse-csi-driver: The sidecar container does not respect terminationGracePeriodSeconds when the Pod restartPolicy is OnFailure or Always
Symptom
When the Pod restartPolicy is OnFailure or Always, and the Pod does not belong to a Job workload, the sidecar container gke-gcsfuse-sidecar does not respect the Pod terminationGracePeriodSeconds. The sidecar container will keep running until the terminationGracePeriodSeconds passed.
For example, the Pod terminationGracePeriodSeconds is set to be 60s. When you run kubectl delete pod xxx, the main workload container may have a logic to catch the SIGTERM signal, clean up, and gracefully exit in a few seconds. At this moment, the sidecar container is supposed to terminate gracefully as well because there will be no I/O that relies on the sidecar container. However, the sidecar container will wait the full terminationGracePeriodSeconds 60 seconds until Kubernetes sends the SIGKILL signal to forcibly kill it.
Note that, if the Pod
restartPolicyisNever, or if the Pod belongs to aJobworkload, after all the main workload containers exit, the sidecar container will automatically terminate immediately.
Root cause
The root cause of this behavior is that, when the sidecar container catches the SIGTERM signal, it does not propagate the signal to gcsfuse processes. Instead, the sidecar container main process will sleep and wait for the SIGKILL signal to forcibly kill all the gcsfuse processes.
Currently, gcsfuse cannot gracefully handle the SIGTERM signal – when it receives the SIGTERM signal , it will terminate immediately, even though there may be in-flight I/O happening. As a result, the sidecar container main process does not propagate the signal to gcsfuse processes in order to avoid any data loss.
Solution
When gcsfuse supports SIGTERM signal graceful handling, the sidecar container will propagate the signal to gcsfuse processes.
Workaround
There are two workarounds to avoid this issue.
- Consider setting the Pod
restartPolicytoNeverif possible. - In your application, add a piece of logic to notify the sidecar container to exit. Below is an example:
apiVersion: v1
kind: Pod
metadata:
name: gcsfuse-csi-grace-test
annotations:
gke-gcsfuse/volumes: "true"
spec:
terminationGracePeriodSeconds: 60
containers:
- name: busybox
image: busybox
command:
- "/bin/sh"
- "-c"
- handle_term() { echo "Caught SIGTERM signal! Finishig up the application!"; sleep 30; touch /gcsfuse-tmp/.volumes/exit; };
trap handle_term SIGTERM;
echo "I am sleeping!";
sleep infinity & wait $!;
volumeMounts:
- name: gcs-fuse-csi-ephemeral
mountPath: /data
- name: gke-gcsfuse-tmp
mountPath: /gcsfuse-tmp
volumes:
- name: gcs-fuse-csi-ephemeral
csi:
driver: gcsfuse.csi.storage.gke.io
volumeAttributes:
bucketName: your-bucket-name
Here is some explanation:
- The
busyboxis your main workload container. - A volume
gke-gcsfuse-tmpis mounted to the mount path/gcsfuse-tmp. You don’t need to define the volumegke-gcsfuse-tmpbecause it is automatically injected by GKE. - In your application, there is a process running in the background to catch the
SIGTERMsignal. - When the
SIGTERMsignal is seen, your application does some clean up work. - After the cleanup is done, your application creates a new file
/gcsfuse-tmp/.volumes/exit. - The sidecar container also has the volume
gke-gcsfuse-tmpmounted, and will detect the file/gcsfuse-tmp/.volumes/exitand exit immediately. - In the above example, after the main application caught the
SIGTERMsignal, it took 30 seconds to do the cleanup, and then notifies the sidecar container to exit. The Pod is able to terminate gracefully in ~ 30 seconds after the Pod delete operation.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 19 (9 by maintainers)
The new GKE version rollout completed. Starting from GKE
1.29.3-gke.1093000, the CSI driver injects the GCSFuse sidecar container as an init container that respects the terminationGracePeriodSeconds.To try out the new feature, please upgrade your GKE cluster to
1.29.3-gke.1093000or later, and make sure ALL your nodes are also upgraded to GKE version 1.29 or later, then re-deploy your workloads.Closing this issue for now.
Hi @timscottbell we released a new CSI version and started the rollout. The new ETA is 4/4 to complete the rollout. I will update the doc to mention the GKE version will you need to upgrade to.
@timscottbell , thanks for the information, this is very helpful for me to understand your use case. I understand your frustration, and I am prioritizing this bug fix. I am thinking about another way to fix this issue. Will keep you posted.
We will update the GKE GCS FUSE CSI Driver doc to guid you which GKE version is available to include the new feature or fix.
Hi @timscottbell , I pushed a fix to properly handle the sidecar auto-termination: https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver/commit/3c9cc36c392962ef6770ae8b363938464834e6a6
Basically, in your use case, when you scale down your Deployment workload, with the new fix, the following steps will happen:
SIGTERMto the Pod. TheterminationGracePeriodSeconds(300s) count down starts.SIGTERMsignal.Terminatingstate.Terminatingstate, AND all the workload containers exited, the CSI driver sends aexit fileto the gcsfuse sidecar container. (This will take 10s.)exit file.I’ve added a test case in our e2e test: https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver/blob/3c9cc36c392962ef6770ae8b363938464834e6a6/test/e2e/testsuites/workloads.go#L110-L157
@timscottbell please let me your thoughts. If this looks good to you, I will start the rollout this week. Thank you!