aws-efs-csi-driver: Mount hanging indefinitely

+1 to setting this with the helm chart. Updating all of the node groups with custom launch templates is quite the chore to configure this one setting.

subscribing as we have a similar issue but killing stunnel doesn’t fix it for us still

Jun 7 01:00:38 [ip] kernel: nfs: server 127.0.0.1 not responding, timed out

is there some other things to check/restart ? (vs killing the node; I’d like to understand how it gets in this state and a workaround to unblock it)

Hi @0nebody, Nice job with the workaround daemonset, just be aware that you may not need to mount /proc since “hostPID” is already set to true. This means that the containers within the pod would use the host PID namespace and will already be able to see and interact with the processes in the host namespace.

If you need to test the effect of the script, you can temporarily update your EFS security group to remove/block access on port 2049. This would cause the NFS timeouts to eventually appear in kernel logs which would then trigger the script to execute.

Hi @Cappuccinuo, Stunnel is not included/installed in the EKS optimised AMI/worker nodes, instead it is installed within the EFS CSI Driver image (see here) which is running as a Daemonset in an EKS cluster. This docker image seems to clone the efs-utils GitHub repo and install the amazon-efs-utils package as per this spec file which specifies stunnel version 4.56. Therefore upgrading it is a bit challenging, it looks like you have to build your own EFS CSI driver image manually to update stunnel. Is it possible for a docker image version/tag of the EFS CSI driver to be available with stunnel upgraded to 5.63?

Adding a workaround here for anyone who needs it, the following script monitors the stunnel processes running on the worker node and kills them if they are unresponsive. The killed processes are then re-created by the EFS CSI driver. The stunnel processes are used to add TLS encryption to NFS traffic, you can read more about it here

#!/bin/bash

WAIT_TIME=60 # Time in seconds to wait before polling stunnel processes
TIMEOUT=2    # Time in seconds to wait for response from remote EFS system connected via stunnel

while true; do
    STUNNEL_NFS_MOUNTS=`mount | grep nfs4 | grep 127.0.0.1 | awk '{print $3}'`
    for STUNNEL_MOUNT in $STUNNEL_NFS_MOUNTS; do
        timeout $TIMEOUT stat -t $STUNNEL_MOUNT >/dev/null 2>&1
        if [ $? -eq 124 ]; then
            echo "[`date`] Timeout when connecting to NFS, killing stunnel process"
            NAME=`echo $STUNNEL_MOUNT | sed -r 's:/:.:g'`
            PROCESS=`ps -ef | grep stunnel | grep $NAME`
            PROCESS_NAME=`echo $PROCESS | awk '{ print $8, $9}'`
            PROCESS_PID=`echo $PROCESS | awk '{print $2}'`
            kill -9 $PROCESS_PID
            echo "[`date`] Killed process $PROCESS_PID '$PROCESS_NAME'"
        fi
    done
    sleep $WAIT_TIME
done

The above script can be invoked in a regular crontab entry or as a systemd service (see below). Running it as a daemonset is also possible but you would need to run the container as root, mount /var/lib/kubelet to see NFS mount points created by the EFS CSI Driver and specify hostPID: true in pod spec to view and kill stunnel processes

[Unit]
Description=Monitor health of stunnel processes created by the EFS CSI driver

[Service]
ExecStart=<SCRIPT_LOCATION>

[Install]
WantedBy=multi-user.target

If you are using EKS managed nodegroups with a launch template specified (see here), you can use EC2 user data to make the above configuration changes as new EKS worker nodes are being provisioned.

If anyone else wants to try this workaround, feel free to do so but please test before deploying to a production environment.

same issue, I noticed this issue because of:

• High cpu load on the node, but with low cpu usage
• No response from df -h
• Hang at efs mounting by strace df
• Reporting nfs: server 127.0.0.1 not responding, still trying by dmesg

eks vesion: 1.21.5 driver version: 1.3.5

Edited: I upgraded driver version to 1.3.6, still the same issue, I think this is a critical P1 bug.