aws-efs-csi-driver: helm chart v2.3.0 -> EFS CSI v1.4.3 -> missing '/usr/bin/stunnel'

/kind bug

What happened?

On EKS 1.23 I tried to upgrade the EFS CSI helm chart from v2.2.9 to v.2.3.0, but the efs-csi-node Pods started to crash like this:

efs-plugin E1102 11:28:25.736669       1 efs_watch_dog.go:285] Process amazon-efs-mount-watchdog exits exit status 1                                                                                                                          
efs-plugin E1102 11:28:25.848044       1 efs_watch_dog.go:331] amazon-efs-mount-watchdog[Error]: Traceback (most recent call last):                                                                                                           
efs-plugin   File "/usr/bin/amazon-efs-mount-watchdog", line 2014, in <module>                                                                                                                                                                
efs-plugin     main()                                                                                                                                                                                                                         
efs-plugin   File "/usr/bin/amazon-efs-mount-watchdog", line 2004, in main                                                                                                                                                                    
efs-plugin     unmount_count_for_consistency,                                                                                                                                                                                                 
efs-plugin   File "/usr/bin/amazon-efs-mount-watchdog", line 1029, in check_efs_mounts                                                                                                                                                        
efs-plugin     restart_tls_tunnel(child_procs, state, state_file_dir, state_file)                                                                                                                                                             
efs-plugin   File "/usr/bin/amazon-efs-mount-watchdog", line 943, in restart_tls_tunnel                                                                                                                                                       
efs-plugin     new_tunnel_pid = start_tls_tunnel(child_procs, state_file, state["cmd"])                                                                                                                                                       
efs-plugin   File "/usr/bin/amazon-efs-mount-watchdog", line 858, in start_tls_tunnel                                                                                                                                                         
efs-plugin     close_fds=True,                                                                                                                                                                                                                
efs-plugin   File "/usr/lib64/python3.7/subprocess.py", line 800, in __init__                                                                                                                                                                 
efs-plugin     restore_signals, start_new_session)                                                                                                                                                                                            
efs-plugin   File "/usr/lib64/python3.7/subprocess.py", line 1551, in _execute_child                                                                                                                                                          
efs-plugin     raise child_exception_type(errno_num, err_msg, err_filename)                                                                                                                                                                   
efs-plugin FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/stunnel': '/usr/bin/stunnel'

I had to roll back to v2.2.9 version.

What you expected to happen?

Smooth helm upgrade.

How to reproduce it (as minimally and precisely as possible)?

Upgrade EFS CSI helm chart from v2.2.9 to v.2.3.0.

Anything else we need to know?:

Environment

Kubernetes version (use kubectl version): v1.23.x
Driver version: v1.4.3

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 10
Comments: 21 (1 by maintainers)

Most upvoted comments

I have the same problem.

EKS 1.22, node OS is Amazon Linux 2.

Looks like the aws-efs-csi-driver:v1.4.2 Docker image still has /usr/bin/stunnel, but in aws-efs-csi-driver:v1.4.3 it’s /usr/bin/stunnel5.

yum list shows:

stunnel.x86_64 4.56-6.amzn2.0.3 in aws-efs-csi-driver:v1.4.2
stunnel5.x86_64 5.58-1.amzn2.0.1 in aws-efs-csi-driver:v1.4.3

MiGrandjean on Nov 4, 2022

Did the revert get reverted? We are seeing this in 1.4.5.

efs-csi-node-vfhxq efs-plugin 2022-11-14T16:57:11.358783111+11:00 E1114 05:57:11.358677       1 efs_watch_dog.go:285] Process amazon-efs-mount-watchdog exits exit status 1
efs-csi-node-vfhxq efs-plugin 2022-11-14T16:57:12.860564756+11:00 E1114 05:57:12.860429       1 efs_watch_dog.go:331] amazon-efs-mount-watchdog[Error]: Traceback (most recent call last):efs-csi-node-vfhxq efs-plugin 2022-11-14T16:57:12.860608387+11:00   File "/usr/bin/amazon-efs-mount-watchdog", line 2014, in <module>
efs-csi-node-vfhxq efs-plugin 2022-11-14T16:57:12.860612737+11:00     main()
efs-csi-node-vfhxq efs-plugin 2022-11-14T16:57:12.860616227+11:00   File "/usr/bin/amazon-efs-mount-watchdog", line 2004, in main
efs-csi-node-vfhxq efs-plugin 2022-11-14T16:57:12.860619467+11:00     unmount_count_for_consistency,
efs-csi-node-vfhxq efs-plugin 2022-11-14T16:57:12.860622467+11:00   File "/usr/bin/amazon-efs-mount-watchdog", line 1029, in check_efs_mounts
efs-csi-node-vfhxq efs-plugin 2022-11-14T16:57:12.860625467+11:00     restart_tls_tunnel(child_procs, state, state_file_dir, state_file)
efs-csi-node-vfhxq efs-plugin 2022-11-14T16:57:12.860628577+11:00   File "/usr/bin/amazon-efs-mount-watchdog", line 943, in restart_tls_tunnel
efs-csi-node-vfhxq efs-plugin 2022-11-14T16:57:12.860641787+11:00     new_tunnel_pid = start_tls_tunnel(child_procs, state_file, state["cmd"])
efs-csi-node-vfhxq efs-plugin 2022-11-14T16:57:12.860645617+11:00   File "/usr/bin/amazon-efs-mount-watchdog", line 858, in start_tls_tunnel
efs-csi-node-vfhxq efs-plugin 2022-11-14T16:57:12.860648957+11:00     close_fds=True,
efs-csi-node-vfhxq efs-plugin 2022-11-14T16:57:12.860659078+11:00   File "/usr/lib64/python3.7/subprocess.py", line 800, in __init__
efs-csi-node-vfhxq efs-plugin 2022-11-14T16:57:12.860663438+11:00     restore_signals, start_new_session)
efs-csi-node-vfhxq efs-plugin 2022-11-14T16:57:12.860666928+11:00   File "/usr/lib64/python3.7/subprocess.py", line 1551, in _execute_child
efs-csi-node-vfhxq efs-plugin 2022-11-14T16:57:12.860670268+11:00     raise child_exception_type(errno_num, err_msg, err_filename)
efs-csi-node-vfhxq efs-plugin 2022-11-14T16:57:12.860674158+11:00 FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/stunnel5': '/usr/bin/stunnel5'
efs-csi-node-vfhxq efs-plugin 2022-11-14T16:57:12.964893690+11:00 E1114 05:57:12.964696       1 efs_watch_dog.go:285] Process amazon-efs-mount-watchdog exits exit status 1

jurgen-weber-deltatre on Nov 14, 2022

I’m kinda amazed how this was released in the first place. This is critical issue that breaks intended driver functionality. Did e2e tests catch this (but results were ignored) or is this a flaw in the test suite?

z0rc on Nov 6, 2022