python: Watcher stuck forever in a case of network disconnect

What happened (please include outputs or screenshots): If the network to Kubernetes API disconnects, watcher just gets stuck forever with no exception.

What you expected to happen: Watcher should raise an exception in a case of any network issues.

How to reproduce it (as minimally and precisely as possible):

Run watcher for any object list with some timeout (60 seconds, for example)
Turn off network adapter on your laptop
Wait for 60 seconds (timeout) and watcher still does not return the control: thread just stuck

Anything else we need to know?:

Environment:

Kubernetes version (kubectl version): 1.17.0
OS (e.g., MacOS 10.13.6): Windows 10
Python version (python --version): 3.7
Python client version (pip list | grep kubernetes): 11.0.0

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 16 (2 by maintainers)

Most upvoted comments

kwargs['timeout_seconds'] = 60 is a polite request to the server, asking it to cleanly close the connection after 60 seconds. If you have a network outage, this does nothing. You can set this number much higher, maybe to 3600 seconds (1h).

kwargs['_request_timeout'] = 60 is a client-side timeout, configuring your local socket. If you have a network outage dropping all packets with no RST/FIN, this is how long your client waits before realizing & dropping the connection. You can keep this number low, maybe 60 seconds.

ellieayla on May 9, 2020