ray: [autoscaler][kubernetes] Cluster launcher broken due to ray:nightly Kubernetes import error

What is the problem?

Autoscaler running in ray-ml:nightly image on Kubernetes fails, with a Kubernetes import error.

ImportError: cannot import name 'client' from 'kubernetes' (/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/kubernetes/__init__.py)

However, directly kubectl execing into the pod and open a Python session, from kubernetes import client works fine.

Also, this issue does not arise for the rayproject/ray:nightly image.

Also, older ray-ml images don’t have this problem – rayproject/ray-ml:73fb5d does not have this problem.

Maybe an issue with the Python path in recent ray-ml images?

Ray version and other system information (Python version, TensorFlow version, OS):

Reproduction (REQUIRED)

Please provide a short code snippet (less than 50 lines if possible) that can be copy-pasted to reproduce the issue. The snippet should have no external library dependencies (i.e., use fake or mock data / environments):

If the code snippet cannot be run by itself, the issue will be closed with “needs-repro-script”.

• I have verified my script runs in a clean environment and reproduces the issue.
• I have verified the issue also occurs with the latest wheels.