kubernetes-client: Upload file/dir to pod does not work in GKE

Describe the bug

Ever since our clusters were upgraded to a GKE version that works only with containerd, the upload method simply does not work. It does work on my local minikube, but not on GKE (1.25.10-gke.2700).

Trying to debug this, I see that the upload process starts since I see a fragment of the file for less then a second in /tmp but it removed immediately after starting and the upload method returns false. Tried to play with kubernetes.upload.connection.timeout & kubernetes.upload.request.timeout but it doesn’t help.

Important to mention that my source is actually a different pod in the same cluster from which I initiate the kubernetes-client. From this pod the tests are executed and the API to deploy and trying to upload the file to the remote pod is ran. A simple kubectl cp command in the cluster does work.

Any ides?

Fabric8 Kubernetes Client version

6.9.2

Steps to reproduce

  1. Deploy a maven pod on a GKE cluster from which you can execute the kubernetes-client API and tests.
  2. Try to deploy a different pod and upload a file to it using kubernetes-client code.

Expected behavior

The upload method will return false and the file/dir will not be copied.

Runtime

other (please specify in additional context)

Kubernetes API Server version

1.25.3@latest

Environment

GKE

Fabric8 Kubernetes Client Logs

No logs from the fabric8 are seen...

Additional context

No response

About this issue

  • Original URL
  • State: open
  • Created 7 months ago
  • Comments: 17 (9 by maintainers)

Most upvoted comments

@shawkins You mean simply running “wc -c < tmpfile” in the pod? The tmpfile is removed shortly after the upload starts, I can see it for just a fragile of second…

I mean on any file to start just to validate that the appropriate size is seen.

The client uses slf4j, so you can configure it with whatever is bound as the logging implementation.

The relevant package / logging context here is io.fabric8.kubernetes.client.dsl.internal.uploadable - but you can go much higher up if you want to see more of what is happening.

Might be the same root-cause as for #5527.

In any case, the same fix should likely apply.