kubernetes: In-cluster configuration fails to be properly evaluated on Windows when the current working directory is not on the system drive

What happened:

Running kubectl from a container running in a Windows-based pod fails if the current working directory is on a different drive than the system drive (C:\):

PS T:\> kubectl --v=999 version
I0819 00:42:45.890463   20432 round_trippers.go:435] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl.exe/v1.22.0 (windows/amd64) kubernet
es/c2b5237" 'http://localhost:8080/version?timeout=32s'
I0819 00:42:47.252500   20432 round_trippers.go:454] GET http://localhost:8080/version?timeout=32s  in 1310 milliseconds
I0819 00:42:47.252500   20432 round_trippers.go:460] Response Headers:
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"c2b5237ccd9c0f1d600d3072634ca66cefdf272f", GitTreeState:"clean", BuildDate:"20
21-08-04T18:03:20Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"windows/amd64"}
I0819 00:42:47.253112   20432 helpers.go:235] Connection error: Get http://localhost:8080/version?timeout=32s: dial tcp [::1]:8080: connectex: No connection could
be made because the target machine actively refused it.
F0819 00:42:47.253112   20432 helpers.go:116] Unable to connect to the server: dial tcp [::1]:8080: connectex: No connection could be made because the target machi
ne actively refused it.
goroutine 1 [running]:
k8s.io/kubernetes/vendor/k8s.io/klog/v2.stacks(0xc000006001, 0xc00015cdc0, 0xbb, 0x125)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1026 +0xbf
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).output(0x2f9ff60, 0xc000000003, 0x0, 0x0, 0xc0009b4fc0, 0x2, 0x26abb69, 0xa, 0x74, 0x24ef00)
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).printDepth(0x2f9ff60, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0x2, 0xc0005ee130, 0x1, 0x1)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:735 +0x190
k8s.io/kubernetes/vendor/k8s.io/klog/v2.FatalDepth(...)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1500
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util.fatal(0xc00055e090, 0x8c, 0x1)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:94 +0x296
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util.checkErr(0x211b000, 0xc000556030, 0x1f93930)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:189 +0x948
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util.CheckErr(...)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:116
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/version.NewCmdVersion.func1(0xc000589680, 0xc000451a20, 0x0, 0x1)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/version/version.go:79 +0x125
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute(0xc000589680, 0xc000451a10, 0x1, 0x1, 0xc000589680, 0xc000451a10)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:856 +0x2c2
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc00093f180, 0xc0000dc060, 0xc0000da040, 0x3)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:960 +0x375
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute(...)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:897
main.main()
        _output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:49 +0x234

goroutine 6 [chan receive]:
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).flushDaemon(0x2f9ff60)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1169 +0x92
created by k8s.io/kubernetes/vendor/k8s.io/klog/v2.init.0
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:420 +0xe5

goroutine 8 [select]:
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x1f93838, 0x2119500, 0xc000556000, 0x1, 0xc0000e0ba0)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:167 +0x119
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x1f93838, 0x12a05f200, 0x0, 0x1, 0xc0000e0ba0)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x9f
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0x1f93838, 0x12a05f200, 0xc0000e0ba0)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x54
created by k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/util/logs.InitLogs
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/util/logs/logs.go:51 +0x9e

(Note that this call is made from a current working directory at the root of the T:\ drive).

Running the same command from a working directory located on the system drive, however, works without issue:

PS C:\> kubectl --v=999 version
I0819 00:28:54.485885   46548 merged_client_builder.go:121] Using in-cluster configuration
I0819 00:28:54.597212   46548 round_trippers.go:435] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl.exe/v1.22.0 (windows/amd64) kubernet
es/c2b5237" -H "Authorization: Bearer <masked>" 'https://172.20.0.1:443/version?timeout=32s'
I0819 00:28:54.604207   46548 round_trippers.go:454] GET https://172.20.0.1:443/version?timeout=32s 200 OK in 5 milliseconds
I0819 00:28:54.604207   46548 round_trippers.go:460] Response Headers:
I0819 00:28:54.604207   46548 round_trippers.go:463]     Content-Type: application/json
I0819 00:28:54.604207   46548 round_trippers.go:463]     Content-Length: 277
I0819 00:28:54.604746   46548 round_trippers.go:463]     Date: Thu, 19 Aug 2021 00:28:54 GMT
I0819 00:28:54.604809   46548 round_trippers.go:463]     Cache-Control: no-cache, private
I0819 00:28:54.608939   46548 request.go:1181] Response Body: {
  "major": "1",
  "minor": "18+",
  "gitVersion": "v1.18.20-eks-8c579e",
  "gitCommit": "8c579edfc914f013ff48b2a2b2c1308fdcacc53f",
  "gitTreeState": "clean",
  "buildDate": "2021-07-31T01:34:13Z",
  "goVersion": "go1.13.15",
  "compiler": "gc",
  "platform": "linux/amd64"
}
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"c2b5237ccd9c0f1d600d3072634ca66cefdf272f", GitTreeState:"clean", BuildDate:"20
21-08-04T18:03:20Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.20-eks-8c579e", GitCommit:"8c579edfc914f013ff48b2a2b2c1308fdcacc53f", GitTreeState:"clean",
BuildDate:"2021-07-31T01:34:13Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.22) and server (1.18) exceeds the supported minor version skew of +/-1

(Note that this command is, this time, invoked from a current working directory at the root of the C:\ drive. The version difference warning is irrelevant to this issue – this is a successful output.)

What you expected to happen:

The current working directory should have no impact on the behavior of kubectl. The command should succeed regardless of the drive the current working directory is located in.

How to reproduce it (as minimally and precisely as possible):

  • Create a Windows image with a secondary drive (e.g. using VOLUME [ 'T:' ] with Docker), ideally with kubectl installed
  • Run this image in container in a pod on a Windows node
  • Download/install kubectl (if not included with the image)
  • Navigating to a directory on the secondary drive (e.g. T:\), run kubectl version
  • kubectl fails because the in-cluster configuration is not properly evaluated

Anything else we need to know?:

Notice that in the working case (when invoking kubectl from C:\), this is the first line of the verbose log:

I0819 00:28:54.485885   46548 merged_client_builder.go:121] Using in-cluster configuration

This line is not present when running kubectl from T:\.

Looking at the client-go project, and specifically, at merged_client_builder.go (link), this log is produced there:

	// check for in-cluster configuration and use it
	if config.icc.Possible() {
		klog.V(4).Infof("Using in-cluster configuration")
		return config.icc.ClientConfig()
	}

Presumably, in the failing case, that branch is not taken. Digging deeper into the implementation of config.icc.Possible() (link), it is clear why:

func (config *inClusterClientConfig) Possible() bool {
	fi, err := os.Stat("/var/run/secrets/kubernetes.io/serviceaccount/token")
	return os.Getenv("KUBERNETES_SERVICE_HOST") != "" &&
		os.Getenv("KUBERNETES_SERVICE_PORT") != "" &&
		err == nil && !fi.IsDir()
}

The path that is being evaluated – /var/run/secrets/kubernetes.io/serviceaccount/token – does not specify a drive letter (as it looks like it is assuming a Unix/Linux system).

On Windows, when a path omits a volume or drive letter, and begins with a directory separator character, it is assumed to be relative to the current drive, taken from the current working directory (reference).

This succeeds when running from a current working directory on the C:\ drive, because the path C:\var\run\secrets\kubernetes.io\serviceaccount\token does exist. But if fails when running from another drive, as no such file exists there.

Environment:

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"c2b5237ccd9c0f1d600d3072634ca66cefdf272f", GitTreeState:"clean", BuildDate:"20
21-08-04T18:03:20Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.20-eks-8c579e", GitCommit:"8c579edfc914f013ff48b2a2b2c1308fdcacc53f", GitTreeState:"clean",
BuildDate:"2021-07-31T01:34:13Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration: Amazon EKS
  • OS (e.g: cat /etc/os-release):
PS C:\> [System.Environment]::OSVersion.Version

Major  Minor  Build  Revision
-----  -----  -----  --------
10     0      17763  0
  • Kernel (e.g. uname -a): n/a
  • Install tools: n/a
  • Network plugin and version (if this is a network-related bug): n/a
  • Others: n/a

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Comments: 23 (9 by maintainers)

Most upvoted comments

I’ve refactored for support in hostprocess and this would fit nicely in there #104490

I am not 100% that the file would always be mapped to the c:\ drive in all cases though. any ideas @marosset @dcantah @kevpar?

It looks like the drive is currently hardcoded to be C: in the kubelet.

When the container request is being created, this check is performed for each mount:

https://github.com/kubernetes/kubernetes/blob/b0bc8adbc2178e15872f9ef040355c51c45d04bb/pkg/kubelet/kubelet_pods.go#L251-L254

Because on Windows a path starting with a slash is not considered absolute, it then calls into volumeutil.MakeAbsolutePath which prepends c::

https://github.com/kubernetes/kubernetes/blob/b0bc8adbc2178e15872f9ef040355c51c45d04bb/pkg/volume/util/util.go#L493-L509

+1
kevpar on Aug 31, 2021
Read more comments on GitHub
← kubernetes: Improve kubectl cp, so it doesn't require the tar binary in the container
kubernetes: Incompatibility between kubelet and kubeadm and kubectl on which label prefix to use →