kubernetes: kubectl proxy not working for kube contexts with base path

What happened:

Given that you have a kube context that looks like this:

apiVersion: v1
kind: Config
clusters:
- cluster:
    server: https://my-url-to-kubernetes.com/any-base-path-to-kubernetes
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
users:
- name: default
  user:
    client-certificate-data: ANY
    client-key-data: ANY

and you try to run kubectl proxy, the request http://localhost:8081/version fails, because internally the URL is not correctly rewritten to https://my-url-to-kubernetes.com/any-base-path-to-kubernetes/version and instead calls https://my-url-to-kubernetes.com/version. The problem is in file (at least what I think) https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/util/proxy/upgradeaware.go#L231 , where the path needs to be passed as well (also for upgrade connections at https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/util/proxy/upgradeaware.go#L273)

You can see the problem via kubectl proxy --v 10:

I0827 14:18:25.042177   30678 loader.go:375] Config loaded from file:  /Users/fabiankramm/.kube/config
Starting to serve on 127.0.0.1:8001
I0827 14:18:28.414766   30678 proxy_server.go:88] /version matched ^.*
I0827 14:18:28.415391   30678 proxy_server.go:88] 127.0.0.1 matched ^127\.0\.0\.1$
I0827 14:18:28.415610   30678 proxy_server.go:128] Filter accepting GET /version 127.0.0.1
I0827 14:18:28.416002   30678 upgradeaware.go:253] Request was not an upgrade
I0827 14:18:28.416841   30678 round_trippers.go:423] curl -k -v -XGET  -H "Cache-Control: max-age=0" -H "Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7" -H "Sec-Fetch-Mode: navigate" -H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.135 Safari/537.36" -H "Accept-Encoding: gzip, deflate, br" -H "X-Forwarded-For: 127.0.0.1" -H "Sec-Fetch-Site: none" -H "Sec-Fetch-User: ?1" -H "Sec-Fetch-Dest: document" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9" -H "Upgrade-Insecure-Requests: 1" 'https://test.localhost/version'
I0827 14:18:28.654599   30678 round_trippers.go:443] GET https://test.localhost/version 404 Not Found in 237 milliseconds
I0827 14:18:28.654637   30678 round_trippers.go:449] Response Headers:
I0827 14:18:28.654644   30678 round_trippers.go:452]     Date: Thu, 27 Aug 2020 12:18:29 GMT
I0827 14:18:28.654649   30678 round_trippers.go:452]     Content-Type: text/plain; charset=utf-8
I0827 14:18:28.654654   30678 round_trippers.go:452]     Content-Length: 21
I0827 14:18:28.654659   30678 round_trippers.go:452]     Server: nginx/1.19.1

As you can see the request is made to ‘https://test.localhost/version’, even though the kube config has a base path of ‘/kubernetes’:

apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority-data: ANY
    server: https://test.localhost/kubernetes
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
users:
- name: default
  user:
    client-certificate-data: ANY
    client-key-data: ANY

What you expected to happen:

The request http://localhost:8081/version works, when using kubectl proxy

How to reproduce it (as minimally and precisely as possible):

  • Spin up a cluster with api root at a different base path like /kubernetes
  • Run kubectl proxy
  • Try http://localhost:8081/version in the webbrowser

Environment:

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"e19964183377d0ec2052d1f1fa930c4d7575bd50", GitTreeState:"clean", BuildDate:"2020-08-26T14:30:33Z", GoVersion:"go1.15", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:43:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration:
minikube version: v1.12.3
commit: 2243b4b97c131e3244c5f014faedca0d846599f5-dirty

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 1
  • Comments: 25 (21 by maintainers)

Most upvoted comments

@eddiezane can you make sure this is part of the next triage?

Well, we could wait for them to triage issues, but I’m not sure what SIG CLI does there-- we can also look them up and just ping @seans3, @soltysh, @pwittrock

@lavalamp thanks for clarifying!

I see your point and I would be fine to make this optional, but I think it is worth a thought of making kubectl proxy always expose the kubernetes api at ‘/’, regardless of how the underlying kube context looks like. I guess this would be easier to understand for the end user and makes the functionality of kubectl proxy somewhat more consistent, if you do not have exact knowledge of how the used kube context looks like. Also the documentation and command description never really mentions this special case.

Our use case is probably a little special, but we provide some sorts of multi Kubernetes cluster API gateway, where users can specify different kube contexts that can be reached from a single endpoint (something like /kubernetes/cluster/name/api/…). Internally we basically just use the kubectl proxy code and then reroute the incoming request to the specific kubectl proxy handler. Our current workaround is that we modified the k8s.io/apimachinery/pkg/util/proxy package to always expose the api at ‘/’, but I guess we could also add the base path at the incoming request itself, but it would just be much better if we could use the package without any adjustments and workarounds.

Regarding the returned URL rewriting, in my opinion this might be not always needed, because often kube contexts that have a base path are targeting some sort of API gateway proxy (like rancher is doing it for example) that just forward the request to a kubernetes cluster without any modification of the returned URLs, so the original selfLinks etc. without any base path would be correct in that scenario (at least in what I have seen, I could be wrong though). Furthermore, I think there is also no rewriting for kubectl proxy --api-prefix, which is kind of similar?

If the decision is made to pursue this, I would be also willing to work on a pull request for this.