kubernetes: kubectl proxy not working for kube contexts with base path
What happened:
Given that you have a kube context that looks like this:
apiVersion: v1
kind: Config
clusters:
- cluster:
server: https://my-url-to-kubernetes.com/any-base-path-to-kubernetes
name: default
contexts:
- context:
cluster: default
user: default
name: default
current-context: default
users:
- name: default
user:
client-certificate-data: ANY
client-key-data: ANY
and you try to run kubectl proxy
, the request http://localhost:8081/version
fails, because internally the URL is not correctly rewritten to https://my-url-to-kubernetes.com/any-base-path-to-kubernetes/version
and instead calls https://my-url-to-kubernetes.com/version
. The problem is in file (at least what I think) https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/util/proxy/upgradeaware.go#L231 , where the path needs to be passed as well (also for upgrade connections at https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/util/proxy/upgradeaware.go#L273)
You can see the problem via kubectl proxy --v 10
:
I0827 14:18:25.042177 30678 loader.go:375] Config loaded from file: /Users/fabiankramm/.kube/config
Starting to serve on 127.0.0.1:8001
I0827 14:18:28.414766 30678 proxy_server.go:88] /version matched ^.*
I0827 14:18:28.415391 30678 proxy_server.go:88] 127.0.0.1 matched ^127\.0\.0\.1$
I0827 14:18:28.415610 30678 proxy_server.go:128] Filter accepting GET /version 127.0.0.1
I0827 14:18:28.416002 30678 upgradeaware.go:253] Request was not an upgrade
I0827 14:18:28.416841 30678 round_trippers.go:423] curl -k -v -XGET -H "Cache-Control: max-age=0" -H "Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7" -H "Sec-Fetch-Mode: navigate" -H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.135 Safari/537.36" -H "Accept-Encoding: gzip, deflate, br" -H "X-Forwarded-For: 127.0.0.1" -H "Sec-Fetch-Site: none" -H "Sec-Fetch-User: ?1" -H "Sec-Fetch-Dest: document" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9" -H "Upgrade-Insecure-Requests: 1" 'https://test.localhost/version'
I0827 14:18:28.654599 30678 round_trippers.go:443] GET https://test.localhost/version 404 Not Found in 237 milliseconds
I0827 14:18:28.654637 30678 round_trippers.go:449] Response Headers:
I0827 14:18:28.654644 30678 round_trippers.go:452] Date: Thu, 27 Aug 2020 12:18:29 GMT
I0827 14:18:28.654649 30678 round_trippers.go:452] Content-Type: text/plain; charset=utf-8
I0827 14:18:28.654654 30678 round_trippers.go:452] Content-Length: 21
I0827 14:18:28.654659 30678 round_trippers.go:452] Server: nginx/1.19.1
As you can see the request is made to ‘https://test.localhost/version’, even though the kube config has a base path of ‘/kubernetes’:
apiVersion: v1
kind: Config
clusters:
- cluster:
certificate-authority-data: ANY
server: https://test.localhost/kubernetes
name: default
contexts:
- context:
cluster: default
user: default
name: default
current-context: default
users:
- name: default
user:
client-certificate-data: ANY
client-key-data: ANY
What you expected to happen:
The request http://localhost:8081/version
works, when using kubectl proxy
How to reproduce it (as minimally and precisely as possible):
- Spin up a cluster with api root at a different base path like /kubernetes
- Run
kubectl proxy
- Try
http://localhost:8081/version
in the webbrowser
Environment:
- Kubernetes version (use
kubectl version
):
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"e19964183377d0ec2052d1f1fa930c4d7575bd50", GitTreeState:"clean", BuildDate:"2020-08-26T14:30:33Z", GoVersion:"go1.15", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:43:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
- Cloud provider or hardware configuration:
minikube version: v1.12.3
commit: 2243b4b97c131e3244c5f014faedca0d846599f5-dirty
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 1
- Comments: 25 (21 by maintainers)
@eddiezane can you make sure this is part of the next triage?
Well, we could wait for them to triage issues, but I’m not sure what SIG CLI does there-- we can also look them up and just ping @seans3, @soltysh, @pwittrock
@lavalamp thanks for clarifying!
I see your point and I would be fine to make this optional, but I think it is worth a thought of making
kubectl proxy
always expose the kubernetes api at ‘/’, regardless of how the underlying kube context looks like. I guess this would be easier to understand for the end user and makes the functionality ofkubectl proxy
somewhat more consistent, if you do not have exact knowledge of how the used kube context looks like. Also the documentation and command description never really mentions this special case.Our use case is probably a little special, but we provide some sorts of multi Kubernetes cluster API gateway, where users can specify different kube contexts that can be reached from a single endpoint (something like /kubernetes/cluster/name/api/…). Internally we basically just use the
kubectl proxy
code and then reroute the incoming request to the specifickubectl proxy
handler. Our current workaround is that we modified thek8s.io/apimachinery/pkg/util/proxy
package to always expose the api at ‘/’, but I guess we could also add the base path at the incoming request itself, but it would just be much better if we could use the package without any adjustments and workarounds.Regarding the returned URL rewriting, in my opinion this might be not always needed, because often kube contexts that have a base path are targeting some sort of API gateway proxy (like rancher is doing it for example) that just forward the request to a kubernetes cluster without any modification of the returned URLs, so the original selfLinks etc. without any base path would be correct in that scenario (at least in what I have seen, I could be wrong though). Furthermore, I think there is also no rewriting for
kubectl proxy --api-prefix
, which is kind of similar?If the decision is made to pursue this, I would be also willing to work on a pull request for this.