kubeadm: Forbidden error when retrieving logs from non-master node's pods
What keywords did you search in kubeadm issues before filing this one?
kubectl logs
logs forbidden curl insecure
Is this a BUG REPORT or FEATURE REQUEST?
BUG REPORT
Versions
kubeadm version (use kubeadm version):
kubeadm version: version.Info{Major:“1”, Minor:“6+”, GitVersion:“v1.6.0-alpha.0.2074+a092d8e0f95f52”, GitCommit:“a092d8e0f95f5200f7ae2cba45c75ab42da36537”, GitTreeState:“clean”, BuildDate:“2016-12-13T17:03:18Z”, GoVersion:“go1.7.4”, Compiler:“gc”, Platform:“linux/amd64”}
Environment:
- Kubernetes version (use
kubectl version):
Client Version: version.Info{Major:“1”, Minor:“5”, GitVersion:“v1.5.5”, GitCommit:“894ff23729bbc0055907dd3a496afb725396adda”, GitTreeState:“clean”, BuildDate:“2017-03-23T16:14:24Z”, GoVersion:“go1.8”, Compiler:“gc”, Platform:“darwin/amd64”} Server Version: version.Info{Major:“1”, Minor:“5”, GitVersion:“v1.5.4”, GitCommit:“7243c69eb523aa4377bce883e7c0dd76b84709a1”, GitTreeState:“clean”, BuildDate:“2017-03-07T23:34:32Z”, GoVersion:“go1.7.4”, Compiler:“gc”, Platform:“linux/amd64”}
- Cloud provider or hardware configuration: Vsphere
- OS (e.g. from /etc/os-release): Ubuntu 16.04
- Kernel (e.g.
uname -a): Linux <hostname> 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux - Others:
Kubernetes cluster consists of a single master node and minion node, joined together by kubeadm.
What happened?
From a remote machine (that is not the master or minion), when doing a kubectl logs on any pods that lives on the minion node, the following error occurs:
Error from server: Get https://<minion_ip>:10250/containerLogs/default/critics-1347287238-wdssk/critics: Forbidden
When doing a kubectl logs on any of the pods that lives on the master node, no error occurs and logs can be retrieved as expected.
When doing a curl of the URL returned in the error above with a --insecure, I am able to pull the logs from the affected node.
What you expected to happen?
Should be able to retrieve logs of a pod from a non-master node.
Anything else we need to know?
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 12
- Comments: 25 (11 by maintainers)
I found the reason. Its the no_proxy that must be set to include all nodes ip otherwise it try to use the proxy, and thats the proxy which answer Forbiden
I’m using Ubuntu 17.04 and Kubernetes 1.8.1 seems to work now. I tried a couple of things, but It think the resolution was to change the no_proxy settings in the yaml files (/etc/kubernetes/manifests/…yaml) to the same as listed in current environment settings in /etc/environment
@tomdee I’m constantly hitting issues where something doesn’t work if person is in isolated network behind proxies, and trying to fix as much as I can. We have several patches that are already merged into 1.9 and some even backported to 1.8.x to get it better. Some PRs are still under review, but hopefully will soon be merged in 1.9. If you hit something, please feel free to open issue and assign to me or CC me.
@jamiehannaford @tomdee Yes, in my case “no_proxy” should be made before k8s cluster is setup. And the forbidden error was solved.
I spent a while trying to use
no_proxyboth with*and with the IP addresses of all the nodes, but it still did not resolve the problem. Any specific guidance would be really usefulkubectl > apiserver > node hosting the pod
@gousse So setting
export NO_PROXY=$no_proxy,<node1-ip>,<node2-ip>,...solved the issue for you?