tailscale: k8s operator: tailscale ingress sometimes tries to connect to 127.0.0.1 instead of ClusterIP, fails with "netstack: could not connect to local server at ..."
What is the issue?
I’m really not sure how to reproduce this but I’ve seen this a couple of times. Restarting the tailscale ingress pods seems to fix it.
I set up two services (docker-registry, and headlamp) with type ClusterIP both listening on port 80. Both services have a tailscale ingress. This is a test cluster with only one node, so everything is on the same node.
When trying to connect, I see errors like this in the tailscale pod:
2024/05/09 22:55:03 Accept: TCP{100.115.199.49:52551 > 100.77.184.113:80} 64 tcp ok
2024/05/09 22:55:03 netstack: could not connect to local server at 127.0.0.1:80: dial tcp 127.0.0.1:80: connect: connection refused
Restarting the tailscale ingress pod seems to fix the issue.
I’m not a kubernetes expert, but it seems supicious that tailscale is trying to connect to the service on 127.0.0.1:80 rather than using the cluster IP.
# kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.233.0.1 <none> 443/TCP 30m
docker-registry docker-registry ClusterIP 10.233.22.50 <none> 80/TCP 17m
headlamp headlamp ClusterIP 10.233.49.34 <none> 80/TCP 27m
kube-system coredns ClusterIP 10.233.0.3 <none> 53/UDP,53/TCP,9153/TCP 29m
tailscale ts-docker-registry-ingress-4w69x ClusterIP None <none> <none> 16m
tailscale ts-headlamp-ingress-6mzrv ClusterIP None <none> <none> 17m
# kubectl get ingress -A
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE
docker-registry docker-registry-ingress tailscale * berkeley-staging-docker.taila1eba.ts.net 80, 443 36m
headlamp headlamp-ingress tailscale * berkeley-staging-headlamp.taila1eba.ts.net 80, 443 36m
I am able to connect to both services simultaneously using kubectl port-forward, so I’m pretty sure this is not an inherit limitation of my kubernetes set-up.
Steps to reproduce
No response
Are there any recent changes that introduced the issue?
No response
OS
Linux
OS version
kubernetes
Tailscale version
1.62.1
Other software
calico CNI
Bug report
BUG-893cd6c8f00a44fda54bd05672e58e601316cc21d2892024305ff3afa3f4c675-20240509231821Z-5c7d259ff6d31521
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 3
- Comments: 34 (16 by maintainers)
Commits related to this issue
- ipn/ipnlocal: log if ServeConfig cannot be configured We have lately seen a bunch of mystery issues where serve config 'disappears' on kube, hopefully these extra logs will help to understand why. U... — committed to tailscale/tailscale by irbekrm 2 years ago
- ipn/ipnlocal: log if ServeConfig cannot be configured We have lately seen a bunch of mystery issues where serve config 'disappears' on kube, which may have been because of transient errors when readi... — committed to tailscale/tailscale by irbekrm 2 years ago
- ipn/store/kubestore: cache state in memory Cache state in memory on writes, read from memory in reads. kubestore was previously always reading state from a Secret. This change should fix bugs caused ... — committed to tailscale/tailscale by deleted user 4 years ago
- ipn/store/kubestore: cache state in memory Cache state in memory on writes, read from memory in reads. kubestore was previously always reading state from a Secret. This change should fix bugs caused ... — committed to tailscale/tailscale by deleted user 4 years ago
- ipn/store/kubestore: cache state in memory (#13918) Cache state in memory on writes, read from memory in reads. kubestore was previously always reading state from a Secret. This change should fix ... — committed to tailscale/tailscale by irbekrm 2 years ago
- cmd/containerboot,kube/kubetypes: refactor containerboot kube client So that the kube client is no longer a global var. Updates tailscale/tailscale#12079 Signed-off-by: Irbe Krumina <irbe@tailscale... — committed to tailscale/tailscale by irbekrm 2 years ago
- cmd/containerboot,kube/kubetypes: refactor containerboot kube client So that the kube client is no longer a global var. Updates tailscale/tailscale#12079 Signed-off-by: Irbe Krumina <irbe@tailscale... — committed to tailscale/tailscale by irbekrm 2 years ago
- cmd/{containerboot,k8s-operator},kube/kubetypes: kube Ingress L7 proxies only advertise HTTPS endpoint when ready (#14171) cmd/containerboot,kube/kubetypes,cmd/k8s-operator: detect if Ingress is crea... — committed to tailscale/tailscale by irbekrm 2 years ago
- ipn/store/kubestore: cache state in memory (#13918) Cache state in memory on writes, read from memory in reads. kubestore was previously always reading state from a Secret. This change should fix ... — committed to ninech/tailscale by irbekrm 2 years ago
- cmd/{containerboot,k8s-operator},kube/kubetypes: kube Ingress L7 proxies only advertise HTTPS endpoint when ready (#14171) cmd/containerboot,kube/kubetypes,cmd/k8s-operator: detect if Ingress is crea... — committed to ninech/tailscale by irbekrm 2 years ago
Thank you @DevPre24 that would explain the erorrs. It looks like the serve config (configuration for the proxy how to route traffic to the backend) was never loaded, so we fall back routing to local server where there is nothing listening.
This is expected, it will be undefined in the file, it’s only interpolated at config set up time.
I can’t tell yet why the config is not loaded.
Would you be able to:
check if the machine has a cert domain available, you can do so via something like
tailscale debug netmap | grep -A 2 CertDomainson the ingress proxyPod, it should show the full MagicDNS name of the proxycheck if it works if you manually load the serve config. You can do that from the proxy Pod:
Curious what is
tailscale serve statusafter running that last command as well as whether that actually makes the routing to backend work?It’s definitely an issue specific to
Ingress. The tun mode proxies that we spin up forServices don’t use netstack, so we don’t hit that path there.@s04 thank you for the logs- for
Ingress, this seems to be the same issue where the proxy backend config is not loaded, you could verify if that is the case by runningtailscale serve statusfrom the proxyPod. If LoadBalancerServiceis also not working for you that would be a separate issue and worth opening a different issue for. I also see that you are on OpenStack, not sure if that has some limitations - the proxies we spin up for LoadBalancerServices need more permissions than theIngressones.Have this problem on a K3s cluster as well. Tried all variants in the docs (service annotations, Ingress…) but all attempts ends up with this log
netstack: could not connect to local server at 127.0.0.1:80: dial tcp 127.0.0.1:80: connect: connection refusedin the operator pod. I have never gotten it to work.I believe this issue should have been fixed in 1.78.
It seems like there were two reasons for this not working:
issues around writing/reading serve config from the Kubernetes Secret. This should have been fixed by caching the contents of the serve config, so that we now read from memory only https://github.com/tailscale/tailscale/pull/13918 Additionally we’ve bumped the timeout for writes and are also now writing events to the proxy Pod so that if there are more issues related to state Secret they should be observable https://github.com/tailscale/tailscale/pull/14112
HTTPS not enabled for tailnet. In 1.78 it should be easier to discover if the Ingress is not working if HTTPS is not enabled for a tailnet. The proxies now log an error if HTTPS is not enabled and the operator does not set the Ingress’s hostname unless the proxies have been able to successfully load configuration https://github.com/tailscale/tailscale/pull/14171
With that I am going to close this issue. However, if there are more issues that folks observe post-1.78, keen to hear about that
@irbekrm D’oH, I’m an idiot. This works in this configuration. So assuming you have your admissioncontrol setup properly, both Ingress and Service endpoints via tailscale work on Talos. Seeing what can be done from your end is already being handled in https://github.com/tailscale/tailscale/issues/7359
@nralbers thank you once again for the detailed logs.
For a
Serviceunlike theIngresswe don’t expose port 443 to tailnet, we just proxy traffic received on any port to the same port on the clusterService. I think in your case herecurl -vv flux-system-capacitor-test.tailbc32d6.ts.net:9000should have worked.Basically
Ingressis a mechanism for exposing a bunch ofServicesover TLS to tailnet on port 443 (L7) whereasServicejust proxies any traffic received on any port to the same port onService(L3/L4)