tailscale: k8s operator: tailscale ingress sometimes tries to connect to 127.0.0.1 instead of ClusterIP, fails with "netstack: could not connect to local server at ..."

What is the issue?

I’m really not sure how to reproduce this but I’ve seen this a couple of times. Restarting the tailscale ingress pods seems to fix it.

I set up two services (docker-registry, and headlamp) with type ClusterIP both listening on port 80. Both services have a tailscale ingress. This is a test cluster with only one node, so everything is on the same node.

When trying to connect, I see errors like this in the tailscale pod:

2024/05/09 22:55:03 Accept: TCP{100.115.199.49:52551 > 100.77.184.113:80} 64 tcp ok
2024/05/09 22:55:03 netstack: could not connect to local server at 127.0.0.1:80: dial tcp 127.0.0.1:80: connect: connection refused

Restarting the tailscale ingress pod seems to fix the issue.

I’m not a kubernetes expert, but it seems supicious that tailscale is trying to connect to the service on 127.0.0.1:80 rather than using the cluster IP.

# kubectl get svc -A
NAMESPACE         NAME                               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                  AGE
default           kubernetes                         ClusterIP   10.233.0.1     <none>        443/TCP                  30m
docker-registry   docker-registry                    ClusterIP   10.233.22.50   <none>        80/TCP                 17m
headlamp          headlamp                           ClusterIP   10.233.49.34   <none>        80/TCP                   27m
kube-system       coredns                            ClusterIP   10.233.0.3     <none>        53/UDP,53/TCP,9153/TCP   29m
tailscale         ts-docker-registry-ingress-4w69x   ClusterIP   None           <none>        <none>                   16m
tailscale         ts-headlamp-ingress-6mzrv          ClusterIP   None           <none>        <none>                   17m
# kubectl get ingress -A
NAMESPACE         NAME                      CLASS       HOSTS   ADDRESS                                      PORTS     AGE
docker-registry   docker-registry-ingress   tailscale   *       berkeley-staging-docker.taila1eba.ts.net     80, 443   36m
headlamp          headlamp-ingress          tailscale   *       berkeley-staging-headlamp.taila1eba.ts.net   80, 443   36m

I am able to connect to both services simultaneously using kubectl port-forward, so I’m pretty sure this is not an inherit limitation of my kubernetes set-up.

Steps to reproduce

No response

Are there any recent changes that introduced the issue?

No response

OS

Linux

OS version

kubernetes

Tailscale version

1.62.1

Other software

calico CNI

Bug report

BUG-893cd6c8f00a44fda54bd05672e58e601316cc21d2892024305ff3afa3f4c675-20240509231821Z-5c7d259ff6d31521

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 3
  • Comments: 34 (16 by maintainers)

Commits related to this issue

Most upvoted comments

/ # tailscale serve status No serve config

Thank you @DevPre24 that would explain the erorrs. It looks like the serve config (configuration for the proxy how to route traffic to the backend) was never loaded, so we fall back routing to local server where there is nothing listening.

notice that TS_CERT_DOMAIN is undefined

This is expected, it will be undefined in the file, it’s only interpolated at config set up time.

I can’t tell yet why the config is not loaded.

Would you be able to:

  • check if the machine has a cert domain available, you can do so via something like tailscale debug netmap | grep -A 2 CertDomains on the ingress proxy Pod, it should show the full MagicDNS name of the proxy

  • check if it works if you manually load the serve config. You can do that from the proxy Pod:

$ apk add curl
$ curl --unix-socket /var/run/tailscale/tailscaled.sock "http://local-tailscaled.sock/localapi/v0/serve-config" -X POST -d '{"TCP":{"443":{"HTTPS":true}},"Web":{"${TS_CERT_DOMAIN}:443":{"Handlers":{"/":{"Proxy":"https+insecure://10.43.95.1:443/"}}}}}', just swap the `${TS_CERT_DOMAIN}` for the full MagicDNS name of the proxy

Curious what is tailscale serve status after running that last command as well as whether that actually makes the routing to backend work?

Could it be due to the different (in this case userspace level) implementation? I have no clue of the tailscale internals, this is a late night hypothesis with no backing whatsover

It’s definitely an issue specific to Ingress. The tun mode proxies that we spin up for Services don’t use netstack, so we don’t hit that path there.

I’ve also tried creating a LoadBalancer service with the LBclass of tailscale or annotating the existing service.

@s04 thank you for the logs- for Ingress, this seems to be the same issue where the proxy backend config is not loaded, you could verify if that is the case by running tailscale serve status from the proxy Pod. If LoadBalancer Service is also not working for you that would be a separate issue and worth opening a different issue for. I also see that you are on OpenStack, not sure if that has some limitations - the proxies we spin up for LoadBalancer Services need more permissions than the Ingress ones.

Have this problem on a K3s cluster as well. Tried all variants in the docs (service annotations, Ingress…) but all attempts ends up with this log netstack: could not connect to local server at 127.0.0.1:80: dial tcp 127.0.0.1:80: connect: connection refused in the operator pod. I have never gotten it to work.

I believe this issue should have been fixed in 1.78.

It seems like there were two reasons for this not working:

With that I am going to close this issue. However, if there are more issues that folks observe post-1.78, keen to hear about that

@irbekrm D’oH, I’m an idiot. This works in this configuration. So assuming you have your admissioncontrol setup properly, both Ingress and Service endpoints via tailscale work on Talos. Seeing what can be done from your end is already being handled in https://github.com/tailscale/tailscale/issues/7359

@nralbers thank you once again for the detailed logs.

Fired off the following curl commands while also tailing the proxy logs.

For a Service unlike the Ingress we don’t expose port 443 to tailnet, we just proxy traffic received on any port to the same port on the cluster Service. I think in your case here curl -vv flux-system-capacitor-test.tailbc32d6.ts.net:9000 should have worked.

Basically Ingress is a mechanism for exposing a bunch of Services over TLS to tailnet on port 443 (L7) whereas Service just proxies any traffic received on any port to the same port on Service (L3/L4)