kubernetes: Cannot connect to service of it's own from inside pod on kubernetes 1.6

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): kubernetes service discovery

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version): 1.6.0

Environment:

  • Cloud provider or hardware configuration: Bare metal
  • OS (e.g. from /etc/os-release): CentOS 7.3
  • Kernel (e.g. uname -a): Linux c4n1 3.10.0-514.10.2.el7.x86_64 #1 SMP Fri Mar 3 00:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools: kubeadm
  • Others:

What happened: I created service and deployment for my web service. Now from inside the pod I’m trying to connect to it’s own service name and port. It gets times out after few minutes. This works perfectly fine on kubenetes v1.5.x but not v1.6. I’m using weave network plugin. Also, I’m able to connect with other services perfectly fine on v1.6.

What you expected to happen: I should be able to connect to the service (of it’s own) from inside pod.

How to reproduce it (as minimally and precisely as possible): Create a small web app (Deployment), create a service for it (NodePort). From the same pod do curl -c http://<service-name>:<port>. It get’s timed out.

Anything else we need to know: cat /etc/resolv.conf

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

cat /etc/hosts

# Kubernetes-managed hosts file.
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
fe00::0	ip6-mcastprefix
fe00::1	ip6-allnodes
fe00::2	ip6-allrouters
10.32.0.7	falkonry-kafka-56017906-9qlg3

Connecting to kafka service from other container

    root@falkonry-redis-0:/data# curl -v http://falkonry-kafka:9092          
    * About to connect() to falkonry-kafka port 9092 (#0)
    *   Trying 10.99.232.10...
    * connected
    * Connected to falkonry-kafka (10.99.232.10) port 9092 (#0)
    > GET / HTTP/1.1
    > User-Agent: curl/7.26.0
    > Host: falkonry-kafka:9092
    > Accept: */*
    > 
    * additional stuff not fine transfer.c:1037: 0 0
    * Recv failure: Connection reset by peer
    * Closing connection #0
    curl: (56) Recv failure: Connection reset by peer

Connecting to kafka service from inside kafka container (Request never finishes)

    root@falkonry-kafka-56017906-9qlg3:/# curl -v http://falkonry-kafka:9092
    * Rebuilt URL to: http://falkonry-kafka:9092/
    * Hostname was NOT found in DNS cache
    *   Trying 10.99.232.10...
    ^C

Attaching cluster dump clusterdump.txt

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 15 (10 by maintainers)

Most upvoted comments

Just to be explicit: 1.6 has two implementations (CRI and non-CRI) and it’s the CRI one that doesn’t set hairpin when using CNI. CRI is the default but you can select the other with --enable-cri=false

1.7 will remove the non-CRI choice.

For Weave Net specifically we will add the hairpin setting at https://github.com/weaveworks/weave/pull/2971

Typically newer issues are closed in favor of older ones 😃