ingress-nginx: nginx-ingress-controller crashes with SIGSEGV

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.):

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

NGINX Ingress controller version: NGINX Ingress controller Release: 0.20.0 Build: git-e8d8103 Repository: https://github.com/kubernetes/ingress-nginx.git Image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller-arm:0.20.0

Kubernetes version (use kubectl version): Client Version: version.Info{Major:“1”, Minor:“10”, GitVersion:“v1.10.3”, GitCommit:“2bba0127d85d5a46ab4b778548be28623b32d0b0”, GitTreeState:“clean”, BuildDate:“2018-05-21T09:17:39Z”, GoVersion:“go1.9.3”, Compiler:“gc”, Platform:“darwin/amd64”}

Server Version: version.Info{Major:“1”, Minor:“11”, GitVersion:“v1.11.4”, GitCommit:“bf9a868e8ea3d3a8fa53cbb22f566771b3f8068b”, GitTreeState:“clean”, BuildDate:“2018-10-25T19:06:30Z”, GoVersion:“go1.10.3”, Compiler:“gc”, Platform:“linux/arm”}

Environment:

• Cloud provider or hardware configuration: Raspberry Pi 3 Model B+
• OS (e.g. from /etc/os-release): Raspbian GNU/Linux 9 (stretch
• Kernel (e.g. uname -a): Linux 4.14.79-v7+ #1159 SMP Sun Nov 4 17:50:20 GMT 2018 armv7l GNU/Linux
• Install tools:
  Client: &version.Version{SemVer:“v2.11.0”, GitCommit:“2e55dbe1fdb5fdb96b75ff144a339489417b146b”, GitTreeState:“clean”} Server: &version.Version{SemVer:“v2.11.0+unreleased”, GitCommit:“158d6dbb746f525bad9a0aacb698af7d370ac3f5”, GitTreeState:“dirty”}
• Others:
  • Docker version 18.04.0-ce, build 3d479c0
  • metallb/controller:v0.7.3-arm

What happened: The nginx-ingress-controller pod crashes with segfault after successful request to service.

1. kubectl get pod nginx-ingress-controller-59cc49954d-b5kch

   nginx-ingress-controller-59cc49954d-b5kch   1/1       Running   26        3d
   

2. Request to service curl --verbose ranger/towel/helloworld --header "Host: ranger"

   *   Trying 2003:c8:9f2e:2100:15d8:1b9f:5a9c:b715...
   * TCP_NODELAY set
   * Connection failed
   * connect to 2003:c8:9f2e:2100:15d8:1b9f:5a9c:b715 port 80 failed: Connection refused
   *   Trying 192.168.178.77...
   * TCP_NODELAY set
   * Connected to ranger (192.168.178.77) port 80 (#0)
   > GET /towel/helloworld HTTP/1.1
   > Host: ranger
   > User-Agent: curl/7.54.0
   > Accept: */*
   >
   < HTTP/1.1 200
   < Server: nginx/1.15.5
   < Date: Tue, 11 Dec 2018 07:46:08 GMT
   < Content-Type: application/json;charset=UTF-8
   < Transfer-Encoding: chunked
   < Connection: keep-alive
   < Vary: Accept-Encoding
   <
   * Connection #0 to host ranger left intact
   {"id":14,"message":"Hello, world!"}%
   

3. nginx-ingress-controller crashes

kubectl get pod nginx-ingress-controller-59cc49954d-b5kc:

nginx-ingress-controller-59cc49954d-b5kch   0/1       Error     27        3d

kubectl logs nginx-ingress-controller-59cc49954d-b5kch:

I1211 07:46:20.803405       8 flags.go:180] Watching for Ingress class: nginx
nginx version: nginx/1.15.5
W1211 07:46:20.814264       8 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1211 07:46:20.817531       8 main.go:196] Creating API client for https://10.96.0.1:443
I1211 07:46:20.947391       8 main.go:240] Running in Kubernetes cluster version v1.11 (v1.11.4) - git (clean) commit bf9a868e8ea3d3a8fa53cbb22f566771b3f8068b - platform linux/arm
I1211 07:46:20.954249       8 main.go:101] Validated default/nginx-ingress-default-backend as the default backend.
I1211 07:46:23.600322       8 nginx.go:256] Starting NGINX Ingress controller
I1211 07:46:23.668833       8 event.go:221] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"default", Name:"nginx-ingress-controller", UID:"9dc046fd-fa2a-11e8-bc35-b827eb5c793c", APIVersion:"v1", Re
sourceVersion:"150947", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap default/nginx-ingress-controller
I1211 07:46:24.710633       8 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"towel-ingress", UID:"23aabc05-fc87-11e8-bc35-b827eb5c793c", APIVersion:"extensions/v1beta1",
 ResourceVersion:"573396", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/towel-ingress
I1211 07:46:24.801389       8 leaderelection.go:185] attempting to acquire leader lease  default/ingress-controller-leader-nginx...
I1211 07:46:24.801457       8 nginx.go:277] Starting NGINX process
I1211 07:46:24.813672       8 controller.go:177] Configuration changes detected, backend reload required.
I1211 07:46:24.825633       8 leaderelection.go:194] successfully acquired lease default/ingress-controller-leader-nginx
I1211 07:46:24.825894       8 status.go:197] new leader elected: nginx-ingress-controller-59cc49954d-b5kch
I1211 07:46:25.298432       8 controller.go:195] Backend successfully reloaded.
I1211 07:46:25.298820       8 controller.go:205] Initial synchronization of the NGINX configuration.
I1211 07:46:26.309819       8 controller.go:212] Dynamic reconfiguration succeeded.
W1211 07:46:50.597452       8 controller.go:824] Service "default/towel-tool" does not have any active Endpoint.
I1211 07:46:50.597887       8 controller.go:177] Configuration changes detected, backend reload required.
I1211 07:46:51.053256       8 controller.go:195] Backend successfully reloaded.
I1211 07:46:51.057831       8 controller.go:212] Dynamic reconfiguration succeeded.
W1211 07:47:18.153682       8 controller.go:824] Service "default/towel-tool" does not have any active Endpoint.
I1211 07:47:21.487755       8 controller.go:177] Configuration changes detected, backend reload required.
I1211 07:47:21.939385       8 controller.go:195] Backend successfully reloaded.
I1211 07:47:21.946202       8 controller.go:212] Dynamic reconfiguration succeeded.
192.168.178.50 - [192.168.178.50] - - [11/Dec/2018:07:55:42 +0000] "GET /towel/helloworld HTTP/1.1" 200 46 "-" "curl/7.54.0" 86 0.022 [default-towel-tool-8080] 10.244.2.27:8080 46 0.020 200 93bea19734a3da7dc93776b3c9a5d2ba
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x11a70]

goroutine 1308 [running]:
runtime/internal/atomic.goXadd64(0x38a652c, 0x2, 0x0, 0x20c49ba, 0x3f96872b)
        /usr/local/go/src/runtime/internal/atomic/atomic_arm.go:96 +0x1c
k8s.io/ingress-nginx/vendor/github.com/prometheus/client_golang/prometheus.(*histogram).Observe(0x38a64d0, 0x20c49ba, 0x3f96872b)
        /go/src/k8s.io/ingress-nginx/vendor/github.com/prometheus/client_golang/prometheus/histogram.go:272 +0x68
k8s.io/ingress-nginx/internal/ingress/metric/collectors.(*SocketCollector).handleMessage(0x38f7c00, 0x407ac00, 0x146, 0x600)
        /go/src/k8s.io/ingress-nginx/internal/ingress/metric/collectors/socket.go:269 +0xb8c
k8s.io/ingress-nginx/internal/ingress/metric/collectors.(*SocketCollector).handleMessage-fm(0x407ac00, 0x146, 0x600)
        /go/src/k8s.io/ingress-nginx/internal/ingress/metric/collectors/socket.go:317 +0x34
k8s.io/ingress-nginx/internal/ingress/metric/collectors.handleMessages(0x66c40d00, 0x3d3adb8, 0x3d3adc0)
        /go/src/k8s.io/ingress-nginx/internal/ingress/metric/collectors/socket.go:435 +0xa8
created by k8s.io/ingress-nginx/internal/ingress/metric/collectors.(*SocketCollector).Start
        /go/src/k8s.io/ingress-nginx/internal/ingress/metric/collectors/socket.go:317 +0xc0

1. Controller then restarts and runs until next request

nginx-ingress-controller-59cc49954d-b5kch   0/1       CrashLoopBackOff   27        3d
nginx-ingress-controller-59cc49954d-b5kch   0/1       Running   28        3d
nginx-ingress-controller-59cc49954d-b5kch   1/1       Running   28        3d

What you expected to happen: The controller not to crash.

How to reproduce it (as minimally and precisely as possible):

ingress.yml used:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: towel-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
  - host: ranger
    http:
      paths:
        - path: /towel
          backend:
            serviceName: towel-tool
            servicePort: 8080

Service:

apiVersion: v1
kind: Service
metadata:
  name: towel-tool
  labels:
    app: towel-tool
spec:
  ports:
    - port: 8080
      protocol: TCP
      targetPort: 8080
  selector:
    app: towel-tool

Deployment:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: towel-tool
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: towel-tool
    spec:
      containers:
      - name: towel-tool
        image: aqube/towel-tool-arm:1.1-SNAPSHOT
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          protocol: TCP

The towel-tool image runs a spring app returning a hello world message at /helloworld.

Anything else we need to know: Only happens when a host is specified in the ingress rule.