ingress-nginx: Sticky Sessions / Affinity Cookie not working as expected

What happened:

I’ve configured sticky sessions as per the documentation but my annotations are ignored and a client’s subsequent http requests were randomly sent to all my pod/deployment’s containers instead of the first one the client hit.

Scroll to the bottom of this issue to see nginx ingress controller logs that show the requests being routed randomly to the 5 pods.

What you expected to happen:

  1. An affinity cookie be returned in responses, which I assume would allow…
  2. Requests from a client be served by the same container that initially served them.
    The expected behaviour is that only 1 pod’s ip address would appear indicating that session affinity is active. I may be wrong in this expectation. If so, how do I achieve pod - client stickiness?

the ingress controller ignores affinity cookie annotations, fails to attach a cookie (route) and without the cookie nginx ingress is unable to keep sending client A’s request to the container A that initially served them.

NGINX Ingress controller version :

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.2.0
  Build:         a2514768cd282c41f39ab06bda17efefc4bd233a
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.10

-------------------------------------------------------------------------------

Kubernetes version :

Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:46:05Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.8-gke.201", GitCommit:"2dca91e5224568a093c27d3589aa0a96fd3ddc9a", GitTreeState:"clean", BuildDate:"2022-05-11T18:39:02Z", GoVersion:"go1.16.14b7", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.24) and server (1.22) exceeds the supported minor version skew of +/-1

Environment:

  • Cloud provider or hardware configuration: GKE

  • OS (e.g. from /etc/os-release): Container-Optimized OS from Google

  • Basic cluster related info:

    • kubectl version (see kubernetes version above)
    • kubectl get nodes -o wide
gke-node-5728b-vcz2   Ready    <none>   5d23h   v1.22.8-gke.201   10.10.0.23    34.228.120.06     Container-Optimized OS from Google   5.10.90+         containerd://1.5.4
gke-node-8d933-853w   Ready    <none>   5d23h   v1.22.8-gke.201   10.10.0.24    34.120.222.214     Container-Optimized OS from Google   5.10.90+         containerd://1.5.4
gke-node-s9933-psb6   Ready    <none>   23m     v1.22.8-gke.201   10.10.0.26    35.222.06.201   Container-Optimized OS from Google   5.10.90+         containerd://1.5.4
gke-node-gr45b-75d4   Ready    <none>   5d23h   v1.22.8-gke.201   10.10.0.25    34.208.06.292     Container-Optimized OS from Google   5.10.90+         containerd://1.5.4

resource "helm_release" "ingress_nginx" {
  chart      = "ingress-nginx"
  name       = "ingress-nginx"
  namespace  = var.ingress_namespace
  repository = "https://kubernetes.github.io/ingress-nginx"
  version    = "4.1.0"
  depends_on = [
    kubernetes_namespace.ingress_namespace
  ]

  values = [
    templatefile("${path.module}/kubernetes/helm/templates/values-ingress-nginx.yaml", {
      controller_publishService_enabled = true
      controller_service_type           = "LoadBalancer"
    })
  ]
}
  • helm ls -A | grep -i ingress-nginx
ingress-nginx   ingress-nginx   1               2022-06-16 09:09:12.29149 +0300 EAT     deployed        ingress-nginx-4.1.0     1.2.0
  • helm -n ingress-nginx get values ingress-nginx
USER-SUPPLIED VALUES:
controller.publishService.enabled: true
controller.service.type: LoadBalancer
  • Current State of the controller:
    • kubectl describe ingress-nginx
Name:             http-svc
Labels:           app=httpsvc
Namespace:        default
Address:          <<ip address>>
Ingress Class:    nginx
Default backend:  httpsvc-service:5000 (10.4.3.13:5000,10.4.4.11:5000,10.4.4.12:5000 + 2 more...)
TLS:
  SNI routes service.io
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           *     httpsvc-service:5000 (10.4.3.13:5000,10.4.4.11:5000,10.4.4.12:5000 + 2 more...)
Annotations:  cert-manager.io/cluster-issuer: letsencrypt-staging
              nginx.ingress.kubernetes.io/affinity: cookie
              nginx.ingress.kubernetes.io/session-cookie-expires: 17250000
              nginx.ingress.kubernetes.io/session-cookie-max-age: 17250000
              nginx.ingress.kubernetes.io/session-cookie-name: route
Events:
  Type     Reason     Age                From                       Message
  ----     ------     ----               ----                       -------
  Normal   Sync       49m (x4 over 24h)  nginx-ingress-controller   Scheduled for sync
  Warning  BadConfig  49m (x4 over 24h)  cert-manager-ingress-shim  Skipped a TLS block: spec.tls[0].secretName: Required value
  • Current state of ingress object:
    • kubectl get svc,ing -A -o wide
NAMESPACE       NAME                                         TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)                                     AGE     SELECTOR
cert-manager    service/cert-manager                         ClusterIP      10.7.246.234   <none>           9402/TCP                                    5d23h   app.kubernetes.io/component=controller,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cert-manager
cert-manager    service/cert-manager-webhook                 ClusterIP      10.7.251.161   <none>           443/TCP                                     5d23h   app.kubernetes.io/component=webhook,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=webhook
default         service/cm-acme-http-solver-lmg8l            NodePort       10.7.252.251   <none>           500089:31608/TCP                              40h     acme.cert-manager.io/http-domain=923862623,acme.cert-manager.io/http-token=1195511342,acme.cert-manager.io/http01-solver=true
default         service/kubernetes                           ClusterIP      10.7.240.1     <none>           443/TCP                                     5d23h   <none>
default         service/httpsvc-service                     ClusterIP      10.7.248.184   <none>           5000/TCP,2525/TCP                           40h     app=httpsvc
ingress-nginx   service/ingress-nginx-controller             LoadBalancer   10.7.253.234   <<ip address>>   5000:3150000/TCP,443:31461/TCP,2525:31509/TCP   5d23h   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
ingress-nginx   service/ingress-nginx-controller-admission   ClusterIP      10.7.243.216   <none>           443/TCP                                     5d23h   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
kube-system     service/default-http-backend                 NodePort       10.7.251.181   <none>           5000:32257/TCP                                5d23h   k8s-app=glbc
kube-system     service/kube-dns                             ClusterIP      10.7.240.10    <none>           53/UDP,53/TCP                               5d23h   k8s-app=kube-dns
kube-system     service/metrics-server                       ClusterIP      10.7.252.39    <none>           443/TCP                                     5d23h   k8s-app=metrics-server

NAMESPACE   NAME                                                  CLASS    HOSTS              ADDRESS          PORTS     AGE
default     ingress.networking.k8s.io/cm-acme-http-solver-hj9mp   <none>   www.service.io   <<ip address>>   5000        40h
default     ingress.networking.k8s.io/httpsvc-ingress            nginx    service.io       <<ip address>>   5000, 443   40h
  • Current state of deployment’s pods:
    • kubectl get pods -o wide
NAME                                   READY   STATUS    RESTARTS   AGE   IP          NODE                                                  NOMINATED NODE   READINESS GATES
httpsvc-deployment-5689c846f4-254cr   1/1     Running   0          15h   10.4.5.20   gke-node-gr45b-75d4   <none>           <none>
httpsvc-deployment-5689c846f4-7r8hd   1/1     Running   0          62m   10.4.4.30   gke-node-8d933-853w   <none>           <none>
httpsvc-deployment-5689c846f4-gtxpk   1/1     Running   0          30m   10.4.6.2    gke-node-s9933-psb6   <none>           <none>
httpsvc-deployment-5689c846f4-h6mjq   1/1     Running   0          62m   10.4.4.29   gke-node-8d933-853w   <none>           <none>
httpsvc-deployment-5689c846f4-phxsm   1/1     Running   0          30m   10.4.5.21   gke-node-gr45b-75d4   <none>           <none>

How to reproduce this issue:

Tested on

Install the ingress controller (v1.2.1) via helm (chart 4.1.4)

  • helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
  • helm install ingress-nginx ingress-nginx/ingress-nginx --version 4.1.4

Install an application that will act as default backend (is just an echo app)

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/http-svc.yaml
PS: httpsvc is actually and elixir app so http-svc is just a placeholder for this issue.

Create an ingress

echo "
  apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
  namespace: default
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-staging"
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/affinity-mode: "persistent"
    nginx.ingress.kubernetes.io/session-cookie-name: "route"
spec:
  ingressClassName: nginx
  defaultBackend:
    service:
      name: http-svc
      port:
        number: 5000
  tls:
    - hosts:
        - service.io
      secretName: elixir-phoenix-tls-secret
  rules:
    - host: service.io
      http:
        paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: http-svc
                port:
                  number: 5000
          - path: /*
            pathType: ImplementationSpecific
            backend:
              service:
                name: http-svc
                port:
                  number: 5000
" | kubectl apply -f -

make a request

(sample curl request)

✗ curl -kIv --cookie cookie.txt --cookie-jar cookie.txt https://service.io/admin
*   Trying <<ip address>>:443...
* Connected to service.io (<<ip address>>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  start date: Jun 16 06:10:30 2022 GMT
*  expire date: Jun 16 06:10:30 2023 GMT
*  issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x14b00fc00)
> HEAD /admin HTTP/2
> Host: service.io
> user-agent: curl/7.79.1
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 302 
HTTP/2 302 
< date: Wed, 22 Jun 2022 05:57:35 GMT
date: Wed, 22 Jun 2022 05:57:35 GMT
< content-type: text/html; charset=utf-8
content-type: text/html; charset=utf-8
< content-length: 79
content-length: 79
< cache-control: max-age=0, private, must-revalidate
cache-control: max-age=0, private, must-revalidate
< cross-origin-window-policy: deny
cross-origin-window-policy: deny
< location: /sessions/new
location: /sessions/new
< x-content-type-options: nosniff
x-content-type-options: nosniff
< x-download-options: noopen
x-download-options: noopen
< x-frame-options: SAMEORIGIN
x-frame-options: SAMEORIGIN
< x-permitted-cross-domain-policies: none
x-permitted-cross-domain-policies: none
< x-request-id: 5dbbe6bd9ecebc02557ccf1d4f65000517
x-request-id: 5dbbe6bd9ecebc02557ccf1d4f65000517
< x-xss-protection: 1; mode=block
x-xss-protection: 1; mode=block
* Added cookie _httpsvc_key="SFMyNTY.g3QAAAACbQAAAA1waG9lbml4X2ZsYXNodAAAAAFtAAAABWVycm9ybQAAACRQbGVhc2Ugc2lnbiBpbiBpbiBvcmRlciB0byBjb250aW51ZS5tAAAADnVzZXJfcmV0dXJuX3RvbQAAABpodHRwczovL211Y2hlcnUudGVjaC9hZG1pbg.HLg8kJzHBxg1q8HT2CIjnPT8xKQREJOn7WQm71r-_1I" for domain service.io, path /, expire 0
< set-cookie: _httpsvc_key=SFMyNTY.g3QAAAACbQAAAA1waG9lbml4X2ZsYXNodAAAAAFtAAAABWVycm9ybQAAACRQbGVhc2Ugc2lnbiBpbiBpbiBvcmRlciB0byBjb250aW51ZS5tAAAADnVzZXJfcmV0dXJuX3RvbQAAABpodHRwczovL211Y2hlcnUudGVjaC9hZG1pbg.HLg8kJzHBxg1q8HT2CIjnPT8xKQREJOn7WQm71r-_1I; path=/; HttpOnly
set-cookie: _httpsvc_key=SFMyNTY.g3QAAAACbQAAAA1waG9lbml4X2ZsYXNodAAAAAFtAAAABWVycm9ybQAAACRQbGVhc2Ugc2lnbiBpbiBpbiBvcmRlciB0byBjb250aW51ZS5tAAAADnVzZXJfcmV0dXJuX3RvbQAAABpodHRwczovL211Y2hlcnUudGVjaC9hZG1pbg.HLg8kJzHBxg1q8HT2CIjnPT8xKQREJOn7WQm71r-_1I; path=/; HttpOnly
< strict-transport-security: max-age=1572450000; includeSubDomains
strict-transport-security: max-age=1572450000; includeSubDomains

< 
* Connection #0 to host service.io left intact

contents of cookie.txt

✗ cat cookie.txt 
# Netscape HTTP Cookie File
# https://curl.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.

#HttpOnly_service.io  FALSE   /       FALSE   0       _httpsvc_key   SFMyNTY.g3QAAAACbQAAAA1waG9lbml4X2ZsYXNodAAAAAFtAAAABWVycm9ybQAAACRQbGVhc2Ugc2lnbiBpbiBpbiBvcmRlciB0byBjb250aW51ZS5tAAAADnVzZXJfcmV0dXJuX3RvbQAAABpodHRwczovL211Y2hlcnUudGVjaC9hZG1pbg.HLg8kJzHBxg1q8HT2CIjnPT8xKQREJOn7WQm71r-_1I

Logs (With 5-pod deployment of httpsvc)

2022/06/22 05:45:26 [error] 1203#1203: *2176182 connect() failed (111: Connection refused) while connecting to upstream, client: 10.10.0.24, server: service.io, request: "POST /sessions HTTP/2.0", upstream: "http://10.4.3.16:5000/sessions", host: "service.io", referrer: "https://service.io/sessions/new"
10.10.0.24 - - [22/Jun/2022:05:45:27 +0000] "POST /sessions HTTP/2.0" 302 92 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 1051 1.018 [default-httpsvc-service-5000] [] 10.4.3.16:5000, 10.4.6.2:5000 0, 92 0.001, 1.017 502, 302 7490e4174ee479341b1b937758189d66
10.10.0.24 - - [22/Jun/2022:05:45:27 +0000] "GET /admin HTTP/2.0" 302 79 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 316 0.002 [default-httpsvc-service-5000] [] 10.4.5.20:5000 79 0.003 302 8a9ab90540b7a0ea0abf7e13125000276a
10.10.0.24 - - [22/Jun/2022:05:45:27 +0000] "GET /sessions/new HTTP/2.0" 200 6451 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 371 0.004 [default-httpsvc-service-5000] [] 10.4.4.29:5000 6451 0.004 200 45f4e01b05018e1bdeb892504d4a3dff
10.10.0.24 - - [22/Jun/2022:05:45:34 +0000] "POST /sessions HTTP/2.0" 302 92 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 559 0.968 [default-httpsvc-service-5000] [] 10.4.5.21:5000 92 0.968 302 852bf0bd4b35c3db0ddfa8fbcf3bfe6c
10.10.0.24 - - [22/Jun/2022:05:45:34 +0000] "GET /admin HTTP/2.0" 302 79 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 315 0.003 [default-httpsvc-service-5000] [] 10.4.4.30:5000 79 0.004 302 0e3b1bd2f4bf4797afd020f892d7d7dd
2022/06/22 05:45:35 [error] 1203#1203: *2176182 connect() failed (111: Connection refused) while connecting to upstream, client: 10.10.0.24, server: service.io, request: "GET /sessions/new HTTP/2.0", upstream: "http://10.4.3.16:5000/sessions/new", host: "service.io", referrer: "https://service.io/sessions/new"
10.10.0.24 - - [22/Jun/2022:05:45:35 +0000] "GET /sessions/new HTTP/2.0" 200 6451 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 372 0.006 [default-httpsvc-service-5000] [] 10.4.3.16:5000, 10.4.6.2:5000 0, 6451 0.002, 0.004 502, 200 e18320fe430b2d887cf9db0d3efb7003
10.10.0.24 - - [22/Jun/2022:05:45:39 +0000] "POST /sessions HTTP/2.0" 302 92 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 558 0.946 [default-httpsvc-service-5000] [] 10.4.5.20:5000 92 0.945 302 4730cf92e3886261e1edcb0fafda645f
10.10.0.24 - - [22/Jun/2022:05:45:39 +0000] "GET /admin HTTP/2.0" 302 79 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 761 0.003 [default-httpsvc-service-5000] [] 10.4.4.29:5000 79 0.004 302 c1eb7e30cfbef2116afcb142a92512a3
10.10.0.24 - - [22/Jun/2022:05:45:39 +0000] "GET /sessions/new HTTP/2.0" 200 6451 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 371 0.002 [default-httpsvc-service-5000] [] 10.4.5.21:5000 6451 0.003 200 1037dee914ed7467940b7c850bce5ebc
10.10.0.24 - - [22/Jun/2022:05:45:44 +0000] "POST /sessions HTTP/2.0" 302 92 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 609 0.942 [default-httpsvc-service-5000] [] 10.4.4.30:5000 92 0.942 302 8ca9779e3d15000012a287f219f6ab5d52
2022/06/22 05:45:44 [error] 1203#1203: *2176182 connect() failed (111: Connection refused) while connecting to upstream, client: 10.10.0.24, server: service.io, request: "GET /admin HTTP/2.0", upstream: "http://10.4.3.16:5000/admin", host: "service.io", referrer: "https://service.io/sessions/new"
10.10.0.24 - - [22/Jun/2022:05:45:44 +0000] "GET /admin HTTP/2.0" 302 79 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 315 0.004 [default-httpsvc-service-5000] [] 10.4.3.16:5000, 10.4.6.2:5000 0, 79 0.002, 0.002 502, 302 4b93216bf9e8c97daca2bd566d075a7b
10.10.0.24 - - [22/Jun/2022:05:45:44 +0000] "GET /sessions/new HTTP/2.0" 200 6451 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 371 0.003 [default-httpsvc-service-5000] [] 10.4.5.20:5000 6451 0.002 200 6b8dc51073490d82671bb2f2f2f6a864
10.10.0.24 - - [22/Jun/2022:05:45:49 +0000] "POST /sessions HTTP/2.0" 302 92 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 559 1.008 [default-httpsvc-service-5000] [] 10.4.4.29:5000 92 1.008 302 c19a2ce5645e650008de6fc31e7102cf5d
10.10.0.24 - - [22/Jun/2022:05:45:50 +0000] "GET /admin HTTP/2.0" 302 79 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 315 0.003 [default-httpsvc-service-5000] [] 10.4.5.21:5000 79 0.003 302 2cbbef83fbd17d54e8d97845501929f9
10.10.0.24 - - [22/Jun/2022:05:45:51 +0000] "GET /sessions/new HTTP/2.0" 200 6451 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 817 0.003 [default-httpsvc-service-5000] [] 10.4.4.30:5000 6451 0.004 200 d3b737936d4f1fc562d403af5fa286df
2022/06/22 05:45:55 [error] 1203#1203: *2176182 connect() failed (111: Connection refused) while connecting to upstream, client: 10.10.0.24, server: service.io, request: "POST /sessions HTTP/2.0", upstream: "http://10.4.3.16:5000/sessions", host: "service.io", referrer: "https://service.io/sessions/new"
10.10.0.24 - - [22/Jun/2022:05:45:56 +0000] "POST /sessions HTTP/2.0" 302 92 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 555 0.977 [default-httpsvc-service-5000] [] 10.4.3.16:5000, 10.4.6.2:5000 0, 92 0.003, 0.973 502, 302 93ad38a7eb0ae1e82ae2ea185305ee26
10.10.0.24 - - [22/Jun/2022:05:45:56 +0000] "GET /admin HTTP/2.0" 302 79 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 315 0.002 [default-httpsvc-service-5000] [] 10.4.5.20:5000 79 0.001 302 3ba040a85000a0140a670b7c4dc8797256
10.10.0.24 - - [22/Jun/2022:05:45:56 +0000] "GET /sessions/new HTTP/2.0" 200 6451 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 371 0.002 [default-httpsvc-service-5000] [] 10.4.4.29:5000 6451 0.002 200 8d940e5ae5065444c7202f13493ca056
10.10.0.24 - - [22/Jun/2022:05:46:00 +0000] "POST /sessions HTTP/2.0" 302 92 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 606 1.000 [default-httpsvc-service-5000] [] 10.4.5.21:5000 92 1.000 302 69bc14b44d8ca310d84b493e3d20c8af
10.10.0.24 - - [22/Jun/2022:05:46:01 +0000] "GET /admin HTTP/2.0" 302 79 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 316 0.003 [default-httpsvc-service-5000] [] 10.4.4.30:5000 79 0.003 302 fe4dac009f6f14f453cf9cbe34702e8e
2022/06/22 05:46:01 [error] 1203#1203: *2176182 connect() failed (111: Connection refused) while connecting to upstream, client: 10.10.0.24, server: service.io, request: "GET /sessions/new HTTP/2.0", upstream: "http://10.4.3.16:5000/sessions/new", host: "service.io", referrer: "https://service.io/sessions/new"
10.10.0.24 - - [22/Jun/2022:05:46:01 +0000] "GET /sessions/new HTTP/2.0" 200 6451 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 371 0.004 [default-httpsvc-service-5000] [] 10.4.3.16:5000, 10.4.6.2:5000 0, 6451 0.002, 0.002 502, 200 3ace9efeeabb77adf1f540e503ecd442
10.10.0.24 - - [22/Jun/2022:05:46:06 +0000] "POST /sessions HTTP/2.0" 302 92 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 1000 0.963 [default-httpsvc-service-5000] [] 10.4.5.20:5000 92 0.963 302 f95000cd18c7376c6544621bf5363c2d21
10.10.0.24 - - [22/Jun/2022:05:46:06 +0000] "GET /admin HTTP/2.0" 302 79 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 315 0.003 [default-httpsvc-service-5000] [] 10.4.4.29:5000 79 0.004 302 c45bcc812168c6efeaf3c35dd4febf39
10.10.0.24 - - [22/Jun/2022:05:46:07 +0000] "GET /sessions/new HTTP/2.0" 200 6451 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 370 0.004 [default-httpsvc-service-5000] [] 10.4.5.21:5000 6451 0.003 200 1dab24e84dbde62ed3a3f7f1a3f74ebf
10.10.0.24 - - [22/Jun/2022:05:46:11 +0000] "POST /sessions HTTP/2.0" 302 92 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 555 0.949 [default-httpsvc-service-5000] [] 10.4.4.30:5000 92 0.949 302 1b156f06ab1e3cf05a994bb687fea84f
2022/06/22 05:46:11 [error] 1203#1203: *2176182 connect() failed (111: Connection refused) while connecting to upstream, client: 10.10.0.24, server: service.io, request: "GET /admin HTTP/2.0", upstream: "http://10.4.3.16:5000/admin", host: "service.io", referrer: "https://service.io/sessions/new"
10.10.0.24 - - [22/Jun/2022:05:46:11 +0000] "GET /admin HTTP/2.0" 302 79 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 315 0.006 [default-httpsvc-service-5000] [] 10.4.3.16:5000, 10.4.6.2:5000 0, 79 0.001, 0.004 502, 302 a202a2d0a11435fccabf5732675271ae
10.10.0.24 - - [22/Jun/2022:05:46:12 +0000] "GET /sessions/new HTTP/2.0" 200 6451 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 370 0.003 [default-httpsvc-service-5000] [] 10.4.5.20:5000 6451 0.002 200 50005e7b94c899d34a9c9447706aa8b684
10.10.0.24 - - [22/Jun/2022:05:46:12 +0000] "GET /favicon/apple-touch-icon-d55623182cfcc7a95f251e71ed2de47c.png?vsn=d HTTP/2.0" 499 0 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 328 0.000 [default-httpsvc-service-5000] [] 10.4.4.29:5000 0 0.000 - 9a60a60451343f5cda72b7dea0cbbc55
10.10.0.24 - - [22/Jun/2022:05:46:17 +0000] "POST /sessions HTTP/2.0" 302 92 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 50009 0.964 [default-httpsvc-service-5000] [] 10.4.5.21:5000 92 0.965 302 61ec9e3e6c48bda4e9f7703b2e62725a
10.10.0.24 - - [22/Jun/2022:05:46:17 +0000] "GET /admin HTTP/2.0" 302 79 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 316 0.003 [default-httpsvc-service-5000] [] 10.4.4.30:5000 79 0.003 302 b92b688d67f0468fd9581f88e13870c7
2022/06/22 05:46:17 [error] 1203#1203: *2176182 connect() failed (111: Connection refused) while connecting to upstream, client: 10.10.0.24, server: service.io, request: "GET /sessions/new HTTP/2.0", upstream: "http://10.4.3.16:5000/sessions/new", host: "service.io", referrer: "https://service.io/sessions/new"
10.10.0.24 - - [22/Jun/2022:05:46:17 +0000] "GET /sessions/new HTTP/2.0" 200 6451 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 371 0.006 [default-httpsvc-service-5000] [] 10.4.3.16:5000, 10.4.6.2:5000 0, 6451 0.001, 0.004 502, 200 1272477154fe5ec6bfe1181815601566
10.10.0.24 - - [22/Jun/2022:05:46:21 +0000] "POST /sessions HTTP/2.0" 302 92 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 554 0.998 [default-httpsvc-service-5000] [] 10.4.5.20:5000 92 0.997 302 79679875fb10f3b3da1d26e54ccd4d7b
10.10.0.24 - - [22/Jun/2022:05:46:21 +0000] "GET /admin HTTP/2.0" 302 79 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 316 0.004 [default-httpsvc-service-5000] [] 10.4.4.29:5000 79 0.004 302 abf062d66b8b2c7b6c2500f82b092627
10.10.0.24 - - [22/Jun/2022:05:46:22 +0000] "GET /sessions/new HTTP/2.0" 200 6451 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 370 0.003 [default-httpsvc-service-5000] [] 10.4.5.21:5000 6451 0.003 200 d0d539aff365364ef7bcfa14a0fd5a85
10.10.0.24 - - [22/Jun/2022:05:46:26 +0000] "POST /sessions HTTP/2.0" 302 92 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 559 0.935 [default-httpsvc-service-5000] [] 10.4.4.30:5000 92 0.935 302 1dd717f6d20c2607c0c46b94534b016a
2022/06/22 05:46:27 [error] 1203#1203: *2176182 connect() failed (111: Connection refused) while connecting to upstream, client: 10.10.0.24, server: service.io, request: "GET /admin HTTP/2.0", upstream: "http://10.4.3.16:5000/admin", host: "service.io", referrer: "https://service.io/sessions/new"
10.10.0.24 - - [22/Jun/2022:05:46:27 +0000] "GET /admin HTTP/2.0" 302 79 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 761 0.004 [default-httpsvc-service-5000] [] 10.4.3.16:5000, 10.4.6.2:5000 0, 79 0.002, 0.002 502, 302 666135aec577a9aab75603f11c17beb3
10.10.0.24 - - [22/Jun/2022:05:46:27 +0000] "GET /sessions/new HTTP/2.0" 200 6451 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 371 0.003 [default-httpsvc-service-5000] [] 10.4.5.20:5000 6451 0.002 200 b3582526e6fce2315188e4db4c3bcbeb
10.10.0.24 - - [22/Jun/2022:05:46:31 +0000] "POST /sessions HTTP/2.0" 302 92 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 608 0.988 [default-httpsvc-service-5000] [] 10.4.4.29:5000 92 0.988 302 ae48fc0830558250004c0ce36d9b20cbb1
10.10.0.24 - - [22/Jun/2022:05:46:31 +0000] "GET /admin HTTP/2.0" 302 79 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 317 0.003 [default-httpsvc-service-5000] [] 10.4.5.21:5000 79 0.002 302 2aaadd4ac6a7e59190d27cefaf1cffc6
10.10.0.24 - - [22/Jun/2022:05:46:31 +0000] "GET /sessions/new HTTP/2.0" 200 6451 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 372 0.003 [default-httpsvc-service-5000] [] 10.4.4.30:5000 6451 0.003 200 d2a67493e1230b9fdf99686f458dae89

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 17 (8 by maintainers)

Most upvoted comments

For anyone who comes looking, this was an issue with my implementation of Kustomize. It’s not a bug on ingress-nginx.

can you please share with us what was the issue and how you fixed it because we’re getting the same thing

+1
sichiba on May 24, 2023

For anyone who comes looking, this was an issue with my implementation of Kustomize. It’s not a bug on ingress-nginx.

+1
cmush on Sep 21, 2022
Read more comments on GitHub
← ingress-nginx: SSL redirect broken when default-ssl-certificate is used
ingress-nginx: Sticky sessions not working in version 1.0.2 k8s 1.21.2 →