ingress-nginx: Cookie affinity does not work when using configuration-snipped annotation

NGINX Ingress controller version: 0.34.1

Kubernetes version (use kubectl version):

Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.6-eks-4e7f64", GitCommit:"4e7f642f9f4cbb3c39a4fc6ee84fe341a8ade94c", GitTreeState:"clean", BuildDate:"2020-06-11T13:55:35Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: Amazon AWS, using EKS
• OS (e.g. from /etc/os-release): Amazon Linux AMI
• Kernel (e.g. uname -a): Linux debug 4.14.186-146.268.amzn2.x86_64 #1 SMP Tue Jul 14 18:16:52 UTC 2020 x86_64 Linux
• Install tools: Helm
• Others: Linkerd 2 as service mesh.

What happened:

We use Linkerd and according to its manual we need to add the following annotation to our ingresses in order to supposedly avoid infinite loops:

nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;

One of our microservices depends on session stickyness, so we added the appropriate annotation to its ingress in order to make it happen (see bellow). And, it turns out that even though the cookie is actually being set by nginx, the requests are randomly (or as per round-robin or whatever) distributed across the instances. If I remove the the previously mentioned annotation (configuration-snipped), the affinity works seamesly.

So, it seems there is an incompatibility between the annotations.

What you expected to happen:

I would expect to be able to use both nginx.ingress.kubernetes.io/configuration-snippet and nginx.ingress.kubernetes.io/affinity and have the affinity working.

How to reproduce it:

Our Helm values

Just in case they are relevant:

controller:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: release
              operator: In
              values:
              - nginx-ingress
            - key: component
              operator: In
              values:
              - controller
          topologyKey: failure-domain.beta.kubernetes.io/zone
        weight: 100
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: release
            operator: In
            values:
            - nginx-ingress
          - key: component
            operator: In
            values:
            - controller
        topologyKey: kubernetes.io/hostname
  config:
    client-body-buffer-size: 128m
    enable-opentracing: "true"
    hsts: "false"
    jaeger-collector-host: localhost
    jaeger-sampler-host: localhost
    jaeger-sampler-type: remote
    proxy-body-size: 50m
  containerPort:
    http: 8080
    https: 8443
  deploymentAnnotations:
    sidecar.jaegertracing.io/inject: jaeger
  extraArgs:
    http-port: 8080
    https-port: 8443
  image:
    allowPrivilegeEscalation: false
  metrics:
    enabled: true
    serviceMonitor:
      enabled: true
  podAnnotations:
    config.linkerd.io/skip-inbound-ports: 8080,8442,8443
    linkerd.io/inject: enabled
  publishService:
    enabled: true
  replicaCount: 3
  resources:
    limits:
      cpu: 2
      memory: 1Gi
    requests:
      cpu: 500m
      memory: 1Gi
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: nlb
    externalTrafficPolicy: Local
  useComponentLabel: true
defaultBackend:
  podAnnotations:
    linkerd.io/inject: enabled
  resources:
    limits:
      cpu: 200m
      memory: 20Mi
    requests:
      cpu: 20m
      memory: 20Mi
  useComponentLabel: true
podSecurityPolicy:
  enabled: true

The ingresses

This works:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: notification-service-rpcm
  annotations:
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/session-cookie-hash: sha1
    nginx.ingress.kubernetes.io/session-cookie-name: ingressrpcm
spec:
  rules:
  - host: my-domain.com
    http:
      paths:
      - backend:
          serviceName: my-service
          servicePort: http
        path: /

This does not:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: notification-service-rpcm
  annotations:
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/configuration-snippet: |
       proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;
    nginx.ingress.kubernetes.io/session-cookie-hash: sha1
    nginx.ingress.kubernetes.io/session-cookie-name: ingressrpcm
spec:
  rules:
  - host: my-domain.com
    http:
      paths:
      - backend:
          serviceName: my-service
          servicePort: http
        path: /

/kind bug