ingress-nginx: Cannot see OpenTelemetry traces

What happened:

I upgraded to ingress-nginx 1.7.0 and Helm chart 4.6.0, and enabled OpenTelemetry configuration, to the best of my knowledge. The following represent the custom values I entered:

controller:
  opentelemetry:
    enabled: true

  config:
    otlp-collector-host: $HOST_IP
    enable-opentelemetry: "true"
    enable-brotli: "true"
  extraEnvs:
  - name: HOST_IP
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP

Firstly, without that first controller.opentelemetry.enabled: true option, it would fail to load the otel dynamic library. I didn’t initially see this documented anywhere, but a kind soul on the OTEL feature ticket told me to add that.

What you expected to happen:

I expected to start seeing traces from the nginx service on my traces feed. I have both Datadog and SignOz hooked up to the OTEL collector (which runs as a daemonset), but I haven’t seen any nginx traces anywhere yet. No errors either.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

nginx-ingress-nginx-controller-56675749b6-pwfn5:/etc/nginx$ nginx-ingress-controller --version
bash: nginx-ingress-controller: command not found

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.7", GitCommit:"723bcdb232300aaf5e147ff19b4df7ec8a20278d", GitTreeState:"archive", BuildDate:"2023-03-02T00:00:00Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.5", GitCommit:"34f89fd3fb1a106e1b23d3454b2f2cbf305602a1", GitTreeState:"clean", BuildDate:"2023-02-10T14:48:59Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: Azure AKS

• OS (e.g. from /etc/os-release):

• Kernel (e.g. uname -a):

• Install tools: AKS managed

  • Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.

• Basic cluster related info:

  • kubectl version
  • kubectl get nodes -o wide

• How was the ingress-nginx-controller installed:

  • If helm was used then please show output of helm ls -A | grep -i ingress

nginx             	nginx         	12      	2023-03-27 16:29:03.278334939 +0300 EEST	deployed	ingress-nginx-4.6.0           	1.7.0 

• If helm was used then please show output of helm -n <ingresscontrollernamepspace> get values <helmreleasename>

USER-SUPPLIED VALUES:
controller:
  config:
    enable-brotli: "true"
    enable-opentelemetry: "true"
    otlp-collector-host: $HOST_IP
  extraEnvs:
  - name: HOST_IP
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP
  opentelemetry:
    enabled: true
  service:
    externalTrafficPolicy: Local
    loadBalancerIP: <redacted>

• If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used

• if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances

• Current State of the controller:

  • kubectl describe ingressclasses

Name:         nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=nginx
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.7.0
              helm.sh/chart=ingress-nginx-4.6.0
Annotations:  meta.helm.sh/release-name: nginx
              meta.helm.sh/release-namespace: nginx
Controller:   k8s.io/ingress-nginx
Events:       <none>

• kubectl -n <ingresscontrollernamespace> get all -A -o wide

Why do you need all resources from all namespaces?

• kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>

Name:             nginx-ingress-nginx-controller-56675749b6-pwfn5
Namespace:        nginx
Priority:         0
Service Account:  nginx-ingress-nginx
Node:             aks-agentpool-27862475-vmss000001/10.240.0.115
Start Time:       Mon, 27 Mar 2023 16:29:11 +0300
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=nginx
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=ingress-nginx
                  app.kubernetes.io/part-of=ingress-nginx
                  app.kubernetes.io/version=1.7.0
                  helm.sh/chart=ingress-nginx-4.6.0
                  pod-template-hash=56675749b6
Annotations:      <none>
Status:           Running
IP:               10.240.0.138
IPs:
  IP:           10.240.0.138
Controlled By:  ReplicaSet/nginx-ingress-nginx-controller-56675749b6
Init Containers:
  opentelemetry:
    Container ID:  containerd://f8d399eb9b7dc97fcec3394ea4c7aea41b19e1346d41cb928cf2769cac153e34
    Image:         registry.k8s.io/ingress-nginx/opentelemetry:v20230312-helm-chart-4.5.2-28-g66a760794@sha256:40f766ac4a9832f36f217bb0e98d44c8d38faeccbfe861fbc1a76af7e9ab257f
    Image ID:      registry.k8s.io/ingress-nginx/opentelemetry@sha256:40f766ac4a9832f36f217bb0e98d44c8d38faeccbfe861fbc1a76af7e9ab257f
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      /usr/local/bin/init_module.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 27 Mar 2023 16:29:13 +0300
      Finished:     Mon, 27 Mar 2023 16:29:13 +0300
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /modules_mount from modules (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-pnmmk (ro)
Containers:
  controller:
    Container ID:  containerd://9223ed6ebf327ec74ffd86ab6940b7981b67bb235ef5168daad5c056a276a85b
    Image:         registry.k8s.io/ingress-nginx/controller:v1.7.0@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7
    Image ID:      registry.k8s.io/ingress-nginx/controller@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7
    Ports:         80/TCP, 443/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/nginx-ingress-nginx-controller
      --election-id=nginx-ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/nginx-ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Running
      Started:      Mon, 27 Mar 2023 16:29:22 +0300
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       nginx-ingress-nginx-controller-56675749b6-pwfn5 (v1:metadata.name)
      POD_NAMESPACE:  nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
      HOST_IP:         (v1:status.hostIP)
    Mounts:
      /modules_mount from modules (rw)
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-pnmmk (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  modules:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  nginx-ingress-nginx-admission
    Optional:    false
  kube-api-access-pnmmk:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From                      Message
  ----    ------     ----  ----                      -------
  Normal  Scheduled  18m   default-scheduler         Successfully assigned nginx/nginx-ingress-nginx-controller-56675749b6-pwfn5 to aks-agentpool-27862475-vmss000001
  Normal  Pulling    18m   kubelet                   Pulling image "registry.k8s.io/ingress-nginx/opentelemetry:v20230312-helm-chart-4.5.2-28-g66a760794@sha256:40f766ac4a9832f36f217bb0e98d44c8d38faeccbfe861fbc1a76af7e9ab257f"
  Normal  Pulled     17m   kubelet                   Successfully pulled image "registry.k8s.io/ingress-nginx/opentelemetry:v20230312-helm-chart-4.5.2-28-g66a760794@sha256:40f766ac4a9832f36f217bb0e98d44c8d38faeccbfe861fbc1a76af7e9ab257f" in 1.340993991s
  Normal  Created    17m   kubelet                   Created container opentelemetry
  Normal  Started    17m   kubelet                   Started container opentelemetry
  Normal  Pulling    17m   kubelet                   Pulling image "registry.k8s.io/ingress-nginx/controller:v1.7.0@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7"
  Normal  Pulled     17m   kubelet                   Successfully pulled image "registry.k8s.io/ingress-nginx/controller:v1.7.0@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7" in 6.859694032s
  Normal  Created    17m   kubelet                   Created container controller
  Normal  Started    17m   kubelet                   Started container controller
  Normal  RELOAD     17m   nginx-ingress-controller  NGINX reload triggered due to a change in configuration

• kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>

Name:                     nginx-ingress-nginx-controller
Namespace:                nginx
Labels:                   app.kubernetes.io/component=controller
                          app.kubernetes.io/instance=nginx
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/part-of=ingress-nginx
                          app.kubernetes.io/version=1.7.0
                          helm.sh/chart=ingress-nginx-4.6.0
Annotations:              meta.helm.sh/release-name: nginx
                          meta.helm.sh/release-namespace: nginx
Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.0.59.87
IPs:                      10.0.59.87
IP:                       20.50.181.64
LoadBalancer Ingress:     20.50.181.64
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  30879/TCP
Endpoints:                10.240.0.138:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  32505/TCP
Endpoints:                10.240.0.138:443
Session Affinity:         None
External Traffic Policy:  Local
HealthCheck NodePort:     30068
Events:                   <none>

• Current state of ingress object, if applicable:

  • kubectl -n <appnnamespace> get all,ing -o wide
  • kubectl -n <appnamespace> describe ing <ingressname>
  • If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

• Others:

  • Any other related information like ;
    • copy/paste of the snippet (if applicable)
    • kubectl describe ... of any custom configmap(s) created and in use
    • Any other related information that may help

How to reproduce this issue:

Anything else we need to know: