ingress-nginx: Ingress v0.35.0 - Failed to load tracing library

NGINX Ingress controller version: v0.35.0

Kubernetes version (use kubectl version):v1.16.2

Environment:

  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:

What happened: I have ingress controller installed via helm chart 2.13.0 with enabled opentracing and with sidecars jaeger opentracing agent In ingress configmap:

  enable-opentracing: "true"
  jaeger-collector-host: localhost
  jaeger-service-name: nginx-ingress

In my ingress pod i have this /etc/nginx/opentracing.json:

{
  "service_name": "nginx-ingress",
  "sampler": {
	"type": "const",
	"param": 1,
	"samplingServerURL": "http://127.0.0.1:5778/sampling"
  },
  "reporter": {
	"localAgentHostPort": "localhost:6831"
  },
  "headers": {
	"TraceContextHeaderName": "",
	"jaegerDebugHeader": "",
	"jaegerBaggageHeader": "",
	"traceBaggageHeaderPrefix": ""
  },
}

I have updated ingress from v0.34.1 to v0.35.0 and now i have a crashed ingress with this error:

E0831 15:10:37.993504       8 controller.go:153] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: exit status 1
2020/08/31 15:10:37 [error] 57#57: Failed to load tracing library /usr/local/lib/libjaegertracing_plugin.so: Error relocating /usr/local/lib/libjaegertracing_plugin.so: _ZN4YAML6detail9node_data12empty_scalarB5cxx11E: symbol not found
nginx: [error] Failed to load tracing library /usr/local/lib/libjaegertracing_plugin.so: Error relocating /usr/local/lib/libjaegertracing_plugin.so: _ZN4YAML6detail9node_data12empty_scalarB5cxx11E: symbol not found
nginx: configuration file /tmp/nginx-cfg335013150 test failed

-------------------------------------------------------------------------------
W0831 15:10:37.993534       8 queue.go:130] requeuing clustersecret/cluster.local-nfs-provisioner-nfs-client-provisioner, err
-------------------------------------------------------------------------------
Error: exit status 1
2020/08/31 15:10:37 [error] 57#57: Failed to load tracing library /usr/local/lib/libjaegertracing_plugin.so: Error relocating /usr/local/lib/libjaegertracing_plugin.so: _ZN4YAML6detail9node_data12empty_scalarB5cxx11E: symbol not found
nginx: [error] Failed to load tracing library /usr/local/lib/libjaegertracing_plugin.so: Error relocating /usr/local/lib/libjaegertracing_plugin.so: _ZN4YAML6detail9node_data12empty_scalarB5cxx11E: symbol not found
nginx: configuration file /tmp/nginx-cfg335013150 test failed

conf file nginx-cfg335013150.txt

What you expected to happen:

working ingress

How to reproduce it:

install ingress-nginx with configmap:

apiVersion: v1
data:
  client-body-buffer-size: 32k
  client-header-buffer-size: 256k
  enable-opentracing: "true"
  http-snippet: proxy_cache_path /tmp/nginx-cache-mp4 levels=1:2 keys_zone=mp4:50m
    max_size=256m inactive=30m use_temp_path=off;
  jaeger-collector-host: localhost
  jaeger-service-name: nginx-ingress
  large-client-header-buffers: 4 256k
  log-format-escape-json: "true"
  log-format-upstream: '{"time":"$time_iso8601","proxy_protocol_addr":"$proxy_protocol_addr","remote_addr":"$remote_addr","x-forward-for":"$proxy_add_x_forwarded_for","request_id":"$req_id","request":"$request","remote_user":"$remote_user","bytes_sent":"$bytes_sent","body_bytes_sent":"$body_bytes_sent","request_time":"$request_time","status":"$status","vhost":"$host","request_proto":"$server_protocol","path":"$uri","request_query":"$args","request_length":"$request_length","method":"$request_method","http_referrer":"$http_referer","http_user_agent":"$http_user_agent","upstream":"$proxy_upstream_name","upstream_ip":"$upstream_addr","upstream_latency":"$upstream_response_time","upstream_status":"$upstream_status"}'
  proxy-set-headers: ingress-nginx/ingress-ingress-nginx-custom-proxy-headers
  ssl-ciphers: ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
  ssl-protocols: TLSv1 TLSv1.1 TLSv1.2
kind: ConfigMap
metadata:
  creationTimestamp: "2020-08-31T15:07:52Z"
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: 0.35.0
    helm.sh/chart: ingress-nginx-2.13.0
  name: ingress-ingress-nginx-controller
  namespace: ingress-nginx

Anything else we need to know:

/kind bug

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 10
  • Comments: 17 (6 by maintainers)

Most upvoted comments

Getting the same error with this image: k8s.gcr.io/ingress-nginx/controller:v0.47.0:

Error: exit status 1
2021/08/03 19:37:45 [warn] 954#954: the "http2_max_field_size" directive is obsolete, use the "large_client_header_buffers" directive instead in /tmp/nginx-cfg592055688:145
nginx: [warn] the "http2_max_field_size" directive is obsolete, use the "large_client_header_buffers" directive instead in /tmp/nginx-cfg592055688:145
2021/08/03 19:37:45 [warn] 954#954: the "http2_max_header_size" directive is obsolete, use the "large_client_header_buffers" directive instead in /tmp/nginx-cfg592055688:146
nginx: [warn] the "http2_max_header_size" directive is obsolete, use the "large_client_header_buffers" directive instead in /tmp/nginx-cfg592055688:146
2021/08/03 19:37:45 [warn] 954#954: the "http2_max_requests" directive is obsolete, use the "keepalive_requests" directive instead in /tmp/nginx-cfg592055688:147
nginx: [warn] the "http2_max_requests" directive is obsolete, use the "keepalive_requests" directive instead in /tmp/nginx-cfg592055688:147
2021/08/03 19:37:45 [error] 954#954: opentracing_propagate_context before tracer loaded
nginx: [error] opentracing_propagate_context before tracer loaded
nginx: configuration file /tmp/nginx-cfg592055688 test failed

Ah nice! I can now access https://console.cloud.google.com/gcr/images/k8s-staging-ingress-nginx/GLOBAL/controller?gcrImageListsize=30

this image works

controller:
  image:
    repository: gcr.io/k8s-staging-ingress-nginx/controller
    tag: "v0.35.0"
    digest: "sha256:97cedc940a4c23621ed307339c3140738228644cbbcf785f1163774d098a87d3"