jaeger-operator: Jaeger Rollover Elastic Search - SSL: CERTIFICATE_VERIFY_FAILED

Describe the bug When deploying the jaeger production operator using elastic search “use-aliases” the kubernetes pod for roll over initialization fails with SSL Certificate verification failure even though the certificate is valid and set. Configuration listed below.

To Reproduce Steps to reproduce the behavior:

1. kubectl apply -f ./jaeger-prod.yaml -n observability
2. kubectl get pods -n observability
3. kubectl logs <rollover-pod> -n observability > output.txt
4. Review output.txt and the logs show SSL certificate verify errors

Expected behavior Rollover job is successful, indexes are created, and jaeger starts up successfully

Version (please complete the following information):

• OS: Ubuntu
• Jaeger version: Latest
• Deployment: Kubernetes
• Cloud: Azure AKS 1.19

What troubleshooting steps did you try? I have tried using different certificates, server names, setting the TLS settings directly on the cron jobs.

Additional context NAME READY STATUS RESTARTS AGE jaeger-operator-6954c97677-nc2jk 1/1 Running 0 19h jaeger-prod-es-rollover-create-mapping-9fk6q 0/1 Error 1 3s

Output Logs Traceback (most recent call last): File “/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py”, line 597, in urlopen httplib_response = self._make_request(conn, method, url, File “/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py”, line 343, in _make_request self._validate_conn(conn) File “/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py”, line 839, in validate_conn conn.connect() File “/usr/local/lib/python3.8/site-packages/urllib3/connection.py”, line 337, in connect self.sock = ssl_wrap_socket( File "/usr/local/lib/python3.8/site-packages/urllib3/util/ssl.py", line 345, in ssl_wrap_socket return context.wrap_socket(sock, server_hostname=server_hostname) File “/usr/local/lib/python3.8/ssl.py”, line 500, in wrap_socket return self.sslsocket_class._create( File “/usr/local/lib/python3.8/ssl.py”, line 1040, in _create self.do_handshake() File “/usr/local/lib/python3.8/ssl.py”, line 1309, in do_handshake self._sslobj.do_handshake() ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1124)

Jaeger Config

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger-prod
  namespace: observability
spec:
  strategy: production
  query:
    options:
      es:
        use-aliases: true
      query:
        base-path: /jaeger
  storage:
    type: elasticsearch
    options:
      log-level: debug
      es:
        server-urls: https://elasticsearch_url/elastic
        index-prefix: jaeger-
        use-aliases: true
        tls:
          enabled: yes
          ca: "/tls/es.crt"
    esIndexCleaner:
        enabled: true
        numberOfDays: 7
        schedule: "55 23 * * *"
    esRollover:
      conditions: "{\"max_age\": \"2d\"}"
      readTTL: 168h
      schedule: "55 23 * * *" 
  collector:
    log-level: debug
    options:
      es:
        use-aliases: true
    maxReplicas: 5
    zipkin:
      host-port: "9411"
    resources:
      limits:
        cpu: 100m
        memory: 128Mi
    ingress:
      enabled: false
  volumeMounts:
    - name: jaeger-tls
      mountPath: /tls
      subPath: ""
      configMap: jaeger-tls
      readOnly: true
  volumes:
    - name: jaeger-tls
      configMap:
        name: jaeger-tls