opentelemetry-operator: How to resolve conflicts between Boto3 in opentelemetry auto-instrument and Kubernetes applications?

Sumary

• I installed OpenTelemetry Operator and OpenTelemetry Collector by helm to implement auto-instrument application which deployed on Kubernetes. But after upgrade, the status of application pod is “crashloopback”.

Describe your environment

• Version of OpenTelemetry Operator helm chart: 0.76.1
• Version of OpenTelemetry Collector helm chart: 0.77.0
• The requirement.txt file of application which deployed on Kubernetes

fastapi[all]==0.88.0 
aiofiles==22.1.0
loguru==0.6.0
tritonclient[all]==2.28.0
python-decouple==3.6
pymongo==4.3.3
opencv-python==4.6.0.66
Pillow==9.3.0
pandas==1.5.2
pyarrow==10.0.1
dask==2022.12.1
boto3==1.26.49
typeguard==2.13.3
scikit-image==0.19.3
s3fs==0.4.2
rocketry==2.5.1
requests==2.25.1

Steps to reproduce

• Install OpenTelemetry Operator helm chart
• Install OpenTelemetry Collector helm chart
• Create and apply instrument file:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: instrumentation
spec:
  resource:
    addK8sUIDAttributes: true
  exporter:
    endpoint: http://opentelemetry-collector:4317
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "0.25"
  python:
    env:
      - name: OTEL_LOG_LEVEL
        value: "debug"
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://opentelemetry-collector:4318

• Add annotations in application yaml file and upgrade:

annotations:
  instrumentation.opentelemetry.io/inject-python: "instrumentation"

What happen? But after upgraded application, the status is “CrashLoopBack”. I logged that application:

Instrumenting of boto3 failed
Traceback (most recent call last):
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 84, in _load_instrumentors
    distro.load_instrumentor(entry_point, skip_dep_check=True)
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/distro.py", line 62, in load_instrumentor
    instrumentor: BaseInstrumentor = entry_point.load()
  File "/otel-auto-instrumentation/pkg_resources/__init__.py", line 2517, in load
    return self.resolve()
  File "/otel-auto-instrumentation/pkg_resources/__init__.py", line 2523, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/boto3sqs/__init__.py", line 34, in <module>
    import boto3
  File "/usr/local/lib/python3.8/dist-packages/boto3/__init__.py", line 17, in <module>
    from boto3.session import Session
  File "/usr/local/lib/python3.8/dist-packages/boto3/session.py", line 17, in <module>
    import botocore.session
  File "/usr/local/lib/python3.8/dist-packages/botocore/session.py", line 26, in <module>
    import botocore.client
  File "/usr/local/lib/python3.8/dist-packages/botocore/client.py", line 15, in <module>
    from botocore import waiter, xform_name
  File "/usr/local/lib/python3.8/dist-packages/botocore/waiter.py", line 18, in <module>
    from botocore.docs.docstring import WaiterDocstring
  File "/usr/local/lib/python3.8/dist-packages/botocore/docs/__init__.py", line 15, in <module>
    from botocore.docs.service import ServiceDocumenter
  File "/usr/local/lib/python3.8/dist-packages/botocore/docs/service.py", line 14, in <module>
    from botocore.docs.client import ClientDocumenter, ClientExceptionsDocumenter
  File "/usr/local/lib/python3.8/dist-packages/botocore/docs/client.py", line 14, in <module>
    from botocore.docs.example import ResponseExampleDocumenter
  File "/usr/local/lib/python3.8/dist-packages/botocore/docs/example.py", line 13, in <module>
    from botocore.docs.shape import ShapeDocumenter
  File "/usr/local/lib/python3.8/dist-packages/botocore/docs/shape.py", line 19, in <module>
    from botocore.utils import is_json_value_header
  File "/usr/local/lib/python3.8/dist-packages/botocore/utils.py", line 37, in <module>
    import botocore.httpsession
  File "/usr/local/lib/python3.8/dist-packages/botocore/httpsession.py", line 22, in <module>
    from urllib3.util.ssl_ import (
ImportError: cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' (/otel-auto-instrumentation/urllib3/util/ssl_.py)
Failed to auto initialize opentelemetry

Additional context After consider problems, I see boto3 in opentelemetry auto-instrument conflicts to boto3 in my applications. How to fix it??