azure-sdk-for-python: OpenTelemetry Container App AzureVMResourceDetector Exception
- azure-monitor-opentelemetry:
- 1.1.0:
- Linux:
- 3.9:
Describe the bug
Auto instrumentation of FastAPI Python in Azure Container App results in a error:
WARNING:Exception in detector <opentelemetry.resource.detector.azure.vm.AzureVMResourceDetector object at 0x7f011e043ac0>, ignoring
The error pointing to the AzureVM detector seems random to me.
To Reproduce Steps to reproduce the behavior:
- Create Container App Environment, configure logging to Azure Monitor and the correct log workspace you want to use for application insights as well
- Create container App with an image with the code in the appendix
- Set Environment variable
APPLICATIONINSIGHTS_CONNECTION_STRING
to a valid app insights instance - The replica will not activate and log :
WARNING:Exception in detector <opentelemetry.resource.detector.azure.vm.AzureVMResourceDetector object at 0x7f011e043ac0>, ignoring
main.py:
from typing import Union
import logging
import os
from opentelemetry.trace import (
SpanKind,
get_tracer_provider,
set_tracer_provider,
)
from opentelemetry.propagate import extract
from fastapi import FastAPI, HTTPException
from azure.monitor.opentelemetry import configure_azure_monitor
# Import the tracing api from the `opentelemetry` package.
from opentelemetry import trace
logging.basicConfig(format = "%(asctime)s:%(levelname)s:%(message)s", level = logging.WARNING)
logger = logging.getLogger(__name__)
# Configure OpenTelemetry to use Azure Monitor with the specified connection
# string.
configure_azure_monitor(
#connection_string = os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"],
connection_string = os.getenv("APPLICATIONINSIGHTS_CONNECTION_STRING")
instrumentation_options = {"azure_sdk": {"enabled": False}}
disable_offline_storage=True
)
tracer = trace.get_tracer(__name__,
tracer_provider=get_tracer_provider())
# Start a new span with the name "hello". This also sets this created span as the current span in this context. This span will be exported to Azure Monitor as part of the trace.
with tracer.start_as_current_span("hello"):
print("Hello, World!")
app = FastAPI()
@app.get("/")
def read_root():
return {"Hello": "World"}
@app.get("/testroute")
def read_root():
with tracer.start_as_current_span(
"error_test_request",
#context=extract(request.headers),
kind=SpanKind.SERVER
):
logger.error("Error test endpoint was reached. . .")
raise HTTPException(status_code=500, detail="Inernal Error")
return {"message": "Error Test!"}
@app.get("/items/{item_id}")
def read_item(item_id: int, q: Union[str, None] = None):
return {"item_id": item_id, "q": q}
Dockerfile:
FROM python:3.9
WORKDIR /
COPY ./requirements.txt /requirements.txt
RUN pip install --no-cache-dir --upgrade -r /requirements.txt
COPY ./ /
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "80"]
requirements.txt:
annotated-types==0.6.0
anyio==3.7.1
asgiref==3.7.2
azure-core==1.29.5
azure-core-tracing-opentelemetry==1.0.0b11
azure-monitor-opentelemetry==1.1.0
azure-monitor-opentelemetry-exporter==1.0.0b19
certifi==2023.11.17
charset-normalizer==3.3.2
click==8.1.7
Deprecated==1.2.14
exceptiongroup==1.2.0
fastapi==0.104.1
fixedint==0.1.6
h11==0.14.0
httptools==0.6.1
idna==3.4
importlib-metadata==6.8.0
isodate==0.6.1
msrest==0.7.1
oauthlib==3.2.2
opentelemetry-api==1.21.0
opentelemetry-instrumentation==0.42b0
opentelemetry-instrumentation-asgi==0.42b0
opentelemetry-instrumentation-dbapi==0.42b0
opentelemetry-instrumentation-django==0.42b0
opentelemetry-instrumentation-fastapi==0.42b0
opentelemetry-instrumentation-flask==0.42b0
opentelemetry-instrumentation-psycopg2==0.42b0
opentelemetry-instrumentation-requests==0.42b0
opentelemetry-instrumentation-urllib==0.42b0
opentelemetry-instrumentation-urllib3==0.42b0
opentelemetry-instrumentation-wsgi==0.42b0
opentelemetry-resource-detector-azure==0.1.0
opentelemetry-sdk==1.21.0
opentelemetry-semantic-conventions==0.42b0
opentelemetry-util-http==0.42b0
packaging==23.2
pydantic==2.5.2
pydantic_core==2.14.5
python-dotenv==1.0.0
PyYAML==6.0.1
requests==2.31.0
requests-oauthlib==1.3.1
six==1.16.0
sniffio==1.3.0
starlette==0.27.0
typing_extensions==4.8.0
urllib3==2.1.0
uvicorn==0.24.0.post1
uvloop==0.19.0
watchfiles==0.21.0
websockets==12.0
wrapt==1.16.0
zipp==3.17.0
Expected behavior The container app sends logs and metrics to Azure Monitor and the replica is healthy.
Screenshots
About this issue
- Original URL
- State: closed
- Created 7 months ago
- Comments: 18 (9 by maintainers)
Regarding the confusing warning message itself, I believe I have discovered the source: The concurrent.futures system that the OpenTelemetry SDK is using to run the resource detectors is not working correctly for processes that can take longer than 5 seconds.
I have made an issue in the OTel repo to track this.
Again this is unrelated to whether your telemetry is exporting or not.
@vipinanandcpp I’ll be a bit clearer: Resource Detector errors such as the AzureVMResourceDetector warning you mentioned are ignored and do not crash the app or prevent telemetry from flowing. There was a bug that could cause the app to hang inside AzureVMResourceDetector preventing it from starting. That should be fixed by the recent release. However, if telemetry is not exporting to Application Insights even after this is fixed, there is likely an unrelated issue with your set up.
@litan1106
Yes set it to the same value for now. This is just a workaround to block the error message from appearing.