opentelemetry-python-contrib: Instrumentation raises uncaught exception on non utf-8 request header sequences

Describe your environment Poetry (version 1.2.2) Python 3.10.4 opentelemetry-instrumentation-fastapi = “>=0.32b0,<1.0dev”

Steps to reproduce Send an HTTP request to a Python ASGI application with opentelemetry-instrumentation-fastapi instrumentation, which contains non utf-8 sequences in its request headers.

What is the expected behavior? REST request does not fail without the instrumentation, and thus should not fail with the instrumentation either.

What is the actual behavior?

ERROR 2022-12-07 00:47:10,938 run_asgi Exception in ASGI application
 
Traceback (most recent call last):
  File "/root/.cache/pypoetry/virtualenvs/--REDACTED--/lib/python3.10/site-packages/opentelemetry/instrumentation/asgi/__init__.py", line 560, in __call__
    collect_custom_request_headers_attributes(scope)
  File "/root/.cache/pypoetry/virtualenvs/--REDACTED--/lib/python3.10/site-packages/opentelemetry/instrumentation/asgi/__init__.py", line 345, in collect_custom_request_headers_attributes
    headers = {
  File "/root/.cache/pypoetry/virtualenvs/--REDACTED--/lib/python3.10/site-packages/opentelemetry/instrumentation/asgi/__init__.py", line 346, in <dictcomp>
    _key.decode("utf8"): _value.decode("utf8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbf in position 0: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.cache/pypoetry/virtualenvs/--REDACTED--/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 372, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/root/.cache/pypoetry/virtualenvs/--REDACTED--/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/root/.cache/pypoetry/virtualenvs/--REDACTED--/lib/python3.10/site-packages/fastapi/applications.py", line 270, in __call__
    await super().__call__(scope, receive, send)
  File "/root/.cache/pypoetry/virtualenvs/--REDACTED--/lib/python3.10/site-packages/starlette/applications.py", line 124, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/root/.cache/pypoetry/virtualenvs/--REDACTED--/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/root/.cache/pypoetry/virtualenvs/--REDACTED--/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/root/.cache/pypoetry/virtualenvs/--REDACTED--/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/root/.cache/pypoetry/virtualenvs/--REDACTED--/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/root/.cache/pypoetry/virtualenvs/--REDACTED--/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/root/.cache/pypoetry/virtualenvs/--REDACTED--/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/root/.cache/pypoetry/virtualenvs/--REDACTED--/lib/python3.10/site-packages/starlette/routing.py", line 706, in __call__
    await route.handle(scope, receive, send)
  File "/root/.cache/pypoetry/virtualenvs/--REDACTED--/lib/python3.10/site-packages/starlette/routing.py", line 443, in handle
    await self.app(scope, receive, send)
  File "/root/.cache/pypoetry/virtualenvs/--REDACTED--/lib/python3.10/site-packages/fastapi/applications.py", line 270, in __call__
    await super().__call__(scope, receive, send)
  File "/root/.cache/pypoetry/virtualenvs/--REDACTED--/lib/python3.10/site-packages/starlette/applications.py", line 124, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/root/.cache/pypoetry/virtualenvs/--REDACTED--/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/root/.cache/pypoetry/virtualenvs/--REDACTED--/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/root/.cache/pypoetry/virtualenvs/--REDACTED--/lib/python3.10/site-packages/opentelemetry/instrumentation/asgi/__init__.py", line 584, in __call__
    duration = max(round((default_timer() - start) * 1000), 0)
UnboundLocalError: local variable 'start' referenced before assignment

Additional context Issue happens in this line, which assumes utf-8 encoding: https://github.com/open-telemetry/opentelemetry-python-contrib/blob/bc57cc029dc0d77a4c0327b856a4f4bb6151d504/instrumentation/opentelemetry-instrumentation-asgi/src/opentelemetry/instrumentation/asgi/__init__.py#L346

About this issue

Original URL
State: open
Created 2 years ago
Reactions: 1
Comments: 19 (7 by maintainers)

Commits related to this issue

Fix UnboundLocalError in asgi handler (#1478) — committed to mike-baker-on/opentelemetry-python-contrib by mike-baker-on a year ago
[asgi] add test for header decoding bug see #1478 #1814 — committed to thomasleveil/opentelemetry-python-contrib by thomasleveil a year ago

Most upvoted comments

That sounds good to me. We can probably have both fallback and try-catch pass through.

srikanthccv on Mar 6, 2023

I think regardless of the byte stream encoding, if the requests didn’t blow up without this middleware, then they shouldn’t blow up with it. I don’t think it’s correct that this middleware enforces a stricter header rule, even if the existing one is more permissive than it should.

Would it be sensible to either not decode them, or try-catch the decode and pass-through, or try more encodings?

marianogappa on Mar 2, 2023