opentelemetry-python-contrib: gRPC client + botocore + datadog exporter instrumentation error

This one is a bit complex - when using the gRPC client instrumentation to make a call to an instrumented gRPC service that then makes a call via boto to the DynamoDB API, the combination of instrumentation leads to this error:

2020-12-17 14:19:39,666 DEBUG    [botocore.httpsession:298] Exception received when sending urllib3 HTTP request
Traceback (most recent call last):
  File "/home/michael/work/myapp/.venv/lib/python3.8/site-packages/botocore/httpsession.py", line 254, in send
    urllib_response = conn.urlopen(
  File "/home/michael/work/myapp/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/home/michael/work/myapp/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/home/michael/work/myapp/.venv/lib/python3.8/site-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib64/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/michael/work/myapp/.venv/lib/python3.8/site-packages/botocore/awsrequest.py", line 91, in _send_request
    rval = super(AWSConnection, self)._send_request(
  File "/usr/lib64/python3.8/http/client.py", line 1296, in _send_request
    self.putheader(hdr, value)
  File "/home/michael/work/myapp/.venv/lib/python3.8/site-packages/urllib3/connection.py", line 219, in putheader
    _HTTPConnection.putheader(self, header, *values)
  File "/usr/lib64/python3.8/http/client.py", line 1232, in putheader
    if _is_illegal_header_value(values[i]):
TypeError: expected string or bytes-like object

Without either client instrumentation, this call works perfectly.

Steps to reproduce As described above - I don’t have a simplified sample, but I’ll try to produce one.

What is the expected behavior? No exception 😃

What is the actual behavior? The above exception.

Additional context

I’m definitely working on this, but any suggestions would certainly be welcome, I’m not quite sure what the invalid header is or where it might get set.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 18 (18 by maintainers)

Commits related to this issue

switch setter for injecting headers w/propagator Occasionally not all of the headers available to botocore are strings. It does allow this, using the `add_header` method rather than `__setitem__`. A... — committed to alertedsnake/opentelemetry-python-contrib by deleted user 4 years ago
Don't set tags which are None This is often `None`, but tags are always strings, and so things get broken when spans get passed along to other client calls. Fixes #262 — committed to alertedsnake/opentelemetry-python-contrib by deleted user 4 years ago
Don't set tags which are None This is often `None`, but tags are always strings, and so things get broken when spans get passed along to other client calls. Fixes #262 — committed to alertedsnake/opentelemetry-python-contrib by deleted user 4 years ago
Don't set tags which are None This is often `None`, but tags are always strings, and so things get broken when spans get passed along to other client calls. Fixes #262 — committed to alertedsnake/opentelemetry-python-contrib by deleted user 4 years ago
Don't set tags which are None This is often `None`, but tags are always strings, and so things get broken when spans get passed along to other client calls. Fixes #262 — committed to alertedsnake/opentelemetry-python-contrib by deleted user 4 years ago
Don't set tags which are None This is often `None`, but tags are always strings, and so things get broken when spans get passed along to other client calls. Fixes #262 — committed to alertedsnake/opentelemetry-python-contrib by deleted user 4 years ago
Don't set tags which are None This is often `None`, but tags are always strings, and so things get broken when spans get passed along to other client calls. Fixes #262 — committed to alertedsnake/opentelemetry-python-contrib by deleted user 4 years ago
Don't set tags which are None This is often `None`, but tags are always strings, and so things get broken when spans get passed along to other client calls. Fixes #262 — committed to alertedsnake/opentelemetry-python-contrib by deleted user 4 years ago
Don't set tags which are None This is often `None`, but tags are always strings, and so things get broken when spans get passed along to other client calls. Fixes #262 — committed to alertedsnake/opentelemetry-python-contrib by deleted user 4 years ago

Most upvoted comments

I’m running the two services locally, so I can hack whatever I need to add debugging. I did try a bunch of printing, but couldn’t get anything useful before the exception was raised.

Have you tried adding a pdb statement to this snippet, within a try except?

  File "/usr/lib64/python3.8/http/client.py", line 1232, in putheader
    if _is_illegal_header_value(values[i])

try: 
   if is_illegal_header_value(...)
except:
   import pdb;
   pdb.set_trace()

You should be able to extract all the information you need from that pdb session. prints work too, just a bit more time consuming.

toumorokoshi on Dec 20, 2020