grpc: python str/unicode error in metadata with grpc 1.8.0

What version of gRPC and what language are you using?

1.8.0, python 2.7

What operating system (Linux, Windows, …) and version?

debian stretch (dockerized)

What runtime / compiler are you using (e.g. python version or version of gcc)

• Python 2.7.13
• grpcio==1.8.0
• google-cloud-datastore==1.4.0

grpcio and google-cloud-datastore installed from pip without a version specified.

What did you do?

If possible, provide a recipe for reproducing the error. Try being specific and include code snippets if helpful.

I’m using google-cloud-datastore in a python flask app, which is dockerized. Starting with today our CI tests fail with an TypeError: Expected str, got unicode. I traced it back to a new version (1.8.0, before 1.7.3) being available via pip.

What did you expect to see?

Google Cloud Datastore queries working with Datastore or Datastore Emulator.

What did you see instead?

No queries or any other requests are possible against Cloud Datastore.

E1213 16:11:08.523933100       1 ev_epollex_linux.cc:1482]   Skipping epollex becuase GRPC_LINUX_EPOLL is not defined.
E1213 16:11:08.524041000       1 ev_epoll1_linux.cc:1261]    Skipping epoll1 becuase GRPC_LINUX_EPOLL is not defined.
E1213 16:11:08.524077200       1 ev_epollsig_linux.cc:1761]  Skipping epollsig becuase GRPC_LINUX_EPOLL is not defined.
TypeError: Expected str, got unicode
Exception TypeError: 'Expected str, got unicode' in 'grpc._cython.cygrpc._store_c_metadata' ignored
E1213 16:11:08.528453300       1 call.cc:1001]               validate_metadata: {"created":"@1513181468.528438800","description":"Metadata keys cannot be zero length","file":"src/core/lib/surface/validate_metadata.cc","file_line":66}
ERROR    2017-12-13 16:11:08,529 service.py: 34] something went bad: RetryError(Exception occurred in retry method that was not classified as transient, caused by metadata was invalid: [(u'x-goog-api-client', 'gl-python/2.7.13 gccl/1.4.0 gapic/0.15.3 gax/0.15.16 grpc/1.8.0')])
Traceback (most recent call last):
...
  File "/usr/local/lib/python2.7/dist-packages/google/api_core/page_iterator.py", line 186, in _items_iter
    for page in self._page_iter(increment=False):
  File "/usr/local/lib/python2.7/dist-packages/google/api_core/page_iterator.py", line 217, in _page_iter
    page = self._next_page()
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/datastore/query.py", line 498, in _next_page
    query=query_pb,
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/datastore/_gax.py", line 134, in run_query
    return super(GAPICDatastoreAPI, self).run_query(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/gapic/datastore/v1/datastore_client.py", line 258, in run_query
    return self._run_query(request, options)
  File "/usr/local/lib/python2.7/dist-packages/google/gax/api_callable.py", line 452, in inner
    return api_caller(api_call, this_settings, request)
  File "/usr/local/lib/python2.7/dist-packages/google/gax/api_callable.py", line 438, in base_caller
    return api_call(*args)
  File "/usr/local/lib/python2.7/dist-packages/google/gax/api_callable.py", line 376, in inner
    return a_func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/google/gax/retry.py", line 127, in inner
    ' classified as transient', exception)
RetryError: RetryError(Exception occurred in retry method that was not classified as transient, caused by metadata was invalid: [(u'x-goog-api-client', 'gl-python/2.7.13 gccl/1.4.0 gapic/0.15.3 gax/0.15.16 grpc/1.8.0')])

versions:

• Python 2.7.13
• grpcio==1.8.0
• google-cloud-datastore==1.4.0

If i add the explicit requirement of grpcio==1.7.3 to my python requirements the tests succeed.

It seems like this one line is key:

Exception occurred in retry method that was not classified as transient, caused by metadata was invalid: [(u'x-goog-api-client', 'gl-python/2.7.13 gccl/1.4.0 gapic/0.15.3 gax/0.15.16 grpc/1.8.0')]

the user agent string x-google-api-client is a unicode string, not a byte string.

Anything else we should know about your project / environment?