google-cloud-python: httplib2.Http is not thread-safe

This is from a StackOverflow post. I’ve done some debugging from the Datastore side but I don’t think these requests ever make it into the Datastore part of the stack. I’d appreciate if someone look at this from the gcloud-python team. Note that this issue was originally reported via gcd-discuss@google.com In July.

I have a Python Django application running on a Google Compute instance. It is using gcloudoem to interface from Django to Google Datastore. gcloudoem uses the same underlying code to communicate with Datastore as gcloud-python 0.5.x

At what seems to be completely random times, I will get SSL errors happening when trying to talk to Datastore. There is no pattern in where in my application code these happen. It’s just during a random call to Datastore. Here are the two flavours of errors:

ERROR:django.request:Internal Server Error: /complete/google-oauth2/
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/cache.py", line 52, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/csrf.py", line 57, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/apps/django_app/utils.py", line 51, in wrapper
    return func(request, backend, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/apps/django_app/views.py", line 28, in complete
    redirect_name=REDIRECT_FIELD_NAME, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/actions.py", line 43, in do_complete
    user = backend.complete(user=user, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/backends/base.py", line 41, in complete
    return self.auth_complete(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/utils.py", line 229, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/backends/oauth.py", line 387, in auth_complete
    *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/utils.py", line 229, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/backends/oauth.py", line 396, in do_auth
    return self.strategy.authenticate(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/strategies/django_strategy.py", line 96, in authenticate
    return authenticate(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/contrib/auth/__init__.py", line 60, in authenticate
    user = backend.authenticate(**credentials)
  File "/usr/local/lib/python2.7/dist-packages/social/backends/base.py", line 82, in authenticate
    return self.pipeline(pipeline, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/backends/base.py", line 85, in pipeline
    out = self.run_pipeline(pipeline, pipeline_index, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/backends/base.py", line 112, in run_pipeline
    result = func(*args, **out) or {}
  File "/usr/local/lib/python2.7/dist-packages/social/pipeline/social_auth.py", line 20, in social_user
    social = backend.strategy.storage.user.get_social_auth(provider, uid)
  File "./social_gc/storage.py", line 105, in get_social_auth
    return cls.objects.get(provider=provider, uid=uid)
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/queryset/__init__.py", line 162, in get
    num = len(clone)
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/queryset/__init__.py", line 126, in __len__
    self._fetch_all()
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/queryset/__init__.py", line 370, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/datastore/query.py", line 480, in __iter__
    self.next_page()
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/datastore/query.py", line 452, in next_page
    transaction_id=transaction and transaction.id,
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/datastore/connection.py", line 249, in run_query
    response = self._rpc('runQuery', request, datastore_pb.RunQueryResponse)
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/datastore/connection.py", line 159, in _rpc
    data=request_pb.SerializeToString()
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/datastore/connection.py", line 134, in _request
    body=data
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 589, in new_request
    redirections, connection_type)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1609, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1351, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1307, in _conn_request
    response = conn.getresponse()
  File "/usr/lib/python2.7/httplib.py", line 1127, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 453, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 409, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 480, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib/python2.7/ssl.py", line 734, in recv
    return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 621, in read
    v = self._sslobj.read(len or 1024)
SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1752)

Unfortunately, for the second, I don’t have a full stacktrace handy:

[SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:1752)

These errors don’t happen when I am using the GCD tool. Does anyone have any idea what is happening here? Is this some sort of networking problem?

About this issue

Original URL
State: closed
Created 9 years ago
Reactions: 2
Comments: 83 (44 by maintainers)

Most upvoted comments

For those still following this issue, #3674 is putting the final nail in our usage of httplib2

dhermes on Jul 27, 2017

As a fast workaround I wrote a custom wrapper around requests to pretend to be httplib2.

class CustomHttp(object):
  def __init__(self, timeout=None):
    self.timeout = timeout

  def request(self, uri, method='GET', body=None, headers=None,
              redirections=None, connection_type=None):
    if connection_type is not None:
      uri = '%s://%s' % (connection_type, uri)
    resp = requests.request(method=method, url=uri, data=body, headers=headers,
                            timeout=self.timeout)
    resp.status = resp.status_code
    return resp, resp.content

credentials.authorize(CustomHttp())

Works great now. Maybe useful to somebody.

sadovnychyi on Feb 23, 2016

Making my use of httplib thread-safe fixed the issue. Not sure what I did to cause this error to pop up, but I was tinkering with some related code, so I may have caused it.

If it helps anyone else, below is the code that I use to get the authentication token. It relies on a dictionary PRIVATE_KEY to lookup the service account email and the PEM file (which should probably just be inputs into the function).

PRIVATE_KEY = {'my-project' : {'email':'serviceaccount@foo.com', 'key':'my_key.pem'}}
auth_cache = threading.local()
auth_cache_lock = threading.Lock()
def get_auth(project, force = False, scopes = None):
    global auth_cache, auth_cache_lock
    # At some point oauth2client had breaking changes. There are different 
    # copies of this library floating around, so handle whichever default imports.  
    try:
        from oauth2client.client import SignedJwtAssertionCredentials
    except ImportError as e:
        SignedJwtAssertionCredentials = None
        from oauth2client import crypt as oauth_crypt
        from oauth2client.service_account import ServiceAccountCredentials
    if project not in PRIVATE_KEY:
        raise Exception("Cannot connect to %s"%project)
    with auth_cache_lock:
        if not hasattr(auth_cache, 'auths'):
            auth_cache.auths = {}

        if not force and project in auth_cache.auths:
            return auth_cache.auths[project]
        f = None
        key = PRIVATE_KEY[project]['key']
        for path in ('updater/', '../updater/', ''):
            pfile = path + key
            try:
                f = file(pfile, 'rb')
            except IOError:
                continue
            else:
                break
        if not f:
            raise Exception("Could not find key file.")
        pem_contents = f.read()
        f.close()
        if scopes == None:
            scopes = DEFAULT_SCOPES
        # The first parameter, service_account_name, is the Email address created 
        # for the Service account. It must be the email address associated with 
        # the key that was created.
        if SignedJwtAssertionCredentials:
            # The old way of doing it.
            credential = SignedJwtAssertionCredentials(
              PRIVATE_KEY[project]['email'], pem_contents, scope = scopes)
        else:
            # The new way (improvement?)
            signer = oauth_crypt.Signer.from_string(pem_contents)
            credential = ServiceAccountCredentials(
                PRIVATE_KEY[project]['email'], signer, scopes = scopes)
            credential._private_key_pkcs8_pem = pem_contents

        http = httplib2.Http()
        http = credential.authorize(http)
        auth_cache.auths[project] = http
        return http

speedplane on Oct 11, 2016

Yes I am taking this on right now, the goal is to get a divorce from httplib2 and allow BYO transport with some nice defaults / examples.

dhermes on Jul 20, 2016

@eric-optimizely That error occurs when multiple threads read bytes from a payload at once, which gives invalid crypto bits. This likely means multiple threads have access to the same httplib2.Http object.

dhermes on Jul 18, 2016

httplib2shim should be a drop-in replacement. I encourage anyone looking for threadsafety in the near term to use and please report any bugs. It’ll help me abstract httplib2 from oauth2client.

On Tue, Feb 23, 2016 at 12:27 PM Danny Hermes notifications@github.com wrote:

Thanks for chiming in @bendemaree https://github.com/bendemaree! @jonparrott https://github.com/jonparrott is simultaneously working on httplib2shim https://github.com/GoogleCloudPlatform/httplib2shim and removing it as a dependency within oauth2client.

— Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/gcloud-python/issues/1214#issuecomment-187888560 .

theacodes on Feb 23, 2016