google-cloud-python: blob.download_to_filename fails with google-cloud-storage==1.3.0

Due to the notable implementation changes in google-cloud-storage 1.3.0, the blob.download_to_filename fails whatever google-cloud from 0.24.0 to 0.27.0 (latest)

Our prod server which using google-cloud==0.24.0 has been broken since last Saturday. I analyzed a bit the error. The problem is come from the upgraded package google-cloud-storage==1.3.0. We are using the google-cloud-storage==1.2.0.

I did several tests which figured out the following problems:

Problem 1: blob.download_to_filename failed when using google-cloud==0.27.0 (included google-cloud-storage==1.3.0 as default, defined in setup.py). Here is a part of stacktrace:

blob.download_to_filename(blob_local_path)
  File "/root/.local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 482, in download_to_filename
    self.download_to_file(file_obj, client=client)
  File "/root/.local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 464, in download_to_file
    self._do_download(transport, file_obj, download_url, headers)
  File "/root/.local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 418, in _do_download
    download.consume(transport)
  File "/root/.local/lib/python2.7/site-packages/google/resumable_media/requests/download.py", line 101, in consume
    self._write_to_stream(result)
  File "/root/.local/lib/python2.7/site-packages/google/resumable_media/requests/download.py", line 62, in _write_to_stream
    with response:
AttributeError: __exit__

Problem 2: blob.download_to_filename failed when using google-cloud==0.24.0 (included google-cloud-storage==1.3.0 as default, defined in setup.py). Here is a part of stacktrace:

File "dev_env/python_venv/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 482, in download_to_filename
    self.download_to_file(file_obj, client=client)
  File "dev_env/python_venv/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 464, in download_to_file
    self._do_download(transport, file_obj, download_url, headers)
  File "dev_env/python_venv/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 418, in _do_download
    download.consume(transport)
  File "dev_env/python_venv/local/lib/python2.7/site-packages/google/resumable_media/requests/download.py", line 96, in consume
    transport, method, url, **request_kwargs)
  File "dev_env/python_venv/local/lib/python2.7/site-packages/google/resumable_media/requests/_helpers.py", line 101, in http_request
    func, RequestsMixin._get_status_code, retry_strategy)
  File "dev_env/python_venv/local/lib/python2.7/site-packages/google/resumable_media/_helpers.py", line 146, in wait_and_retry
    response = func()
  File "dev_env/python_venv/local/lib/python2.7/site-packages/google_auth_httplib2.py", line 198, in request
    uri, method, body=body, headers=request_headers, **kwargs)
TypeError: request() got an unexpected keyword argument 'data'

So, why it happened? I realized that in the REQUIREMENTS of the package google-cloud which is defined in the file setup.py. An example for google-cloud==0.27.0 https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/setup.py#L67

• google-cloud==0.24.0 requires google-cloud-storage >= 1.0.0 and < 2.0dev
• google-cloud==0.25.0 requires google-cloud-storage >= 1.1.0 and < 2.0dev => https://github.com/GoogleCloudPlatform/google-cloud-python/commit/bd9fdfba07fac63a91847628613928a569250c0f#diff-2eeaed663bd0d25b7e608891384b7298R66
• google-cloud==0.25.1 requires google-cloud-storage >= 1.1.0 and < 1.2dev => https://github.com/GoogleCloudPlatform/google-cloud-python/commit/ad4b02c88088666e298aa53cd61687b461c09191#diff-2eeaed663bd0d25b7e608891384b7298R66
• google-cloud==0.26.0 requires google-cloud-storage >= 1.2.0 and < 2.0dev => https://github.com/GoogleCloudPlatform/google-cloud-python/commit/214aba604fbcaa9e4936fa1798efde050389992a#diff-2eeaed663bd0d25b7e608891384b7298R67
• google-cloud==0.26.1 requires google-cloud-storage >= 1.2.0 and < 1.3dev => https://github.com/GoogleCloudPlatform/google-cloud-python/commit/ea5245ad27e777c654448a259a9fa5f6bf5d758e#diff-2eeaed663bd0d25b7e608891384b7298R67
• google-cloud==0.27.0 requires google-cloud-storage >= 1.3.0 and < 1.4dev => https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/setup.py#L67

We can see that if we only define google-cloud==[0.24.0 to 0.27.0], “pip install” always try to install google-cloud-storage 1.3.0 which requires the dependencies google-cloud-core ~= 0.26. That’s why the bug happens.

Solution:

• Downgrade to google-cloud < 0.27.0 and force the requirements google-cloud-storage < 1.3.0
• The problem seems still there with google-cloud==0.27.0 and google-cloud-storage==1.3.0 (to check but at least it has been broken for my case, see Problem 2 above)

Suggestion: better manage the deps for google-cloud in setup.py.

Thanks.