boto3: Boto3 incompatible with python zip import
One of Python’s useful features is its ability to load modules from a .zip archive (PEP here), allowing you to package up multiple dependencies into a single file.
Boto breaks when trying to import it from a .zip, throwing:
File "C:\code sandbox\boto.zip\boto3\session.py", line 263, in client
File "C:\code sandbox\boto.zip\botocore\session.py", line 799, in create_client
File "C:\code sandbox\boto.zip\botocore\session.py", line 668, in _get_internal_component
File "C:\code sandbox\boto.zip\botocore\session.py", line 870, in get_component
File "C:\code sandbox\boto.zip\botocore\session.py", line 150, in create_default_resolver
File "C:\code sandbox\boto.zip\botocore\loaders.py", line 132, in _wrapper
File "C:\code sandbox\boto.zip\botocore\loaders.py", line 424, in load_data
botocore.exceptions.DataNotFoundError: Unable to load data for: endpoints
How to Reproduce:
- Create a .zip containing boto3 and botocore
- Create a .py file in the same directory as the zip (access keys removed for obvious reasons):
sys.path.insert(0, 'boto.zip')
import boto3
s3 = boto3.client('s3', aws_access_key_id='access_key', aws_secret_access_key='secret_key')
- Run
Tested on Python 3.6.7 boto3 1.9.39 botocore 1.12.39
About this issue
- Original URL
- State: open
- Created 6 years ago
- Reactions: 25
- Comments: 22 (6 by maintainers)
Commits related to this issue
- Use pkg_resources to eliminate __file__ Using pkg_resources allows for loading modules from a zip. Original author: Gábor Lipták <gliptak@gmail.com> See: - https://github.com/boto/boto3/issues/177... — committed to roverdotcom/botocore by philipkimmey 4 years ago
- Eliminate __file__ for importlib.resources See: - https://github.com/boto/boto3/issues/1770 - https://github.com/boto/botocore/pull/1969 — committed to roverdotcom/botocore by philipkimmey 4 years ago
- Eliminate __file__ for importlib.resources See: - https://github.com/boto/boto3/issues/1770 - https://github.com/boto/botocore/pull/1969 — committed to roverdotcom/botocore by philipkimmey 4 years ago
- Tests for __file__ zip loading See: - https://github.com/boto/boto3/issues/1770 - https://github.com/boto/botocore/pull/1969 — committed to roverdotcom/botocore by philipkimmey 4 years ago
- Make PySpark dependency distribution configurable Some python modules should not be distributed via PySpark for various reasons. For example, Boto doesn't work when distributed as a zip file (which i... — committed to polynote/polynote by jonathanindig 3 years ago
What are the odds of getting this implemented? Its preventing us from distributing boto3, which makes it very hard to provide a package that depends on it in PySpark.
Found a work around for this. You can pass spark conf args to have spark unzip the dependencies and include in path, something like this,
Worked with EMR 6.2.0 and Python 3.7.9