boto3: Boto3 incompatible with python zip import

One of Python’s useful features is its ability to load modules from a .zip archive (PEP here), allowing you to package up multiple dependencies into a single file.

Boto breaks when trying to import it from a .zip, throwing:

  File "C:\code sandbox\boto.zip\boto3\session.py", line 263, in client
  File "C:\code sandbox\boto.zip\botocore\session.py", line 799, in create_client
  File "C:\code sandbox\boto.zip\botocore\session.py", line 668, in _get_internal_component
  File "C:\code sandbox\boto.zip\botocore\session.py", line 870, in get_component
  File "C:\code sandbox\boto.zip\botocore\session.py", line 150, in create_default_resolver
  File "C:\code sandbox\boto.zip\botocore\loaders.py", line 132, in _wrapper
  File "C:\code sandbox\boto.zip\botocore\loaders.py", line 424, in load_data
botocore.exceptions.DataNotFoundError: Unable to load data for: endpoints

How to Reproduce:

  1. Create a .zip containing boto3 and botocore
  2. Create a .py file in the same directory as the zip (access keys removed for obvious reasons):
sys.path.insert(0, 'boto.zip')
import boto3

s3 = boto3.client('s3', aws_access_key_id='access_key', aws_secret_access_key='secret_key')
  1. Run

Tested on Python 3.6.7 boto3 1.9.39 botocore 1.12.39

About this issue

  • Original URL
  • State: open
  • Created 6 years ago
  • Reactions: 25
  • Comments: 22 (6 by maintainers)

Commits related to this issue

Most upvoted comments

What are the odds of getting this implemented? Its preventing us from distributing boto3, which makes it very hard to provide a package that depends on it in PySpark.

Found a work around for this. You can pass spark conf args to have spark unzip the dependencies and include in path, something like this,

	--conf spark.yarn.dist.archives=s3://<bucket+path>/sparkapp.zip#deps" \
	--conf spark.yarn.appMasterEnv.PYTHONPATH=deps" \
	--conf spark.executorEnv.PYTHONPATH=deps" \

Worked with EMR 6.2.0 and Python 3.7.9