chalice: chalice deployment package missing libraries

When my Bitbucket pipeline runs chalice deploy --stage dev, the deployment works just fine. When I run chalice deploy --stage dev from my command line, the package deployed to AWS is missing most libraries… so I get errors like Unable to import module 'app': No module named 'phpserialize' when the Lambda is executed.

I’ve verified that the deployment packages differ by running deployments, downloading the deployment packages from AWS (using the Lambda console), and diff’ing the contents of the .zip files.

Clearly there’s some difference between my personal environment and the Bitbucket pipeline environment that causes a difference in how the deployment .zip is built, but I can’t figure it out. Here’s what I’ve tried:

• Changing the commands to/from chalice deploy && chalice deploy --stage dev. No affect.
• Rebuilding my vendor dependencies: In the past I’ve been able to ‘fix’ deployments missing modules by manually rebuilding the wheels in the vendor directory, but this hasn’t worked in the past few days: pip download x && pip wheel x-y.y.y.tar.gz && rm *.gz
• python/pip versions: Bitbucket was using python 3.6.3 & pip 9.0.1 and my personal machine was python 3.6.5 & pip 9.0.3. I changed my versions 3.6.3/9.0.1 and my deployment packages were still missing libraries. I changed the Bitbucket pipeline to 3.6.5/10.0.1 and the deployment was still perfect.
• I’ve also reviewed issues: #106, #155, #189

Anyone else seen this problem? It’s plaguing my office here and some of us have turned to pushing all changes to Bitbucket because we can’t trust a local chalice deploy to work.

Bitbucket OS: Linux 44faaabc-f42c-4803-993e-f72fa3365e1e 4.14.48-coreos-r2 #1 SMP Thu Jun 14 08:23:03 UTC 2018 x86_64 GNU/Linux Laptop OS: Darwin Macbook.localdomain 17.6.0 Darwin Kernel Version 17.6.0: Tue May 8 15:22:16 PDT 2018; root:xnu-4570.61.1~1/RELEASE_X86_64 x86_64

Here’s my requirements.txt file:

chalice==1.3.0
phpserialize==1.3
yoyo-migrations==5.1.5

Here’re the contents of my vendor directory:

MarkupSafe-1.0-cp36-cp36m-macosx_10_13_x86_64.whl	idna-2.7-py2.py3-none-any.whl
PyYAML-3.13-cp36-cp36m-macosx_10_13_x86_64.whl		iniherit-0.3.9-py3-none-any.whl
argh-0.26.2-py2.py3-none-any.whl			jmespath-0.9.3-py2.py3-none-any.whl
argparse-1.4.0-py2.py3-none-any.whl			pathtools-0.1.2-py3-none-any.whl
asn1crypto-0.24.0-py2.py3-none-any.whl			phpserialize-1.3-py3-none-any.whl
bidict-0.17.2-py3-none-any.whl				pycparser-2.18-py2.py3-none-any.whl
boto3-1.7.42-py2.py3-none-any.whl			python_dateutil-2.7.3-py2.py3-none-any.whl
botocore-1.10.42-py2.py3-none-any.whl			pytz-2018.4-py2.py3-none-any.whl
cffi-1.11.5-cp36-cp36m-macosx_10_6_intel.whl		s3transfer-0.1.13-py2.py3-none-any.whl
ciso8601-2.0.1.tar.gz					schemapi-0.3.0-py3-none-any.whl
credstash-1.14.0-py3-none-any.whl			simplejson-3.15.0-cp36-cp36m-macosx_10_13_x86_64.whl
cryptography-2.0.3-cp36-cp36m-macosx_10_6_intel.whl	six-1.11.0-py2.py3-none-any.whl
dateutils-0.6.6-py3-none-any.whl			typing-3.5.3.0-py3-none-any.whl
docutils-0.14-py3-none-any.whl				watchdog-0.8.3-cp36-cp36m-macosx_10_13_x86_64.whl
expiringdict-1.1.4-py3-none-any.whl

Here’s a sanitized copy of my Bitbucket pipeline file:

image: python:3.6.5

pipelines:
  branches:
    develop:
      - step:
          script:
            - export AWS_ACCESS_KEY_ID=$DEV_AWS_ACCESS_KEY_ID
            - export AWS_SECRET_ACCESS_KEY=$DEV_AWS_SECRET_ACCESS_KEY
            - pip install -r requirements.txt
            - pip install pytest
            - python -m pytest tests/
            - chalice deploy --stage dev
          deployment: staging

Here’s a diff of the deployment package (Bitbucket deployed vs. manually deployed):

Only in notifications-poller-dev-Bitbucket: MarkupSafe-1.0.dist-info
Only in notifications-poller-dev-Bitbucket: PyYAML-3.13.dist-info
Only in notifications-poller-dev-Bitbucket: _yaml.cpython-36m-x86_64-linux-gnu.so
Only in notifications-poller-dev-Bitbucket: bidict
Only in notifications-poller-dev-Bitbucket: bidict-0.17.2.dist-info
Only in notifications-poller-dev-Bitbucket: ciso8601-2.0.1.dist-info
Only in notifications-poller-dev-Bitbucket: ciso8601.cpython-36m-x86_64-linux-gnu.so
Only in notifications-poller-dev-Bitbucket: dateutils
Only in notifications-poller-dev-Bitbucket: dateutils-0.6.6.dist-info
Only in notifications-poller-dev-Bitbucket: iniherit
Only in notifications-poller-dev-Bitbucket: iniherit-0.3.9.dist-info
Only in notifications-poller-dev-Bitbucket: markupsafe
Only in notifications-poller-dev-Bitbucket: pathtools
Only in notifications-poller-dev-Bitbucket: pathtools-0.1.2.dist-info
Only in notifications-poller-dev-Bitbucket: phpserialize-1.3.dist-info
Only in notifications-poller-dev-Bitbucket: phpserialize.py
Only in notifications-poller-dev-Bitbucket: pycparser
Only in notifications-poller-dev-Bitbucket: pycparser-2.18.dist-info
Only in notifications-poller-dev-Bitbucket: schemapi
Only in notifications-poller-dev-Bitbucket: schemapi-0.3.0.dist-info
Only in notifications-poller-dev-Bitbucket: watchdog
Only in notifications-poller-dev-Bitbucket: watchdog-0.8.3.dist-info
Only in notifications-poller-dev-Bitbucket: yaml