serverless-python-requirements: serverless-python-requirements doesn't seem to remove .pyc, .pyo, or other files from lambda zip, irrespective of options specified in serverless.yml
It appears that the serverless-python-requirements plugin is exhibiting some odd behavior. It doesn’t appear to be slimming the contents of the package that’s deployed to Lambda, and includes the entire contents of the venv directory in the package, irrespective of whether or not athe zip option was selected.
Here’s what our serverless.yml looks like:
plugins:
- serverless-python-requirements
custom:
pythonRequirements:
useDownloadCache: false
useStaticCache: false
invalidateCaches: true
zip: true
slim: true
slimPatternsAppendDefaults: false # we need to overwrite default dist-info
slimPatterns:
# "normal" exclusions:
- '**/tests/*'
# default dist-info with !(lib_name*) - matches only if name is not present:
- "**/!(lib_name*)*.dist-info/*"
# the rest of defaults:
- '**/*.py[c|o]'
- '**/__pycache__*'
stage: 'dev'
package:
individually: true
provider:
name: aws
runtime: python3.8
timeout: 20
region: us-west-2
functions:
train-model:
handler: src/python/aws/lambda/train_model.handler
The reason for the slimPatterns is the following comment:
I have found a workaround, sadly it’s a bit inconvenient to use At least it works
slim: true
slimPatternsAppendDefaults: false # we need to overwrite default dist-info
slimPatterns:
# "normal" exclusions:
- '**/tests/*'
# default dist-info with !(lib_name*) - matches only if name is not present:
- "**/!(lib_name*)*.dist-info/*"
# the rest of defaults:
- '**/*.py[c|o]'
- '**/__pycache__*'
Okay fine, but even that doesn’t seem to work. Now with the zip: true above, I do get the .requirements.zip. Woo hoo! It’s ~59 megs. BUT, here’s my Serverless package:
$ ls -alh train-model.zip
-rw-r--r-- 1 ryancammer ryancammer 281M Dec 8 20:27 train-model.zip
And why is it so huge?
$ unzip train-model.zip
inflating: venv/lib64/python3.8/site-packages/websocket/tests/__pycache__/test_cookiejar.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/websocket/tests/__pycache__/test_websocket.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/websocket/tests/data/header01.txt
inflating: venv/lib64/python3.8/site-packages/websocket/tests/data/header02.txt
inflating: venv/lib64/python3.8/site-packages/websocket/tests/test_cookiejar.py
inflating: venv/lib64/python3.8/site-packages/websocket/tests/test_websocket.py
inflating: venv/lib64/python3.8/site-packages/websocket_client-0.56.0.dist-info/INSTALLER
inflating: venv/lib64/python3.8/site-packages/websocket_client-0.56.0.dist-info/LICENSE
inflating: venv/lib64/python3.8/site-packages/websocket_client-0.56.0.dist-info/METADATA
inflating: venv/lib64/python3.8/site-packages/websocket_client-0.56.0.dist-info/RECORD
inflating: venv/lib64/python3.8/site-packages/websocket_client-0.56.0.dist-info/WHEEL
inflating: venv/lib64/python3.8/site-packages/websocket_client-0.56.0.dist-info/top_level.txt
inflating: venv/lib64/python3.8/site-packages/yaml/__init__.py
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/__init__.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/composer.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/constructor.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/cyaml.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/dumper.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/emitter.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/error.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/events.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/loader.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/nodes.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/parser.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/reader.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/representer.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/resolver.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/scanner.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/serializer.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/tokens.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/composer.py
inflating: venv/lib64/python3.8/site-packages/yaml/constructor.py
inflating: venv/lib64/python3.8/site-packages/yaml/cyaml.py
inflating: venv/lib64/python3.8/site-packages/yaml/dumper.py
inflating: venv/lib64/python3.8/site-packages/yaml/emitter.py
inflating: venv/lib64/python3.8/site-packages/yaml/error.py
inflating: venv/lib64/python3.8/site-packages/yaml/events.py
inflating: venv/lib64/python3.8/site-packages/yaml/loader.py
inflating: venv/lib64/python3.8/site-packages/yaml/nodes.py
inflating: venv/lib64/python3.8/site-packages/yaml/parser.py
inflating: venv/lib64/python3.8/site-packages/yaml/reader.py
inflating: venv/lib64/python3.8/site-packages/yaml/representer.py
inflating: venv/lib64/python3.8/site-packages/yaml/resolver.py
inflating: venv/lib64/python3.8/site-packages/yaml/scanner.py
inflating: venv/lib64/python3.8/site-packages/yaml/serializer.py
inflating: venv/lib64/python3.8/site-packages/yaml/tokens.py
So basically, the plugin is including the venv directory anyway, .pyc and all. Using dockerizePip: true, this is the result on OSX:
✗ ls -h
-rw-r--r-- 1 cory staff 64M Dec 8 20:46 .requirements.zip
-rw-r--r-- 1 cory staff 76M Dec 8 20:46 train-model.zip
Again, that train-model.zip file was chock full of .pyc files. I’ve tried with and without the slimPatterns, with and without zip: true, but no matter what, the end result is a massive lambda zip that’s full of .pyc files.
About this issue
- Original URL
- State: open
- Created 7 years ago
- Comments: 15 (4 by maintainers)
@bsamuel-ui
I am also hitting this issue. I don’t know if it helps but can describe how it happened:
And this point everything is working fine 🚀
slim: true):And I noticed that I don’t have Docker running anymore (I am using macOS +
dockerizePip: non-linux) and also useless files don’t get removed from the ZIP according to the documentation ofslim: trueoption.serverless-python-requirementsplugin looks at changes inrequirements.txtand makes a decision to use a cached version from the previous deployment or not. And this is incorrect because it completely ignores any changes to config inserverless.yml.Exactly this command helped me (and I have to do it every time I change config of the plugin):
serverless requirements cleanCache. Docker got running and also useless files got removed byslim: trueoption and made mypythonRequirements.zipfile5 MBlighter 👍Also, it’s a pity that this command isn’t documented in README.md… I found it wandering in the source code.
@AndrewFarley Maybe because you didn’t have cache on your machine for that project you couldn’t reproduce the issue here
Thanks @GarrisonD,
serverless requirements cleanCachedid the trick!I had the inverse of this issue show its ugly head. I was using slim: true and it was removing the dist-info directories I needed because Langchain in their newer version used a new version of metadata package that relies on the .dist-info directories.
See issue #441 and Freek Wiekmeijer comments at the bottom.
Despite the new changes to exclude the .dist-info directories from being removed, this kept excluding them due to the cached dependencies this was leaning on. Running
serverless requirements cleanCacheif I make any requirements.txt changes has helped thus far. It does take a lot longer to run deployment but it has solved my issue thus far. Thanks @GarrisonD !!!I hope this information helps someone save time. Time wasted: 11 hours. (Mostly waiting on serverless deploy to do its thing.)
This was my exact error
serverless requirements cleanCachedefinitely should be mentioned in the README. without this commandslimdidn’t worked for me and my several teammates. so we just never usedslimat all…@dashmug I tried your repo, the zip looks good to me. It slimmed perfectly. Admittedly, I did modify the serverless.yml a bit though. Also, I didn’t use the version of serverless from your package.json, I used the latest release. My serverless.yml has…
@AndrewFarley I’ll post a repro repo later today.
I’m having this issue as well.
slim: truedoes not seem to do anything.