jsonschema: Package version mechanism incompatible with AWS Lambda
Tested on AWS Lambda. Following code does not work on AWS Lambda due to lack of package manager.
It has beed added in Julian/jsonschema@b07d0f1d893f4a21008e0c8922959ddcf0614b73 commit.
Stacktrace I’m getting
Traceback (most recent call last):
File "/var/task/wsgi_handler.py", line 44, in import_app
wsgi_module = importlib.import_module(wsgi_fqn_parts[-1])
File "/var/lang/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/var/task/index.py", line 11, in <module>
(...)
from jsonschema.exceptions import ValidationError
File "/var/task/jsonschema/__init__.py", line 33, in <module>
__version__ = get_distribution(__name__).version
File "/var/task/pkg_resources/__init__.py", line 481, in get_distribution
dist = get_provider(dist)
File "/var/task/pkg_resources/__init__.py", line 357, in get_provider
return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
File "/var/task/pkg_resources/__init__.py", line 900, in require
needed = self.resolve(parse_requirements(requirements))
File "/var/task/pkg_resources/__init__.py", line 786, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'jsonschema' distribution was not found and is required by the application
START RequestId: xxx-xxx-xxx Version: $LATEST
module initialization error: Unable to import index.api
END RequestId: xxx-xxx-xxx
module initialization error
Unable to import index.api
Traceback (most recent call last):
File "/var/task/wsgi_handler.py", line 44, in import_app
wsgi_module = importlib.import_module(wsgi_fqn_parts[-1])
File "/var/lang/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/var/task/index.py", line 11, in <module>
(...)
from jsonschema.exceptions import ValidationError
File "/var/task/jsonschema/__init__.py", line 33, in <module>
__version__ = get_distribution(__name__).version
File "/var/task/pkg_resources/__init__.py", line 481, in get_distribution
dist = get_provider(dist)
File "/var/task/pkg_resources/__init__.py", line 357, in get_provider
return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
File "/var/task/pkg_resources/__init__.py", line 900, in require
needed = self.resolve(parse_requirements(requirements))
File "/var/task/pkg_resources/__init__.py", line 786, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'jsonschema' distribution was not found and is required by the application
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 1
- Comments: 20 (6 by maintainers)
Commits related to this issue
- Make the `jsonschema` package work Make necessary packaging tweaks (taken from the discussion at https://github.com/Julian/jsonschema/issues/584) to make the `jsonschema` package work when running in... — committed to oslokommune/okdata-pipeline by simenheg 4 years ago
- Squashed 'json/' changes from 597b1fb0d..b3c8672a3 b3c8672a3 Merge pull request #587 from json-schema-org/cross-draft-tests e2a681ac6 $comment -> comment 52eb27902 add draft 2020-12 -> draft 2019-09 ... — committed to python-jsonschema/jsonschema by Julian 2 years ago
@Julian Perhaps a blurb can be added to the docs related to deploying this library using Serverless?
If deploying with Serverless framework, ensure that
*.dist-infofiles are included in the deployed .zip file using something like this:Otherwise you be unable to import the library. By default, the
serverless-python-requirementsplugin ignores these files when usingslimmode.Can you please add a note to the front page of the documentation that this library is not Lambda-compatible? I just threw a few hours down the drain that could have easily been avoided were this prominently noted.
Also, on a side-note, labeling Lambda an “uncommon” platform is perhaps a bit out-of-touch. My current project is about 90% lambda code vs traditional server instances and I don’t believe my approach is unusual.
In any case, breaking backwards-compatibility without a really good reason is always a bad idea.
For my own use, I’ve pinned this package to 3.0.2 and I guess I will be forced to maintain a fork going forward.
@glyph If I had time to contribute, I would concur. Unfortunately I’m engaged in developing a startup where I am the only developer, so I don’t have the resources to commit to any of the above. This is doubly true since Python packaging isn’t an area I have any particular expertise in. I selected this library at a time it was compatible, only to have it break underneath of me during an update.
I’m certainly willing to file a bug report with AWS and will do so, but at this time, a fork appears to be the least time-consuming option, since cherry-picking commits to the library is much more straightforward than diving into the mess that is Python packaging.
I’d also add that of the Python libraries I utilize (which are numerous), this is the only one to cause any sort of problem, hence my skepticism about the value of the change.
@cwells The change which broke this was a migration from
vcversioner(abandoned; last released on 2016-04-12; maintained by an individual) toimportlib-metadataandsetuptools-scm(supported; recently released; maintained by the usual suspects in the python packaging cabal). Using actively maintained tools for packaging has a whole slew of benefits.Rather than maintaining a fork, your energy would probably be better spent contributing upstream fixes to those libraries to make them compatible with Lambda - or, better yet, asking the Lambda Python team to engage more actively with the packaging community and developing tooling integration that works with Python’s evolving PEP-specified toolchain that solves a whole bunch of problems for tons of other platforms, rather than mandating the somewhat odd “installed but not installed” artifact shape for Python which creates lots of these problems. I’m mostly familiar with the problems rather than the solutions in this space (which is why I avoid lambda) but it looks like there are others actively engaged with this problem, for example maybe the right place for the fix to land is in https://pypi.org/project/lambda-setuptools/ .
Ok, it seems we were barking up the wrong tree. The problem stems from the
serverless-python-requirementsplugin. I came across this https://github.com/UnitedIncome/serverless-python-requirements/issues/441Changing my serverless.yml to the following appears to work:
Kudos to @glyph for shaming me into a bit more effort.
I can sympathize with these sentiments. Based on my understanding of the Lambda & Python Packaging ecosystems, unless something changes to make them work together more smoothly, I would expect to see this sort of breakage propagate through more libraries as people adopt more modern packaging standards. (Probably if you just ignore it, eventually someone on either the Lambda or the open source side will fix it, since the platform will get less and less useful if lots of libraries start breaking it.)
Hi @danie1k , I also had the same error. In my case it was because I enabled slim mode in serverless configuration. Disabling the slim mode in serverless config fixed it for me. Hope it helps!