kedro: Error to deploy spaceflights tutorial in AWS Steps Function
Description
I tried to follow the tutorial to deploy the spaceflights template project on AWS Step Functions (https://antonymilneqb.github.io/kedro-docs/10_deployment/10_aws_step_functions.html). I follow the steps written in the tutorial but I get an error when I try to run the service.
Context
Steps to Reproduce
- python -m venv venv & source venv/bin/activate
- pip install kedro==0.17.1 & kedro info
- kedro new --starter=spaceflights
- cd space (project folder)
- create new configuration environment to prepare a compatible data catalog (conf/aws/catalog.yml) with the aws file paths
- pip install -r src/requirements.txt
- kedro package
- create lambda_handler.py with a copy the code available in the tutorial and changed the row 7 in
configure_project("space") - create Dockerfile with a copy the code available in the tutorial
- push the project on ECR
- pip install deploy_requirements.txt
- go in the script deploy.py with a copy the code available in the tutorial and changed the rows 31 to
s3_data_bucket_name = ("<bucket-name>")with my bucket name - create the file cdk.json with a copy the code available in the tutorial & run cdk deploy
- go to aws and try to run the step function, which is deployed but it throws an error
Expected Result
The service step function starts the execution correctly.
Actual Result
The step functions throws an error in the first step, it seems that the lambda can’t find the functions implemented.
– Error Received.
{
"errorMessage": "[Errno 38] Function not implemented",
"errorType": "OSError",
"requestId": "xxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"stackTrace": [
" File \"/home/app/lambda_handler.py\", line 7, in handler\n configure_project(\"space\")\n",
" File \"/home/app/kedro/framework/project/__init__.py\", line 211, in configure_project\n settings.configure(settings_module)\n",
" File \"/home/app/dynaconf/base.py\", line 223, in configure\n self._wrapped = Settings(settings_module=settings_module, **kwargs)\n",
" File \"/home/app/dynaconf/base.py\", line 271, in __init__\n self.validators.validate()\n",
" File \"/home/app/dynaconf/validator.py\", line 318, in validate\n validator.validate(self.settings)\n",
" File \"/home/app/dynaconf/validator.py\", line 172, in validate\n self._validate_items(settings, settings.current_env)\n",
" File \"/home/app/kedro/framework/project/__init__.py\", line 58, in _validate_items\n super()._validate_items(settings, env)\n",
" File \"/home/app/dynaconf/validator.py\", line 183, in _validate_items\n self.default(settings, self)\n",
" File \"/home/app/kedro/framework/project/__init__.py\", line 49, in validator_func\n return getattr(importlib.import_module(module), class_name)\n",
" File \"/usr/local/lib/python3.8/importlib/__init__.py\", line 127, in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n",
" File \"<frozen importlib._bootstrap>\", line 1014, in _gcd_import\n",
" File \"<frozen importlib._bootstrap>\", line 991, in _find_and_load\n",
" File \"<frozen importlib._bootstrap>\", line 961, in _find_and_load_unlocked\n",
" File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\n",
" File \"<frozen importlib._bootstrap>\", line 1014, in _gcd_import\n",
" File \"<frozen importlib._bootstrap>\", line 991, in _find_and_load\n",
" File \"<frozen importlib._bootstrap>\", line 975, in _find_and_load_unlocked\n",
" File \"<frozen importlib._bootstrap>\", line 671, in _load_unlocked\n",
" File \"<frozen importlib._bootstrap_external>\", line 843, in exec_module\n",
" File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\n",
" File \"/home/app/kedro/framework/session/__init__.py\", line 32, in <module>\n from .session import KedroSession, get_current_session\n",
" File \"/home/app/kedro/framework/session/session.py\", line 54, in <module>\n from kedro.framework.session.store import BaseSessionStore\n",
" File \"/home/app/kedro/framework/session/store.py\", line 73, in <module>\n class ShelveStore(BaseSessionStore):\n",
" File \"/home/app/kedro/framework/session/store.py\", line 76, in ShelveStore\n _lock = Lock()\n",
" File \"/usr/local/lib/python3.8/multiprocessing/context.py\", line 68, in Lock\n return Lock(ctx=self.get_context())\n",
" File \"/usr/local/lib/python3.8/multiprocessing/synchronize.py\", line 162, in __init__\n SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)\n",
" File \"/usr/local/lib/python3.8/multiprocessing/synchronize.py\", line 57, in __init__\n sl = self._semlock = _multiprocessing.SemLock(\n"
]
}
Your Environment
Include as many relevant details about the environment in which you experienced the bug:
- Kedro version used: 0.17.1
- Python version used: 3.8.10
- Operating system and version: Linux Ubuntu 20.04 LTS
- cdk version used: 1.128.0
- npm version used: 6.14.15
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 15 (9 by maintainers)
Commits related to this issue
- [KED-2426] Update spaceflights tutorial documentation (#1006) — committed to Galileo-Galilei/kedro by antonymilne 3 years ago
Hello @MerelTheisenQB My tools version:
This issue comes from the import of
ShelveStorewhich usesmultiprocessing.lockwhich isn’t supported on AWS Lambda functions.ShelveStorehas been moved out to its own module in #1614, therefore, will not be imported, which will resolve this issue in the upcoming release ofKedro 0.18.4.also, the tutorial should be updated to use cdk v2.