aws-lambda-builders: Missing PKG-INFO in case of python-pip workflow

Description: There are cases where there is no PKG-INFO file in the package’s egg-info directory. Instead it exists in the package’s root folder. I assume this is the case for those packages which doesn’t have binary distribution. The problem rises with SAM CLI when I try to build my lambda app.

Steps to reproduce the issue:

  1. Download the following zip: reprosamissue.zip - it contains a sample project
  2. Unzip it in a folder.
  3. Create a virtual env.
  4. Activate virtual env in a terminal
  5. Navigate to the project folder
  6. cd hello-world
  7. pip install -r requirements.txt
  8. cd ..
  9. sam build -b .\dist

Observed result:

PS C:\Users\pushrbx\PycharmProjects\reprosamissue> sam build --debug -b .\dist
2019-05-16 15:26:05 Using SAM Template at C:\Users\pushrbx\PycharmProjects\reprosamissue\template.yaml
2019-05-16 15:26:05 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-05-16 15:26:05 Changing event name from before-call.apigateway to before-call.api-gateway
2019-05-16 15:26:05 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-05-16 15:26:05 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-05-16 15:26:05 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-05-16 15:26:05 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-05-16 15:26:05 Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section 2019-05-16 15:26:05 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-05-16 15:26:05 Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-05-16 15:26:05 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-05-16 15:26:05 Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2019-05-16 15:26:05 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-05-16 15:26:05 Changing event name from before-call.apigateway to before-call.api-gateway
2019-05-16 15:26:05 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-05-16 15:26:05 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-05-16 15:26:05 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-05-16 15:26:05 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-05-16 15:26:05 Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section 2019-05-16 15:26:05 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-05-16 15:26:05 Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-05-16 15:26:05 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-05-16 15:26:05 Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2019-05-16 15:26:05 'build' command is called
2019-05-16 15:26:05 No Parameters detected in the template
2019-05-16 15:26:05 2 resources found in the template
2019-05-16 15:26:05 Found Serverless function with name='HelloWorldFunction' and CodeUri='hello_world/'
2019-05-16 15:26:05 Building resource 'HelloWorldFunction'
2019-05-16 15:26:05 Loading workflow module 'aws_lambda_builders.workflows'
2019-05-16 15:26:05 Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2019-05-16 15:26:05 Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
2019-05-16 15:26:05 Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
2019-05-16 15:26:05 Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
2019-05-16 15:26:05 Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
2019-05-16 15:26:05 Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
2019-05-16 15:26:05 Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
2019-05-16 15:26:05 Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
2019-05-16 15:26:05 Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2019-05-16 15:26:05 Running workflow 'PythonPipBuilder'
2019-05-16 15:26:05 Running PythonPipBuilder:ResolveDependencies
2019-05-16 15:26:05 calling pip download -r C:\Users\pushrbx\PycharmProjects\reprosamissue\hello_world\requirements.txt --dest C:\Users\TAMAS~1.SZE\AppData\Local\Temp\tmp2hcw4uih
2019-05-16 15:26:14 PythonPipBuilder:ResolveDependencies raised unhandled exception
Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflow.py", line 248, in run
    action.execute()
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\actions.py", line 39, in execute
    self.scratch_dir
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 143, in build_dependencies
    requirements_path, artifacts_dir_path, scratch_dir_path)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 209, in build_site_packages
    scratch_directory, requirements_filepath)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 234, in _download_dependencies
    requirements_filename, directory)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 332, in _download_all_dependencies
    in self._osutils.get_directory_contents(directory)}
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 331, in <setcomp>
    deps = {Package(directory, filename) for filename
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 437, in __init__
    self._name, self._version = self._calculate_name_and_version()
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 480, in _calculate_name_and_version
    sdist_path)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 541, in get_package_name_and_version
    pkg_info_filepath = self._generate_egg_info(package_dir)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 524, in _generate_egg_info
    egg_info_dir, info_contents[0], 'PKG-INFO')
IndexError: list index out of range

Build Failed
Error: PythonPipBuilder:ResolveDependencies - list index out of range

Expected result:

C:\Users\pushrbx\PycharmProjects\reprosamissue> sam build -b .\dist
2019-05-16 15:17:06 Building resource 'HelloWorldFunction'
2019-05-16 15:17:06 Running PythonPipBuilder:ResolveDependencies
2019-05-16 15:17:16 Running PythonPipBuilder:CopySource

Build Succeeded

Built Artifacts  : dist
Built Template   : dist\template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke -t dist\template.yaml
[*] Package: sam package --s3-bucket <yourbucket> --template-file dist\template.yaml

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

OS: Windows 10 Attempted in a powershell console. Python version: 3.6 (for the virtualenv)

Possible fix The following change completely cured the problem: https://github.com/pushrbx/aws-lambda-builders/commit/5449ac38f54de50b9c780a8c29c861d4322101b4

I’m not clear on the environment requirements to run builds with SAM CLI but this change only worked after I’ve installed the wheel package in my virtualenv.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 18
  • Comments: 27 (2 by maintainers)

Commits related to this issue

Most upvoted comments

I’m also hitting this problem.

Specifically, it happens in a Python 3.6 SAM project, when running sam build with --use-container, on the dependency cryptography==3.4.6. When I don’t use --use-container, the build works (on macOS 10.15.7).

I dug into it a little bit more and it looks like, when trying to _generate_egg_info in a containerized SAM build for the cryptography 3.4.6 package, the setup.py invocation here ( https://github.com/aws/aws-lambda-builders/blob/develop/aws_lambda_builders/workflows/python_pip/packager.py#L498 ) is failing. Grabbing the stdout and stderr output from the command, it’s failing because setuptools_rust is missing. Making sure that setuptools_rust was installed in the container solved the problem.

I’m assuming that I didn’t hit the error on macOS because the ‘manylinux’ package is used instead, thereby skipping the need for Rust compilation.

Since the cryptography package is a big deal, does it make sense to install setuptools_rust in the container image so that trying to _generate_egg_info for this package doesn’t fail? I’d submit a pull request but I can’t find the Dockerfile for amazon/aws-sam-cli-build-image-python3.6.

Furthermore, it seems like it would be a good idea to check the returncode on the setup.py command and fail or show some debug output if it didn’t succeed, since it probably means that we won’t be able to locate the PKG-INFO.

+2
robertsigler on Mar 12, 2021

Same here… any update on the solution?

+2
arnathan2k on Dec 24, 2020

Closing as this was fixed in Lambda Builders 1.7.0. Feel free to re-open if this is not the case.

+1
hawflau on Mar 30, 2022

Yes, it will.

+1
prenx4x on Aug 30, 2021
Read more comments on GitHub
← aws-iot-device-sdk-python-v2: Publish future raises AWS_ERROR_MQTT_CONNECTION_DESTROYED
aws-lambda-builders: Unable to build Python lambda with package dependency in a private Git repository →