aws-lambda-builders: Unable to build Python lambda with package dependency in a private Git repository

Also getting this error, with version 1.40.1, when specifying a private git repo in requirements.txt. I am using python 3.8.

To follow up on my last comment, I’m quite certain that the issue I’m experiencing on my system is caused by Python’s sys.executable returning the path of the SAM CLI executable as opposed to the path of the Python interpreter when executing sam build. The stacktrace from sam build --debug reveals that the issue happens on the following lines:

cmd = [sys.executable, "-c", script, "--no-user-cfg", "egg_info", "--egg-base", "egg-info"]
p = subprocess.Popen(
    cmd, cwd=package_dir, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=self._osutils.original_environ()
)
_, stderr = p.communicate()
info_contents = self._osutils.get_directory_contents(egg_info_dir)
if p.returncode != 0:
    LOG.debug("Non zero rc (%s) from the setup.py egg_info command: %s", p.returncode, stderr)

Here is why I believe this is the case:

1. My output from sam build --debug includes the following log message:

Non zero rc (2) from the setup.py egg_info command: b"Usage: sam [OPTIONS] COMMAND [ARGS]…\nTry ‘sam -h’ for help.\n\nError: no such option: -c\n"

This, along with the stack trace included in the output, makes it apparent that the above call to subprocess.Popen() is trying to execute the command against the SAM CLI.

1. The Python docs indicate that sys.executable returns:

A string giving the absolute path of the executable binary for the Python interpreter, on systems where this makes sense. If Python is unable to retrieve the real path to its executable, sys.executable will be an empty string or None.

I would expect for the Python interpreter path to be undefined within a compiled binary such as the SAM CLI executable. What is odd, is that the sys.executable is not returning an empty string or None, as per the documentation, which would result in a different and perhaps clearer error message. However, it actually makes sense that sys.executable is returning the path to the compiled binary itself.

1. After following the instructions here to clone the aws-sam-cli repo and set up a local development environment, I was able to successfully build my stack using samdev build, thereby confirming that the command works as expected when not in a compiled binary.

I think that’s an accurate diagnosis of the issue. How to solve it? I’m not sure at this point. It seems that sys.executable may need to be substituted by a different approach for reliably obtaining the Python interpreter path.

I am getting the same error: Error: PythonPipBuilder:ResolveDependencies - Unable to retrieve name/version for package: with version SAM CLI, version 1.37.0 (latest) during the build process when specifying a dependency from a public github repo.

I am able to reproduce this issue. We will work on a solution for this issue.

Any news on this? We get the Error: PythonPipBuilder:ResolveDependencies - Unable to retrieve name/version for package:… and the workaround setting up a local development environment (in no 3 above) seems a little cumbersome.

In my case sam build expects a setup.py file to coexist with my pyproject.toml file before it will process the dependency.

I used the dephell utility to convert my pyproject.toml file to a setup.py file like this.

pipx install dephell
env --chdir=lib -- dephell deps convert --from=pyproject.toml --to=setup.py

With setup.py, it works. sam build gives this debug output. No error after the first pip download command.

Running workflow 'PythonPipBuilder'
 Running PythonPipBuilder:ResolveDependencies
calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch --exists-action i

Without setup.py, it fails. sam build gives this debug output that shows an error.

Running workflow 'PythonPipBuilder'
 Running PythonPipBuilder:ResolveDependencies
calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch --exists-action i
Non zero rc (1) from the setup.py egg_info command: b'Traceback (most recent call last):\n  File "<string>", line 1, in <module>\n  File "/var/lang/lib/python3.8/tokenize.py", line 392, in open\n    buffer = _builtin_open(filename, \'rb\')\nFileNotFoundError: [Errno 2] No such file or directory: \'/tmp/tmphnl2j1yp/aws_sso_lib/setup.py\'\n'
Using fallback location for PKG-INFO file in package directory: /tmp/tmphnl2j1yp/aws_sso_lib
PythonPipBuilder:ResolveDependencies raised unhandled exception

To add to the previous comment, I’m able to successfully build this package inside Docker using the following command:

docker run --rm -v ~/.ssh/known_hosts:/etc/ssh/ssh_known_hosts -v $SSH_AUTH_SOCK:/ssh-agent --env SSH_AUTH_SOCK=/ssh-agent -v $(pwd):/var/task "amazon/aws-sam-cli-build-image-python3.7" sam build

The SSH related params in the above command are necessary for Docker to use my local system’s SSH keys to fetch the sample-pkg package from my private repo. But the key thing to note is that sam build executes successfully within that Docker container.

This leads me to think that the problem is in some way operating-system related or configuration related. I’m on Ubuntu 20.04.3 LTS.

Here is another thing I noticed. Please note how one of the debug messages says:

Non zero rc (2) from the setup.py egg_info command: b"Usage: sam [OPTIONS] COMMAND [ARGS]...\nTry 'sam -h' for help.\n\nError: no such option: -c\n"

This error message is a result of executing this command.

The strange thing is that this command should be executing within the context of the Python interpreter, like so:

/usr/bin/python -c [other params]

But instead, it appears to be executing within the context of sam build as indicated by the log message:

Usage: sam [OPTIONS] COMMAND [ARGS]... 
Try 'sam -h' for help.

Error: no such option: -c

So it appears that sys.executable points to the SAM CLI executable on my system instead of the Python interpreter. Which is strange, because it doesn’t under normal circumstances. Here is the output from a simple test script:

~$ python -c "import sys; print(sys.executable)"
/usr/bin/python

This is how far I’ve gotten so far troubleshooting this. Maybe someone can pick up where I left off and identify the root cause.

Hi @yasntrk, could you provide more information on your error? If its the one above to do with setup.py files thats because currently sam build doesn’t support pyproject.toml files and requires a setup.py instead.

If its to do with the original issue could you try running the commands again but using one of the versions of sam linked here https://github.com/sidhujus/aws-sam-cli/actions/runs/7011049269

The version of sam there includes the fix that was previously mentioned in this thread and did fix the error for me. Just make sure to run the command as sam-beta build

@iainelder

here’s a setup.cfg file that works for me with the setup.py above. There’s other metadata fields that are optional you might want to include like author & description. (replaced private info with {{}} placeholders)

all the source code is located in “src/{{name}}” from the root of my repository where setup.cfg is located.

# setup.cfg

[metadata]
name = {{name}}
version = {{version}}
classifiers =
    Programming Language :: Python :: 3
    Programming Language :: Python :: 3.8
    Programming Language :: Python :: 3.9
    Programming Language :: Python :: 3.10

[options]
package_dir = 
    =src
packages = find:
python_requires = >=3.8
install_requires =
    {{your versioned pip requirements here e.g.:}}
    {{gql[aiohttp, websockets, requests]>=3.0,<4}}

[options.packages.find]
where = src

adding a setup.py to my private packages fixed the issue for me. I was using just setup.cfg and pyproject.toml - thinking it wasn’t needed b/c of PEP 517/518

I just needed to add the following to a setup.py to the project root (requires setup.cfg):

from setuptools import setup

if __name__ == "__main__":
    setup()

Any idea when this gets merged? We have this issue happening to us both locally and within our pipelines. It’s really frustrating to use SAM with this.

Any updates on this?

Contrary to what I wrote earlier, I’m still facing this issue (or a related one) with SAM CLI version 1.34.1. The error message changed. Please see attached output of sam build --debug below.

I’m on Ubuntu 20.04.3 LTS.

Here is what my requirements.txt looks like:

requests
git+ssh://git@github.com-p/Berciq/sample-pkg.git@v0.0.4

And here is the output from sam build --debug:

2021-11-17 18:42:36,429 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-11-17 18:42:36,429 | Using config file: samconfig.toml, config environment: default
2021-11-17 18:42:36,429 | Expand command line arguments to:
2021-11-17 18:42:36,429 | --template_file=/home/bartek/sandbox/sample-sam-app/template.yaml --build_dir=.aws-sam/build --cache_dir=.aws-sam/cache 
2021-11-17 18:42:36,516 | 'build' command is called
2021-11-17 18:42:36,520 | No Parameters detected in the template
2021-11-17 18:42:36,538 | 2 stacks found in the template
2021-11-17 18:42:36,539 | No Parameters detected in the template
2021-11-17 18:42:36,556 | 2 resources found in the stack 
2021-11-17 18:42:36,556 | No Parameters detected in the template
2021-11-17 18:42:36,573 | Found Serverless function with name='HelloWorldFunction' and CodeUri='hello_world/'
2021-11-17 18:42:36,573 | --base-dir is not presented, adjusting uri hello_world/ relative to /home/bartek/sandbox/sample-sam-app/template.yaml
2021-11-17 18:42:36,573 | No Parameters detected in the template
2021-11-17 18:42:36,591 | Instantiating build definitions
2021-11-17 18:42:36,595 | Same function build definition found, adding function (Previous: BuildDefinition(python3.7, /home/bartek/sandbox/sample-sam-app/hello_world, Zip, , f5cc0c06-8e35-4ddf-90c9-e24b594ef076, {}, {}, x86_64, []), Current: BuildDefinition(python3.7, /home/bartek/sandbox/sample-sam-app/hello_world, Zip, , 96314c54-ef15-494e-a22a-0fa2ab73ab93, {}, {}, x86_64, []), Function: Function(name='HelloWorldFunction', functionname='HelloWorldFunction', runtime='python3.7', memory=None, timeout=3, handler='app.lambda_handler', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='/home/bartek/sandbox/sample-sam-app/hello_world', environment=None, rolearn=None, layers=[], events={'HelloWorld': {'Type': 'Api', 'Properties': {'Path': '/hello', 'Method': 'get', 'RestApiId': 'ServerlessRestApi'}}}, metadata=None, inlinecode=None, codesign_config_arn=None, architectures=None, stack_path=''))
2021-11-17 18:42:36,596 | Building codeuri: /home/bartek/sandbox/sample-sam-app/hello_world runtime: python3.7 metadata: {} architecture: x86_64 functions: ['HelloWorldFunction']
2021-11-17 18:42:36,596 | Building to following folder /home/bartek/sandbox/sample-sam-app/.aws-sam/build/HelloWorldFunction
2021-11-17 18:42:36,597 | Loading workflow module 'aws_lambda_builders.workflows'
2021-11-17 18:42:36,599 | Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2021-11-17 18:42:36,600 | Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
2021-11-17 18:42:36,600 | Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
2021-11-17 18:42:36,601 | Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
2021-11-17 18:42:36,602 | Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
2021-11-17 18:42:36,603 | Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
2021-11-17 18:42:36,604 | Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
2021-11-17 18:42:36,605 | Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
2021-11-17 18:42:36,605 | Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
2021-11-17 18:42:36,606 | Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2021-11-17 18:42:36,617 | Running workflow 'PythonPipBuilder'
2021-11-17 18:42:36,617 | Running PythonPipBuilder:ResolveDependencies
2021-11-17 18:42:36,634 | calling pip download -r /home/bartek/sandbox/sample-sam-app/hello_world/requirements.txt --dest /tmp/tmpdpnjclo8 --exists-action i
2021-11-17 18:42:40,910 | Non zero rc (2) from the setup.py egg_info command: b"Usage: sam [OPTIONS] COMMAND [ARGS]...\nTry 'sam -h' for help.\n\nError: no such option: -c\n"
2021-11-17 18:42:40,910 | Using fallback location for PKG-INFO file in package directory: /tmp/tmpe8l5xmx9/sample-pkg
2021-11-17 18:42:40,911 | PythonPipBuilder:ResolveDependencies raised unhandled exception
Traceback (most recent call last):
  File "aws_lambda_builders/workflow.py", line 291, in run
  File "aws_lambda_builders/workflows/python_pip/actions.py", line 54, in execute
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 157, in build_dependencies
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 258, in build_site_packages
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 282, in _download_dependencies
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 366, in _download_all_dependencies
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 366, in <setcomp>
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 519, in __init__
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 561, in _calculate_name_and_version
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 632, in get_package_name_and_version
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 615, in _get_pkg_info_filepath
aws_lambda_builders.workflows.python_pip.packager.UnsupportedPackageError: Unable to retrieve name/version for package: sample-pkg

Build Failed
2021-11-17 18:42:40,912 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': 'ba585dc5-963d-40ed-b31c-d03ac9cc812d', 'installationId': '8b2c648e-3b81-425e-8ddd-9194c667999e', 'sessionId': '639e019f-22c6-4df5-b747-2799e8ce1cc1', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.7.10', 'samcliVersion': '1.34.1', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 4482, 'exitReason': 'WorkflowUnknownError', 'exitCode': 1}}]}
2021-11-17 18:42:41,826 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Error: PythonPipBuilder:ResolveDependencies - Unable to retrieve name/version for package: sample-pkg