aws-sam-cli: sam build fails when required to install some 3rd party libraries in requirements.txt

Description

I’m trying to build a lambda using aws sam cli which requires the ‘paramiko’ package. However when I add it to the requirements.txt file in the normal way it fails at the build step with the following error output:

`2019-08-28 10:26:09 Building resource ‘HelloWorldFunction’ 2019-08-28 10:26:09 Running PythonPipBuilder:ResolveDependencies

Build Failed Error: PythonPipBuilder:ResolveDependencies - {pynacl==1.3.0(wheel), bcrypt==3.1.7(wheel), cryptography==2.7(wheel)}`

Steps to reproduce

sam init --runtime python3.6 --name TestParamiko

in requirements.txt add:

paramiko

cd TestParamiko

sam build

Observed result (output with --debug)

2019-08-28 10:27:20 Using SAM Template at C:\Users\ww\workspace\AWS-SAM-CLI-Lambdas-and-Layers\TestParamiko\template.yaml 2019-08-28 10:27:20 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane 2019-08-28 10:27:20 Changing event name from before-call.apigateway to before-call.api-gateway 2019-08-28 10:27:20 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict 2019-08-28 10:27:20 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration 2019-08-28 10:27:20 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53 2019-08-28 10:27:20 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search 2019-08-28 10:27:20 Changing event name from docs..autoscaling.CreateLaunchConfiguration.complete-section to docs..auto-scaling.CreateLaunchConfiguration.complete-section 2019-08-28 10:27:20 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask 2019-08-28 10:27:20 Changing event name from docs..logs.CreateExportTask.complete-section to docs..cloudwatch-logs.CreateExportTask.complete-section 2019-08-28 10:27:20 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search 2019-08-28 10:27:20 Changing event name from docs..cloudsearchdomain.Search.complete-section to docs..cloudsearch-domain.Search.complete-section 2019-08-28 10:27:20 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane 2019-08-28 10:27:20 Changing event name from before-call.apigateway to before-call.api-gateway 2019-08-28 10:27:20 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict 2019-08-28 10:27:20 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration 2019-08-28 10:27:20 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53 2019-08-28 10:27:20 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search 2019-08-28 10:27:20 Changing event name from docs..autoscaling.CreateLaunchConfiguration.complete-section to docs..auto-scaling.CreateLaunchConfiguration.complete-section 2019-08-28 10:27:20 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask 2019-08-28 10:27:20 Changing event name from docs..logs.CreateExportTask.complete-section to docs..cloudwatch-logs.CreateExportTask.complete-section 2019-08-28 10:27:20 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search 2019-08-28 10:27:20 Changing event name from docs..cloudsearchdomain.Search.complete-section to docs..cloudsearch-domain.Search.complete-section 2019-08-28 10:27:20 ‘build’ command is called 2019-08-28 10:27:20 No Parameters detected in the template 2019-08-28 10:27:20 2 resources found in the template 2019-08-28 10:27:20 Found Serverless function with name=‘HelloWorldFunction’ and CodeUri=‘hello_world/’ 2019-08-28 10:27:21 Building resource ‘HelloWorldFunction’ 2019-08-28 10:27:21 Loading workflow module ‘aws_lambda_builders.workflows’ 2019-08-28 10:27:21 Registering workflow ‘PythonPipBuilder’ with capability ‘Capability(language=‘python’, dependency_manager=‘pip’, application_framework=None)’ 2019-08-28 10:27:21 Registering workflow ‘NodejsNpmBuilder’ with capability ‘Capability(language=‘nodejs’, dependency_manager=‘npm’, application_framework=None)’ 2019-08-28 10:27:21 Registering workflow ‘RubyBundlerBuilder’ with capability ‘Capability(language=‘ruby’, dependency_manager=‘bundler’, application_framework=None)’ 2019-08-28 10:27:21 Registering workflow ‘GoDepBuilder’ with capability ‘Capability(language=‘go’, dependency_manager=‘dep’, application_framework=None)’ 2019-08-28 10:27:21 Registering workflow ‘GoModulesBuilder’ with capability ‘Capability(language=‘go’, dependency_manager=‘modules’, application_framework=None)’ 2019-08-28 10:27:21 Registering workflow ‘JavaGradleWorkflow’ with capability ‘Capability(language=‘java’, dependency_manager=‘gradle’, application_framework=None)’ 2019-08-28 10:27:21 Registering workflow ‘JavaMavenWorkflow’ with capability ‘Capability(language=‘java’, dependency_manager=‘maven’, application_framework=None)’ 2019-08-28 10:27:21 Registering workflow ‘DotnetCliPackageBuilder’ with capability ‘Capability(language=‘dotnet’, dependency_manager=‘cli-package’, application_framework=None)’ 2019-08-28 10:27:21 Found workflow ‘PythonPipBuilder’ to support capabilities ‘Capability(language=‘python’, dependency_manager=‘pip’, application_framework=None)’ 2019-08-28 10:27:21 Running workflow ‘PythonPipBuilder’ 2019-08-28 10:27:21 Running PythonPipBuilder:ResolveDependencies 2019-08-28 10:27:21 calling pip download -r C:\Users\ww\workspace\AWS-SAM-CLI-Lambdas-and-Layers\TestParamiko\hello_world\requirements.txt --dest C:\Users\ww\AppData\Local\Temp\tmpbjd2m3hp 2019-08-28 10:27:31 Full dependency closure: {paramiko==2.6.0(wheel), pycparser==2.19(sdist), bcrypt==3.1.7(wheel), certifi==2019.6.16(wheel), requests==2.22.0(wheel), cffi==1.12.3(wheel), urllib3==1.25.3(wheel), chardet==3.0.4(wheel), asn1crypto==0.24.0(wheel), cryptography==2.7(wheel), pynacl==1.3.0(wheel), idna==2.8(wheel), six==1.12.0(wheel)} 2019-08-28 10:27:31 initial compatible: {paramiko==2.6.0(wheel), requests==2.22.0(wheel), certifi==2019.6.16(wheel), urllib3==1.25.3(wheel), chardet==3.0.4(wheel), asn1crypto==0.24.0(wheel), idna==2.8(wheel), six==1.12.0(wheel)} 2019-08-28 10:27:31 initial incompatible: {pycparser==2.19(sdist), bcrypt==3.1.7(wheel), cryptography==2.7(wheel), pynacl==1.3.0(wheel), cffi==1.12.3(wheel)} 2019-08-28 10:27:31 Downloading missing wheels: {pycparser==2.19(sdist), bcrypt==3.1.7(wheel), cryptography==2.7(wheel), pynacl==1.3.0(wheel), cffi==1.12.3(wheel)} 2019-08-28 10:27:31 calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp36m --dest C:\Users\ww\AppData\Local\Temp\tmpbjd2m3hp pycparser==2.19 2019-08-28 10:27:34 calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp36m --dest C:\Users\ww\AppData\Local\Temp\tmpbjd2m3hp bcrypt==3.1.7 2019-08-28 10:27:36 calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp36m --dest C:\Users\ww\AppData\Local\Temp\tmpbjd2m3hp cryptography==2.7 2019-08-28 10:27:39 calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp36m --dest C:\Users\ww\AppData\Local\Temp\tmpbjd2m3hp pynacl==1.3.0 2019-08-28 10:27:41 calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp36m --dest C:\Users\ww\AppData\Local\Temp\tmpbjd2m3hp cffi==1.12.3 2019-08-28 10:27:43 compatible wheels after second download pass: {paramiko==2.6.0(wheel), certifi==2019.6.16(wheel), requests==2.22.0(wheel), cffi==1.12.3(wheel), urllib3==1.25.3(wheel), chardet==3.0.4(wheel), asn1crypto==0.24.0(wheel), idna==2.8(wheel), six==1.12.0(wheel)} 2019-08-28 10:27:43 Build missing wheels from sdists (C compiling True): {pycparser==2.19(sdist)} 2019-08-28 10:27:43 calling pip wheel --no-deps --wheel-dir C:\Users\ww\AppData\Local\Temp\tmpbjd2m3hp C:\Users\ww\AppData\Local\Temp\tmpbjd2m3hp\pycparser-2.19.tar.gz 2019-08-28 10:27:50 compatible after building wheels (no C compiling): {paramiko==2.6.0(wheel), pycparser==2.19(wheel), certifi==2019.6.16(wheel), requests==2.22.0(wheel), cffi==1.12.3(wheel), urllib3==1.25.3(wheel), chardet==3.0.4(wheel), asn1crypto==0.24.0(wheel), idna==2.8(wheel), six==1.12.0(wheel)} 2019-08-28 10:27:50 Build missing wheels from sdists (C compiling False): set() 2019-08-28 10:27:50 compatible after building wheels (C compiling): {paramiko==2.6.0(wheel), pycparser==2.19(wheel), certifi==2019.6.16(wheel), requests==2.22.0(wheel), cffi==1.12.3(wheel), urllib3==1.25.3(wheel), chardet==3.0.4(wheel), asn1crypto==0.24.0(wheel), idna==2.8(wheel), six==1.12.0(wheel)} 2019-08-28 10:27:50 Final compatible: {paramiko==2.6.0(wheel), pycparser==2.19(wheel), certifi==2019.6.16(wheel), requests==2.22.0(wheel), cffi==1.12.3(wheel), urllib3==1.25.3(wheel), chardet==3.0.4(wheel), asn1crypto==0.24.0(wheel), idna==2.8(wheel), six==1.12.0(wheel)} 2019-08-28 10:27:50 Final incompatible: {cffi==1.12.3(wheel), cryptography==2.7(wheel), bcrypt==3.1.7(wheel), pynacl==1.3.0(wheel)} 2019-08-28 10:27:50 Final missing wheels: {bcrypt==3.1.7(wheel), cryptography==2.7(wheel), pynacl==1.3.0(wheel)} 2019-08-28 10:27:50 PythonPipBuilder:ResolveDependencies failed Traceback (most recent call last): File “c:\users\ww\appdata\local\programs\python\python36\lib\site-packages\aws_lambda_builders\workflows\python_pip\actions.py”, line 39, in execute self.scratch_dir File “c:\users\ww\appdata\local\programs\python\python36\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py”, line 143, in build_dependencies requirements_path, artifacts_dir_path, scratch_dir_path) File “c:\users\ww\appdata\local\programs\python\python36\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py”, line 212, in build_site_packages raise MissingDependencyError(packages_without_wheels) aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {bcrypt==3.1.7(wheel), cryptography==2.7(wheel), pynacl==1.3.0(wheel)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File “c:\users\ww\appdata\local\programs\python\python36\lib\site-packages\aws_lambda_builders\workflow.py”, line 248, in run action.execute() File “c:\users\ww\appdata\local\programs\python\python36\lib\site-packages\aws_lambda_builders\workflows\python_pip\actions.py”, line 42, in execute raise ActionFailedError(str(ex)) aws_lambda_builders.actions.ActionFailedError: {bcrypt==3.1.7(wheel), cryptography==2.7(wheel), pynacl==1.3.0(wheel)}

Build Failed Error: PythonPipBuilder:ResolveDependencies - {bcrypt==3.1.7(wheel), cryptography==2.7(wheel), pynacl==1.3.0(wheel)}

Expected result

I expected the build to complete as usual for packaging and deployment

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: windows 10
2. sam --version: SAM CLI, version 0.17.0

I wondered if it would work if I ran the build command with the --use-container flag but this also failed:

sam build --template template.yaml --use-container

OUTPUT

2019-08-28 10:49:07 Starting Build inside a container 2019-08-28 10:49:07 Building resource ‘HelloWorldFunction’

Fetching lambci/lambda:build-python3.6 Docker container image… 2019-08-28 10:49:09 Mounting C:\Users\ww\workspace\AWS-SAM-CLI-Lambdas-and-Layers\TestParamiko\hello_world as /tmp/samcli/source:ro,delegated inside runtime container

Build Failed Error: PythonPipBuilder:ResolveDependencies - Requirements file not found: /tmp/samcli/source/requirements.txt Running PythonPipBuilder:ResolveDependencies

Any help how to resolve this would be great, I posted on stackoverflow but no-one has picked it up https://stackoverflow.com/questions/57672859/aws-sam-build-not-able-to-build-packages-which-require-paramiko-due-to-error-p?noredirect=1#comment101801423_57672859. The steps posted here reproduce the exact issue.

If there is another library other than paramiko or pysftp that could be used for SFTP in a lambda that I could use that as a work around? However it would need to have different dependencies.

Many thanks for any help.