aws-sam-cli: Unable to build ruby function requiring a layer

Description

I have a ruby function that requires the “pg” gem which in turn requires having the PostgreSQL libraries to build since it’s a native extension. Without SAM I’ve gotten this work building a layer and then using the docker image locally that built this layer to build the ruby gems. Works just great in production as well. Using sam build --use-container the build fails with the only error indicated: "2019-01-04 14:05:22 Build inside container returned response {“jsonrpc”: “2.0”, “id”: 1, “error”: {“message”: "RubyBundlerBuilder:RubyBundle - Bundler Failed: “, “code”: 400}}”.

I think there two possible causes:

It’s not using the layer to build the gems.
The gem is not able to find the PostgreSQL libraries. In this case I’ve had to use environment variables in the past to set bundler configuration (ex: env BUNDLE_BUILD__PG="--with-pg-config=/opt/bin/pg_config"). It’s not clear to me how this would work with SAM since the environment variables appear to be for runtime not build.

Here’s how I do this process without SAM (happy to share more working code if this is helpful): https://gist.github.com/pdlug/5aba2555756b8f03ec3bc736912270fa

Steps to reproduce

Start a ruby project with sam, add gem 'pg' to your Gemfile'. Publish a PostgreSQL layer with the instructions above and add it to the layers. Execute sam build --use-container`.

Observed result

2019-01-04 14:04:47 Using SAM Template at /Users/paul/parse-xml-lambda/template.yaml
2019-01-04 14:04:47 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-01-04 14:04:47 Changing event name from before-call.apigateway to before-call.api-gateway
2019-01-04 14:04:47 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-01-04 14:04:47 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-01-04 14:04:47 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-01-04 14:04:47 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-01-04 14:04:47 Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2019-01-04 14:04:47 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-01-04 14:04:47 Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-01-04 14:04:47 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-01-04 14:04:47 Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2019-01-04 14:04:47 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-01-04 14:04:47 Changing event name from before-call.apigateway to before-call.api-gateway
2019-01-04 14:04:47 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-01-04 14:04:47 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-01-04 14:04:47 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-01-04 14:04:47 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-01-04 14:04:47 Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2019-01-04 14:04:47 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-01-04 14:04:47 Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-01-04 14:04:47 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-01-04 14:04:47 Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2019-01-04 14:04:47 'build' command is called
2019-01-04 14:04:47 Starting Build inside a container
2019-01-04 14:04:47 Looking for credentials via: env
2019-01-04 14:04:47 Looking for credentials via: assume-role
2019-01-04 14:04:47 Looking for credentials via: shared-credentials-file
2019-01-04 14:04:47 Found credentials in shared credentials file: ~/.aws/credentials
2019-01-04 14:04:47 Loading JSON file: /usr/local/Cellar/aws-sam-cli/0.10.0/libexec/lib/python3.7/site-packages/botocore/data/endpoints.json
2019-01-04 14:04:47 Event choose-service-name: calling handler <function handle_service_name_alias at 0x103a01730>
2019-01-04 14:04:47 Loading JSON file: /usr/local/Cellar/aws-sam-cli/0.10.0/libexec/lib/python3.7/site-packages/botocore/data/serverlessrepo/2017-09-08/service-2.json
2019-01-04 14:04:47 Event creating-client-class.serverlessapplicationrepository: calling handler <function add_generate_presigned_url at 0x1039c7620>
2019-01-04 14:04:47 The s3 config key is not a dictionary type, ignoring its value of: None
2019-01-04 14:04:48 Setting serverlessrepo timeout as (60, 60)
2019-01-04 14:04:48 Loading JSON file: /usr/local/Cellar/aws-sam-cli/0.10.0/libexec/lib/python3.7/site-packages/botocore/data/_retry.json
2019-01-04 14:04:48 Registering retry handlers for service: serverlessrepo
2019-01-04 14:04:48 No Parameters detected in the template
2019-01-04 14:04:48 2 resources found in the template
2019-01-04 14:04:48 Found Serverless function with name='HelloWorldFunction' and CodeUri='hello_world/'
2019-01-04 14:04:48 Trying paths: ['/Users/paul/.docker/config.json', '/Users/paul/.dockercfg']
2019-01-04 14:04:48 Found file at path: /Users/paul/.docker/config.json
2019-01-04 14:04:48 Couldn't find auth-related section ; attempting to interpret as auth-only file
2019-01-04 14:04:48 Config entry for key stackOrchestrator is not auth config
2019-01-04 14:04:48 Building resource 'HelloWorldFunction'
2019-01-04 14:04:48 Trying paths: ['/Users/paul/.docker/config.json', '/Users/paul/.dockercfg']
2019-01-04 14:04:48 Found file at path: /Users/paul/.docker/config.json
2019-01-04 14:04:48 Couldn't find auth-related section ; attempting to interpret as auth-only file
2019-01-04 14:04:48 Config entry for key stackOrchestrator is not auth config
2019-01-04 14:04:48 http://localhost:None "GET /v1.35/images/lambci/lambda:build-ruby2.5/json HTTP/1.1" 200 None
2019-01-04 14:04:48 Looking for auth config
2019-01-04 14:04:48 Looking for auth entry for 'docker.io'
2019-01-04 14:04:48 No entry found
2019-01-04 14:04:48 No auth config found
2019-01-04 14:04:48 http://localhost:None "POST /v1.35/images/create?tag=build-ruby2.5&fromImage=lambci%2Flambda HTTP/1.1" 200 None

Fetching lambci/lambda:build-ruby2.5 Docker container image......
2019-01-04 14:04:48 Mounting /Users/paul/parse-xml-lambda/hello_world as /tmp/samcli/source:ro inside runtime container
2019-01-04 14:04:48 http://localhost:None "POST /v1.35/containers/create HTTP/1.1" 201 90
2019-01-04 14:04:48 http://localhost:None "GET /v1.35/containers/2be52a08add11e3582cdf5942c4d6c787a3ad3ac3925c7f3bdb4c42db0b1f14f/json HTTP/1.1" 200 None
2019-01-04 14:04:48 http://localhost:None "GET /v1.35/containers/2be52a08add11e3582cdf5942c4d6c787a3ad3ac3925c7f3bdb4c42db0b1f14f/json HTTP/1.1" 200 None
2019-01-04 14:04:49 http://localhost:None "POST /v1.35/containers/2be52a08add11e3582cdf5942c4d6c787a3ad3ac3925c7f3bdb4c42db0b1f14f/start HTTP/1.1" 204 0
2019-01-04 14:04:49 http://localhost:None "GET /v1.35/containers/2be52a08add11e3582cdf5942c4d6c787a3ad3ac3925c7f3bdb4c42db0b1f14f/json HTTP/1.1" 200 None
2019-01-04 14:04:49 http://localhost:None "POST /containers/2be52a08add11e3582cdf5942c4d6c787a3ad3ac3925c7f3bdb4c42db0b1f14f/attach?stdout=1&stderr=1&logs=1&stream=1&stdin=0 HTTP/1.1" 101 0
Using the request object from command line argument
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Found workflow 'RubyBundlerBuilder' to support capabilities 'Capability(language=u'ruby', dependency_manager=u'bundler', application_framework=None)'
'ruby' runtime has not been validated!
Running workflow 'RubyBundlerBuilder'
Running RubyBundlerBuilder:CopySource
RubyBundlerBuilder:CopySource succeeded
Running RubyBundlerBuilder:RubyBundle
Running bundle install in /tmp/samcli/artifacts
executing Bundler: ['bundle', 'install', '--without', 'development', 'test']
RubyBundlerBuilder:RubyBundle failed
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/aws_lambda_builders/workflow.py", line 164, in run
    action.execute()
  File "/usr/local/lib/python2.7/site-packages/aws_lambda_builders/workflows/ruby_bundler/actions.py", line 35, in execute
    raise ActionFailedError(str(ex))
ActionFailedError: Bundler Failed: 
Builder workflow failed
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/aws_lambda_builders/__main__.py", line 98, in main
    options=params["options"])
  File "/usr/local/lib/python2.7/site-packages/aws_lambda_builders/builder.py", line 108, in build
    return workflow.run()
  File "/usr/local/lib/python2.7/site-packages/aws_lambda_builders/workflow.py", line 173, in run
    reason=str(ex))
WorkflowFailedError: RubyBundlerBuilder:RubyBundle - Bundler Failed: 
2019-01-04 14:05:22 Build inside container returned response {"jsonrpc": "2.0", "id": 1, "error": {"message": "RubyBundlerBuilder:RubyBundle - Bundler Failed: ", "code": 400}}
Build Failed
Error: RubyBundlerBuilder:RubyBundle - Bundler Failed:

Expected result

Expected gems for ruby function to be built using the ruby runtime and layers specified in template.yaml

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

OS: Mac
sam --version: 0.10.0

Add --debug flag to command you are running

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 24 (6 by maintainers)

Most upvoted comments

@CaDs I’m so sorry I’ve taken a month to respond. Here’s a gist that will hopefully help. https://gist.github.com/mycargus/d86e9201feb43e39b3c7dfb4df451fe5

Lemme know if I can clarify anything!

mycargus on Mar 1, 2019

Basically exactly the same thing I ran into with the mysql2 gem. The issue is that the shared libraries that rubygems with native extensions require aren’t there.

I put together some scripts to automatically build a precompiled version of the gem and a runtime layer for lambda. You can find them here: https://github.com/jasonmk/lambda-gem-compiler. However, I would love to see this become part of the SAM build process. I know some people consider layers to be separate from a function, but if it’s necessary for the function to run then it seems to me that it would make sense to have them share a build process.

jasonmk on Jan 30, 2019

@mycargus Docker does not have access to your localhost on your machine. You can use host.docker.internal has the host name to allow docker to talk to your machine. This is currently only supported by Docker for Windows and Docker for Mac. You can follow: https://github.com/docker/for-linux/issues/264 for linux (there are workarounds there), or you can use --docker-network host to force the docker images to be on your computers host network.

jfuss on Jan 8, 2019