serverless: No way to set contentHandling (example in docs is wrong and no workaround)

Are you certain it’s a bug?

Yes, it looks like a bug

Is the issue caused by a plugin?

It is not a plugin issue

Are you using the latest version?

Yes, I’m using the latest version

Is there an existing issue for this?

I have searched existing issues, it hasn’t been reported yet

Issue description

Using the example here: https://www.serverless.com/framework/docs/providers/aws/events/apigateway/#binary-media-types

functions:
  create:
    handler: posts.create
    events:
      - http:
          path: posts/create
          method: post
          request:
            contentHandling: CONVERT_TO_TEXT
          response:
            contentHandling: CONVERT_TO_TEXT

If I deploy, I get these warnings:

Serverless: Warning! You're using the AWS_PROXY in combination with a request configuration in your function "app". Only the request.parameters, request.schema, request.schemas configs are available in conjunction with AWS_PROXY. Serverless will remove this configuration automatically before deployment.
Serverless: Warning! You're using the AWS_PROXY in combination with response configuration in your function "app". Serverless will remove this configuration automatically before deployment.

This does not allow contentHandling to be set and base64 content returned from the Lambda cannot be transformed into binary data. It appears that serving binary files from serverless via Lambda and Api Gateway is totally broken.

Service configuration (serverless.yml) content

# serverless.yml

service: myapp

provider:
  name: aws
  runtime: nodejs14.x
  stage: dev
  region: us-west-2
  logs:
    restApi: True
  apiGateway:
    binaryMediaTypes:
      - 'image/jpeg'
      - 'image/gif'
      - 'image/png'
      - 'image/x-icon'
      - 'multipart/form-data'
      - 'text/css'
      - 'text/html'
      - 'text/jsx'
      - 'application/javascript'

functions:
  app:
    handler: posts.create
    events:
      - http:
          path: posts/create
          method: post
          request:
            contentHandling: CONVERT_TO_TEXT
          response:
            contentHandling: CONVERT_TO_TEXT

Command name and used flags

serverless deploy

Command output

bash-3.2$ SLS_DEBUG=* make deploy-dev
Serverless: Deprecation warning: Detected ".env" files. In the next major release variables from ".env" files will be automatically loaded into the serverless build process. Set "useDotenv: true" to adopt that behavior now.
            More Info: https://www.serverless.com/framework/docs/deprecations/#LOAD_VARIABLES_FROM_ENV_FILES
Serverless: Running "serverless" installed locally (in service node_modules)
Serverless: To ensure safe major version upgrades ensure "frameworkVersion" setting in service configuration (recommended setup: "frameworkVersion: ^2.66.2")

Serverless: Load command interactiveCli
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command config:tabcompletion
Serverless: Load command config:tabcompletion:install
Serverless: Load command config:tabcompletion:uninstall
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command upgrade
Serverless: Load command uninstall
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command generate-event
Serverless: Load command test
Serverless: Load command dashboard
Serverless: Load command output
Serverless: Load command output:get
Serverless: Load command output:list
Serverless: Load command param
Serverless: Load command param:get
Serverless: Load command param:list
Serverless: Load command studio
Serverless: Skipping variables resolution with old resolver (new resolver reported no more variables to resolve)
Serverless: Deprecation warning: Resolution of lambda version hashes was improved with better algorithm, which will be used in next major release.
            Switch to it now by setting "provider.lambdaHashingVersion" to "20201221".
            While it is highly encouraged to upgrade to new algorithm, you can still use the old approach by setting "provider.lambdaHashingVersion" to "20200924".
            More Info: https://www.serverless.com/framework/docs/deprecations/#LAMBDA_HASHING_VERSION_V2
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: [AWS sts 200 0.622s 0 retries] getCallerIdentity({})
Serverless: Warning! You're using the AWS_PROXY in combination with a request configuration in your function "app". Only the request.parameters, request.schema, request.schemas configs are available in conjunction with AWS_PROXY. Serverless will remove this configuration automatically before deployment.
Serverless: Warning! You're using the AWS_PROXY in combination with response configuration in your function "app". Serverless will remove this configuration automatically before deployment.
Serverless: Installing dependencies for custom CloudFormation resources...
Serverless: Invoke aws:package:finalize
Serverless: Invoke aws:common:moveArtifactsToPackage
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:deploy:deploy
Serverless: [AWS cloudformation 200 0.279s 0 retries] describeStacks({ StackName: 'lasttweetinaws-dev' })
Serverless: [AWS cloudformation 200 0.188s 0 retries] describeStackResource({
  StackName: 'lasttweetinaws-dev',
  LogicalResourceId: 'ServerlessDeploymentBucket'
})
Serverless: [AWS s3 200 0.187s 0 retries] listObjectsV2({
  Bucket: 'lasttweetinaws-dev-serverlessdeploymentbucket-1e8jyh92zhnvc',
  Prefix: 'serverless/lasttweetinaws/dev'
})
Serverless: [AWS s3 200 0.121s 0 retries] headObject({
  Bucket: 'lasttweetinaws-dev-serverlessdeploymentbucket-1e8jyh92zhnvc',
  Key: 'serverless/lasttweetinaws/dev/1637750603831-2021-11-24T10:43:23.831Z/compiled-cloudformation-template.json'
})
Serverless: [AWS lambda 200 0.172s 0 retries] getFunction({ FunctionName: 'lasttweetinaws-dev-app' })
Serverless: Uploading CloudFormation file to S3...

Environment information

Serverless: Running "serverless" installed locally (in service node_modules)
Framework Core: 2.66.2 (local)
Plugin: 5.5.1
SDK: 4.3.0
Components: 3.18.1

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 16 (8 by maintainers)

Most upvoted comments

So here is how binary files works.

provider:
  apiGateway:
    binaryMediaTypes:
      - '*/*'

functions:
  app:
    handler: src/server.handler
    events:
      - http:
          path: /
          method: ANY
      - http:
          path: /{any+}
          method: ANY

And then in your lambda, whenever isBase64Encoded: true is returned by the Lambda, API Gateway will decode it back into binary, and return it to the client.

If you’re using serverless-http, you’ll want to use something like this:

// These will be base64 encoded when returned from Lambda -> APIGateway.
// Because provider.apiGateway.binaryMediaTypes is "*/*",
// we'll always decode the lambda's base64 back into
// a binary response whenever the Lambda response
// is base64 encoded.
const BINARY_CONTENT_TYPES = [
  "image/jpeg",
  "image/gif",
  "image/png",
  "image/x-icon",
  "multipart/form-data",
];

// Using binary: true is broken, see https://github.com/dougmoscrop/serverless-http/issues/222
module.exports.handler = serverless(app, { binary: BINARY_CONTENT_TYPES });

Similarly for other web framework -> serverless wrappers.

The frustrating part is there are so many mismatches that can go wrong, with no clear indication of what the solution.

This is not explained holistically anywhere- not in AWS docs, not in Serverless docs, not in serverless-http, nowhere. Figuring this out required comprehensively understanding API Gateway, Serverless’s integrations, and digging into how serverless-http works. This to me largely defeats the purpose of using SLS over working with the provider directly.

The SLS documentation is both wrong, in the sense that the example YAML is invalid, and unclear, in the sense that, there are two ways to do this:

Using lambda-proxy (default) integration as I described, and
Using lambda integration, specifying many more details, like explicit content handling

The existing docs are worse than no docs because they propose a third, invalid option: lambda-proxy integration with explicit content handling.

In the end, the solution makes perfect sense, and is in line with API Gateway design/docs, Serverless design/docs, and serverless-http’s design/docs (though there is a bug there I filed in their repo). But these docs (at the very least, and probably the UX of using binary files entirely, insofar as SLS wants to abstract it) desperately needs to be revisited with the eyes of a normal customer/user, who doesn’t have a comprehensive understanding of API Gateway, Serverless, and Lambda. I have built a dozen serverless projects and many dozens of web servers and this was incredibly difficult to figure out.

rgalanakis on Nov 24, 2021