serverless: Dashboard breaks deployments and cause lambda to timeout

Bug Report

Description

What did you do? I can’t pinpoint exactly what the problem is, but it started when I deleted my serverless app from the dashboard UI then created a new one. After updating my serverless.yml file and running sls deploy my lambda function was timing out and was unreachable.
What happened? it took me a while to figure out what was happening, but it seems like sending traces to serverless API was the part causing it to timeout. After commenting out the dashboard specific fields from my serverless.yml (app and org) and deploying it worked again. This morning I tried doing sls remove then deploying everything again to start with a clean state with the dashboard integrated but I was getting this error when running sls deploy

  An error occurred: ApiGatewayLogGroup - /aws/api-gateway/omniview-backend-dev already exists.

So I deleted all resources and the cloudformation stack manually from AWS UI then tried deploying again with sls deploy and received this error

  Error: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
  <HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
  <TITLE>ERROR: The request could not be satisfied</TITLE>
  </HEAD><BODY>
  <H1>504 ERROR</H1>
  <H2>The request could not be satisfied.</H2>
  <HR noshade size="1px">
  CloudFront attempted to establish a connection with the origin, but either the attempt failed or the origin closed the connection.
  <BR clear="all"/>
  If you received this error while trying to use an app or access a website, please contact the provider or website owner for assistance.
  <BR clear="all">
  If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by following steps in the CloudFront documentation
  (<A href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/http-504-gateway-timeout.html">https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/http-504-gateway-timeout.html</A>).
  <BR clear="all">
  <HR noshade size="1px">
  <PRE>
  Generated by cloudfront (CloudFront)
  Request ID: LgMy8NXtw34GmqDYCLHbZdxTa-g5gR4--JemQlZUjywDSgC38w1MGw==
  </PRE>
  <ADDRESS>
  </ADDRESS>
  </BODY></HTML>
      at _callee$ (/usr/local/lib/node_modules/serverless/node_modules/@serverless/platform-sdk/src/utils/checkHttpResponse.js:19:9)
      at tryCatch (/usr/local/lib/node_modules/serverless/node_modules/regenerator-runtime/runtime.js:45:40)
      at Generator.invoke [as _invoke] (/usr/local/lib/node_modules/serverless/node_modules/regenerator-runtime/runtime.js:271:22)
      at Generator.prototype.<computed> [as next] (/usr/local/lib/node_modules/serverless/node_modules/regenerator-runtime/runtime.js:97:21)
      at step (/usr/local/lib/node_modules/serverless/node_modules/@serverless/platform-sdk/dist/utils/checkHttpResponse.js:7:191)
      at /usr/local/lib/node_modules/serverless/node_modules/@serverless/platform-sdk/dist/utils/checkHttpResponse.js:7:361
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (internal/process/task_queues.js:93:5)

So I disabled the dashboard from serverless.yml and deployed and everything worked again.

What should’ve happened? Dashboard should not break sls deploy or cause a lambda function to timeout if it can’t communicate with the dashboard API or better error messages should be thrown.
What’s the content of your serverless.yml file?

service:
  name: ovs-backend
# app: <dashboard app name>
# org: <dashboard org name>

custom:
  aws_profile: "ovs"
  environment: ${file(env.yml):${self:provider.stage}, file(env.yml):default}
  warmup:
    enabled: true
    prewarm: true
    concurrency: 1
    
# Add the serverless-webpack plugin
plugins:
  - serverless-offline
  - serverless-plugin-warmup

provider:
  name: aws
  runtime: nodejs10.x
  stage: ${opt:stage, 'dev'}
  profile: ${self:custom.aws_profile} # aws credentials profile to use
  region: ${opt:region, 'eu-central-1'}
  apiGateway:
    minimumCompressionSize: 1024
  tracing:
    apiGateway: true
    lambda: true
  environment:
    MONGO_URL: ${self:custom.environment.MONGO_URL}
    ENGINE_API_KEY: ${self:custom.environment.ENGINE_API_KEY}
    BASE_URL: ${self:custom.environment.BASE_URL}
    NODE_ENV: PRODUCTION
    AWS_STAGE: ${self:provider.stage}

iamRoleStatements:
  - Effect: 'Allow'
    Action:
      - "lambda:InvokeFunction"
    Resource:
      - Fn::Join:
        - ':'
        - - arn:aws:lambda
          - Ref: AWS::Region
          - Ref: AWS::AccountId
          - function:${self:service}-${self:provider.stage}-*
  

functions:
  hello:
    handler: api/handler.hello
    events:
      - http:
          path: hello
          method: get

  graphql:
    handler: api/graphqlHandler.graphqlHandler
    events:
      - http:
          path: playground
          method: get
          cors: true
      - http:
          path: graphql
          method: post
          cors: true

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 15 (11 by maintainers)

Commits related to this issue

Use a proxy instead of a plain object so writes to context pass through This fixes serverless/serverless#6761 — committed to serverless/dashboard-plugin by dschep 5 years ago
Use a proxy instead of a plain object so writes to context pass through This fixes serverless/serverless#6761 — committed to serverless/dashboard-plugin by dschep 5 years ago
Use a proxy instead of a plain object so writes to context pass through This fixes serverless/serverless#6761 — committed to serverless/dashboard-plugin by dschep 5 years ago
Use a proxy instead of a plain object so writes to context pass through This fixes serverless/serverless#6761 — committed to serverless/dashboard-plugin by dschep 5 years ago
Use a proxy instead of a plain object so writes to context pass through This fixes serverless/serverless#6761 — committed to serverless/dashboard-plugin by dschep 5 years ago

Most upvoted comments

Short term fix I’ve found to deploy a service, have it connected to the serverless dashboard, and use callback functions to return lambda functions:

uninstall serverless npm uninstall -g serverless
install serverless v1.52.0 npm i -g serverless@1.52.0
comment out app and org from serverless.yml
redeploy service without app & org with v1.52.0
uncomment / add back app and org to serverless.yml
Redeploy service again with v1.52.0

The reason to remove app and org and redeploy with v1.52.0: If you’ve previously had a service connected to dashboard and you’ve run into this issue then there is an cloudwatch log group for the service that will prevent the deployment. Redeploying without app & org removes this log group for you so you can reconnect to the dashboard.

The lambdas that utilize callback functions also operate and terminate as expected when deployed with v1.52.0 (Also the logs will display on the dashboard instead of saying you need to install the enterprise plugin )

Looks to me like using app and org to utilize the serverless dashboard starts to break after v1.52.0 with the abstraction of the Framework Core, Plugin, and SDK into more pieces.

sankhagowit on Nov 7, 2019

@dschep I still can reproduce it with 1.53.0. Short term solution is to remove app and org from serverless.yml.

vtatsakovich on Nov 4, 2019

@dschep I think I have a reproduction for this with a recent version:

Framework Core: 1.54.0
Plugin: 3.1.2
SDK: 2.1.2

It happens when I removed a service under 1.53 and tried to redeploy it with 1.54.

Somehow I think the log group isn’t being deleted properly or in a timely manner?

fernando-mc on Oct 17, 2019

I’ve figured out the issue & PR’d a fix. I’ll release an update soon.

dschep on Sep 27, 2019