aws-sdk-js-v3: Secrets Manager EPROTO error

Describe the bug

We’re using Secrets Manager to initialize lambda state, and are frequently getting write EPROTO failure messages. It started happening recently after we upgraded from v3.41.0 to v3.58.0

Your environment

SDK version number

@aws-sdk/client-secrets-manager@3.58.0

Is the issue in the browser/Node.js/ReactNative?

Node.js

Details of the browser/Node.js/ReactNative version

Node.js 14.x Lambda 😃

Steps to reproduce

Here’s tl;dr of the lambda handler code

const { SecretsManagerClient, GetSecretValueCommand } = require('@aws-sdk/client-secrets-manager')

const promiseEnv = new SecretsManagerClient({
  region: process.env.AWS_ENV_SECRET_REGION,
}).send(
  new GetSecretValueCommand({
    SecretId: process.env.AWS_ENV_SECRET_ID,
  })
)

async function handler(event, context) {
  console.log('Requesting environment variables')
  const env = await promiseEnv
  console.log('Got environment variables')
  // ....
}

module.exports = { handler }

Observed behavior

Most of the times, everything works, but then unexpectedly crashes at await promiseEnv, and Got environment variables is never logged

Expected behavior

Secrets Manager would keep working

Screenshots

N/A

Additional context

Here's the raw logs:
[TS] [UUID] INFO	Requesting environment variables
[TS] [UUID] ERROR	Invoke Error 	{"errorType":"Error","errorMessage":"write EPROTO","code":"EPROTO","errno":-71,"syscall":"write","$metadata":{"attempts":1,"totalRetryDelay":0},"stack":["Error: write EPROTO","    at WriteWrap.onWriteComplete [as oncomplete] (internal/stream_base_commons.js:94:16)","    at WriteWrap.callbackTrampoline (internal/async_hooks.js:130:17)"]}
[TS] [UUID] ERROR	(node:9) PromiseRejectionHandledWarning: Promise rejection was handled asynchronously (rejection id: 14)\n(Use `node --trace-warnings ...` to show where the warning was created)
END RequestId: [UUID]
REPORT RequestId: [UUID]	Duration: 33.91 ms	Billed Duration: 34 ms	Memory Size: 1536 MB	Max Memory Used: 99 MB	Init Duration: 1213.34 ms

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 7
  • Comments: 26 (2 by maintainers)

Most upvoted comments

Has anyone found that using a newer version of Node makes this issue go away? I am planning on upgrading my version of Node, but I was curious if anyone else has already tried this.

Like the OP i am also using 14.X, but I am planning on updating to 18.X

@dgoemans Thanks for the heads up! Looking forward to upgrading my lambdas next month and will follow up if I see success on my end!

On quick revisit during review meeting for issues with p1 labels, we noticed that this issue is likely in Node.js. Search results https://github.com/search?q=repo%3Anodejs%2Fnode+EPROTO&type=issues

We need to find out whether the issue is with the Node.js setup which Lambda follows, or some Node.jsconfiguration which SDK sets, or a bug is Node.js core itself.

The requirement is to provide a minimal repro code which makes multiple secret manager getSecretValue calls. This will help us to log more information, and find out if the issue is specific to Lambda, Node.js or SDK.

For reference, here is a package which attempted to repro npm ping test failure from CodeBuild https://github.com/trivikr/aws-codebuild-npm-ping-test