aws-sdk-js-v3: Secrets Manager EPROTO error
Describe the bug
We’re using Secrets Manager to initialize lambda state, and are frequently getting write EPROTO
failure messages. It started happening recently after we upgraded from v3.41.0 to v3.58.0
Your environment
SDK version number
@aws-sdk/client-secrets-manager@3.58.0
Is the issue in the browser/Node.js/ReactNative?
Node.js
Details of the browser/Node.js/ReactNative version
Node.js 14.x
Lambda 😃
Steps to reproduce
Here’s tl;dr of the lambda handler code
const { SecretsManagerClient, GetSecretValueCommand } = require('@aws-sdk/client-secrets-manager')
const promiseEnv = new SecretsManagerClient({
region: process.env.AWS_ENV_SECRET_REGION,
}).send(
new GetSecretValueCommand({
SecretId: process.env.AWS_ENV_SECRET_ID,
})
)
async function handler(event, context) {
console.log('Requesting environment variables')
const env = await promiseEnv
console.log('Got environment variables')
// ....
}
module.exports = { handler }
Observed behavior
Most of the times, everything works, but then unexpectedly crashes at await promiseEnv
, and Got environment variables
is never logged
Expected behavior
Secrets Manager would keep working
Screenshots
N/A
Additional context
Here's the raw logs:
[TS] [UUID] INFO Requesting environment variables
[TS] [UUID] ERROR Invoke Error {"errorType":"Error","errorMessage":"write EPROTO","code":"EPROTO","errno":-71,"syscall":"write","$metadata":{"attempts":1,"totalRetryDelay":0},"stack":["Error: write EPROTO"," at WriteWrap.onWriteComplete [as oncomplete] (internal/stream_base_commons.js:94:16)"," at WriteWrap.callbackTrampoline (internal/async_hooks.js:130:17)"]}
[TS] [UUID] ERROR (node:9) PromiseRejectionHandledWarning: Promise rejection was handled asynchronously (rejection id: 14)\n(Use `node --trace-warnings ...` to show where the warning was created)
END RequestId: [UUID]
REPORT RequestId: [UUID] Duration: 33.91 ms Billed Duration: 34 ms Memory Size: 1536 MB Max Memory Used: 99 MB Init Duration: 1213.34 ms
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 7
- Comments: 26 (2 by maintainers)
Has anyone found that using a newer version of Node makes this issue go away? I am planning on upgrading my version of Node, but I was curious if anyone else has already tried this.
Like the OP i am also using 14.X, but I am planning on updating to 18.X
@dgoemans Thanks for the heads up! Looking forward to upgrading my lambdas next month and will follow up if I see success on my end!
On quick revisit during review meeting for issues with
p1
labels, we noticed that this issue is likely in Node.js. Search results https://github.com/search?q=repo%3Anodejs%2Fnode+EPROTO&type=issuesWe need to find out whether the issue is with the Node.js setup which Lambda follows, or some Node.jsconfiguration which SDK sets, or a bug is Node.js core itself.
The requirement is to provide a minimal repro code which makes multiple secret manager getSecretValue calls. This will help us to log more information, and find out if the issue is specific to Lambda, Node.js or SDK.
For reference, here is a package which attempted to repro npm ping test failure from CodeBuild https://github.com/trivikr/aws-codebuild-npm-ping-test