aws-sdk-js: [AWS-Lambda] - NetworkingError: Client network socket disconnected before secure TLS connection was established

I was having this issue with Node and it turned out I was making a bunch of asynchronous calls within a forEach() which doesn’t work. The lambda would finish before some calls could complete. Switching to a a for…of worked beautifully.

You just save my life btw 😃

It seems to be a problem at Lambda.

Our support from aws: I also want to confirm that we are already aware of this issue in Nodejs Lambda function and can confirm that it is affecting a very small set of invocations for Nodejs runtime. There are some mitigation steps that we suggest for it, meanwhile the Lambda internal team is doing the investigation for the root cause.

I was having this issue with Node and it turned out I was making a bunch of asynchronous calls within a forEach() which doesn’t work. The lambda would finish before some calls could complete. Switching to a for…of worked beautifully.

Experiencing this with a NuxtJS application hosted on Vercel (who use Lambda). Approximately 1% of invocations are failing.

I have been looking into this as well for over a month and also realized it was because the lambda terminated execution after the response was sent, but before some of my scripts finished running. ( I was attempting to return a response early to quicken the response time for the user while leaving the lambda to continue updating the database for example because I thought it would stay active till the script finished.)

Many people may find this only occurs a small percentage of the time and the reason for it in this case was that the different parts of the code where finishing so close together that the script usually sent a response after all the code had run, but at times ran a little quicker and sent a response prior to the scripts finishing.

I have now changed all my scripts to run asynchronously with promises so all the code has completed running before sending a response to the client. This has solved my very frequent issue with this error.

I am unaware if this is a solution for all cases(if it were I would also agree with @fagiani that this is not actually a bug but a planned feature) but it was an absolute win for my case.

FWIW we had this issue a few months ago and discovered it was due to an unhandled promise. We exported the function asynchronously and did not return a promise in any part of the subsequent code. What was weird however, was that the code worked fine until one day it didn’t. It hadn’t been touched in months and decided to just error out one day. Take from that what you will and double check to make sure you are handling the async flow of your lambdas with care. The sdk can behave quite strangely at times.

I’ve received the following response from the support. I’d request someone from AWS SDK team please address this.

Thank you for being patient while I was getting back with the response from the internal Lambda team.

The team has noted that being AWS SDK issue, they cannot comment on ETA. So their suggestion is to raise a separate GitHub Issue on AWS Javascript SDK repository (refer 1) to get direction attention of the AWS SDK developers and maintainers. 

Therefore I would request you to raise a GitHub Issue and share it with me so that I could contact the concerned team internally for quick response.

any update on this?

Still experiencing this with Nuxt on Vercel which is a shame because it’s really fast. I’m not using Promise.all() anywhere in my code right now but it’s possibly being used in the Nuxt core or in a package I’m using. I find that I usually hit the error when clicking a link that has metadata loaded by SSR.

In case it’s helpful, our issue is related to writes to S3.

Same issue happening for my ALB gateway on us-east-2.

It’s intermittently rejecting requests from my consumer services:

  "parsedLambdaMessage": {
    "lines": [
      {
        "data": {
          "errorMessage": "Error: Client network socket disconnected before secure TLS connection was established",
          "errorType": "Runtime.UnhandledPromiseRejection",
          "reason": {
            "code": "ECONNRESET",
            "config": {
              "data": {},
              "headers": {
                "Accept": "application/json, text/plain, */*",
                "_Content_Length": 315,
                "_Content_Type": "application/json",
                "_User_Agent": "axios/0.21.4",
                "authorization": "123123123"
              },
              "maxBodyLength": -1,
              "maxContentLength": -1,
              "method": "post",
              "timeout": 0,
              "transitional": {
                "clarifyTimeoutError": false,
                "forcedJSONParsing": true,
                "silentJSONParsing": true
              },
              "url": "https://*******.com/queue",
              "xsrfCookieName": "XSRF-TOKEN",
              "xsrfHeaderName": "X-XSRF-TOKEN"
            },
            "message": "Client network socket disconnected before secure TLS connection was established",
            "name": "Error",
            "stack": "Error: Client network socket disconnected before secure TLS connection was established\n    at connResetException (internal/errors.js:639:14)\n    at TLSSocket.onConnectEnd (_tls_wrap.js:1570:19)\n    at TLSSocket.emit (events.js:412:35)\n    at TLSSocket.emit (domain.js:537:15)\n    at endReadableNT (internal/streams/readable.js:1334:12)\n    at processTicksAndRejections (internal/process/task_queues.js:82:21)"
          },
          "stack": [
            "Runtime.UnhandledPromiseRejection: Error: Client network socket disconnected before secure TLS connection was established",
            "    at process.<anonymous> (/var/runtime/index.js:35:15)",
            "    at process.emit (events.js:412:35)",
            "    at process.emit (domain.js:475:12)",
            "    at processPromiseRejections (internal/process/promises.js:245:33)",
            "    at processTicksAndRejections (internal/process/task_queues.js:96:32)"
          ]
        },
        "level": "error",
        "message": "Unhandled Promise Rejection \t",
        "timestamp": "2022-03-11T16:54:41.709Z"
      }
    ],

We also started seeing this a couple of days ago. Before that there was absolutely no problem. The lambda does not run in a VPC but is invoked by API Gateway

I was not able to get any updates as of yet, I have contacted the engineers associated and will update you guys once I have some more insight from Lambda team.

Reached out to the Lambda team again, would update the issue shortly? @robsonkades was there a word from Lambda team?

Same issue on Lambda running nodejs 12

Is there anything we can do to solve it?

Best regards

We’ve also got the same issue with AppSync - happening on a somewhat sensitive service.

Any idea how we can mitigate this until there is a proper fix?

Same for us. Seems to affect quite significantly our Lambdas. Thanks @ajredniwja for the update.

2021-02-05T01:27:25.910Z	1f0f65f6-4194-4948-b322-981bb95c4466	ERROR	Unhandled Promise Rejection 	{
    "errorType": "Runtime.UnhandledPromiseRejection",
    "errorMessage": "NetworkingError: Client network socket disconnected before secure TLS connection was established",
    "reason": {
        "errorType": "NetworkingError",
        "errorMessage": "Client network socket disconnected before secure TLS connection was established",
        "code": "NetworkingError",
        "message": "Client network socket disconnected before secure TLS connection was established",
        "path": null,
        "host": "dynamodb.ap-southeast-1.amazonaws.com",
        "port": 443,
        "region": "ap-southeast-1",
        "hostname": "dynamodb.ap-southeast-1.amazonaws.com",
        "retryable": true,
        "time": "2021-02-05T01:27:25.908Z",
        "stack": [
            "Error: Client network socket disconnected before secure TLS connection was established",
            "    at connResetException (internal/errors.js:609:14)",
            "    at TLSSocket.onConnectEnd (_tls_wrap.js:1549:19)",
            "    at Object.onceWrapper (events.js:420:28)",
            "    at TLSSocket.emit (events.js:326:22)",
            "    at TLSSocket.EventEmitter.emit (domain.js:483:12)",
            "    at endReadableNT (_stream_readable.js:1241:12)",
            "    at processTicksAndRejections (internal/process/task_queues.js:84:21)"
        ]
    },
    "promise": {},
    "stack": [
        "Runtime.UnhandledPromiseRejection: NetworkingError: Client network socket disconnected before secure TLS connection was established",
        "    at process.<anonymous> (/var/runtime/index.js:35:15)",
        "    at process.emit (events.js:314:20)",
        "    at process.EventEmitter.emit (domain.js:483:12)",
        "    at process.emit (/var/task/node_modules/source-map-support/source-map-support.js:495:21)",
        "    at processPromiseRejections (internal/process/promises.js:209:33)",
        "    at processTicksAndRejections (internal/process/task_queues.js:98:32)"
    ]
}

I face same error in aws lambda, I am using nodejs looks like aws-sdk lib problem?? using "aws-sdk": "^2.835.0", try upgrade version see how

We came across this issue and it turned out to be due to a missing await for an outbound http request using axios.get(). The lambda returned before the request had finished, so the connection was then closed before the call to axios.get() had terminated 💡 Make sure you got no pending requests or promises at the time of returning from the lambda

@Mektrode I can confirm that!

FWIW we had this issue a few months ago and discovered it was due to an unhandled promise. We exported the function asynchronously and did not return a promise in any part of the subsequent code. What was weird however, was that the code worked fine until one day it didn’t. It hadn’t been touched in months and decided to just error out one day. Take from that what you will and double check to make sure you are handling the async flow of your lambdas with care. The sdk can behave quite strangely at times.

After finding an asynchronous call that did not respond with a proper promise, although it worked just fine locally and within an EC2 environment, lambda would just let it go and skip it’s return and end the function execution early, which after understanding properly one of the lambda’s goals I think it actually makes sense as its pricing includes execution time charges as a relevant part of the offer.

@ajredniwja it may not be actually related to the SDK but AWS could definitely do a better job here explaining clearly and providing some good examples with proposed refactors as it looks like a product feature and not a bug considering the explanation given above.

FWIW, I’d recommend anyone facing this issue to review the implementation and find where it could be improved promise wise in order to achieve its goal within lambda. For that, this stack overflow topic has been quite helpful to me as it explains both serialized and parallelized execution approaches!

Keep Rocking!

We had this issue with our Nuxt.js application. An intermittent crash which happened around 5% of the time.

We were calling a function which used useRuntimeConfig() on created. Moving this function from created to mounted seems to have fixed it for us.

Also facing this issue when connecting to an FTP using basic-ftp

const client = new ftp.Client()
client.ftp.verbose = true

try {
     await client.access({
	    host,
	    user,
	    password,
	    secure: true,
      })
      console.log(await client.list())
} catch (err) {
      console.log(err)
}

INFO	Connected to XXX.XXX.XX.X:XX (No encryption)
INFO	< 220 { Welcome message }

INFO	> AUTH TLS
INFO	< 234 Proceed with negotiation.

INFO Client network socket disconnected before secure TLS connection was established
    at connResetException (internal/errors.js:639:14)
    at TLSSocket.onConnectEnd (_tls_wrap.js:1570:19)
    at TLSSocket.emit (events.js:412:35)
    at TLSSocket.emit (domain.js:475:12)
    at endReadableNT (internal/streams/readable.js:1334:12)
    at processTicksAndRejections (internal/process/task_queues.js:82:21) {
  code: 'ECONNRESET',
  path: undefined,
  host: { host },
  port: undefined,
  localAddress: undefined
}

I was having this issue with Node and it turned out I was making a bunch of asynchronous calls within a forEach() which doesn’t work. The lambda would finish before some calls could complete. Switching to a a for…of worked beautifully.

FYI - That’s going to be fine if you’re happy for them to happen sequentially. If you want them to happen in parallel, use .map and then await Promise.all() on the returned array.

For everyone else having this issue, I’m working around it by using S3 triggers. Essentially you’d take your SQS message and put it into a json or similar onto a s3 bucket that has a trigger on object create. The triggered lambda can then delete the file afterwards if you wish. This should be a usable stopgap until this issue gets some attention.

Have not had the same issue with s3 as I did with SQS. Hope this helps (for now)

@brucemackenzie the lambda functions I have do not run in a VPC so I doubt this is a VPC specific issue. The primary function where this is causing us a concern has the issue does run on a cron - every minute - so not sure if that has any impact.

However, I am possibly more concerned should this could happen on webhook requests. Especially as I may not be guaranteed a retry by a third-party service should the function fail.

We had the same issue reading from secretsmanager but also reading from s3 today. Our lambdas are running in a VPC so that is likely the issue in our case. https://docs.aws.amazon.com/lambda/latest/dg/configuration-vpc.html#vpc-samples

@robsonkades, thanks for providing that, I can reach out to the team related to that and can update once I hear back from them.

@ajredniwja, Thank you for your help!

Case 7831443621