aws-cdk: (cli): cdk deploy "fail: socket hang up" error on Big Sur w/ AnyConnect

Deployment using cdk deploy fails with fail: socket hang up. This has started happening since upgrading to Big Sur (MacOS). I opened a support ticket with AWS, and they directed me to here.

I have upgraded npm and node and aws utils to the latest versions as asked by the AWS support.

Reproduction Steps

cdk deploy

Output: -

[0%] start: Publishing 53bc1c8ff3460cde96f053e9ef268efbccac51185d8c46bf5acb064ff9afdf1c:current
[0%] check: Check s3://[bucket-name]/assets/53bc1c8ff3460cde96f053e9ef268efbccac51185d8c46bf5acb064ff9afdf1c.jar
[0%] upload: Upload s3://[bucket-name]/assets/53bc1c8ff3460cde96f053e9ef268efbccac51185d8c46bf5acb064ff9afdf1c.jar
[25%] fail: socket hang up
[25%] start: Publishing 67b7823b74bc135986aa72f889d6a8da058d0c4a20cbc2dfc6f78995fdd2fc24:current
[25%] check: Check s3://[bucket-name]/assets/67b7823b74bc135986aa72f889d6a8da058d0c4a20cbc2dfc6f78995fdd2fc24.zip
[25%] found: Found s3://[bucket-name]/assets/67b7823b74bc135986aa72f889d6a8da058d0c4a20cbc2dfc6f78995fdd2fc24.zip
[50%] success: Published 67b7823b74bc135986aa72f889d6a8da058d0c4a20cbc2dfc6f78995fdd2fc24:current
[50%] start: Publishing 0540b0b3a6863c6d68b73f1f5368b2832a4ffa5dfa77d4308941ee46eff41d21:current
[50%] check: Check s3://[bucket-name]/assets/0540b0b3a6863c6d68b73f1f5368b2832a4ffa5dfa77d4308941ee46eff41d21.jar
[50%] upload: Upload s3://[bucket-name]/assets/0540b0b3a6863c6d68b73f1f5368b2832a4ffa5dfa77d4308941ee46eff41d21.jar
[75%] fail: socket hang up
[75%] start: Publishing 692a0f095ccf744c65ed666353d5a527a0a8a36fa75759113c1da6ccad12f359:current
[75%] check: Check s3://[bucket-name]/assets/692a0f095ccf744c65ed666353d5a527a0a8a36fa75759113c1da6ccad12f359.zip
[75%] found: Found s3://[bucket-name]/assets/692a0f095ccf744c65ed666353d5a527a0a8a36fa75759113c1da6ccad12f359.zip
[100%] success: Published 692a0f095ccf744c65ed666353d5a527a0a8a36fa75759113c1da6ccad12f359:current

 ❌  [Stackname] failed: Error: Failed to publish one or more assets. See the error messages above for more information.
    at Object.publishAssets (/usr/local/lib/node_modules/aws-cdk/lib/util/asset-publishing.ts:25:11)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Object.deployStack (/usr/local/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:232:3)
    at CdkToolkit.deploy (/usr/local/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:180:24)
    at initCommandLine (/usr/local/lib/node_modules/aws-cdk/bin/cdk.ts:210:9)
Failed to publish one or more assets. See the error messages above for more information.
Error: Failed to publish one or more assets. See the error messages above for more information.
    at Object.publishAssets (/usr/local/lib/node_modules/aws-cdk/lib/util/asset-publishing.ts:25:11)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Object.deployStack (/usr/local/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:232:3)
    at CdkToolkit.deploy (/usr/local/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:180:24)
    at initCommandLine (/usr/local/lib/node_modules/aws-cdk/bin/cdk.ts:210:9)

What did you expect to happen?

Successful deployment of artifacts.

What actually happened?

The deployment fails due to upload timeout from cdk to S3

Environment

CDK CLI Version : 1.108.1 (build ae24d8a)
Framework Version: aws-cli/2.2.12 Python/3.9.5 Darwin/20.5.0 source/x86_64 prompt/off
Node.js Version: v16.3.0
OS : MaOS Big Sur v. 11.4
Language (Version): Java: openjdk 11.0.10 2021-01-19. Python: Python 3.9.5

Other

This is 🐛 Bug Report

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 4
Comments: 57 (26 by maintainers)

Most upvoted comments

@SeriousAnt, same. The proxy works for me, and I’ll use it as a workaround for now.

whiskeylover on Jul 2, 2021

You might also trying using https://mitmproxy.org/ to try and capture the traffic in readable form and seeing if that turns anything up.

Quick-fire instructions:
$ pip3 install mitmproxy
$ mitmdump -p 8080 --ssl-insecure -v -ddd

# different tab
$ export https_proxy=http://localhost:8080/
$ export NODE_TLS_REJECT_UNAUTHORIZED=0
$ cdk deploy ...

This is really interesting @rix0rrr. I was able to publish and deploy all assets using the proxy as you have described. Without the proxy I get socket hangup or Inaccessible host ... errors

OSX: 11.4 (20F71) Node: v14.15.3 CDK: 1.85

Will use this as a workaround till I figure it out 🙈

SeriousAnt on Jul 2, 2021

@jsauter, I’m very sorry, but I don’t know how to help anymore.

Did you all open a ticket with Apple and/or Cisco?

jsauter on Jun 29, 2021

I am running Cisco AnyConnect v4.9.05042 and am getting the error.

Also note, a colleague @john-shaskin did a parking lot deploy on our network without the VPN and received the same issue.

I comparison on the #15278, I receive the disconnect error after asset uploading when the changeset is being created.

jsauter on Jun 24, 2021

FYI, I installed MacOS 11.5.2 update yesterday, and now the issue has gone away. I no longer need to run mitmproxy in the background for my cdk deploys. Hope that helps anyone else who’s also facing the issue.

Awesome, thank you for letting us know! I can finally stop putting off my upgrade. 😄

jsauter on Aug 20, 2021

FYI, I installed MacOS 11.5.2 update yesterday, and now the issue has gone away. I no longer need to run mitmproxy in the background for my cdk deploys. Hope that helps anyone else who’s also facing the issue.

whiskeylover on Aug 20, 2021

The sequence numbers in that trace are slightly off from the other one. For whatever reason, the sequence numbers of packets in the general vicinity of where the Big Sure trace would cut off are [419, 545, 1089] instead of [519, 1199]. No idea if that’s even relevant, though.

@jsauter, I’m very sorry, but I don’t know how to help anymore.

rix0rrr on Jun 29, 2021

Could this be related? nodejs/node#36826

Are you on M1 or Intel architecture?

We have been all on Intel thus far.

jsauter on Jun 28, 2021

@rix0rrr

Here are both sides of the conversation with the failure, exported as a wireshark file.

AWSConversationWithIssue.pcapng.zip

jsauter on Jun 28, 2021

@whiskeylover our mac expert said that in Big Sur AnyConnect starts using a different mechanism to hook into the kernel.

So you would only see it when both factors are true: recent AnyConnect PLUS Big Sur, and you will see it starting with whatever update comes last.

rix0rrr on Jun 28, 2021

Hi Ryan,

Thanks for responding. The Cisco version is something pushed by IT and not something I can choose.

Here are the answers to your questions.

CDK code is written in TS
I just built the bootstrap stack and retried the deployment. It failed.
Yes, CDK is installed globally.

Also

I tried deploying the code with npx cdk deploy and I get a slightly different error.
- Instead of fail: socket hang up or fail: write EPIPE when deploying artifacts to S3 (see my original post), it now says fail: Inaccessible host: '[staging-bucket-name].s3.us-west-2.amazonaws.com'. This service may not be available in the 'us-west-2' region..
- I tried deploying it using the -v switch for verbose output and noticed that the deploy was failing when trying to upload the built artifacts (jar and zip files) to S3. It uploads the zip files successfully, but fails when uploading jar files, for some reason. If I manually upload the jar files to the same location and try again, it goes past the error points, and then fails at deploying cloud formation templates.
- I’ve found that other people are also having cdk deploy to S3 issues after upgrading to Big Sur. Here’s a link.
I updated brew and installed the latest CDK version, and still get the same error.
The sample CDK project built and deployed fine. But I noticed it’s not uploading any built artifacts to S3. Our project builds a java project, and uploads the JAR to a lambda. That’s where it fails.

Thanks again for responding. Let me know what else I can provide you with.

Ashish

whiskeylover on Jun 23, 2021