sst: Tunnel not working after upgrade to 3.12.3
I updated from 3.9.4 to 3.12.3 which may or may not be related, but since then I can’t get the tunnel to work.
// npx sst tunnel --stage production --print-logs
time=2025-04-03T09:46:47.644-04:00 level=INFO msg="checking for pulumi" path="/Users/andrew/Library/Application Support/sst/bin/pulumi"
time=2025-04-03T09:46:48.541-04:00 level=INFO msg="checking for bun" path="/Users/andrew/Library/Application Support/sst/bin/bun"
time=2025-04-03T09:46:49.155-04:00 level=INFO msg="starting tunnel"
time=2025-04-03T09:46:49.155-04:00 level=INFO msg="initializing project" version=3.12.3
time=2025-04-03T09:46:49.156-04:00 level=INFO msg="esbuild building" out=/Users/andrew/Projects/xxx/.sst/platform/sst.config.1743688009156.mjs
time=2025-04-03T09:46:49.166-04:00 level=INFO msg="esbuild built" outfile=/Users/andrew/Projects/xxx/.sst/platform/sst.config.1743688009156.mjs
time=2025-04-03T09:46:49.167-04:00 level=INFO msg="evaluating config"
time=2025-04-03T09:46:49.202-04:00 level=INFO msg="config evaluated"
time=2025-04-03T09:46:49.204-04:00 level=INFO msg="checking platform"
time=2025-04-03T09:46:49.205-04:00 level=INFO msg="checking provider" name=command version=1.0.2 compare=1.0.2
time=2025-04-03T09:46:49.205-04:00 level=INFO msg="loading home"
time=2025-04-03T09:46:50.007-04:00 level=INFO msg="aws credentials found" region=us-east-2 profile=xxx-prod
time=2025-04-03T09:46:50.007-04:00 level=INFO msg="fetching bootstrap"
time=2025-04-03T09:46:50.251-04:00 level=INFO msg="found existing bootstrap" data="{\"version\":5,\"asset\":\"sst-asset-xxx\",\"assetEcrRegistryId\":\"xxx\",\"assetEcrUrl\":\"xxx.dkr.ecr.us-east-2.amazonaws.com/sst-asset\",\"state\":\"sst-state-xxx\",\"appsyncHttp\":\"\",\"appsyncRealtime\":\"\"}"
time=2025-04-03T09:46:50.251-04:00 level=INFO msg="loaded config" app=xxx stage=production
time=2025-04-03T09:46:50.252-04:00 level=INFO msg="INFO getting passphrase app=xxx stage=production"
time=2025-04-03T09:46:50.432-04:00 level=INFO msg="INFO pulling state app=xxx stage=production out=/Users/andrew/Projects/xxx/.sst/pulumi/xxx/.pulumi/stacks/xxx/production.json"
time=2025-04-03T09:46:51.045-04:00 level=INFO msg="starting tunnel" cmd="[sudo -n -E /opt/sst/tunnel tunnel start --subnets 10.0.4.0/22,10.0.12.0/22,10.0.0.0/22,10.0.8.0/22 --host xxx --user ec2-user --print-logs]"
Tunnel
➜ Forwarding ranges
10.0.4.0/22
10.0.12.0/22
10.0.0.0/22
10.0.8.0/22
Waiting for connections...
time=2025-04-03T09:48:06.371-04:00 level=INFO msg="killing process" pid=2173
time=2025-04-03T09:48:06.372-04:00 level=INFO msg="killing process" pid=2173
time=2025-04-03T09:48:06.372-04:00 level=INFO msg="process killed with term" pid=2173
time=2025-04-03T09:48:06.372-04:00 level=INFO msg="untracked process" pid=2173
time=2025-04-03T09:48:06.372-04:00 level=ERROR msg="failed to send sigterm" pid=2173
As you can see the tunnel seemingly runs for about 2 minutes before the process is killed. During those 2 minutes, trying to connect to my database from npx sst shell --stage production and then psql -h <hostname> gives me a server closed connection error.
Trying to run tunnel again it exits immediately.
I’ve tried restarting my computer, tried multiple networks that are working fine, and no luck getting this to work.
additional logs from sst diagnostic:
time=2025-04-03T09:31:49.882-04:00 level=INFO msg="tunnel started"
time=2025-04-03T09:33:04.855-04:00 level=ERROR msg="failed to start tunnel" error="dial tcp xxx:22: connect: operation timed out"
time=2025-04-03T09:33:04.857-04:00 level=INFO msg="running command" command="[ifconfig utun69 destroy]"
time=2025-04-03T09:33:04.865-04:00 level=ERROR msg="failed to execute command" command="[ifconfig utun69 destroy]"
Not sure if relevant, but for
sudo -n -E /opt/sst/tunnel tunnel start --subnets 10.0.4.0/22,10.0.12.0/22,10.0.0.0/22,10.0.8.0/22 --host x.x.x.x --user ec2-user --print-logs
the value printed after --host does not match any of the IP’s for EC2 instances that I can see in the console.
my sst.config.ts has
const vpc = new sst.aws.Vpc('Vpc', {
bastion: true,
nat: 'ec2',
});
ETA: I renamed my VPC and tried re-deploying. This failed, so I undid the rename on the VPC. This looks like it created some new EC2 instances in my account and the tunnel is now working and outputting an existing EC2 instance IP address after --host.
While the problem is resolved for now, I’m still curious to know why SST was trying to use the wrong IP value for --host after I upgraded to 3.12.3 and re-deployed.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 14
- Comments: 47 (6 by maintainers)
I’ve been able to fix the issue, see #5820.
I had the same issue (only on a deployed stage on prod not on dev while running sst)
I try to do it like that:
sst tunnel --stage prod --print-logsOn Drizzle Studio I get a
ECONNRESETand I don’t see any IPs forwardedAfter some time the tunnel was killed:
Tried the following until it worked:
.serverfiles -> no available in.sst.sstfolder and dosst install-> Same issue{ bastion: true, nat: 'ec2' }to{ bastion: true, nat: 'ec2' }tonew sst.aws.Vpc('MyVpc', { bastion: false })to remove EC2 and add it back again -> Same Issueec2tomanaged-> Error while deployingwaiting for EC2 NAT Gateway (nat-076890c52b4b5a002) create: unexpected state 'failed', wanted target 'available'. last error: Resource.AlreadyAssociated: Elastic IP address [eipalloc-07ba5cda18b230d4c] is already associated: provider=aws@6.66.2-> WORKSAfter step 5 it worked. But I don’t want to be in this weird state of wanting to deploy a managed nat gateway (which is too expensive for this case anyway) + facing a deployment error. So I try going back to EC2
ec2-> Same error as beforeMyVpctoMyVpc2-> Error while deployingoperation error RDS: ModifyDBSubnetGroup, https response error StatusCode: 400, RequestID: ada5ea40-1b4a-4381-8cf4-eb9f84ac8f11, api error InvalidParameterValue: The new Subnets are not in the same Vpc as the existing subnet group: provider=aws@6.66.2sst refresh --stage prod-> Maybe its updating the IPs? Not sure where they are in the state exactly -> Now the tunnel is immediately failingSo maybe it was just a thing of
I probably triggered something somehow to update the IPs it should connect to. But now it looks fine (at the moment).
Maybe that helps somebody
we have an issue that i’m working on a fix for - some of your issues is caused by hanging
.serverfiles in the.sstfolderif you delete those it may start to work
I think I traced the issue: in
platform/src/components/aws/vpc.tsNot sure, but I think the issue is that the
_tunnelobject is using the bastion.publicIp instead of theeipalloc.I found a workaround by tunneling to your personal stage. The trick is to deploy your personal stage that points to the same VPC used in the production stage.
Here’s how to reproduce:
Grab the production’s VPC ID (skip the sub-steps if you already know it):
Update
sst.config.tsto expose your production’s VPC IDOutput the VPD ID by running
sst refresh --stage productionUpdate your
sst.config.tsto reuse production’s VPCDeploy your personal stage:
sst deployUndo any changes to
sst.config.tsTunnel using your personal stage
sst tunnelRun whatever you need on the production stage
sst shell --stage production ...Note: both the production stage and your personal must be deployed in the same AWS account.
I keep running into this problem. Doing this will fix it consistently.
First, you need to make sure you don’t have any other sst running. I have found that I need to run this a lot.
Next, we need to remove the local config so we can reset it.
Then we can reset our local config by running another deploy.
You can verify this issue by running the following command. You’ll see the host IP being used in the logs. Then go to AWS Console and look for the EC2 instance with the tag sst:is-bastion = true. You’ll see the public IP doesn’t match the IP being used for the tunnel. Running the above commands seems to force a refresh.
Note that this might be the same thing as running
refreshand thendeployto your stage. You can try that first. The key is that you have to redeploy.can confirm doing 9 and 10 fixed it for me as well 🚀
note: before i did 9, i removed
.sstfolder, but im not sure if that’s relatedThank you for the detailed steps, I had done them all but it was #9 and #10 that finally did the trick for me
I think I narrowed it down, at least for my case. The issue seems to be with reusing the
natec2 instance for the bastion. When I remove thenatfrom thevpcand it redeploys a standalone bastion instance then the tunnel works fine. When I add back thenat, the tunnel seems to not resolve the right IP for either of thenatinstancesHello, I just started using SST. However, I only successfully used the tunnel once. The scenario was that after updating the Mac system version and restarting the system, the unexpectedly worked.
But shortly after that, it stopped working properly again because the remote IP of the tunnel SSH was incorrect. I didn’t find this bastion machine IP in my AWS console.
This problem has been bothering me for a long time because I can’t access the table structure of the remote RDS database.
I was facing the same issue using SST 3.16.0. After downgrading to 3.9.39 (an earlier version that was working for me), I can now see the tunnel picking up the correct IP, I guess something change between versions.
in v3.14.29 the tunnel command prints out the host ip it’s connected to
if this does NOT match what you are seeing in aws console can you share
sst export --stage=<stage-vpc-is-in>Deleting the
.sstfolder and moving to managed fix it for me. Still an issue on3.13.14. This was how I got it to finally work:First, I flipped nat from
ec2tomanagedand got some errors about resource conflicts. Then I named it toVpc2and that made more errors like:The new Subnets are not in the same Vpc as the existing subnet group. Then I renamed it back toVpcand it worked worked itself out 🤷.Another interesting change around this in the past few months is that I have to run
sst tunnelseparately from my dev process. I remember that I didn’t need to do that at one time.fwiw, i was able to fix this today by doing
npx sst deploy --stage production --target MyVpcName. after running that, the IPs started matching up properly.On
"sst": "3.13.9"(on a mac). The removal of.sstworked yesterday but weirdly not today, the change tonat: 'managed'does work as others said. The IP in the sst tunnel logs withnat: 'ec2'doesn’t match any EC2 instance public IP.In the mix, random immediate exits/crashes of
sst tunnellike in https://github.com/sst/sst/issues/5008, nothing useful in--print-logsthough.I’ve seen this too. Tunneling failed due to wrong IP.
I downgraded to
3.10.7and tested with personal stage. Only after remove and redeploy the IP changed. Having a3.12.3deployment and tunneling with a3.10.7without redeploying shows the wrong IP.EDIT: Tried with
sst tunnel --stage mystageinstead ofsst devtunnel today. No luck. IP is wrong again. Tunneling to personal stage via justsst tunnelresolves a correct IP.This is fixed for me after upgrading to
3.12.6🎉