sst: Tunnel not working after upgrade to 3.12.3

I updated from 3.9.4 to 3.12.3 which may or may not be related, but since then I can’t get the tunnel to work.

// npx sst tunnel --stage production --print-logs

time=2025-04-03T09:46:47.644-04:00 level=INFO msg="checking for pulumi" path="/Users/andrew/Library/Application Support/sst/bin/pulumi"
time=2025-04-03T09:46:48.541-04:00 level=INFO msg="checking for bun" path="/Users/andrew/Library/Application Support/sst/bin/bun"
time=2025-04-03T09:46:49.155-04:00 level=INFO msg="starting tunnel"
time=2025-04-03T09:46:49.155-04:00 level=INFO msg="initializing project" version=3.12.3
time=2025-04-03T09:46:49.156-04:00 level=INFO msg="esbuild building" out=/Users/andrew/Projects/xxx/.sst/platform/sst.config.1743688009156.mjs
time=2025-04-03T09:46:49.166-04:00 level=INFO msg="esbuild built" outfile=/Users/andrew/Projects/xxx/.sst/platform/sst.config.1743688009156.mjs
time=2025-04-03T09:46:49.167-04:00 level=INFO msg="evaluating config"
time=2025-04-03T09:46:49.202-04:00 level=INFO msg="config evaluated"
time=2025-04-03T09:46:49.204-04:00 level=INFO msg="checking platform"
time=2025-04-03T09:46:49.205-04:00 level=INFO msg="checking provider" name=command version=1.0.2 compare=1.0.2
time=2025-04-03T09:46:49.205-04:00 level=INFO msg="loading home"
time=2025-04-03T09:46:50.007-04:00 level=INFO msg="aws credentials found" region=us-east-2 profile=xxx-prod
time=2025-04-03T09:46:50.007-04:00 level=INFO msg="fetching bootstrap"
time=2025-04-03T09:46:50.251-04:00 level=INFO msg="found existing bootstrap" data="{\"version\":5,\"asset\":\"sst-asset-xxx\",\"assetEcrRegistryId\":\"xxx\",\"assetEcrUrl\":\"xxx.dkr.ecr.us-east-2.amazonaws.com/sst-asset\",\"state\":\"sst-state-xxx\",\"appsyncHttp\":\"\",\"appsyncRealtime\":\"\"}"
time=2025-04-03T09:46:50.251-04:00 level=INFO msg="loaded config" app=xxx stage=production
time=2025-04-03T09:46:50.252-04:00 level=INFO msg="INFO getting passphrase app=xxx stage=production"
time=2025-04-03T09:46:50.432-04:00 level=INFO msg="INFO pulling state app=xxx stage=production out=/Users/andrew/Projects/xxx/.sst/pulumi/xxx/.pulumi/stacks/xxx/production.json"
time=2025-04-03T09:46:51.045-04:00 level=INFO msg="starting tunnel" cmd="[sudo -n -E /opt/sst/tunnel tunnel start --subnets 10.0.4.0/22,10.0.12.0/22,10.0.0.0/22,10.0.8.0/22 --host xxx --user ec2-user --print-logs]"
Tunnel

➜  Forwarding ranges
   10.0.4.0/22
   10.0.12.0/22
   10.0.0.0/22
   10.0.8.0/22

Waiting for connections...

time=2025-04-03T09:48:06.371-04:00 level=INFO msg="killing process" pid=2173
time=2025-04-03T09:48:06.372-04:00 level=INFO msg="killing process" pid=2173
time=2025-04-03T09:48:06.372-04:00 level=INFO msg="process killed with term" pid=2173
time=2025-04-03T09:48:06.372-04:00 level=INFO msg="untracked process" pid=2173
time=2025-04-03T09:48:06.372-04:00 level=ERROR msg="failed to send sigterm" pid=2173

As you can see the tunnel seemingly runs for about 2 minutes before the process is killed. During those 2 minutes, trying to connect to my database from npx sst shell --stage production and then psql -h <hostname> gives me a server closed connection error.

Trying to run tunnel again it exits immediately.

I’ve tried restarting my computer, tried multiple networks that are working fine, and no luck getting this to work.

additional logs from sst diagnostic:

time=2025-04-03T09:31:49.882-04:00 level=INFO msg="tunnel started"
time=2025-04-03T09:33:04.855-04:00 level=ERROR msg="failed to start tunnel" error="dial tcp xxx:22: connect: operation timed out"
time=2025-04-03T09:33:04.857-04:00 level=INFO msg="running command" command="[ifconfig utun69 destroy]"
time=2025-04-03T09:33:04.865-04:00 level=ERROR msg="failed to execute command" command="[ifconfig utun69 destroy]" 

Not sure if relevant, but for

sudo -n -E /opt/sst/tunnel tunnel start --subnets 10.0.4.0/22,10.0.12.0/22,10.0.0.0/22,10.0.8.0/22 --host x.x.x.x --user ec2-user --print-logs

the value printed after --host does not match any of the IP’s for EC2 instances that I can see in the console.

my sst.config.ts has

const vpc = new sst.aws.Vpc('Vpc', {
  bastion: true,
  nat: 'ec2',
});

ETA: I renamed my VPC and tried re-deploying. This failed, so I undid the rename on the VPC. This looks like it created some new EC2 instances in my account and the tunnel is now working and outputting an existing EC2 instance IP address after --host.

While the problem is resolved for now, I’m still curious to know why SST was trying to use the wrong IP value for --host after I upgraded to 3.12.3 and re-deployed.

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 14
  • Comments: 47 (6 by maintainers)

Most upvoted comments

I’ve been able to fix the issue, see #5820.

I had the same issue (only on a deployed stage on prod not on dev while running sst)

I try to do it like that:

  • Start sst tunnel --stage prod --print-logs
  • Start drizzle studio

On Drizzle Studio I get a ECONNRESET and I don’t see any IPs forwarded

Image

After some time the tunnel was killed:

time=2025-04-25T07:24:23.461+02:00 level=INFO msg="killing process" pid=35807
time=2025-04-25T07:24:23.462+02:00 level=INFO msg="process killed with term" pid=35807
time=2025-04-25T07:24:23.462+02:00 level=INFO msg="untracked process" pid=35807
time=2025-04-25T07:24:23.462+02:00 level=INFO msg="killing process" pid=35807
time=2025-04-25T07:24:23.462+02:00 level=ERROR msg="failed to send sigterm" pid=35807

Tried the following until it worked:

  1. Remove .server files -> no available in .sst
  2. Remove .sst folder and do sst install-> Same issue
  3. Upgraded from 3.13.14 to 3.13.18 -> Same issue
  4. Changing my VPC from { bastion: true, nat: 'ec2' } to { bastion: true, nat: 'ec2' } to new sst.aws.Vpc('MyVpc', { bastion: false }) to remove EC2 and add it back again -> Same Issue
  5. Changing VPC Bastion from ec2 to managed -> Error while deploying waiting for EC2 NAT Gateway (nat-076890c52b4b5a002) create: unexpected state 'failed', wanted target 'available'. last error: Resource.AlreadyAssociated: Elastic IP address [eipalloc-07ba5cda18b230d4c] is already associated: provider=aws@6.66.2 -> WORKS

After step 5 it worked. But I don’t want to be in this weird state of wanting to deploy a managed nat gateway (which is too expensive for this case anyway) + facing a deployment error. So I try going back to EC2

  1. Changing it back to ec2 -> Same error as before
  2. Renaming VPC from MyVpc to MyVpc2 -> Error while deploying operation error RDS: ModifyDBSubnetGroup, https response error StatusCode: 400, RequestID: ada5ea40-1b4a-4381-8cf4-eb9f84ac8f11, api error InvalidParameterValue: The new Subnets are not in the same Vpc as the existing subnet group: provider=aws@6.66.2
  3. Going back to original name of VPC -> same error
  4. sst refresh --stage prod -> Maybe its updating the IPs? Not sure where they are in the state exactly -> Now the tunnel is immediately failing
time=2025-04-25T07:44:21.718+02:00 level=INFO msg="starting tunnel" cmd="[sudo -n -E /opt/sst/tunnel tunnel start --subnets 10.0.4.0/22,10.0.12.0/22,10.0.0.0/22,10.0.8.0/22 --host 3.122.53.251 --user ec2-user --print-logs]"
Tunnel

➜  Forwarding ranges
   10.0.4.0/22
   10.0.12.0/22
   10.0.0.0/22
   10.0.8.0/22

Waiting for connections...

time=2025-04-25T07:44:21.814+02:00 level=INFO msg="killing process" pid=41189
time=2025-04-25T07:44:21.814+02:00 level=INFO msg="killing process" pid=41189
time=2025-04-25T07:44:21.814+02:00 level=ERROR msg="failed to send sigterm" pid=41189
time=2025-04-25T07:44:21.814+02:00 level=INFO msg="process killed with term" pid=41189
time=2025-04-25T07:44:21.814+02:00 level=INFO msg="untracked process" pid=41189
  1. Deployed prod again And it works

So maybe it was just a thing of

  • updating SST
  • refreshing the state
  • deploying again

I probably triggered something somehow to update the IPs it should connect to. But now it looks fine (at the moment).

Maybe that helps somebody

we have an issue that i’m working on a fix for - some of your issues is caused by hanging .server files in the .sst folder

if you delete those it may start to work

I think I traced the issue: in platform/src/components/aws/vpc.ts

function registerOutputs() {
  self.registerOutputs({
    _tunnel: all([
      self.bastionInstance,
      self.privateKeyValue,
      self._privateSubnets,
      self._publicSubnets,
    ]).apply(
      ([bastion, privateKeyValue, privateSubnets, publicSubnets]) => {
        if (!bastion) return;
        return {
          ip: bastion.publicIp,
          username: "ec2-user",
          privateKey: privateKeyValue!,
          subnets: [...privateSubnets, ...publicSubnets].map(
            (s) => s.cidrBlock,
          ),
        };
      },
    ),
  });
}

Not sure, but I think the issue is that the _tunnel object is using the bastion.publicIp instead of the eipalloc.

I found a workaround by tunneling to your personal stage. The trick is to deploy your personal stage that points to the same VPC used in the production stage.

Here’s how to reproduce:

  1. Grab the production’s VPC ID (skip the sub-steps if you already know it):

    1. Update sst.config.ts to expose your production’s VPC ID

      export default $config({
        // ...
        async run() {
          const vpc = new sst.aws.Vpc('vpc', { bastion: true,nat: 'ec2' })
          // ...
          return { vpcId: vpc.id }
        }
      }
      
    2. Output the VPD ID by running sst refresh --stage production

  2. Update your sst.config.ts to reuse production’s VPC

    export default $config({
      async run() {
        sst.aws.Vpc.get("vpc", "VPC's ID")
        // I recommend removing all other components since we don't want to deploy to production
      }
    }
    
  3. Deploy your personal stage: sst deploy

  4. Undo any changes to sst.config.ts

  5. Tunnel using your personal stage sst tunnel

  6. Run whatever you need on the production stage sst shell --stage production ...

Note: both the production stage and your personal must be deployed in the same AWS account.

I keep running into this problem. Doing this will fix it consistently.

First, you need to make sure you don’t have any other sst running. I have found that I need to run this a lot.

ps aux | grep sst | awk '{print $2}' | xargs kill -9

Next, we need to remove the local config so we can reset it.

rm -rf .sst/platform
rm -rf .sst/pulumi

Then we can reset our local config by running another deploy.

sst deploy --stage <stage_name>

You can verify this issue by running the following command. You’ll see the host IP being used in the logs. Then go to AWS Console and look for the EC2 instance with the tag sst:is-bastion = true. You’ll see the public IP doesn’t match the IP being used for the tunnel. Running the above commands seems to force a refresh.

npx sst tunnel --stage <stage_name> --print-logs

Note that this might be the same thing as running refresh and then deploy to your stage. You can try that first. The key is that you have to redeploy.

  1. sst refresh --stage prod -> Maybe its updating the IPs? Not sure where they are in the state exactly -> Now the tunnel is
  2. Deployed prod again And it works

can confirm doing 9 and 10 fixed it for me as well 🚀

note: before i did 9, i removed .sst folder, but im not sure if that’s related

  1. sst refresh --stage prod -> Maybe its updating the IPs? Not sure where they are in the state exactly -> Now the tunnel is immediately failing
  2. Deployed prod again And it works

So maybe it was just a thing of

  • updating SST
  • refreshing the state
  • deploying again

I probably triggered something somehow to update the IPs it should connect to. But now it looks fine (at the moment).

Maybe that helps somebody

Thank you for the detailed steps, I had done them all but it was #9 and #10 that finally did the trick for me

I think I narrowed it down, at least for my case. The issue seems to be with reusing the nat ec2 instance for the bastion. When I remove the nat from the vpc and it redeploys a standalone bastion instance then the tunnel works fine. When I add back the nat, the tunnel seems to not resolve the right IP for either of the nat instances

Image

Image

Hello, I just started using SST. However, I only successfully used the tunnel once. The scenario was that after updating the Mac system version and restarting the system, the unexpectedly worked.

But shortly after that, it stopped working properly again because the remote IP of the tunnel SSH was incorrect. I didn’t find this bastion machine IP in my AWS console.

This problem has been bothering me for a long time because I can’t access the table structure of the remote RDS database.

I was facing the same issue using SST 3.16.0. After downgrading to 3.9.39 (an earlier version that was working for me), I can now see the tunnel picking up the correct IP, I guess something change between versions.

Image

in v3.14.29 the tunnel command prints out the host ip it’s connected to

if this does NOT match what you are seeing in aws console can you share sst export --stage=<stage-vpc-is-in>

Deleting the .sst folder and moving to managed fix it for me. Still an issue on 3.13.14. This was how I got it to finally work:

export const vpc = new sst.aws.Vpc('Vpc', { bastion: true, nat: 'managed' });

First, I flipped nat from ec2 to managed and got some errors about resource conflicts. Then I named it to Vpc2 and that made more errors like: The new Subnets are not in the same Vpc as the existing subnet group. Then I renamed it back to Vpc and it worked worked itself out 🤷.

Another interesting change around this in the past few months is that I have to run sst tunnel separately from my dev process. I remember that I didn’t need to do that at one time.

fwiw, i was able to fix this today by doing npx sst deploy --stage production --target MyVpcName. after running that, the IPs started matching up properly.

On "sst": "3.13.9" (on a mac). The removal of .sst worked yesterday but weirdly not today, the change to nat: 'managed' does work as others said. The IP in the sst tunnel logs with nat: 'ec2' doesn’t match any EC2 instance public IP.

In the mix, random immediate exits/crashes of sst tunnel like in https://github.com/sst/sst/issues/5008, nothing useful in --print-logs though.

Not sure if relevant, but for

sudo -n -E /opt/sst/tunnel tunnel start --subnets 10.0.4.0/22,10.0.12.0/22,10.0.0.0/22,10.0.8.0/22 --host x.x.x.x --user ec2-user --print-logs

the value printed after --host does not match any of the IP’s for EC2 instances that I can see in the console.

I’ve seen this too. Tunneling failed due to wrong IP.

I downgraded to 3.10.7 and tested with personal stage. Only after remove and redeploy the IP changed. Having a 3.12.3 deployment and tunneling with a 3.10.7 without redeploying shows the wrong IP.

EDIT: Tried with sst tunnel --stage mystage instead of sst dev tunnel today. No luck. IP is wrong again. Tunneling to personal stage via just sst tunnel resolves a correct IP.

This is fixed for me after upgrading to 3.12.6 🎉