tailscale: userspace-networking incoming TCP doesn't always work right away

Found it!

func (f *Forwarder) HandlePacket(id stack.TransportEndpointID, pkt *stack.PacketBuffer) bool {
        s := newIncomingSegment(id, f.stack.Clock(), pkt)
        defer s.decRef()

        // We only care about well-formed SYN packets.                                                                        
        if !s.parse(pkt.RXTransportChecksumValidated) || !s.csumValid || s.flags != header.TCPFlagSyn {
                return false
        }

Note the s.flags != header.TCPFlagSyn.

😂

I CAN REPRODUCE IT!!! Only on Linux, but good enough to convince me that https://github.com/tailscale/tailscale/pull/3770 is the fix.

On Linux:

$ sudo sysctl net.ipv4.tcp_ecn=1

(For me, it was set to 2 before, which means “Enable ECN when requested by incoming connections but do not request ECN on outgoing connections”)

But once you force ECN on on Linux, then modified that PR to log when it removed it:

        if p.RemoveECNBits() { // Issue 2642 
                ns.logf("XXX netstack removed ECN")
        }

Then you’ll see curl to the userspace tailscaled IP work every time, with the logging that it removed ECN:

2022/01/19 18:41:23.014481 XXX netstack removed ECN
2022/01/19 18:41:23.048391 netstack: forwarder connection to 127.0.0.1:8080 closed
2022/01/19 18:41:23.496088 XXX netstack removed ECN
2022/01/19 18:41:23.531139 netstack: forwarder connection to 127.0.0.1:8080 closed
2022/01/19 18:41:23.812847 XXX netstack removed ECN
2022/01/19 18:41:23.846999 netstack: forwarder connection to 127.0.0.1:8080 closed

But if you comment out the RemoveECNBits line instead, it fails every time:

curl: (7) Failed to connect to 100.70.219.120 port 8080: Connection refused
root@tsbi:/home/bradfitz# curl 100.70.219.120:8080
curl: (7) Failed to connect to 100.70.219.120 port 8080: Connection refused
root@tsbi:/home/bradfitz# curl 100.70.219.120:8080
curl: (7) Failed to connect to 100.70.219.120 port 8080: Connection refused
root@tsbi:/home/bradfitz# curl 100.70.219.120:8080
curl: (7) Failed to connect to 100.70.219.120 port 8080: Connection refused
root@tsbi:/home/bradfitz# curl 100.70.219.120:8080
curl: (7) Failed to connect to 100.70.219.120 port 8080: Connection refused
root@tsbi:/home/bradfitz# curl 100.70.219.120:8080
curl: (7) Failed to connect to 100.70.219.120 port 8080: Connection refused
root@tsbi:/home/bradfitz# curl 100.70.219.120:8080
curl: (7) Failed to connect to 100.70.219.120 port 8080: Connection refused
root@tsbi:/home/bradfitz# curl 100.70.219.120:8080
curl: (7) Failed to connect to 100.70.219.120 port 8080: Connection refused

Seems 100% reproducible, which is great.

macOS does its weird 5% or 50% or automatic thing with ECN which makes it really hard to debug when ECN is actually being used. I think there’s a ioctl/sockopt to force ECN, but I’m not exactly sure what it is. But for now I don’t care, since I can reproduce perfectly from Linux.

Still not sure why gVisor hates ECN so much, but at least we have a workaround.

cc @danderson @DentonGentry

When the 1.22 tree is branched I’ll remove our ECN bit masking workaround on main and close this bug.

Fixed (with workaround) in 1.20.2 and 1.21.x unstable builds at the same time.

Keeping this open to track removing our workarounds and using the upstream fix (https://github.com/google/gvisor/commit/0b81b32c95f6643936d0894671d0b00809fe22d6) instead.

I dug through the git history on google/gvisor and it goes all the way back to the first code dump from the Google tree to github when the code at the time was:

commit d02b74a5dcfed4bfc8f2f8e545bca4d2afabb296
Author: Googler <noreply@google.com>
Date:   Fri Apr 27 10:37:02 2018 -0700

    Check in gVisor.
    
    PiperOrigin-RevId: 194583126
    Change-Id: Ica1d8821a90f74e7e745962d71801c598c652463

…

func (f *Forwarder) HandlePacket(r *stack.Route, id stack.TransportEndpointID, vv *buffer.VectorisedView) bool {
       s := newSegment(r, id, vv)
       defer s.decRef()

       // We only care about well-formed SYN packets.
       if !s.parse() || s.flags != flagSyn {
               return false
       }

@bradfitz Happy to help - your product is probably the single most magical piece of software I use on a daily basis these days 😃

Yes, I can reproduce 100% of the time when connecting to at least one particular host. Let me know if it’s easiest to connect live for a minute. Happy to demo.