core: NPTv6 return traffic exiting WAN with wrong IP

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

Describe the bug

I have NPTv6 setup so I can use my limited IPv6 /64 prefix on any network be hide OPNsense. When traffic originates from the machine behide OPNsense (fd37:c611:72fb:80::10) I can curl and ping the internet fine and curl -6 ifconfig.co returns 2001:41d0:800:2647::123:123:123 so I know its working. Monitoring the traffic with tcpdump -i vtnet0 icmp6 on OPNsense I can see the traffic leave the firewall with the correct IP.

Now the bug, when I try access 2001:41d0:800:2647::123:123:123 from the internet, good example is ping6 tool. I can see the traffic go through OPNsense, translate to the fd37:c611:72fb:80::10, then see it on the host be hide OPNsense using tcpdump as well. Then when it goes back through OPNsense it doesn’t get translated back to 2001:41d0:800:2647::123:123:123 when it exits the WAN atleast tcpdump isn’t showing this.

This used to work fine, and I heavy tested it, I believe its broken since going to 21.1, but can’t be 100% sure, since I wasn’t externally monitoring the IP, only internally monitoring.

To Reproduce Create NPT rule

NPT Rule | interface | WAN | | Internal IPv6 Source | fd37:c611:72fb:80::10/128 | | Destination IPv6 Prefix | 2001:41d0:800:2647::123:123:123 | | Description | Mailcow |

Firewall Rule | interface | WAN | | Direction | IN | | TCP/IP Version | IPv6+IPv6 | | Source | ANY | | Destination | ALIAS (contains fd37:c611:72fb:80::10 and 10.111.1.11[One-to-One NAT]) | | Log | checked | | Description | Mailcow ICMP | | Gateway | Default |

Expected behaviour

Traffic to get translated back to the public IPv6, before exiting the WAN.

Describe alternatives you considered

Removing the NPTv6 rule and adding it again.

Relevant log files

Ping start on external (TCPDUMP monitored on PFsense WAN). Doesn’t work

18:24:08.169244 IP6 (flowlabel 0xafd41, hlim 49, next-header ICMPv6 (58) payload length: 64) 2a03:b0c0:3:d0::101:4001 > 2001:41d0:800:2647::123:123:123: [icmp6 sum ok] ICMP6, echo request, seq 1
18:24:08.169588 IP6 (flowlabel 0xce9b9, hlim 62, next-header ICMPv6 (58) payload length: 64) fd37:c611:72fb:80::10 > 2a03:b0c0:3:d0::101:4001: [icmp6 sum ok] ICMP6, echo reply, seq 1
18:24:09.187685 IP6 (flowlabel 0xafd41, hlim 49, next-header ICMPv6 (58) payload length: 64) 2a03:b0c0:3:d0::101:4001 > 2001:41d0:800:2647::123:123:123: [icmp6 sum ok] ICMP6, echo request, seq 2
18:24:09.187912 IP6 (flowlabel 0xce9b9, hlim 62, next-header ICMPv6 (58) payload length: 64) fd37:c611:72fb:80::10 > 2a03:b0c0:3:d0::101:4001: [icmp6 sum ok] ICMP6, echo reply, seq 2
18:24:10.211634 IP6 (flowlabel 0xafd41, hlim 49, next-header ICMPv6 (58) payload length: 64) 2a03:b0c0:3:d0::101:4001 > 2001:41d0:800:2647::123:123:123: [icmp6 sum ok] ICMP6, echo request, seq 3
18:24:10.211935 IP6 (flowlabel 0xce9b9, hlim 62, next-header ICMPv6 (58) payload length: 64) fd37:c611:72fb:80::10 > 2a03:b0c0:3:d0::101:4001: [icmp6 sum ok] ICMP6, echo reply, seq 3

Ping start on Guest. Works

18:24:45.375525 IP6 (flowlabel 0xb6b18, hlim 63, next-header ICMPv6 (58) payload length: 64) 2001:41d0:800:2647::123:123:123 > 2a03:b0c0:3:d0::101:4001: [icmp6 sum ok] ICMP6, echo request, seq 1
18:24:45.390534 IP6 (flowlabel 0x818e3, hlim 49, next-header ICMPv6 (58) payload length: 64) 2a03:b0c0:3:d0::101:4001 > 2001:41d0:800:2647::123:123:123: [icmp6 sum ok] ICMP6, echo reply, seq 1
18:24:46.376990 IP6 (flowlabel 0xb6b18, hlim 63, next-header ICMPv6 (58) payload length: 64) 2001:41d0:800:2647::123:123:123 > 2a03:b0c0:3:d0::101:4001: [icmp6 sum ok] ICMP6, echo request, seq 2
18:24:46.391635 IP6 (flowlabel 0x818e3, hlim 49, next-header ICMPv6 (58) payload length: 64) 2a03:b0c0:3:d0::101:4001 > 2001:41d0:800:2647::123:123:123: [icmp6 sum ok] ICMP6, echo reply, seq 2
18:24:47.378108 IP6 (flowlabel 0xb6b18, hlim 63, next-header ICMPv6 (58) payload length: 64) 2001:41d0:800:2647::123:123:123 > 2a03:b0c0:3:d0::101:4001: [icmp6 sum ok] ICMP6, echo request, seq 3
18:24:47.392740 IP6 (flowlabel 0x818e3, hlim 49, next-header ICMPv6 (58) payload length: 64) 2a03:b0c0:3:d0::101:4001 > 2001:41d0:800:2647::123:123:123: [icmp6 sum ok] ICMP6, echo reply, seq 3

Environment

Software version used and hardware type if relevant, e.g.:

OPNsense 21.1.3_3-amd64 on Proxmox FreeBSD 12.1-RELEASE-p14-HBSD OpenSSL 1.1.1j 16 Feb 2021 Intel® Xeon® E-2236 CPU @ 3.40GHz (2 cores) VTNET (Vertio)

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 33 (25 by maintainers)

Most upvoted comments

Thanks all! The second rule might have been a workaround and broke when pf(4) changes in FreeBSD made it obsolete at some point, maybe going from 11 to 12 in 20.7?

I tried to reproduce the return traffic translation issue but it seems to be intermittent. Maybe related to firewall state? And I’m using the development version (21.7.a_384) so there might be other differences.

But I’m now pretty confident that the reverse binat rule is erroneous. Not sure whether it’s actually responsible for the reply-to incompatibility, but it’s pointless. Binat is by definition bidirectional and a single address can’t be both internal and external.

Patch 3fbf5371dc94d49b676fabfa4f96b9e1d1418699 omits the reverse binat rule. NPT keeps working for me and return traffic gets translated correctly, even with reply-to enabled. @FingerlessGlov3s, could you test opnsense-patch 3fbf537?