core: NPTv6 return traffic exiting WAN with wrong IP
Important notices
Before you add a new report, we ask you kindly to acknowledge the following:
- I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
- I have searched the existing issues and I am convinced that mine is new.
Describe the bug
I have NPTv6 setup so I can use my limited IPv6 /64 prefix on any network be hide OPNsense. When traffic originates from the machine behide OPNsense (fd37:c611:72fb:80::10) I can curl and ping the internet fine and curl -6 ifconfig.co
returns 2001:41d0:800:2647::123:123:123
so I know its working. Monitoring the traffic with tcpdump -i vtnet0 icmp6
on OPNsense I can see the traffic leave the firewall with the correct IP.
Now the bug, when I try access 2001:41d0:800:2647::123:123:123
from the internet, good example is ping6 tool. I can see the traffic go through OPNsense, translate to the fd37:c611:72fb:80::10
, then see it on the host be hide OPNsense using tcpdump as well. Then when it goes back through OPNsense it doesn’t get translated back to 2001:41d0:800:2647::123:123:123
when it exits the WAN atleast tcpdump isn’t showing this.
This used to work fine, and I heavy tested it, I believe its broken since going to 21.1, but can’t be 100% sure, since I wasn’t externally monitoring the IP, only internally monitoring.
To Reproduce Create NPT rule
NPT Rule | interface | WAN | | Internal IPv6 Source | fd37:c611:72fb:80::10/128 | | Destination IPv6 Prefix | 2001:41d0:800:2647::123:123:123 | | Description | Mailcow |
Firewall Rule | interface | WAN | | Direction | IN | | TCP/IP Version | IPv6+IPv6 | | Source | ANY | | Destination | ALIAS (contains fd37:c611:72fb:80::10 and 10.111.1.11[One-to-One NAT]) | | Log | checked | | Description | Mailcow ICMP | | Gateway | Default |
Expected behaviour
Traffic to get translated back to the public IPv6, before exiting the WAN.
Describe alternatives you considered
Removing the NPTv6 rule and adding it again.
Relevant log files
Ping start on external (TCPDUMP monitored on PFsense WAN). Doesn’t work
18:24:08.169244 IP6 (flowlabel 0xafd41, hlim 49, next-header ICMPv6 (58) payload length: 64) 2a03:b0c0:3:d0::101:4001 > 2001:41d0:800:2647::123:123:123: [icmp6 sum ok] ICMP6, echo request, seq 1
18:24:08.169588 IP6 (flowlabel 0xce9b9, hlim 62, next-header ICMPv6 (58) payload length: 64) fd37:c611:72fb:80::10 > 2a03:b0c0:3:d0::101:4001: [icmp6 sum ok] ICMP6, echo reply, seq 1
18:24:09.187685 IP6 (flowlabel 0xafd41, hlim 49, next-header ICMPv6 (58) payload length: 64) 2a03:b0c0:3:d0::101:4001 > 2001:41d0:800:2647::123:123:123: [icmp6 sum ok] ICMP6, echo request, seq 2
18:24:09.187912 IP6 (flowlabel 0xce9b9, hlim 62, next-header ICMPv6 (58) payload length: 64) fd37:c611:72fb:80::10 > 2a03:b0c0:3:d0::101:4001: [icmp6 sum ok] ICMP6, echo reply, seq 2
18:24:10.211634 IP6 (flowlabel 0xafd41, hlim 49, next-header ICMPv6 (58) payload length: 64) 2a03:b0c0:3:d0::101:4001 > 2001:41d0:800:2647::123:123:123: [icmp6 sum ok] ICMP6, echo request, seq 3
18:24:10.211935 IP6 (flowlabel 0xce9b9, hlim 62, next-header ICMPv6 (58) payload length: 64) fd37:c611:72fb:80::10 > 2a03:b0c0:3:d0::101:4001: [icmp6 sum ok] ICMP6, echo reply, seq 3
Ping start on Guest. Works
18:24:45.375525 IP6 (flowlabel 0xb6b18, hlim 63, next-header ICMPv6 (58) payload length: 64) 2001:41d0:800:2647::123:123:123 > 2a03:b0c0:3:d0::101:4001: [icmp6 sum ok] ICMP6, echo request, seq 1
18:24:45.390534 IP6 (flowlabel 0x818e3, hlim 49, next-header ICMPv6 (58) payload length: 64) 2a03:b0c0:3:d0::101:4001 > 2001:41d0:800:2647::123:123:123: [icmp6 sum ok] ICMP6, echo reply, seq 1
18:24:46.376990 IP6 (flowlabel 0xb6b18, hlim 63, next-header ICMPv6 (58) payload length: 64) 2001:41d0:800:2647::123:123:123 > 2a03:b0c0:3:d0::101:4001: [icmp6 sum ok] ICMP6, echo request, seq 2
18:24:46.391635 IP6 (flowlabel 0x818e3, hlim 49, next-header ICMPv6 (58) payload length: 64) 2a03:b0c0:3:d0::101:4001 > 2001:41d0:800:2647::123:123:123: [icmp6 sum ok] ICMP6, echo reply, seq 2
18:24:47.378108 IP6 (flowlabel 0xb6b18, hlim 63, next-header ICMPv6 (58) payload length: 64) 2001:41d0:800:2647::123:123:123 > 2a03:b0c0:3:d0::101:4001: [icmp6 sum ok] ICMP6, echo request, seq 3
18:24:47.392740 IP6 (flowlabel 0x818e3, hlim 49, next-header ICMPv6 (58) payload length: 64) 2a03:b0c0:3:d0::101:4001 > 2001:41d0:800:2647::123:123:123: [icmp6 sum ok] ICMP6, echo reply, seq 3
Environment
Software version used and hardware type if relevant, e.g.:
OPNsense 21.1.3_3-amd64 on Proxmox FreeBSD 12.1-RELEASE-p14-HBSD OpenSSL 1.1.1j 16 Feb 2021 Intel® Xeon® E-2236 CPU @ 3.40GHz (2 cores) VTNET (Vertio)
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 33 (25 by maintainers)
Thanks all! The second rule might have been a workaround and broke when pf(4) changes in FreeBSD made it obsolete at some point, maybe going from 11 to 12 in 20.7?
I tried to reproduce the return traffic translation issue but it seems to be intermittent. Maybe related to firewall state? And I’m using the development version (21.7.a_384) so there might be other differences.
But I’m now pretty confident that the reverse binat rule is erroneous. Not sure whether it’s actually responsible for the reply-to incompatibility, but it’s pointless. Binat is by definition bidirectional and a single address can’t be both internal and external.
Patch 3fbf5371dc94d49b676fabfa4f96b9e1d1418699 omits the reverse binat rule. NPT keeps working for me and return traffic gets translated correctly, even with reply-to enabled. @FingerlessGlov3s, could you test
opnsense-patch 3fbf537
?