core: NAT reflection is broken

Note: It is perhaps not “new”, but there are no open tickets relating to it. Some very similar, potentially the same, have been closed with unclear reasons. It may be that this ticket provides additional material which will help achieve a resolution.

Describe the bug

Using a clean, brand-new installation of the latest OPNsense, NAT reflection does not work.

Background

It has been a few years since I last set up pfSense, and in the intervening time it appears OPNsense has grown in popularity (I had not previously heard of it). The debacle with the pfSense codebase (dodgy Wireguard patches from Netgate to the BSD kernel, that kinda thing) made interesting reading, and I therefore deleted my freshly-downloaded copy of pfSense and instead turned to OPNsense. Setup was painless, and within a short space of time I was up and running, and had replaced my Ubiquiti EdgeRouter with equivalent settings for Internet connectivity and DHCP leases. Just some simple port forwarding to set up to have a complete replacement, and…

The problem

Having configured a pretty vanilla setup, with a PPPoE WAN and a single-subnet LAN, I was surprised when the port forwards I set up did not work. Reading a little around the subject I realised that they were actually working, and external access was fine, but I had to enable NAT reflection in the main/global settings in order to get internal resolution. I dutifully did this, and descended into a rabbit-hole for several hours, trying lots of things with no success. Along the way I found several reports by other people who have apparently encountered the same, or a very similar, issue.

In a nutshell (TLDR)

If you set up NAT port forwarding, even if you have NAT reflection enabled in the main settings and on the forwarding rule, there is no internal resolution of traffic that is directed towards the WAN interface.

To reproduce

Bear with me, as this is a detailed account of how to start from scratch and verify the issue, along with notes and data.

Initial setup

Define VM

Create VM configuration

This setup is running on a VM using KVM on Linux, specifically, Ubuntu Server 23.04.

Create a new virtual machine using virt-manager. Mine is:

  • 4 CPUs
  • 4GB RAM
  • Assign one virtual NIC against br0 (this is a host bridge that is present on the LAN)
  • Assign a physical PCI card for the enp60s0 Intel NIC (this is the interface for the WAN)
  • Assign disk device to the ZFS zvol (the host runs ZFS, and OPNsense is installed into a zvol)

Ensure there are no references to enp60s0 in the host’s netplan config (as this card is a passthrough device to be owned by the OPNsense VM).

Autostart

This is so it will always come to life with the host machine.

virsh autostart OPNsense

Install OPNsense

Boot VM

Install by following the steps laid out, but notably:

  • Use UFS (as this system is backed by ZFS, so no point having ZFS on ZFS)
  • Interfaces should get assigned automatically, but best to specify manually:
    • igc0 to WAN (this is the passthrough physical NIC)
    • vtnet0 to LAN (this is the br0 bridge)

Configure via CLI

Using the CLI, log in and, using the menu:

  • Assign an IP address of 10.0.0.1 to the LAN interface
  • Enable DHCP when prompted
  • Disable HTTPS when prompted

Configure via web GUI

Log into the web UI and run through the setup wizard.

  • Hostname: router
  • Domain: lan
  • Do not allow ISP to override DNS (i.e. disable this)
  • Set DNS server to 10.0.0.2 (the network currently runs a separate DNSmasq server that will later be moved into OPNsense)
  • PPPoE
    • Username: xxxxx
    • Password: yyyyy

It should connect and assign the correct public IP address from the ISP, which will be referred to from now on as 1.2.3.4.

Configure OPNsense

Static IPs for DHCP clients

  • Services → DHCPv4 → [LAN]
    • Set range to 10.0.0.150-239
    • Add all static mappings:
      • MAC address
      • IP address
      • Hostname
      • ARP table static entry: enabled

Port forwarding

  • Firewall → Settings → Advanced
    • Reflection for port forwards: Enabled
    • Reflection for 1:1: Enabled (I am not sure this one should be strictly necessary, but I tried with and without)
    • Automatic outbound NAT for Reflection: Enabled
  • Firewall → NAT → Port Forward
    • Interface: WAN, LAN
    • TCP/IP version: IPv4
    • Protocol: TCP/UDP
    • Destination: WAN address
    • Destination port range: (as required - here I have 80, 443, and 22, so three rules in total)
    • Redirect target IP: Single host or Network
      • 10.0.0.5
    • Redirect target port: (as required - here I have 80, 443, and 2222)
    • Description: (as required)
    • NAT reflection: Enable
    • Filter rule association: Add associated filter rule

I added three rules, for:

  • 80 → 80 (HTTP -> HTTP)
  • 443 → 443 (HTTPS -> HTTPS)
  • 22 → 2222 (SSH -> Other: 2222)

Examination of configuration

The steps above represent my latest, most complete attempt, which is I believe the most accurate and therefore the best to represent as a test case for reproduction.

It is worth noting that, along the way, I tried multiple variations:

  1. My first attempt added the port forwarding rules with no NAT reflection. I then enabled this globally (no effect), and then enabled it in each rule as well (no effect). I then deleted all the rules and recreated them (no effect). This attempt notably only got applied to the WAN interface, as indicated by most documentation.
  2. A later attempt (reverting config and indeed disk snapshot in the meantime) ensured to set up the global NAT reflection config from the start (thinking, perhaps the order mattered, and caused the problem). This made no difference.
  3. I then played with other options, eventually arriving at a position where the LAN interface was specified along with the WAN interface. Paranoid that my fiddling had caused unseen issues, I rolled back the ZFS disk snapshot several times, and made a number of careful comparisons of the config changes.

Note: Although it seems reasonable that the LAN interface might need specifying as well as the WAN interface, I stumbled upon this by accident when reading around the subject. Some comments somewhere. It does make sense, and as the target is the WAN address, should not conflict with the LAN address services. Unfortunately this still didn’t work for me, but if it is indeed a requirement then maybe it should be documented more clearly - for instance in the UI, which itself suggests the WAN interface is what is usually needed.

First config changes

These are the changes resulting from my first attempt, i.e.:

  • Adding the rules to WAN, without reflection
  • Adding the global reflection config
  • Enabling reflection in the rules

I am including these lines for completeness, because I believe that path should work - if not done properly first time, a correction through editing should surely be sufficient.

Configuration diff from 7/4/23 08:08:45 to 7/4/23 10:25:06
--- /conf/backup/config-1688458125.1867.xml	2023-07-04 08:08:45.187386000 +0000
+++ /conf/backup/config-1688466306.9975.xml	2023-07-04 10:25:06.999178000 +0000
@@ -228,14 +228,13 @@
       <protocol>http</protocol>
       <ssl-certref>64a3c924280cb</ssl-certref>
     </webgui>
-    <disablenatreflection>yes</disablenatreflection>
     <usevirtualterminal>1</usevirtualterminal>
     <disableconsolemenu/>
     <disablevlanhwfilter>1</disablevlanhwfilter>
     <disablechecksumoffloading>1</disablechecksumoffloading>
     <disablesegmentationoffloading>1</disablesegmentationoffloading>
     <disablelargereceiveoffloading>1</disablelargereceiveoffloading>
-    <ipv6allow/>
+    <ipv6allow>1</ipv6allow>
     <powerd_ac_mode>hadp</powerd_ac_mode>
     <powerd_battery_mode>hadp</powerd_battery_mode>
     <powerd_normal_mode>hadp</powerd_normal_mode>
@@ -255,6 +254,11 @@
     </firmware>
     <language>en_US</language>
     <dnsserver>10.0.0.2</dnsserver>
+    <enablenatreflectionhelper>yes</enablenatreflectionhelper>
+    <maximumstates/>
+    <maximumfrags/>
+    <aliasesresolveinterval/>
+    <maximumtableentries/>
   </system>
   <interfaces>
     <wan>
@@ -619,6 +623,96 @@
     <outbound>
       <mode>automatic</mode>
     </outbound>
+    <rule>
+      <protocol>tcp/udp</protocol>
+      <interface>wan</interface>
+      <category/>
+      <ipprotocol>inet</ipprotocol>
+      <descr>Gitea HTTP</descr>
+      <tag/>
+      <tagged/>
+      <poolopts/>
+      <associated-rule-id>nat_64a3d5cec4db08.05189375</associated-rule-id>
+      <target>10.0.0.5</target>
+      <local-port>80</local-port>
+      <source>
+        <any>1</any>
+      </source>
+      <destination>
+        <network>wanip</network>
+        <port>80</port>
+      </destination>
+      <updated>
+        <username>root@10.0.0.40</username>
+        <time>1688466293.9502</time>
+        <description>/firewall_nat_edit.php made changes</description>
+      </updated>
+      <created>
+        <username>root@10.0.0.40</username>
+        <time>1688458702.8064</time>
+        <description>/firewall_nat_edit.php made changes</description>
+      </created>
+    </rule>
+    <rule>
+      <protocol>tcp/udp</protocol>
+      <interface>wan</interface>
+      <category/>
+      <ipprotocol>inet</ipprotocol>
+      <descr>Gitea HTTPS</descr>
+      <tag/>
+      <tagged/>
+      <poolopts/>
+      <associated-rule-id>nat_64a3d5f4d81474.92278692</associated-rule-id>
+      <target>10.0.0.5</target>
+      <local-port>443</local-port>
+      <source>
+        <any>1</any>
+      </source>
+      <destination>
+        <network>wanip</network>
+        <port>443</port>
+      </destination>
+      <updated>
+        <username>root@10.0.0.40</username>
+        <time>1688466302.5817</time>
+        <description>/firewall_nat_edit.php made changes</description>
+      </updated>
+      <created>
+        <username>root@10.0.0.40</username>
+        <time>1688458740.8851</time>
+        <description>/firewall_nat_edit.php made changes</description>
+      </created>
+    </rule>
+    <rule>
+      <protocol>tcp/udp</protocol>
+      <interface>wan</interface>
+      <category/>
+      <ipprotocol>inet</ipprotocol>
+      <descr>Gitea SSH</descr>
+      <tag/>
+      <tagged/>
+      <poolopts/>
+      <associated-rule-id>nat_64a3d6261c0f80.88097215</associated-rule-id>
+      <target>10.0.0.5</target>
+      <local-port>2222</local-port>
+      <source>
+        <any>1</any>
+      </source>
+      <destination>
+        <network>wanip</network>
+        <port>22</port>
+      </destination>
+      <updated>
+        <username>root@10.0.0.40</username>
+        <time>1688466306.9404</time>
+        <description>/firewall_nat_edit.php made changes</description>
+      </updated>
+      <created>
+        <username>root@10.0.0.40</username>
+        <time>1688458790.115</time>
+        <description>/firewall_nat_edit.php made changes</description>
+      </created>
+    </rule>
   </nat>
   <filter>
     <rule>
@@ -645,6 +739,69 @@
         <any/>
       </destination>
     </rule>
+    <rule>
+      <source>
+        <any>1</any>
+      </source>
+      <interface>wan</interface>
+      <statetype>keep state</statetype>
+      <protocol>tcp/udp</protocol>
+      <ipprotocol>inet</ipprotocol>
+      <destination>
+        <address>10.0.0.5</address>
+        <port>80</port>
+      </destination>
+      <descr>Gitea HTTP</descr>
+      <category/>
+      <associated-rule-id>nat_64a3d5cec4db08.05189375</associated-rule-id>
+      <created>
+        <username>root@10.0.0.40</username>
+        <time>1688458702.8063</time>
+        <description>/firewall_nat_edit.php made changes</description>
+      </created>
+    </rule>
+    <rule>
+      <source>
+        <any>1</any>
+      </source>
+      <interface>wan</interface>
+      <statetype>keep state</statetype>
+      <protocol>tcp/udp</protocol>
+      <ipprotocol>inet</ipprotocol>
+      <destination>
+        <address>10.0.0.5</address>
+        <port>443</port>
+      </destination>
+      <descr>Gitea HTTPS</descr>
+      <category/>
+      <associated-rule-id>nat_64a3d5f4d81474.92278692</associated-rule-id>
+      <created>
+        <username>root@10.0.0.40</username>
+        <time>1688458740.8851</time>
+        <description>/firewall_nat_edit.php made changes</description>
+      </created>
+    </rule>
+    <rule>
+      <source>
+        <any>1</any>
+      </source>
+      <interface>wan</interface>
+      <statetype>keep state</statetype>
+      <protocol>tcp/udp</protocol>
+      <ipprotocol>inet</ipprotocol>
+      <destination>
+        <address>10.0.0.5</address>
+        <port>2222</port>
+      </destination>
+      <descr>Gitea SSH</descr>
+      <category/>
+      <associated-rule-id>nat_64a3d6261c0f80.88097215</associated-rule-id>
+      <created>
+        <username>root@10.0.0.40</username>
+        <time>1688458790.1149</time>
+        <description>/firewall_nat_edit.php made changes</description>
+      </created>
+    </rule>
   </filter>
   <rrd>
     <enable/>
@@ -700,9 +857,9 @@
     <column_count>2</column_count>
   </widgets>
   <revision>
-    <username>root@10.0.0.161</username>
-    <time>1688458125.1867</time>
-    <description>/services_dhcp_edit.php made changes</description>
+    <username>root@10.0.0.40</username>
+    <time>1688466306.9975</time>
+    <description>/firewall_nat_edit.php made changes</description>
   </revision>
   <OPNsense>
     <IPsec version="1.0.1">

Latest config changes

These are the changes resulting from my latest attempt, which is the most complete and correct, i.e. as described in my steps to reproduce:

  • Adding the global reflection config first, including all three options being enabled
  • Adding the rules to WAN plus LAN, ensuring to enable reflection

I believe that these steps should have resulted in a correct outcome.

Configuration diff from 7/4/23 08:08:45 to 7/4/23 13:25:07
--- /conf/backup/config-1688458125.1867.xml	2023-07-04 08:08:45.187386000 +0000
+++ /conf/config.xml	2023-07-04 13:25:07.469589000 +0000
@@ -228,14 +228,13 @@
       <protocol>http</protocol>
       <ssl-certref>64a3c924280cb</ssl-certref>
     </webgui>
-    <disablenatreflection>yes</disablenatreflection>
     <usevirtualterminal>1</usevirtualterminal>
     <disableconsolemenu/>
     <disablevlanhwfilter>1</disablevlanhwfilter>
     <disablechecksumoffloading>1</disablechecksumoffloading>
     <disablesegmentationoffloading>1</disablesegmentationoffloading>
     <disablelargereceiveoffloading>1</disablelargereceiveoffloading>
-    <ipv6allow/>
+    <ipv6allow>1</ipv6allow>
     <powerd_ac_mode>hadp</powerd_ac_mode>
     <powerd_battery_mode>hadp</powerd_battery_mode>
     <powerd_normal_mode>hadp</powerd_normal_mode>
@@ -255,6 +254,12 @@
     </firmware>
     <language>en_US</language>
     <dnsserver>10.0.0.2</dnsserver>
+    <enablebinatreflection>yes</enablebinatreflection>
+    <enablenatreflectionhelper>yes</enablenatreflectionhelper>
+    <maximumstates/>
+    <maximumfrags/>
+    <aliasesresolveinterval/>
+    <maximumtableentries/>
   </system>
   <interfaces>
     <wan>
@@ -619,6 +624,99 @@
     <outbound>
       <mode>automatic</mode>
     </outbound>
+    <rule>
+      <protocol>tcp/udp</protocol>
+      <interface>lan,wan</interface>
+      <category/>
+      <ipprotocol>inet</ipprotocol>
+      <descr>Gitea HTTP</descr>
+      <tag/>
+      <tagged/>
+      <poolopts/>
+      <associated-rule-id>nat_64a41d269dddd8.11732562</associated-rule-id>
+      <target>10.0.0.5</target>
+      <local-port>80</local-port>
+      <source>
+        <any>1</any>
+      </source>
+      <destination>
+        <network>wanip</network>
+        <port>80</port>
+      </destination>
+      <natreflection>purenat</natreflection>
+      <updated>
+        <username>root@10.0.0.40</username>
+        <time>1688476966.6467</time>
+        <description>/firewall_nat_edit.php made changes</description>
+      </updated>
+      <created>
+        <username>root@10.0.0.40</username>
+        <time>1688476966.6467</time>
+        <description>/firewall_nat_edit.php made changes</description>
+      </created>
+    </rule>
+    <rule>
+      <protocol>tcp/udp</protocol>
+      <interface>lan,wan</interface>
+      <category/>
+      <ipprotocol>inet</ipprotocol>
+      <descr>Gitea HTTPS</descr>
+      <tag/>
+      <tagged/>
+      <poolopts/>
+      <associated-rule-id>nat_64a41d7112c513.18400326</associated-rule-id>
+      <target>10.0.0.5</target>
+      <local-port>443</local-port>
+      <source>
+        <any>1</any>
+      </source>
+      <destination>
+        <network>wanip</network>
+        <port>443</port>
+      </destination>
+      <natreflection>purenat</natreflection>
+      <updated>
+        <username>root@10.0.0.40</username>
+        <time>1688477041.0769</time>
+        <description>/firewall_nat_edit.php made changes</description>
+      </updated>
+      <created>
+        <username>root@10.0.0.40</username>
+        <time>1688477041.0769</time>
+        <description>/firewall_nat_edit.php made changes</description>
+      </created>
+    </rule>
+    <rule>
+      <protocol>tcp/udp</protocol>
+      <interface>lan,wan</interface>
+      <category/>
+      <ipprotocol>inet</ipprotocol>
+      <descr>Gitea SSH</descr>
+      <tag/>
+      <tagged/>
+      <poolopts/>
+      <associated-rule-id>nat_64a41dafcab5c5.02730287</associated-rule-id>
+      <target>10.0.0.5</target>
+      <local-port>2222</local-port>
+      <source>
+        <any>1</any>
+      </source>
+      <destination>
+        <network>wanip</network>
+        <port>22</port>
+      </destination>
+      <natreflection>purenat</natreflection>
+      <updated>
+        <username>root@10.0.0.40</username>
+        <time>1688477103.8303</time>
+        <description>/firewall_nat_edit.php made changes</description>
+      </updated>
+      <created>
+        <username>root@10.0.0.40</username>
+        <time>1688477103.8303</time>
+        <description>/firewall_nat_edit.php made changes</description>
+      </created>
+    </rule>
   </nat>
   <filter>
     <rule>
@@ -645,6 +743,75 @@
         <any/>
       </destination>
     </rule>
+    <rule>
+      <source>
+        <any>1</any>
+      </source>
+      <interface>lan,wan</interface>
+      <statetype>keep state</statetype>
+      <protocol>tcp/udp</protocol>
+      <ipprotocol>inet</ipprotocol>
+      <destination>
+        <address>10.0.0.5</address>
+        <port>80</port>
+      </destination>
+      <floating>1</floating>
+      <quick>yes</quick>
+      <descr>Gitea HTTP</descr>
+      <category/>
+      <associated-rule-id>nat_64a41d269dddd8.11732562</associated-rule-id>
+      <created>
+        <username>root@10.0.0.40</username>
+        <time>1688476966.6466</time>
+        <description>/firewall_nat_edit.php made changes</description>
+      </created>
+    </rule>
+    <rule>
+      <source>
+        <any>1</any>
+      </source>
+      <interface>lan,wan</interface>
+      <statetype>keep state</statetype>
+      <protocol>tcp/udp</protocol>
+      <ipprotocol>inet</ipprotocol>
+      <destination>
+        <address>10.0.0.5</address>
+        <port>443</port>
+      </destination>
+      <floating>1</floating>
+      <quick>yes</quick>
+      <descr>Gitea HTTPS</descr>
+      <category/>
+      <associated-rule-id>nat_64a41d7112c513.18400326</associated-rule-id>
+      <created>
+        <username>root@10.0.0.40</username>
+        <time>1688477041.0769</time>
+        <description>/firewall_nat_edit.php made changes</description>
+      </created>
+    </rule>
+    <rule>
+      <source>
+        <any>1</any>
+      </source>
+      <interface>lan,wan</interface>
+      <statetype>keep state</statetype>
+      <protocol>tcp/udp</protocol>
+      <ipprotocol>inet</ipprotocol>
+      <destination>
+        <address>10.0.0.5</address>
+        <port>2222</port>
+      </destination>
+      <floating>1</floating>
+      <quick>yes</quick>
+      <descr>Gitea SSH</descr>
+      <category/>
+      <associated-rule-id>nat_64a41dafcab5c5.02730287</associated-rule-id>
+      <created>
+        <username>root@10.0.0.40</username>
+        <time>1688477103.8303</time>
+        <description>/firewall_nat_edit.php made changes</description>
+      </created>
+    </rule>
   </filter>
   <rrd>
     <enable/>
@@ -700,9 +867,9 @@
     <column_count>2</column_count>
   </widgets>
   <revision>
-    <username>root@10.0.0.161</username>
-    <time>1688458125.1867</time>
-    <description>/services_dhcp_edit.php made changes</description>
+    <username>root@10.0.0.40</username>
+    <time>1688477107.4679</time>
+    <description>/firewall_nat.php made changes</description>
   </revision>
   <OPNsense>
     <IPsec version="1.0.1">

Difference between config changes

What’s strange is that there’s very little changed between the two.

When specifying WAN alone, the rules end up in Firewall -> Rules -> WAN. When specifying WAN plus LAN, the same rules get shown in Firewall -> Rules -> Floating. I don’t know if that’s expected, or what “floating” means exactly - I was perhaps expecting to see some rules under both WAN and LAN, but other than appearing in a slightly different place, the rules seem the same.

The later config has <natreflection>purenat</natreflection> added to each of the rules, and the rules also say they are “floating” and “quick”. They also obviously specify lan,wan and not just wan. But these differences do not appear to have actually changed anything, and I am not sure if they are needed. What is clear is that either a) the rules in place are not working correctly, or b) there are some missing.

Expected behavior

My expectation was as follows:

  • Given that I have a working public IP of 1.2.3.4, and a DNS entry defined against a domain, pointing at it, external traffic should see it and route to it. This does indeed happen, correctly, and external clients can use the Gitea system without issue.
  • Given that I have set up NAT reflection, internal traffic directed toward the WAN address of 1.2.3.4 should be translated and redirected to 10.0.0.5. This is not happening.
  • Given that I have moreover specified the rules to apply to the LAN interface (but not address) as well as the WAN interface, this should definitely be enough to route/redirect all internal traffic sent to the WAN address. This is not happening.

Additionally, we could say that I expect more rules - but I am not 100% sure of that, as it may be that the ones in place are not working correctly.

Describe alternatives you considered

I have considered using unbound DNS in order to get ahead of the requests and make the clients send those requests to the internal 10.0.0.5 IP instead of the external IP. However, this is not particularly practicable, as a) there are various TLS certificates and similar in place that complicate that approach, and b) there are some setups (moving away from the simple Gitea setup here) that need to correctly use proper DNS as part of their automated testing. So this is not really a solution for me.

Another alternative might be to go back to my EdgeRouter, which worked fine… or, maybe, swap over to pfSense, which apparently does not have this specific issue. But I am reluctant to do either of those things.

Screenshots

There are some screenshots that might be useful; not sure how much they add to the steps and config above, but here goes:

List of rules: Firewall -> NAT -> Port Forward

image

Rule configuration: Firewall -> NAT -> Port Forward -> (Rule)

image

List of rules: Firewall -> Rules -> Floating

image

Main/global settings: Firewall -> Settings -> Advanced

image

Relevant log files

I’m not sure what log files would be of use here - I have trawled through what is available but not seen anything notable to submit. Please let me know if you would like something specific.

Additional context

It’s worth noting that this is a brand-new setup on a brand-new system. It’s not an upgrade, it’s not a port or import of settings from elsewhere, and it doesn’t have anything unusual going on - i.e. it should not be an edge case. It’s just bog-standard port forwarding. Therefore it should hopefully be easier to validate than situations where there are additional factors taking effect.

Environment

Host system

  • Ubuntu Server 23.04
  • AMD Ryzen 9 5950X
  • 128GB RAM
  • NVMe drives using ZFS
  • KVM hypervisor

VM configured for OPNsense

  • OPNsense 23.1 OpenSSL AMD64s (using the ISO)
  • 4 CPU cores
  • 4GB RAM
  • UFS chosen for storage
  • LAN: bridge interface present on physical network (10GbE NIC)
  • WAN: PCI card passthrough (Intel 2.5GbE NIC)
  • Internet: PPPoE, gigabit FTTP

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 74 (34 by maintainers)

Most upvoted comments

@NunoHiggs

I have written a Tutorial in the Opnsense Forum as result of this github thread: https://forum.opnsense.org/index.php?topic=34925.0

@AdSchellevis I have checked the rules.debug, and I saw that the nat rules were there, but not in pfctl. So I just hit apply again and then they appeared. It was probably a user error on my side.

The rules look fine. I commented the SNAT rule and its exactly whats needed.

root@opn01:/tmp # cat rules.debug | grep -i nat

# NAT Redirects
nat on pppoe1 inet from (hn4:network) to any -> (pppoe1:0) port 1024:65535 # Automatic outbound rule
rdr on pppoe1 inet proto tcp from {any} to {(pppoe1)} port {33333} -> 10.110.44.252 port 33333 # Nat Reflection Test PPPoE
nat on pppoe1 inet proto tcp from (pppoe1:network) to {10.110.44.252} port {33333} -> (pppoe1) port 1024:65535 # Nat Reflection Test PPPoE
rdr on hn4 inet proto tcp from {any} to {(pppoe1)} port {33333} -> 10.110.44.252 port 33333 # Nat Reflection Test PPPoE
nat on hn4 inet proto tcp from (hn4:network) to {10.110.44.252} port {33333} -> (hn4) port 1024:65535 # Nat Reflection Test

root@opn01:/tmp # pfctl -s nat

# Automatic Outbound NAT
nat on pppoe1 inet from (hn4:network) to any -> (pppoe1:0) port 1024:65535

# SNAT
# If a packet is received by interface hn4 with protocol TCP from the source ip hn4:network (10.110.44.0/24) to destination
# ip 10.110.44.252/32 and destination port 33333 -> rewrite the source ip to the interface ip hn4 (10.110.44.254/32) and the
# source port to the upper port range 1024:65535, with no static port and round robin.
nat on hn4 inet proto tcp from (hn4:network) to 10.110.44.252 port = 33333 -> (hn4) port 1024:65535 round-robin

# DNAT
rdr on pppoe1 inet proto tcp from any to (pppoe1) port = 33333 -> 10.110.44.252 port 33333
rdr on hn4 inet proto tcp from any to (pppoe1) port = 33333 -> 10.110.44.252 port 33333

After all the tests I did, even testing special use cases explained above, there still doesn’t seem to be a bug. Maybe I could contribute sometime to the documentation of the NAT Reflection feature, to explain things like troubleshooting, policy creation, and testing of traffic scenarios.

But for me, I’m kinda done, can’t continue cause there’s nothing left to prove. I think it’s proven that NAT Reflection works, except in some very specific weird edge cases that are out of scope for OPNsense.

I have done enough. Anybody can replicate the setup I did above and mess around with the rules themselves. From all the different setups I tested, I’m pretty sure there is no bug. If I continue now, there will always be a “but what about this scenario” and things will never end.

@Monviech I’m confused why your floating firewall rule has the “WAN address” as target and not the redirect target IP (192.168.1.10) from the port forward.

I made a test kvm hypervisor with Open vSwitch

Network Configuration:

:/etc/libvirt/qemu/networks$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.2 LTS
Release:        22.04
Codename:       jammy
:/etc/libvirt/qemu/networks$ sudo kvm --version
QEMU emulator version 6.2.0 (Debian 1:6.2+dfsg-2ubuntu6.11)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
:/etc/libvirt/qemu/networks$ sudo ovs-vsctl show
42054917-45dd-4ed4-aa93-44b3c94bbb78
    ovs_version: "2.17.7"
:/etc/libvirt/qemu/networks$ sudo virsh --version
8.0.0
:/etc/libvirt/qemu/networks$ sudo nano br0.xml 
<network>
  <name>br0</name>
  <uuid>1106d833-19c8-4eb8-bb43-c241838eca22</uuid>
  <forward mode='bridge'/>
  <bridge name='br0'/>
  <virtualport type='openvswitch'/>
</network>
:/etc/libvirt/qemu/networks$ sudo nano br1.xml 
<network>
  <name>br1</name>
  <uuid>a6eafd21-cd84-4849-bfb0-f0d20288bf03</uuid>
  <forward mode='bridge'/>
  <bridge name='br1'/>
  <virtualport type='openvswitch'/>
</network>
:/etc/libvirt/qemu/networks$ sudo ovs-vsctl show
42054917-45dd-4ed4-aa93-44b3c94bbb78
    Bridge br1
        Port vnet3
            Interface vnet3
        Port br1
            Interface br1
                type: internal
    Bridge br0
        Port br0
            Interface br0
                type: internal
        Port vnet2
            Interface vnet2
    ovs_version: "2.17.7"
13: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether e2:c7:4d:f4:75:47 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.2/24 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::e0c7:4dff:fef4:7547/64 scope link 
       valid_lft forever preferred_lft forever
14: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 7a:9b:20:52:ef:4d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::789b:20ff:fe52:ef4d/64 scope link 
       valid_lft forever preferred_lft forever
17: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UNKNOWN group default qlen 1000
    link/ether fe:54:00:5f:d5:aa brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe5f:d5aa/64 scope link 
       valid_lft forever preferred_lft forever
18: vnet3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UNKNOWN group default qlen 1000
    link/ether fe:54:00:0b:83:2d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe0b:832d/64 scope link 
       valid_lft forever preferred_lft forever

Opnsense Configuration

grafik grafik grafik grafik grafik

EDITED: image EDIT END

As Test client I use the hypervisor with the ip address 192.168.1.2 on br0

The test is pinging 1.2.3.4 and checking with tcpdump if the rules are matched accordingly for NAT Reflection.

host ping

:/etc/libvirt/qemu/networks$ ping 1.2.3.4
PING 1.2.3.4 (1.2.3.4) 56(84) bytes of data.
64 bytes from 1.2.3.4: icmp_seq=1 ttl=63 time=0.602 ms
^C
--- 1.2.3.4 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.602/0.602/0.602/0.000 ms

opnsense tcpdump grafik

host tcpdump

:~$ sudo tcpdump -i br0 -n
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on br0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
20:54:22.936178 IP 192.168.1.2 > 1.2.3.4: ICMP echo request, id 11, seq 1, length 64
20:54:22.936613 IP 1.2.3.4 > 192.168.1.2: ICMP echo request, id 11, seq 1, length 64
20:54:22.936659 IP 192.168.1.2 > 1.2.3.4: ICMP echo reply, id 11, seq 1, length 64
20:54:22.936749 IP 1.2.3.4 > 192.168.1.2: ICMP echo reply, id 11, seq 1, length 64

ANALYSIS:

The Nat reflection works, I can’t find the bug.

Let’s see if this is indeed the issue people are having, documentation might help, but unfortunately this (reflection) is indeed a convoluted concept which we probably wouldn’t have added it it wasn’t already there.