core: IPv6 Track Interface sometimes fails to assign GUAs after PPPoE reconnect

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug This bug report is based upon the discussion in https://forum.opnsense.org/index.php?topic=31443.0. By now, I believe this is a bug.

When using DHCPv6 with a /56 prefix request on the WAN together with v4 PPPoE and a local interface in track interface mode, opnsense sometimes looses the prefix after a PPPoE reconnect, making it impossible for the other interfaces to generate GUAs. Currently, all the cases that I could find/observe are based on Deutsche Telekom as ISP with full Dual Stack.

Unfortunately, this issue seems to be hard to reproduce as in many cases, there are no problems. However, if you have a daily cron for a reconnect, you can probably observe the behavior about 2 times a week (I would estimate).

To Reproduce

1. Set the WAN exactly according to https://docs.opnsense.org/manual/how-tos/ipv6_dsl.html with a /56 prefix.

2. Set one (or multiple) interfaces to “Track Interface - WAN”, correctly generating GUAs for the interfaces as well as the hosts in there (everything fine so far)

3. Trigger a WAN reconnect (either via Overview or e.g. via a Cron)

4. Sometimes(!): Loose basically all IPv6 connectivity for the hosts 4.1: See that the live view FW log is filling up with default denys from the LAN (probably the old IPv6 addresses with invalid states on that interface) 4.2 See that the line with “prefix” on the WAN interface in the overview is missing (i.e. probable root cause: no prefix is assigned) 4.3 See that the Interface overview on the Dashboard only shows “track6” instead of GUAs 4.4 See that the DHCPv6 service is down and cannot be restarted (probably due to not being able to generate GUAs)

5. Trigger another reconnect with the same settings on the WAN or reboot FW and usually regain connectivity

Expected behavior

GUAs (based on the /56 prefix on the WAN) should be assigned to the interfaces in track6 mode.

Describe alternatives you considered

As mentioned, the issue seems to be related to the fact that no prefix on the WAN is obtained after the PPPoE reconnect and therefore basically everything relying on v6 GUAs breaks in the process. Currently, it seems hard to tell if it’s an Opnsense issue, but to be honest I have been running a pfsense on the same line up until a few weeks ago with no such behavior. As a workaround: maybe, it would be possible to implement a check in the newwanip script to see if a prefix has been obtained (when DHCPv6 is active on WAN and prefix request is ticked) and if not, trigger another reconnect until a prefix is assigned? I’m not sure if that is a feasible idea (e.g. I don’t know if some ISPs enforce a rate limiting etc.).

Screenshots

All that can be seen is already described above.

Relevant log files So far, no really relevant log entries could be found. Here is one example. Another (by another user with the same problem can be found at https://forum.opnsense.org/index.php?topic=21682.msg148182#msg148182:

2022-12-31T03:30:23	Error	opnsense	/usr/local/etc/rc.newwanipv6: The command '/usr/local/sbin/radvd -p /var/run/radvd.pid -C /var/etc/radvd.conf -m syslog' returned exit code '255', the output was ''	
2022-12-31T03:30:23	Error	opnsense	/usr/local/etc/rc.newwanipv6: Warning! dhcpd_radvd_configure(auto) found no suitable IPv6 address on ix0	
2022-12-31T03:30:23	Error	opnsense	/usr/local/etc/rc.newwanipv6: Warning! dhcpd_radvd_configure(manual) found no suitable IPv6 address on vlan011	
2022-12-31T03:30:23	Error	opnsense	/usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'wan'	
2022-12-31T03:30:23	Error	opnsense	/usr/local/etc/rc.newwanipv6: Warning! dhcpd_radvd_configure(auto) found no suitable IPv6 address on ix0	
2022-12-31T03:30:23	Error	opnsense	/usr/local/etc/rc.newwanipv6: Warning! dhcpd_radvd_configure(manual) found no suitable IPv6 address on vlan011	
2022-12-31T03:30:08	Notice	opnsense	plugins_configure newwanip (execute task : webgui_configure_do(,wan))	
2022-12-31T03:30:08	Notice	opnsense	plugins_configure newwanip (execute task : vxlan_configure_do())	
2022-12-31T03:30:07	Notice	opnsense	/usr/local/sbin/pluginctl: plugins_configure dns_reload (execute task : system_hosts_generate(1))	
2022-12-31T03:30:07	Notice	opnsense	/usr/local/sbin/pluginctl: plugins_configure dns_reload (execute task : system_resolvconf_generate(1))	
2022-12-31T03:30:07	Notice	opnsense	/usr/local/sbin/pluginctl: plugins_configure dns_reload (1)	
2022-12-31T03:30:06	Notice	dhcp6c		dhcp6c RELEASE on pppoe0 - running dns reload	
2022-12-31T03:30:06	Notice	opnsense	plugins_configure newwanip (execute task : unbound_configure_do(,wan))	
2022-12-31T03:30:06	Notice	opnsense	plugins_configure newwanip (execute task : openssh_configure_do(,wan))	
2022-12-31T03:30:06	Notice	opnsense	plugins_configure newwanip (execute task : opendns_configure_do())	
2022-12-31T03:30:06	Notice	opnsense	/usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : ntpd_configure_do())	
2022-12-31T03:30:06	Notice	opnsense	/usr/local/etc/rc.newwanip: plugins_configure newwanip (execute task : dnsmasq_configure_do())	
2022-12-31T03:30:06	Notice	opnsense	/usr/local/etc/rc.newwanip: plugins_configure newwanip (,wan)	
2022-12-31T03:30:06	Error	opnsense	/usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN.	
2022-12-31T03:30:06	Notice	opnsense	/usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : openvpn_configure_do(,wan))	
2022-12-31T03:30:06	Notice	opnsense	/usr/local/etc/rc.newwanip: plugins_configure vpn (execute task : ipsec_configure_do(,wan))	
2022-12-31T03:30:06	Notice	opnsense	/usr/local/etc/rc.newwanip: plugins_configure vpn (,wan)	
2022-12-31T03:30:06	Error	opnsense	/usr/local/etc/rc.newwanip: IP address renew, killing all previous states	
2022-12-31T03:30:05	Notice	opnsense	/usr/local/etc/rc.newwanipv6: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(,inet6))	
2022-12-31T03:30:05	Notice	opnsense	/usr/local/etc/rc.newwanipv6: plugins_configure dhcp (,inet6)	
2022-12-31T03:30:05	Error	opnsense	/usr/local/etc/rc.newwanipv6: On (IP address: XXXX:XX:XXXX:XXXX:XXXX:XXX:XXXX:XXXX) (interface: WAN[wan]) (real interface: pppoe0).	
2022-12-31T03:30:05	Error	opnsense	/usr/local/etc/rc.newwanipv6: IPv6 renewal is starting on 'pppoe0'	
2022-12-31T03:30:05	Notice	dhcp6c		dhcp6c REQUEST on pppoe0 - running newipv6	
2022-12-31T03:30:05	Notice	opnsense	/usr/local/etc/rc.newwanip: plugins_run return_gateways_status (execute task : dpinger_status())	
2022-12-31T03:30:05	Notice	opnsense	/usr/local/etc/rc.newwanip: plugins_run return_gateways_status ()	
2022-12-31T03:30:04	Error	opnsense	/usr/local/etc/rc.newwanip: Adding static route for monitor 8.8.8.8 via xx.xxx.xxx.xxx	
2022-12-31T03:30:04	Error	opnsense	/usr/local/etc/rc.newwanip: Removing static route for monitor 8.8.8.8 via xx.xxx.xxx.xxx	
2022-12-31T03:30:04	Error	opnsense	/usr/local/etc/rc.newwanip: Chose to bind WAN_PPPOE on xxx.xxx.xxx.xxx since we could not find a proper match.	
2022-12-31T03:30:04	Notice	opnsense	/usr/local/etc/rc.newwanip: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_PPPOE))	
2022-12-31T03:30:04	Notice	opnsense	/usr/local/etc/rc.newwanip: plugins_configure monitor (,WAN_PPPOE)	
2022-12-31T03:30:04	Error	opnsense	/usr/local/etc/rc.newwanip: The WAN_DHCP6 monitor address is empty, skipping.	
2022-12-31T03:30:04	Notice	opnsense	/usr/local/etc/rc.newwanip: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_DHCP6))	
2022-12-31T03:30:04	Notice	opnsense	/usr/local/etc/rc.newwanip: plugins_configure monitor (,WAN_DHCP6)	
2022-12-31T03:30:04	Error	opnsense	/usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway 'feXX::XXXX:XXXX:XXXX:XXXX%pppoe0'	
2022-12-31T03:30:04	Error	opnsense	/usr/local/etc/rc.newwanip: ROUTING: setting IPv6 default route to feXX::XXXX:XXXX:XXXX:XXXX	
2022-12-31T03:30:04	Error	opnsense	/usr/local/etc/rc.newwanip: ROUTING: IPv6 default gateway set to wan	
2022-12-31T03:30:04	Error	opnsense	/usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway 'xx.xxx.xxx.xxx'	
2022-12-31T03:30:04	Error	opnsense	/usr/local/etc/rc.newwanip: ROUTING: setting IPv4 default route to xx.xxx.xxx.xxx	
2022-12-31T03:30:04	Error	opnsense	/usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to wan	
2022-12-31T03:30:03	Notice	opnsense	/usr/local/etc/rc.newwanipv6: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(,inet6))	
2022-12-31T03:30:03	Notice	opnsense	/usr/local/etc/rc.newwanipv6: plugins_configure dhcp (,inet6)	
2022-12-31T03:30:03	Error	opnsense	/usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan'	
2022-12-31T03:30:03	Notice	dhcp6c		RTSOLD script - Sending SIGHUP to dhcp6c	
2022-12-31T03:30:03	Notice	dhcp6c		RTSOLD script - Sending SIGHUP to dhcp6c	
2022-12-31T03:30:03	Notice	opnsense	/usr/local/etc/rc.configure_interface: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(1))	
2022-12-31T03:30:03	Notice	opnsense	/usr/local/etc/rc.configure_interface: plugins_configure dhcp (1)	
2022-12-31T03:30:03	Notice	opnsense	/usr/local/etc/rc.configure_interface: plugins_configure ipsec (execute task : ipsec_configure_do(1,wan))	
2022-12-31T03:30:03	Notice	opnsense	/usr/local/etc/rc.configure_interface: plugins_configure ipsec (1,wan)	
2022-12-31T03:30:03	Error	opnsense	/usr/local/etc/rc.newwanipv6: On (IP address: XXXX:XX:XXXX:XXXX:XXXX:XXX:XXXX:XXXX) (interface: WAN[wan]) (real interface: pppoe0).	
2022-12-31T03:30:03	Error	opnsense	/usr/local/etc/rc.configure_interface: Adding static route for monitor 8.8.8.8 via xx.xxx.xxx.xxx	
2022-12-31T03:30:03	Error	opnsense	/usr/local/etc/rc.configure_interface: Removing static route for monitor 8.8.8.8 via xx.xxx.xxx.xxx	
2022-12-31T03:30:03	Error	opnsense	/usr/local/etc/rc.configure_interface: Chose to bind WAN_PPPOE on xxx.xxx.xxx.xxx since we could not find a proper match.	
2022-12-31T03:30:03	Error	opnsense	/usr/local/etc/rc.newwanipv6: IPv6 renewal is starting on 'pppoe0'	
2022-12-31T03:30:03	Notice	opnsense	/usr/local/etc/rc.configure_interface: plugins_configure monitor (execute task : dpinger_configure_do(1,WAN_PPPOE))	
2022-12-31T03:30:03	Notice	opnsense	/usr/local/etc/rc.configure_interface: plugins_configure monitor (1,WAN_PPPOE)	
2022-12-31T03:30:03	Error	opnsense	/usr/local/etc/rc.configure_interface: The WAN_DHCP6 monitor address is empty, skipping.	
2022-12-31T03:30:03	Error	opnsense	/usr/local/etc/rc.newwanip: On (IP address: xxx.xxx.xxx.xxx) (interface: WAN[wan]) (real interface: pppoe0).	
2022-12-31T03:30:03	Error	opnsense	/usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'pppoe0'	
2022-12-31T03:30:03	Notice	opnsense	/usr/local/etc/rc.configure_interface: plugins_configure monitor (execute task : dpinger_configure_do(1,WAN_DHCP6))	
2022-12-31T03:30:03	Notice	opnsense	/usr/local/etc/rc.configure_interface: plugins_configure monitor (1,WAN_DHCP6)	
2022-12-31T03:30:03	Error	opnsense	/usr/local/etc/rc.configure_interface: ROUTING: skipping IPv6 default route	
2022-12-31T03:30:03	Error	opnsense	/usr/local/etc/rc.configure_interface: ROUTING: IPv6 default gateway set to wan	
2022-12-31T03:30:03	Error	opnsense	/usr/local/etc/rc.configure_interface: ROUTING: skipping IPv4 default route	
2022-12-31T03:30:03	Error	opnsense	/usr/local/etc/rc.configure_interface: ROUTING: IPv4 default gateway set to opt19	
2022-12-31T03:30:02	Error	opnsense	/usr/local/etc/rc.configure_interface: ROUTING: entering configure using 'wan'

Additional context

No other context available.

Environment

Software version used and hardware type if relevant, e.g.:

OPNsense 22.7.10_2-amd64 (but also observed at least with 22.7.9) FreeBSD 13.1-RELEASE-p5 OpenSSL 1.1.1s 1 Nov 2022

Intel® NUC9i5QNX Network Intel® I210-AT