core: Unbound crashing when WAN IP refreshes

Important notices Before you add a new report, we ask you kindly to acknowledge the following:

[x] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.mdxx

[x] I have searched the existing issues and I’m convinced that mine is new.

Describe the bug Unbound is failing to properly restart after a WAN interface refreshes it’s IP address (PPPoE typically).

Tip: to validate your setup was working with the previous version, use opnsense-revert (https://docs.opnsense.org/manual/opnsense_tools.html#opnsense-revert) This has been a problem since at least 20.1, and maybe earlier as well.

To Reproduce Steps to reproduce the behavior:

1. Have a PPPoE interface refresh it’s IPv4 address
2. Watch as unbound fails to properly restart (it seems like pluginctl thinks it restarted, but I think unbound has an invalid configuration or something because there’s no log messages from unbound in this state)

Expected behavior Unbound should be reliable and not crash.

Relevant log files From latest crash (manually restarted unbound when problem was identified an hour later)

2020-08-17T13:17:55 | configd.py[31629] | [f689addc-3582-41d8-827b-86b43dcd3d43] Show log
-- | -- | --
2020-08-17T13:16:41 | configd.py[31629] | [7ceae22f-86e0-4ce2-b71b-dba618182fe7] Show log
2020-08-17T13:16:37 | configd.py[31629] | [a37eb653-c9d0-4a91-9907-6b4c8d35554d] request pfctl byte/packet counters
2020-08-17T13:16:32 | configd.py[31629] | [72ea340e-2788-4042-a2f5-c6d29826e60c] request pfctl byte/packet counters
2020-08-17T13:16:22 | configd.py[31629] | message 6c6af745-a637-4e23-a96c-cf674996ef9e [unbound.start] returned OK
2020-08-17T13:16:22 | configd.py[31629] | [6c6af745-a637-4e23-a96c-cf674996ef9e] Start Unbound
2020-08-17T13:16:22 | configd.py[31629] | OPNsense/Unbound/* generated //var/unbound/etc/lists.inc
2020-08-17T13:16:22 | configd.py[31629] | OPNsense/Unbound/* generated //var/unbound/etc/dot.conf
2020-08-17T13:16:22 | configd.py[31629] | OPNsense/Unbound/* generated //var/unbound/etc/miscellaneous.conf
2020-08-17T13:16:22 | configd.py[31629] | OPNsense/Unbound/* generated //var/unbound/etc/whitelist.inc
2020-08-17T13:16:22 | configd.py[31629] | OPNsense/Unbound/* generated //var/unbound/etc/dnsbl.inc
2020-08-17T13:16:22 | configd.py[31629] | OPNsense/Unbound/* generated //var/unbound/root.hints
2020-08-17T13:16:21 | configd.py[31629] | generate template container OPNsense/Unbound/core
2020-08-17T13:16:21 | configd.py[31629] | [1fa89335-b59a-47ef-a61f-c07b50ba33ff] generate template OPNsense/Unbound/*
2020-08-17T13:05:22 | configd.py[31629] | [406637c5-717c-4f38-8a5b-1c5b5573422c] update IPv6 prefixes
2020-08-17T12:48:32 | configd.py[31629] | [78c5fbcd-1887-479f-860f-693879760ff9] update IPv6 prefixes
2020-08-17T12:26:38 | configd.py[31629] | [52b4cad6-a379-466d-89b1-7a47b12a387b] update IPv6 prefixes
2020-08-17T12:26:25 | configd.py[31629] | message 28bcc99e-20a5-41d9-899f-6eca16d91c50 [filter.refresh_aliases] returned {"status": "ok"}
2020-08-17T12:26:24 | configd.py[31629] | [28bcc99e-20a5-41d9-899f-6eca16d91c50] refresh url table aliases
2020-08-17T12:26:24 | configd.py[31629] | OPNsense/Filter generated //usr/local/etc/filter_geoip.conf
2020-08-17T12:26:24 | configd.py[31629] | OPNsense/Filter generated //usr/local/etc/filter_tables.conf
2020-08-17T12:26:24 | configd.py[31629] | generate template container OPNsense/Filter
2020-08-17T12:26:24 | configd.py[31629] | [924bc198-0861-4afd-a7e9-281ba15f324c] generate template OPNsense/Filter
2020-08-17T12:26:23 | configd.py[31629] | [ed51d27c-cacc-4c33-afa6-54faac782b8c] Reloading filter
2020-08-17T12:26:14 | configd.py[31629] | message 9931bf21-bbb3-4957-945a-c50c7745338b [filter.refresh_aliases] returned {"status": "ok"}
2020-08-17T12:26:13 | configd.py[31629] | [9931bf21-bbb3-4957-945a-c50c7745338b] refresh url table aliases
2020-08-17T12:26:13 | configd.py[31629] | OPNsense/Filter generated //usr/local/etc/filter_geoip.conf
2020-08-17T12:26:13 | configd.py[31629] | OPNsense/Filter generated //usr/local/etc/filter_tables.conf
2020-08-17T12:26:13 | configd.py[31629] | generate template container OPNsense/Filter
2020-08-17T12:26:13 | configd.py[31629] | [9b12fe8c-44d7-4b48-938f-fdfa45cdd4c7] generate template OPNsense/Filter
2020-08-17T12:26:12 | configd.py[31629] | [5211e3d8-8caf-40f3-a198-02509c078528] Reloading filter
2020-08-17T12:26:11 | configd.py[31629] | message 7ac9c736-4b63-4735-a158-4dbe12eb9199 [unbound.start] returned Error (1)
2020-08-17T12:26:11 | configd.py[31629] | [7ac9c736-4b63-4735-a158-4dbe12eb9199] returned exit status 1
2020-08-17T12:26:11 | configd.py[31629] | message 8730ddb3-a641-4e04-8814-254ef05be4b4 [filter.refresh_aliases] returned {"status": "ok"}
2020-08-17T12:26:11 | configd.py[31629] | [7ac9c736-4b63-4735-a158-4dbe12eb9199] Start Unbound
2020-08-17T12:26:11 | configd.py[31629] | OPNsense/Unbound/* generated //var/unbound/etc/lists.inc
2020-08-17T12:26:11 | configd.py[31629] | OPNsense/Unbound/* generated //var/unbound/etc/dot.conf
2020-08-17T12:26:11 | configd.py[31629] | OPNsense/Unbound/* generated //var/unbound/etc/miscellaneous.conf
2020-08-17T12:26:11 | configd.py[31629] | OPNsense/Unbound/* generated //var/unbound/etc/whitelist.inc
2020-08-17T12:26:11 | configd.py[31629] | OPNsense/Unbound/* generated //var/unbound/etc/dnsbl.inc
2020-08-17T12:26:11 | configd.py[31629] | OPNsense/Unbound/* generated //var/unbound/root.hints
2020-08-17T12:26:11 | configd.py[31629] | generate template container OPNsense/Unbound/core
2020-08-17T12:26:11 | configd.py[31629] | [d29f94ad-aab6-47df-8fa2-0a0463ea1f74] generate template OPNsense/Unbound/*
2020-08-17T12:26:11 | configd.py[31629] | [8730ddb3-a641-4e04-8814-254ef05be4b4] refresh url table aliases
2020-08-17T12:26:11 | configd.py[31629] | OPNsense/Filter generated //usr/local/etc/filter_geoip.conf
2020-08-17T12:26:11 | configd.py[31629] | OPNsense/Filter generated //usr/local/etc/filter_tables.conf
2020-08-17T12:26:11 | configd.py[31629] | generate template container OPNsense/Filter
2020-08-17T12:26:11 | configd.py[31629] | [a0f36546-822f-4296-8190-b6d08f3871db] generate template OPNsense/Filter
2020-08-17T12:26:09 | configd.py[31629] | [c0f97fc8-a78c-4b44-8f75-aeb9dae39b95] New IPv6 on pppoe0
2020-08-17T12:26:09 | configd.py[31629] | message ced8cc6a-815b-40a3-959a-624de6000b00 [filter.refresh_aliases] returned {"status": "ok"}
2020-08-17T12:26:09 | configd.py[31629] | message 155f4ce9-cc9e-4c72-9789-ac5c55c8bf98 [unbound.start] returned OK
2020-08-17T12:26:08 | configd.py[31629] | [155f4ce9-cc9e-4c72-9789-ac5c55c8bf98] Start Unbound
2020-08-17T12:26:08 | configd.py[31629] | OPNsense/Unbound/* generated //var/unbound/etc/lists.inc
2020-08-17T12:26:08 | configd.py[31629] | OPNsense/Unbound/* generated //var/unbound/etc/dot.conf
2020-08-17T12:26:08 | configd.py[31629] | OPNsense/Unbound/* generated //var/unbound/etc/miscellaneous.conf
2020-08-17T12:26:08 | configd.py[31629] | OPNsense/Unbound/* generated //var/unbound/etc/whitelist.inc
2020-08-17T12:26:08 | configd.py[31629] | OPNsense/Unbound/* generated //var/unbound/etc/dnsbl.inc
2020-08-17T12:26:08 | configd.py[31629] | OPNsense/Unbound/* generated //var/unbound/root.hints
2020-08-17T12:26:08 | configd.py[31629] | generate template container OPNsense/Unbound/core
2020-08-17T12:26:08 | configd.py[31629] | [211389dd-d4c0-4a84-9aad-a96ef2aaa054] generate template OPNsense/Unbound/*
2020-08-17T12:26:08 | configd.py[31629] | [ced8cc6a-815b-40a3-959a-624de6000b00] refresh url table aliases
2020-08-17T12:26:08 | configd.py[31629] | OPNsense/Filter generated //usr/local/etc/filter_geoip.conf
2020-08-17T12:26:08 | configd.py[31629] | OPNsense/Filter generated //usr/local/etc/filter_tables.conf
2020-08-17T12:26:08 | configd.py[31629] | generate template container OPNsense/Filter
2020-08-17T12:26:08 | configd.py[31629] | [ea127dde-ab39-458d-a56e-dc6273309286] generate template OPNsense/Filter
2020-08-17T12:26:07 | configd.py[31629] | message 20e6c440-a216-4f0f-b06a-c417efceca64 [unbound.start] returned Error (1)
2020-08-17T12:26:07 | configd.py[31629] | [20e6c440-a216-4f0f-b06a-c417efceca64] returned exit status 1
2020-08-17T12:26:07 | configd.py[31629] | message ce1b9dfa-eb83-4697-a4df-172e0f7e8999 [filter.refresh_aliases] returned {"status": "ok"}
2020-08-17T12:26:07 | configd.py[31629] | [20e6c440-a216-4f0f-b06a-c417efceca64] Start Unbound
2020-08-17T12:26:07 | configd.py[31629] | OPNsense/Unbound/* generated //var/unbound/etc/lists.inc
2020-08-17T12:26:07 | configd.py[31629] | OPNsense/Unbound/* generated //var/unbound/etc/dot.conf
2020-08-17T12:26:07 | configd.py[31629] | OPNsense/Unbound/* generated //var/unbound/etc/miscellaneous.conf
2020-08-17T12:26:07 | configd.py[31629] | OPNsense/Unbound/* generated //var/unbound/etc/whitelist.inc
2020-08-17T12:26:07 | configd.py[31629] | OPNsense/Unbound/* generated //var/unbound/etc/dnsbl.inc
2020-08-17T12:26:07 | configd.py[31629] | OPNsense/Unbound/* generated //var/unbound/root.hints
2020-08-17T12:26:07 | configd.py[31629] | generate template container OPNsense/Unbound/core
2020-08-17T12:26:07 | configd.py[31629] | [7cc546e4-0ee7-4532-ba7c-613f7c60b02e] generate template OPNsense/Unbound/*
2020-08-17T12:26:06 | configd.py[31629] | [4ca75370-df52-4f60-a352-e2a7cd64a34e] New IPv6 on igb2
2020-08-17T12:26:06 | configd.py[31629] | [ce1b9dfa-eb83-4697-a4df-172e0f7e8999] refresh url table aliases
2020-08-17T12:26:06 | configd.py[31629] | OPNsense/Filter generated //usr/local/etc/filter_geoip.conf
2020-08-17T12:26:06 | configd.py[31629] | OPNsense/Filter generated //usr/local/etc/filter_tables.conf
2020-08-17T12:26:06 | configd.py[31629] | generate template container OPNsense/Filter
2020-08-17T12:26:06 | configd.py[31629] | [1ef1235a-72fb-4c7a-8418-d3dcaa314e27] generate template OPNsense/Filter
2020-08-17T12:26:05 | configd.py[31629] | [f42a2d6d-2ff8-4a89-991f-5fc9224db39b] New IPv6 on igb2
2020-08-17T12:26:05 | configd.py[31629] | [2e0d4f1e-2b32-4668-9c16-7b349e0a0820] New IPv6 on igb2
2020-08-17T12:26:04 | configd.py[31629] | [08b7bba2-1f78-4964-87f0-c22987e3ac7b] New IPv6 on pppoe0
2020-08-17T12:26:04 | configd.py[31629] | [fa3fc997-506b-44b0-b275-32a465856a67] New IPv4 on pppoe0

Additional context This may be related to the previous issue I filed at #4149 but I’m not certain, and because it’s a different impact, still present in 20.7, I think a new bug is a good idea. It might also be the same problem as #4106 - I do not have unbound listening on any WAN interfaces because I need to have a separate DNS server listening there.

I tried to set up monit to restart unbound, but that seemed to cause pluginctl to freeze completely, so I think there might be a problem getting pluginctl to work with monit, sadly.

Environment Software version used and hardware type if relevant.

OPNsense 20.7.1-amd64 FreeBSD 12.1-RELEASE-p8-HBSD LibreSSL 3.0.2 Intel® Core™ i3-4030U CPU @ 1.90GHz (4 cores)