core: WebGUI: can't bind to socket: 127.0.0.1:443: Address already in use

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug

After some time running opnsense I’ll get a “503 - service unavailable” error when trying to access the WebGUI. Opnsense itself seems to still do its job, however the WebGUI is gone. Killing lighttpd and restarting services via console solves it (as mentioned here: https://forum.opnsense.org/index.php?topic=32540.0)

I noticed this after the last upgrade.

To Reproduce

Steps to reproduce the behavior:

Run below mentioned version
Wait a few days
Try to access WbGUI
See error

Expected behavior

Would expect regular WebGUI operation and login window.

Describe alternatives you considered

Relevant log files

As in Forum post mentioned, I found the below part in the logs as well. Could not spot any other error related to webgui. 2023-02-20T07:20:07 Error opnsense /usr/local/etc/rc.restart_webgui: The command '/usr/local/sbin/lighttpd -f /var/etc/lighty-webConfigurator.conf' returned exit code '255', the output was '2023-02-20 07:20:07: (network.c.537) can't bind to socket: 127.0.0.1:443: Address already in use'

Environment

OPNsense 23.1.1_2-amd64 FreeBSD 13.1-RELEASE-p6 OpenSSL 1.1.1t 7 Feb 2023 AMD GX-412TC SOC

About this issue

Original URL
State: closed
Created a year ago
Comments: 86 (50 by maintainers)

Commits related to this issue

system: switch to SIGINT for lighttpd for #6351 — committed to opnsense/core by fichtner a year ago
system: fix historic oversight in pid vs. name/pidfile reading related to #6351 If we kill a process and want to wait for it we will have to cache the PID file and check this one until it's gone. — committed to opnsense/core by fichtner a year ago
system: adjust timeout #6351 — committed to opnsense/core by fichtner a year ago
system: bring stop/start a little closer together #6351 — committed to opnsense/core by fichtner a year ago
system: use flock while starting lighttpd #6351 — committed to opnsense/core by fichtner a year ago
system: address a number of web GUI startup problems #6351 — committed to opnsense/core by fichtner a year ago
system: defer start of lighttpd on newwanip event #6351 Looks like improvements in 23.1.4 made the problem worse, indicating that the direction this is going is a bit doubtful. Instead, funnel the r... — committed to opnsense/core by fichtner a year ago
system: do not detach on defer ##6351 Same as filter_configure() now. — committed to opnsense/core by fichtner a year ago

Most upvoted comments

So far the WebGUI seems to work. Errors haven’t showed up. Only having the following in the system log at pppoe reconnect:

2023-03-30T05:00:51+02:00 opnsense.<redacted> opnsense 59790 - [meta sequenceId="259"] /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : webgui_configure_do(,lan))
2023-03-30T05:00:52+02:00 opnsense.<redacted> opnsense 86374 - [meta sequenceId="273"] /usr/local/etc/rc.restart_webgui: The command '/usr/local/bin/flock -ne /var/run/lighty-webConfigurator.pid /usr/local/sbin/lighttpd -f /var/etc/lighty-webConfigurator.conf' returned exit code '1', the output was ''
2023-03-30T05:00:52+02:00 opnsense.<redacted> opnsense 86374 - [meta sequenceId="274"] /usr/local/etc/rc.restart_webgui: The command '/bin/kill -'TERM' '9643'' returned exit code '1', the output was 'kill: 9643: No such process'
2023-03-30T05:00:59+02:00 opnsense.<redacted> opnsense 82595 - [meta sequenceId="278"] /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : webgui_configure_do(,opt6))

chris42 on Mar 30, 2023

23.1.6 was released today.

fichtner on Apr 20, 2023

So far webGUI is still alive and kicking. lighttpd logs are clean with only server stopping/starting messages.

Within system logs I found the following on yesterday’s reconnect:

2023-02-26T05:00:16+01:00 opnsense.<redacted> opnsense 65690 - [meta sequenceId="52"] /usr/local/etc/rc.newwanip: webgui_id_63fad950a227e: start webgui_configure_do
2023-02-26T05:00:38+01:00 opnsense.<redacted> opnsense 96761 - [meta sequenceId="136"] /usr/local/etc/rc.newwanipv6: webgui_id_63fad96634fba: start webgui_configure_do
2023-02-26T05:00:39+01:00 opnsense.<redacted> opnsense 24122 - [meta sequenceId="146"] /usr/local/etc/rc.newwanipv6: webgui_id_63fad967269eb: start webgui_configure_do
2023-02-26T05:00:44+01:00 opnsense.<redacted> opnsense 96761 - [meta sequenceId="160"] /usr/local/etc/rc.newwanipv6: webgui_id_63fad96c315ea: start webgui_configure_do
2023-02-26T05:00:44+01:00 opnsense.<redacted> opnsense 96761 - [meta sequenceId="161"] /usr/local/etc/rc.newwanipv6: webgui_id_63fad96c315ea: starting generate_config
2023-02-26T05:00:44+01:00 opnsense.<redacted> opnsense 96761 - [meta sequenceId="162"] /usr/local/etc/rc.newwanipv6: webgui_id_63fad96c315ea: reload gui templates
2023-02-26T05:00:44+01:00 opnsense.<redacted> opnsense 96761 - [meta sequenceId="163"] /usr/local/etc/rc.newwanipv6: webgui_id_63fad96c315ea: php-cgi HUP skipped. realy need it? (lighty spawns FastCGI) (besides, it only kills one instance (and may kill captive portal?)?
2023-02-26T05:00:44+01:00 opnsense.<redacted> opnsense 96761 - [meta sequenceId="164"] /usr/local/etc/rc.newwanipv6: webgui_id_63fad96c315ea: config ready. TERM lighty
2023-02-26T05:00:44+01:00 opnsense.<redacted> opnsense 96761 - [meta sequenceId="165"] /usr/local/etc/rc.newwanipv6: webgui_id_63fad96c315ea: start lighty (lighty will remove pid file of working instance if start fail because of 'can't bind'. thats not funny)

On today’s reconnect (5am) and firmware check (7am) I get the following:

2023-02-27T05:00:20+01:00 opnsense.<redacted> opnsense 625 - [meta sequenceId="178"] /usr/local/etc/rc.newwanip: webgui_id_63fc2ad400c28: start webgui_configure_do
2023-02-27T05:00:37+01:00 opnsense.<redacted> opnsense 27197 - [meta sequenceId="209"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2ae55aa63: start webgui_configure_do
2023-02-27T05:00:37+01:00 opnsense.<redacted> opnsense 6073 - [meta sequenceId="219"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2ae5bfe1c: start webgui_configure_do
2023-02-27T05:00:50+01:00 opnsense.<redacted> opnsense 27197 - [meta sequenceId="240"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2af2439f4: start webgui_configure_do
2023-02-27T05:00:50+01:00 opnsense.<redacted> opnsense 27197 - [meta sequenceId="242"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2af2439f4: starting generate_config
2023-02-27T05:00:50+01:00 opnsense.<redacted> opnsense 27197 - [meta sequenceId="243"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2af2439f4: reload gui templates
2023-02-27T05:00:50+01:00 opnsense.<redacted> opnsense 6073 - [meta sequenceId="245"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2af25c4a9: start webgui_configure_do
2023-02-27T05:00:50+01:00 opnsense.<redacted> opnsense 6073 - [meta sequenceId="246"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2af25c4a9: starting generate_config
2023-02-27T05:00:50+01:00 opnsense.<redacted> opnsense 6073 - [meta sequenceId="247"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2af25c4a9: reload gui templates
2023-02-27T05:00:50+01:00 opnsense.<redacted> opnsense 6073 - [meta sequenceId="248"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2af25c4a9: php-cgi HUP skipped. realy need it? (lighty spawns FastCGI) (besides, it only kills one instance (and may kill captive portal?)?
2023-02-27T05:00:50+01:00 opnsense.<redacted> opnsense 6073 - [meta sequenceId="249"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2af25c4a9: config ready. TERM lighty
2023-02-27T05:00:51+01:00 opnsense.<redacted> opnsense 6073 - [meta sequenceId="250"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2af25c4a9: start lighty (lighty will remove pid file of working instance if start fail because of 'can't bind'. thats not funny)
2023-02-27T05:00:51+01:00 opnsense.<redacted> opnsense 27197 - [meta sequenceId="252"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2af2439f4: php-cgi HUP skipped. realy need it? (lighty spawns FastCGI) (besides, it only kills one instance (and may kill captive portal?)?
2023-02-27T05:00:51+01:00 opnsense.<redacted> opnsense 27197 - [meta sequenceId="253"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2af2439f4: config ready. TERM lighty
2023-02-27T05:00:51+01:00 opnsense.<redacted> opnsense 27197 - [meta sequenceId="260"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2af2439f4: start lighty (lighty will remove pid file of working instance if start fail because of 'can't bind'. thats not funny)
2023-02-27T05:00:55+01:00 opnsense.<redacted> opnsense 55596 - [meta sequenceId="274"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2af7a123b: start webgui_configure_do
2023-02-27T05:01:09+01:00 opnsense.<redacted> opnsense 55596 - [meta sequenceId="288"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2b05b1080: start webgui_configure_do
2023-02-27T05:01:09+01:00 opnsense.<redacted> opnsense 55596 - [meta sequenceId="289"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2b05b1080: starting generate_config
2023-02-27T05:01:09+01:00 opnsense.<redacted> opnsense 55596 - [meta sequenceId="290"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2b05b1080: reload gui templates
2023-02-27T05:01:10+01:00 opnsense.<redacted> opnsense 55596 - [meta sequenceId="291"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2b05b1080: php-cgi HUP skipped. realy need it? (lighty spawns FastCGI) (besides, it only kills one instance (and may kill captive portal?)?
2023-02-27T05:01:10+01:00 opnsense.<redacted> opnsense 55596 - [meta sequenceId="292"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2b05b1080: config ready. TERM lighty
2023-02-27T05:01:10+01:00 opnsense.<redacted> opnsense 55596 - [meta sequenceId="293"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2b05b1080: start lighty (lighty will remove pid file of working instance if start fail because of 'can't bind'. thats not funny)
2023-02-27T05:01:19+01:00 opnsense.<redacted> opnsense 6073 - [meta sequenceId="322"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2b0fcdc3b: start webgui_configure_do
2023-02-27T05:01:19+01:00 opnsense.<redacted> opnsense 27197 - [meta sequenceId="325"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2b0fe1e93: start webgui_configure_do
2023-02-27T05:01:19+01:00 opnsense.<redacted> opnsense 73445 - [meta sequenceId="326"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2b0fe2abc: start webgui_configure_do
2023-02-27T05:01:24+01:00 opnsense.<redacted> opnsense 73445 - [meta sequenceId="340"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2b14e553b: start webgui_configure_do
2023-02-27T05:01:24+01:00 opnsense.<redacted> opnsense 73445 - [meta sequenceId="341"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2b14e553b: starting generate_config
2023-02-27T05:01:24+01:00 opnsense.<redacted> opnsense 73445 - [meta sequenceId="342"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2b14e553b: reload gui templates
2023-02-27T05:01:25+01:00 opnsense.<redacted> opnsense 73445 - [meta sequenceId="343"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2b14e553b: php-cgi HUP skipped. realy need it? (lighty spawns FastCGI) (besides, it only kills one instance (and may kill captive portal?)?
2023-02-27T05:01:25+01:00 opnsense.<redacted> opnsense 73445 - [meta sequenceId="344"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2b14e553b: config ready. TERM lighty
2023-02-27T05:01:25+01:00 opnsense.<redacted> opnsense 73445 - [meta sequenceId="345"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2b14e553b: start lighty (lighty will remove pid file of working instance if start fail because of 'can't bind'. thats not funny)
2023-02-27T05:01:38+01:00 opnsense.<redacted> opnsense 73445 - [meta sequenceId="359"] /usr/local/etc/rc.newwanipv6: webgui_id_63fc2b2241aec: start webgui_configure_do
2023-02-27T07:06:15+01:00 opnsense.<redacted> opnsense 44368 - [meta sequenceId="1"] /usr/local/etc/rc.restart_webgui: webgui_id_63fc4857e4d35: start webgui_configure_do
2023-02-27T07:06:15+01:00 opnsense.<redacted> opnsense 44368 - [meta sequenceId="2"] /usr/local/etc/rc.restart_webgui: webgui_id_63fc4857e4d35: starting generate_config
2023-02-27T07:06:15+01:00 opnsense.<redacted> opnsense 44368 - [meta sequenceId="3"] /usr/local/etc/rc.restart_webgui: webgui_id_63fc4857e4d35: reload gui templates
2023-02-27T07:06:16+01:00 opnsense.<redacted> opnsense 44368 - [meta sequenceId="4"] /usr/local/etc/rc.restart_webgui: webgui_id_63fc4857e4d35: php-cgi HUP skipped. realy need it? (lighty spawns FastCGI) (besides, it only kills one instance (and may kill captive portal?)?
2023-02-27T07:06:16+01:00 opnsense.<redacted> opnsense 44368 - [meta sequenceId="5"] /usr/local/etc/rc.restart_webgui: webgui_id_63fc4857e4d35: config ready. TERM lighty
2023-02-27T07:06:16+01:00 opnsense.<redacted> opnsense 44368 - [meta sequenceId="6"] /usr/local/etc/rc.restart_webgui: webgui_id_63fc4857e4d35: start lighty (lighty will remove pid file of working instance if start fail because of 'can't bind'. thats not funny)

Does that help?

chris42 on Feb 27, 2023

Thanks a lot! Did a reboot before applying, so any silly state is hopefully cleared up. Applied the patch (I guess reboot won’t be necessary to activate) We probably will see tomorrow or on Monday how it works.

chris42 on Feb 25, 2023