supervised-installer: Bug Report: installation steps don't mention enabling and mapping `systemd-journal-gatewayd`

OS Version

Debian 11

System Information

unsure

What happened?

The ADR mentions this step in additional supported conditions:

Systemd journal gateway is enabled and mapped into supervisor as /run/systemd-journal-gatewayd.sock

The installation guide does not include any info on setting this up, it simply says to install systemd-journal-remote. It also doesn’t currently seem to handle this as part of running the script, based on discussions in here.

Think there either needs to be guidance about how to do this in the step-by-step install guide or the script needs to handle this part.

Machine Type

odroid-n2

Installer output

No response

Relevant log output

No response

ADR

  • I have read through the ADR and have confirmed that my system is compliant with the requirements
  • I understand that if my system is found to not be compliant, my issue will be closed immediately without further investigation

Code of Conduct

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 10
  • Comments: 103 (13 by maintainers)

Commits related to this issue

Most upvoted comments

blendrik’s post was the last piece of the puzzle for me!

Running debian, I also found my socket file in /lib/systemd/system/systemd-journal-gatewayd.socket , unlike in sdotter’s post. I then made the same changes to ListenStream and then followed jfparis’s advice so that the sock file just referenced, actually exists:

[Socket]
#ListenStream=19531
ListenStream=/run/systemd-journal-gatewayd.sock

sudo rm -rf /run/systemd-journal-gatewayd.sock
sudo touch /run/systemd-journal-gatewayd.sock
sudo reboot

I can now see host logs and the journal-gatewayd repair is gone. Of course, the TCP errors are still there while viewing host logs via HA.

(The above assumes the pkgs were already updated as the HA docs instruct)

Thanks everyone.

after installation => os-agent_1.4.1_linux_x86_64.deb => systemd-journal-remote => homeassistant-supervised-1.3.1.deb, no errors in HA,

Supervisor: Connected
Supported: Supported
Healthy: Healthy

in file: /usr/lib/systemd/system/systemd-journal-gatewayd.socket

[Socket]
ListenStream=/run/systemd-journal-gatewayd.sock

systemd-journal-gatewayd starts automatically after reboot. systemctl status systemd-journal-gatewayd.socket

● systemd-journal-gatewayd.socket - Journal Gateway Service Socket
     Loaded: loaded (/etc/systemd/system/systemd-journal-gatewayd.socket; enabled; preset: disabled)
     Active: active (running) since Tue 2022-11-08 13:47:58 MSK; 4min 49s ago
      Until: Tue 2022-11-08 13:47:58 MSK; 4min 49s ago
   Triggers: ● systemd-journal-gatewayd.service
       Docs: man:systemd-journal-gatewayd(8)
     Listen: /run/systemd-journal-gatewayd.sock (Stream)
     CGroup: /system.slice/systemd-journal-gatewayd.socket

ноя 08 13:47:58 da4a systemd[1]: Listening on Journal Gateway Service Socket.

after reboot i see the Host logs, before the phrase “Started Journal Gateway Service” Following are some errors:

Nov 07 11:31:42 da4a wpa_supplicant[456]: nl80211: send_and_recv->nl_recvmsgs failed: -33
Nov 07 11:44:53 da4a systemd[1]: Started Journal Gateway Service.
Nov 07 11:44:53 da4a systemd-journal-gatewayd[4031]: microhttpd: Setting TCP_NODELAY option to ON state failed: Operation not supported
Nov 07 11:44:53 da4a systemd-journal-gatewayd[4031]: microhttpd: Setting TCP_CORK option to OFF state failed: Operation not supported

It confuses me that I don’t have /etc/systemd/system/systemd-journal-gatewayd.socket file as written above there is only /usr/lib/systemd/system/systemd-journal-gatewayd.socket i did:

root@da4a:~# cp /usr/lib/systemd/system/systemd-journal-gatewayd.socket /etc/systemd/system/systemd-journal-gatewayd.socket
root@da4a:~# reboot

but it did not help, the result is the same:

Nov 08 13:48:58 da4a systemd-journal-gatewayd[3759]: microhttpd: Setting TCP_NODELAY option to ON state failed: Operation not supported
Nov 08 13:48:58 da4a systemd-journal-gatewayd[3759]: microhttpd: Setting TCP_CORK option to OFF state failed: Operation not supported
Nov 08 13:48:58 da4a systemd-journal-gatewayd[3759]: microhttpd: Failed to push the data from buffers to the network. Client may experience some delay (usually in range 200ms - 5 sec).
Nov 08 13:48:58 da4a systemd-journal-gatewayd[3759]: microhttpd: Setting TCP_NODELAY option to ON state failed: Operation not supported
Nov 08 13:48:58 da4a systemd-journal-gatewayd[3759]: microhttpd: Setting TCP_CORK option to OFF state failed: Operation not supported

I have a Supervised installation on Debian and a reinstallation of the supervisor, removed the “unsupported” notification in home assistant. (and i did confirm that it is the right version) but the host log and the status of the systemd-journal-gatewayd.service is filled with:

● systemd-journal-gatewayd.service - Journal Gateway Service
     Loaded: loaded (/lib/systemd/system/systemd-journal-gatewayd.service; indirect; vendor preset: disabled)
     Active: active (running) since Fri 2022-11-04 07:49:56 CET; 11min ago
TriggeredBy: ● systemd-journal-gatewayd.socket
       Docs: man:systemd-journal-gatewayd(8)
   Main PID: 10267 (systemd-journal)
      Tasks: 2 (limit: 19037)
     Memory: 188.1M
     CGroup: /system.slice/systemd-journal-gatewayd.service
             └─10267 /lib/systemd/systemd-journal-gatewayd

Nov 04 07:51:06 HA-ROOT systemd-journal-gatewayd[10267]: microhttpd: Setting TCP_CORK option to OFF state failed: Operation not supported
Nov 04 07:51:06 HA-ROOT systemd-journal-gatewayd[10267]: microhttpd: Failed to push the data from buffers to the network. Client may experience some delay (usually in range 200ms - 5 sec).
Nov 04 07:51:06 HA-ROOT systemd-journal-gatewayd[10267]: microhttpd: Setting TCP_NODELAY option to ON state failed: Operation not supported
Nov 04 07:51:06 HA-ROOT systemd-journal-gatewayd[10267]: microhttpd: Setting TCP_CORK option to OFF state failed: Operation not supported
Nov 04 07:51:06 HA-ROOT systemd-journal-gatewayd[10267]: microhttpd: Failed to push the data from buffers to the network. Client may experience some delay (usually in range 200ms - 5 sec).
Nov 04 07:51:06 HA-ROOT systemd-journal-gatewayd[10267]: microhttpd: Setting TCP_NODELAY option to ON state failed: Operation not supported
Nov 04 07:51:06 HA-ROOT systemd-journal-gatewayd[10267]: microhttpd: Setting TCP_CORK option to OFF state failed: Operation not supported
Nov 04 07:51:06 HA-ROOT systemd-journal-gatewayd[10267]: microhttpd: Failed to push the data from buffers to the network. Client may experience some delay (usually in range 200ms - 5 sec).
Nov 04 07:51:06 HA-ROOT systemd-journal-gatewayd[10267]: microhttpd: Setting TCP_NODELAY option to ON state failed: Operation not supported
Nov 04 07:51:06 HA-ROOT systemd-journal-gatewayd[10267]: microhttpd: Setting TCP_CORK option to OFF state failed: Operation not supported

just like what @dandek01 is seeing. so something is still not right

I am on debian bullseye.

I have installed journal-remote via sudo apt install systemd-journal-remote and install the new version of homeassistant-supervised.deb package and rebooted

At the end of that

  • /run/systemd-journal-gatewayd.sock is a directory (instaed of a unix socket)
  • It is mapped into the supervisor container (here again this is a directory)

And Obviously fix is then

  • sudo rm -rf /run/systemd-journal-gatewayd.sock
  • sudo touch /run/systemd-journal-gatewayd.sock
  • sudo reboot

And Obviously fix is then

* `sudo rm -rf /run/systemd-journal-gatewayd.sock`

* `sudo touch /run/systemd-journal-gatewayd.sock`

* `sudo reboot`

Tried, did not help… 😦

There hasn’t been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

> [Socket]
> #ListenStream=19531
> ListenStream=/run/systemd-journal-gatewayd.sock
> 
> sudo rm -rf /run/systemd-journal-gatewayd.sock
> sudo touch /run/systemd-journal-gatewayd.sock
> sudo reboot

It works but not for long After starting the Journal Gateway Service, there are errors:

Nov 07 11:31:25 da4a systemd[1]: systemd-hostnamed.service: Succeeded.
Nov 07 11:31:25 da4a systemd[1]: systemd-timedated.service: Succeeded.
Nov 07 11:31:42 da4a wpa_supplicant[456]: nl80211: send_and_recv->nl_recvmsgs failed: -33
Nov 07 11:44:53 da4a systemd[1]: Started Journal Gateway Service.
Nov 07 11:44:53 da4a systemd-journal-gatewayd[4031]: microhttpd: Setting TCP_NODELAY option to ON state failed: Operation not supported
Nov 07 11:44:53 da4a systemd-journal-gatewayd[4031]: microhttpd: Setting TCP_CORK option to OFF state failed: Operation not supported

I have tried systemd-journal-remote versions 247.3-7+deb11u1 and 251.3-1~bpo11+1 from bullseye-backports. The result is the same

It sounds like people were installing an older pkg of supervisor when rerunning the script. When they would wget, it would create a homeassistant-supervised.deb.1, then homeassistant-supervised.deb.2 and so on, but when they tried to install, it would install the original/older version.

The installation guide does not include any info on setting this up, it simply says to install systemd-journal-remote. It also doesn’t currently seem to handle this as part of running the script, based on discussions in here.

It is documented in the readme and required as a dependency in the debian control file you must have version 1.3.0 or later of the HA supervised debian package installed on your system if for whatever reason it is not functioning try re-running the installer and rebooting your host

There are currently two issues tracking the “log spam” on its own: https://github.com/home-assistant/core/issues/87148 https://github.com/home-assistant/supervised-installer/issues/255

I find the relatively low volume of complaints somewhat surprising given the performance impact this must have on some SoC devices. Perhaps the issue isn’t as wide-spread and is a manifestation of some sort of an edge-case?

I managed to fix original problem with this - /run/systemd-journal-gatewayd.sock was a directory and the service was listening on network port. However, the correct way of fixing this is probably service override, not editing the original unit file, it should go like this:

root@host:~# systemctl edit systemd-journal-gatewayd.socket
### editor opens, fill in what's below in /etc/systemd/system/systemd-journal-gatewayd.socket.d/override.conf and save it
root@host:~# systemctl cat systemd-journal-gatewayd.socket
# /lib/systemd/system/systemd-journal-gatewayd.socket
#  SPDX-License-Identifier: LGPL-2.1-or-later
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Journal Gateway Service Socket
Documentation=man:systemd-journal-gatewayd(8)

[Socket]
ListenStream=19531

[Install]
WantedBy=sockets.target

# /etc/systemd/system/systemd-journal-gatewayd.socket.d/override.conf
[Socket]
ListenStream=
ListenStream=/run/systemd-journal-gatewayd.sock

root@host:~# systemctl restart systemd-journal-gatewayd.socket
root@host:~# systemctl status systemd-journal-gatewayd.socket
● systemd-journal-gatewayd.socket - Journal Gateway Service Socket
     Loaded: loaded (/lib/systemd/system/systemd-journal-gatewayd.socket; enabled; preset: disabled)
    Drop-In: /etc/systemd/system/systemd-journal-gatewayd.socket.d
             └─override.conf
     Active: active (running) since Tue 2023-02-14 21:02:53 CET; 1s ago
      Until: Tue 2023-02-14 21:02:53 CET; 1s ago
   Triggers: ● systemd-journal-gatewayd.service
       Docs: man:systemd-journal-gatewayd(8)
     Listen: /run/systemd-journal-gatewayd.sock (Stream)
     CGroup: /system.slice/systemd-journal-gatewayd.socket

Feb 14 21:02:53 host systemd[1]: Listening on systemd-journal-gatewayd.socket - Journal Gateway Service Socket.

The issue with excessive logs exist when I open Settings -> System -> Logs -> Host

For RaspberryPi OS:

The file /usr/lib/systemd/system/systemd-journal-gatewayd.socket installed by homeassistant-supervised.deb is placed in the wrong directory. On my system it’s located in /lib/systemd/system/systemd-journal-gatewayd.socket. So I commented out the line with the port and added the sock-file as so

[Socket]
#ListenStream=19531
ListenStream=/run/systemd-journal-gatewayd.sock

With this, at least I got rid of the systemd-journal-issues and can see system logs within HA.

What do you see when running: wget https://github.com/home-assistant/supervised-installer/releases/latest/download/homeassistant-supervised.deb

Hi @DChe47 Yes. After upgrading Debian to 12, this problem disappears.

I use HA Supervised on Debian 11. I had more spam messages in log of Host and i try to this: sudo rm -rf /run/systemd-journal-gatewayd.sock sudo touch /run/systemd-journal-gatewayd.sock After full reboot log of Host does no present and I have problem:

`● systemd-journal-gatewayd.service - Journal Gateway Service Loaded: loaded (/lib/systemd/system/systemd-journal-gatewayd.service; indirect; vendor preset: disabled) Active: failed (Result: exit-code) since Sun 2024-01-28 22:32:16 MSK; 40min ago TriggeredBy: ● systemd-journal-gatewayd.socket Docs: man:systemd-journal-gatewayd(8) Process: 7943 ExecStart=/lib/systemd/systemd-journal-gatewayd (code=exited, status=1/FAILURE) Main PID: 7943 (code=exited, status=1/FAILURE)

янв 28 22:32:16 ha-master systemd[1]: Started Journal Gateway Service. янв 28 22:32:16 ha-master systemd-journal-gatewayd[7943]: Can’t listen on more than one socket. янв 28 22:32:16 ha-master systemd[1]: systemd-journal-gatewayd.service: Main process exited, code=exited, status=1/FAI> янв 28 22:32:16 ha-master systemd[1]: systemd-journal-gatewayd.service: Failed with result ‘exit-code’. ` Can I fix it? Help me please

Yes you are right! I updated my system to Debian12, this error disappeared)

This is a different problem not strictly related to original bug description

I don’t think that you are right with this

One is /lib/systemd/system/systemd-journal-gatewayd.socket, but the correct one is /usr/lib/systemd/system/systemd-journal-gatewayd.socket, belonging to two different packages. Activating systemd-journal-gatewayd.socket will then activate the wrong one. The /lib directory seems to have precedence over the /usr/lib directory. Amazingly, there is zero information available anywhere about this precedence.

For me LIB is a shortcut for /usr/lib>So that’s why the both path shows the same file, and both .socket is the same.

In my case even after activating systemd-journal-remote.socket it kept recreating a directory after reboot.

Turns out there are two systemd socket configuration files.

One is /lib/systemd/system/systemd-journal-gatewayd.socket, but the correct one is /usr/lib/systemd/system/systemd-journal-gatewayd.socket, belonging to two different packages. Activating systemd-journal-gatewayd.socket will then activate the wrong one. The /lib directory seems to have precedence over the /usr/lib directory. Amazingly, there is zero information available anywhere about this precedence.

$ dpkg -S {,/usr}/lib/systemd/system/systemd-journal-gatewayd.socket
systemd-journal-remote: /lib/systemd/system/systemd-journal-gatewayd.socket
homeassistant-supervised: /usr/lib/systemd/system/systemd-journal-gatewayd.socket

$ systemctl status systemd-journal-gatewayd.socket
● systemd-journal-gatewayd.socket - Journal Gateway Service Socket
     Loaded: loaded (/lib/systemd/system/systemd-journal-gatewayd.socket; disabled;
[...]

I then tried symlinking /etc/systemd/system/systemd-journal-gatewayd.socket to /usr/lib/systemd/system/systemd-journal-gatewayd.socket, (of course followed by systemctl daemon-reload) but this also did not work (probably some security reason). So now I have hard linked it, which seems to do the job.

I then rebooted, and finally it works.

Awesome! Glad it helped someone else at least!

I see homeassistant-supervised.deb and if I run wget again I see homeassistant-supervised.deb.1 But in my case it was impossible to make that mistake because I downloaded it to tmp dir that is deleted on every reboot. And when I installed file I saw version I was installing.

@moto3001 your conclusion is not correct, at least not in my case. I installed debian-supervised twice just in case. In both cases I installed v1.3.1. For some reason after reboot service systemd-journal-gatewayd and systemd-journal-gatewad.socket didn’t started after reboot. I enabled both services. I have to run them manually and then they run. I don’t know what the problem is but there is a problem. I’m running debian 11.

Yes, it is, but it does not resolve the issue.