node-problem-detector: npd yaml failed in centos for journal path: /run/log/journal not /var/log/journal

https://github.com/helm/charts/issues/21817#issuecomment-611510731

Root cause issue

On CentOS log path is /run/log/journal but by default use /var/log/journal https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/node-problem-detector/npd.yaml#L69-L88

/sig cloud-provider /kind bug

Fix

add in daemonSet

      volumes:
      - hostPath:
          path: /run/log/journal
          type: ""
        name: journal

and

       volumeMounts:
        - mountPath: /var/log/journal
          name: journal

error messages

I0926 19:38:38.481196 1 custom_plugin_monitor.go:276] New status generated: &{Source:systemd-monitor Events:[{Severity:info Timestamp:2021-09-26 19:38:38.481139626 +0800 CST m=+0.605519482 Reason:NoFrequentContainerdRestart Message:Node condition FrequentContainerdRestart is now: Unknown, reason: NoFrequentContainerdRestart}] Conditions:[{Type:FrequentKubeletRestart Status:Unknown Transition:2021-09-26 19:38:38.279765139 +0800 CST m=+0.404145014 Reason:NoFrequentKubeletRestart Message:error watching journald: failed to stat the log path “/var/log/journal”: stat /v} {Type:FrequentDockerRestart Status:Unknown Transition:2021-09-26 19:38:38.382463904 +0800 CST m=+0.506843870 Reason:NoFrequentDockerRestart Message:error watching journald: failed to stat the log path “/var/log/journal”: stat /v} {Type:FrequentContainerdRestart Status:Unknown Transition:2021-09-26 19:38:38.481139626 +0800 CST m=+0.605519482 Reason:NoFrequentContainerdRestart Message:error watching journald: failed to stat the log path “/var/log/journal”: stat /v}]}

/cc @Random-Liu @wangzhen127

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 15 (10 by maintainers)

Most upvoted comments

My suggestion would be mount /run/log/journal as well and fix it in NPD to use /run/log/journal if /var/log/journal not exist.

pacoxu on Sep 28, 2021

adding this to helm values is a bit of a hack, but solves my issue (this allows me to keep using config files under the /config directory, while at the same time allowing for node problem detector to work with centos hosts):

logDir:
  host: /run/log/
  pod: "/var/log/"

L1ghtman2k on Jan 18, 2022

I would like to work on this /assign

shivanshu1333 on Sep 26, 2021