splunk-connect-for-kubernetes: Logging Pods (DeamonSet) have Permission Issue

I setup the Splunk Connect on Openshift with Below Steps:

Followed the Steps provided here  https://github.com/splunk/splunk-connect-for-kubernetes

Installed using Helm 3 with below commands.

$ Kubectl create ns splunk-connect

$ helm install splunk-connect -f esf_aro_splunk_values.yaml splunk/splunk-connect-for-kubernetes -n splunk-connect

$ oc adm policy add-scc-to-user privileged “system:serviceaccount:splunk-connect:splunk-connect-splunk-kubernetes-logging”

Object and metrics pods working fine and able to see the events in Splunk. But Container logs are uploaded to Splunk.

Further Investigation found below error in Logging Pods:

2021-08-05 17:45:37 +0000 [info]: #0 starting fluentd worker pid=279 ppid=1 worker=0 2021-08-05 17:45:37 +0000 [info]: #0 listening port port=24224 bind=“0.0.0.0” /usr/share/gems/gems/fluentd-1.11.5/lib/fluent/plugin_helper/http_server/compat/server.rb:84: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call /usr/share/gems/gems/fluentd-1.11.5/lib/fluent/plugin_helper/http_server/compat/webrick_handler.rb:26: warning: The called method build' is defined here **2021-08-05 17:45:37 +0000 [error]: #0 unexpected error error_class=Errno::EACCES error="Permission denied @ rb_sysopen - /var/log/splunk-fluentd-kube-audit.pos"** 2021-08-05 17:45:37 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.11.5/lib/fluent/plugin/in_tail.rb:215:in initialize’ 2021-08-05 17:45:37 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.11.5/lib/fluent/plugin/in_tail.rb:215:in open' 2021-08-05 17:45:37 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.11.5/lib/fluent/plugin/in_tail.rb:215:in start’ 2021-08-05 17:45:37 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.11.5/lib/fluent/root_agent.rb:200:in block in start' 2021-08-05 17:45:37 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.11.5/lib/fluent/root_agent.rb:189:in block (2 levels) in lifecycle’ 2021-08-05 17:45:37 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.11.5/lib/fluent/root_agent.rb:188:in each' 2021-08-05 17:45:37 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.11.5/lib/fluent/root_agent.rb:188:in block in lifecycle’ 2021-08-05 17:45:37 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.11.5/lib/fluent/root_agent.rb:175:in each' 2021-08-05 17:45:37 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.11.5/lib/fluent/root_agent.rb:175:in lifecycle’ 2021-08-05 17:45:37 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.11.5/lib/fluent/root_agent.rb:199:in start' 2021-08-05 17:45:37 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.11.5/lib/fluent/engine.rb:248:in start’ 2021-08-05 17:45:37 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.11.5/lib/fluent/engine.rb:147:in run' 2021-08-05 17:45:37 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.11.5/lib/fluent/supervisor.rb:607:in block in run_worker’ 2021-08-05 17:45:37 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.11.5/lib/fluent/supervisor.rb:845:in main_process' 2021-08-05 17:45:37 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.11.5/lib/fluent/supervisor.rb:598:in run_worker’ 2021-08-05 17:45:37 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.11.5/lib/fluent/command/fluentd.rb:361:in <top (required)>' 2021-08-05 17:45:37 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.11.5/bin/fluentd:8:in require’ 2021-08-05 17:45:37 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.11.5/bin/fluentd:8:in <top (required)>' 2021-08-05 17:45:37 +0000 [error]: #0 /usr/bin/fluentd:23:in load’ 2021-08-05 17:45:37 +0000 [error]: #0 /usr/bin/fluentd:23:in `<main>’ 2021-08-05 17:45:37 +0000 [error]: #0 unexpected error error_class=Errno::EACCES error=“Permission denied @ rb_sysopen - /var/log/splunk-fluentd-kube-audit.pos” 2021-08-05 17:45:37 +0000 [error]: #0 suppressed same stacktrace 2021-08-05 17:45:37 +0000 [info]: Worker 0 finished unexpectedly with status 1 2021-08-05 17:45:38 +0000 [info]: adding filter in @CONCAT pattern=“tail.containers.var.log.containers.dns-controllerdns-controller.log” type=“concat” 2021-08-05 17:45:38 +0000 [info]: adding filter in @CONCAT pattern=“tail.containers.var.log.containers.kube-dnssidecar.log” type=“concat” 2021-08-05 17:45:38 +0000 [info]: adding filter in @CONCAT pattern=“tail.containers.var.log.containers.kube-dnsdnsmasq.log” type=“concat” 2021-08-05 17:45:38 +0000 [info]: adding filter in @CONCAT pattern=“tail.containers.var.log.containers.kube-apiserverkube-apiserver.log” type=“concat” 2021-08-05 17:45:38 +0000 [info]: adding filter in @CONCAT pattern=“tail.containers.var.log.containers.kube-controller-managerkube-controller-manager.log” type=“concat” 2021-08-05 17:45:38 +0000 [info]: adding filter in @CONCAT pattern=“tail.containers.var.log.containers.kube-dns-autoscalerautoscaler.log” type=“concat” 2021-08-05 17:45:38 +0000 [info]: adding filter in @CONCAT pattern=“tail.containers.var.log.containers.kube-proxykube-proxy.log” type=“concat” 2021-08-05 17:45:38 +0000 [info]: adding filter in @CONCAT pattern=“tail.containers.var.log.containers.kube-schedulerkube-scheduler.log” type=“concat” 2021-08-05 17:45:38 +0000 [info]: adding filter in @CONCAT pattern=“tail.containers.var.log.containers.kube-dnskubedns.log” type=“concat” 2021-08-05 17:45:38 +0000 [info]: adding filter in @CONCAT pattern=“journald.kube:kubelet” type=“concat” 2021-08-05 17:45:38 +0000 [info]: adding match in @CONCAT pattern=“**” type=“relabel”

Environment: Openshift 4.7 Running in Azure

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 15 (1 by maintainers)

Most upvoted comments

The debug logs seem fine too, with chunks being uploaded. I would recommend to start from scratch. Your values.yaml seems to be for an older version than 1.4.9, it uses older container images and you deleted all the snippets from the logging section, I guess something got lost on the way.

Just do an helm uninstall, get the new values.yaml from the 1.4.9 release and customize it to your needs. Do not delete stuff, just activate it or fix it (OpenShift stuff).

This is a diff from my values.yaml for SCK 1.4.9 showing the global and logging section only(!) with changes to the original values.yaml, running OCP 4.6.x Some caveats:

My splunk guys had a different entry in the global section for indexName than in the sections for logging, metrics, objects. I don’t remember why. I’m just the OpenShift guy. In the end it depends on your setup in the backend
We’ve also set indexFields: to something. I believe it is optional. The links in the comments weren’t helpful…
journallogs are not active because the default setting journalLogPath: /run/log/journal will not work for OpenShift. You really should think twice if you want that kubelet output in splunk as the amount of logging is immense. This might need some more configuration for systemd-units I guess.
That audit-log stuff for oauth-audit hasn’t been tested much by us. Just be careful who can access it in Splunk.

# diff <(grep -v '#' sck_149_official_values_yaml)  <(grep -v '#' sck_my_ocp_values.yaml) 
6c6
<       host:
---
>       host: $HOST
8c8
<       token:
---
>       token: $TOKEN
10,11c10,11
<       indexName:
<       insecureSSL:
---
>       indexName: $INDEXNAME
>       insecureSSL: true
17c17
<     clusterName: "cluster_name"
---
>     clusterName: $CLUSTERNAME
40,41c40,41
<     pathDest: /var/lib/docker/containers
<     logFormatType: json
---
>     pathDest: /var/log/pods
>     logFormatType: cri
57c57
<     openshiftPrivilegedSccBinding: false
---
>     openshiftPrivilegedSccBinding: true
64,65c64,65
<     create: false
<     apparmor_security: true
---
>     create: true
>     apparmor_security: false
74,75c74,75
<       indexName:
<       insecureSSL:
---
>       indexName: $INDEXNAME_LOGGING
>       insecureSSL: true
187c187
<           path: /var/log/kube-apiserver-audit.log
---
>           path: /var/log/kube-apiserver/audit.log
190a191,204
>     oauth-audit:
>       from:
>         file:
>           path: /var/log/oauth-apiserver/audit.log
>       timestampExtraction:
>         format: "%Y-%m-%dT%H:%M:%SZ"
>       sourcetype: kube:oauth-apiserver-audit
>     openshift-audit:
>       from:
>         file:
>           path: /var/log/openshift-apiserver/audit.log
>       timestampExtraction:
>         format: "%Y-%m-%dT%H:%M:%SZ"
>       sourcetype: kube:apiserver-audit-openshift
238c252
<     securityContext: false
---
>     securityContext: true

gnarc on Aug 20, 2021