splunk-connect-for-kubernetes: Logging Pods are crashing (crashloopbackoff)

What happened: Logging pods are getting crashed within few seconds of creation and the status is crashloopbackoff

What you expected to happen: we have this integration completed on 2 clusters and we followed the same process. But here we are having issues with Logging pods. Object & Metric pods are coming up.

How to reproduce it (as minimally and precisely as possible): redeploy the sck app OR upgrade sck app

Anything else we need to know?: Below is the Error

oc logs -f stageapi-splunk-splunk-kubernetes-logging-bprjj 2022-03-02 00:07:50 +0000 [info]: parsing config file is succeeded path=“/fluentd/etc/fluent.conf” 2022-03-02 00:07:50 +0000 [info]: gem ‘fluentd’ version ‘1.9.1’ 2022-03-02 00:07:50 +0000 [info]: gem ‘fluent-plugin-concat’ version ‘2.4.0’ 2022-03-02 00:07:50 +0000 [info]: gem ‘fluent-plugin-jq’ version ‘0.5.1’ 2022-03-02 00:07:50 +0000 [info]: gem ‘fluent-plugin-kubernetes_metadata_filter’ version ‘2.5.2’ 2022-03-02 00:07:50 +0000 [info]: gem ‘fluent-plugin-prometheus’ version ‘1.7.0’ 2022-03-02 00:07:50 +0000 [info]: gem ‘fluent-plugin-splunk-hec’ version ‘1.2.4’ **2022-03-02 00:07:50 +0000 [info]: gem ‘fluent-plugin-systemd’ version ‘1.0.2’ bundler: failed to load command: fluentd (/usr/bin/fluentd) /usr/share/gems/gems/fluentd-1.9.1/lib/fluent/supervisor.rb:532:in rescue in run_supervisor': failed to create root directory:/tmp/fluentd, #<Errno::EROFS: Read-only file system @ dir_s_mkdir - /tmp/fluentd> (Fluent::InvalidRootDirectory)** from /usr/share/gems/gems/fluentd-1.9.1/lib/fluent/supervisor.rb:529:in run_supervisor’ from /usr/share/gems/gems/fluentd-1.9.1/lib/fluent/command/fluentd.rb:330:in <top (required)>' from /usr/share/gems/gems/fluentd-1.9.1/bin/fluentd:8:in require’ from /usr/share/gems/gems/fluentd-1.9.1/bin/fluentd:8:in <top (required)>' from /usr/bin/fluentd:23:in load’ from /usr/bin/fluentd:23:in <top (required)>' from /usr/local/share/gems/gems/bundler-2.2.15/lib/bundler/cli/exec.rb:63:in load’ from /usr/local/share/gems/gems/bundler-2.2.15/lib/bundler/cli/exec.rb:63:in kernel_load' from /usr/local/share/gems/gems/bundler-2.2.15/lib/bundler/cli/exec.rb:28:in run’ from /usr/local/share/gems/gems/bundler-2.2.15/lib/bundler/cli.rb:494:in exec' from /usr/local/share/gems/gems/bundler-2.2.15/lib/bundler/vendor/thor/lib/thor/command.rb:27:in run’ from /usr/local/share/gems/gems/bundler-2.2.15/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in invoke_command' from /usr/local/share/gems/gems/bundler-2.2.15/lib/bundler/vendor/thor/lib/thor.rb:392:in dispatch’ from /usr/local/share/gems/gems/bundler-2.2.15/lib/bundler/cli.rb:30:in dispatch' from /usr/local/share/gems/gems/bundler-2.2.15/lib/bundler/vendor/thor/lib/thor/base.rb:485:in start’ from /usr/local/share/gems/gems/bundler-2.2.15/lib/bundler/cli.rb:24:in start' from /usr/local/share/gems/gems/bundler-2.2.15/exe/bundle:49:in block in <top (required)>’ from /usr/local/share/gems/gems/bundler-2.2.15/lib/bundler/friendly_errors.rb:130:in with_friendly_errors' from /usr/local/share/gems/gems/bundler-2.2.15/exe/bundle:37:in <top (required)>’ from /usr/local/bin/bundle:23:in load' from /usr/local/bin/bundle:23:in <main>’

Environment:

• Kubernetes version (use kubectl version): 1.4.3 & 1.4.12
• Ruby version (use ruby --version):
• OS (e.g: cat /etc/os-release): OpenShift Container Platform Version – 4.5.28
• Splunk version: 8.2.4
• Splunk Connect for Kubernetes helm chart version: 1.4.3 & 1.4.12
• Others: