splunk-connect-for-kubernetes: Failed parsing container metadata

What happened: After upgrade on 1.4.1 we’ve faced with some problems. Parsing container metadata information is failing , I show much notices in Fluentd pod logs, like: [debug]: parsing container meta information failed for: NAMESPACE/POD_NAME

What you expected to happen:

Correct parsing containers metadata

How to reproduce it (as minimally and precisely as possible): Upgrade to 1.4.1

Environment:

Kubernetes version (use kubectl version): 1.11
OS (e.g: cat /etc/os-release): RHEL7
OpenShift 3.11.98

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 25

Most upvoted comments

@rockb1017 for fyi ^

Shouldnt impact anything, may just take longer for annotation changes etc to show up…

matthewmodestino on Oct 9, 2020

@matthewmodestino i see, is it the right config map update?

<label @SPLUNK>
  # Enrich log with k8s metadata
  <filter tail.containers.**>
    @type kubernetes_metadata
    annotation_match [ "^splunk\.com" ]
    watch false                                                    ---> new line added
    cache_ttl 600                                                 ---> new line added
    de_dot false
  </filter>

kish3007 on Oct 7, 2020

I do see the same issue in Ibm Kubernetes Cluster running Splunk-connect 1.4.0 Kubernetes version: 1.15.10

where the splunk logging pod restarts multiple times with the error:

2020-05-28 18:09:56 +0000 [info]: gem 'fluentd' version '1.9.1'
2020-05-28 18:09:56 +0000 [info]: gem 'fluent-plugin-concat' version '2.4.0'
2020-05-28 18:09:56 +0000 [info]: gem 'fluent-plugin-jq' version '0.5.1'
2020-05-28 18:09:56 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.4.5'
2020-05-28 18:09:56 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.7.0'
2020-05-28 18:09:56 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.2.2'
2020-05-28 18:09:56 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.2'
#<Thread:0x000055c778cb0fd0@/usr/share/gems/gems/fluent-plugin-kubernetes_metadata_filter-2.4.5/lib/fluent/plugin/filter_kubernetes_metadata.rb:277 run> terminated with exception (report_on_exception is true):/usr/share/gems/gems/fluent-plugin-kubernetes_metadata_filter-2.4.5/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:43:in `rescue in set_up_namespace_thread': undefined method `<' for nil:NilClass (NoMethodError)
	from /usr/share/gems/gems/fluent-plugin-kubernetes_metadata_filter-2.4.5/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:38:in `set_up_namespace_thread'	from /usr/share/gems/gems/fluent-plugin-kubernetes_metadata_filter-2.4.5/lib/fluent/plugin/filter_kubernetes_metadata.rb:277:in `block in configure'2020-05-28 18:09:57 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Could not parse jq filter: .record.source = \"/run/log/journal/\" + .record.source | .record.sourcetype = (.tag | ltrimstr(\"journald.\")) | .record.cluster_name = \"iksprodclusterussouth\" | .record.index = \"devops_kubernetes_prod\" |.record, error: undefined method `<' for nil:NilClass"```

kish3007 on May 28, 2020