fluent-bit: Unable to get k8s metadata from systemd input

I am attempting to get the Kubernetes filter metadata from journald logs and output to Elasticsearch. (From the thread https://github.com/fluent/fluent-bit/issues/332 I assume this is possible.)

We have an existing k8s setup with a number of pods logging to journald.

I created a new pod for fluent-bit:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: fluent-bit
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: fluent-bit
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:0.12.14
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
        - name: fluent-bit-log
          mountPath: /fluent-bit/log/
      terminationGracePeriodSeconds: 10
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluent-bit-config
        hostPath:
          path: /var/Davra/fluent-bit
      - name: fluent-bit-log
        hostPath:
          path: /data

This uses the following fluent-bit config:

 # cat /var/Davra/fluent-bit/fluent-bit.conf
[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    info
    Log_File     /fluent-bit/log/fluent-bit.log
    Parsers_File parsers.conf

[INPUT]
    Name            systemd
    Path            /var/log/journal
    Tag             k8s.*

[FILTER]
    Name            kubernetes
    Match           k8s.*

[OUTPUT]
    Name            es
    Match           *
    Host            192.168.10.203
    Port            9200
    Logstash_Format On
    Retry_Limit     False

The resultant docs in Elasticsearch show systemd metadata but no kubernetes metadata. If I flip my log-driver’s to use json-file and tail /var/log/containers/*.log on the input instead, I do see the kubernetes metadata.

So, should I be able to get kubernetes metadata from journald input and, if so, any ideas of what may be wrong here?

Thanks.

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 21 (11 by maintainers)

Commits related to this issue

Most upvoted comments

My expectation is to release it by the third week of this month:

https://github.com/fluent/fluent-bit/milestone/5

I’ve found the root cause of the problem. I will provide an update shorlty

@garysmith-github hmm yeah there is a problem, parsing CONTAINER_NAME we should get:

name_prefix  = k8s
container_name = announce-api
container_hash  = 
pod_name = announce-api-2527845369-9ttmj
namespace_name = default

likely the hash key is not set correctly, I will troubleshoot.