fluent-bit: Kubernetes filter can't find pods anymore after upgrading from 0.14.9 -> 1.0.2

Bug Report

Describe the bug Configuration is working in 0.14.9 but after upgrade to 1.0.2 the kubernetes filter complains with the following error:

[2019/01/21 12:32:37] [ info] [filter_kube] https=1 host=10.233.0.1 port=443 [2019/01/21 12:32:37] [ info] [filter_kube] local POD info OK [2019/01/21 12:32:37] [ info] [filter_kube] testing connectivity with API server... [2019/01/21 12:32:37] [debug] [filter_kube] API Server (ns=kube-system, pod=fluent-bit-2bsvt) http_do=0, HTTP Status: 200 [2019/01/21 12:32:37] [ info] [filter_kube] API server connectivity OK [2019/01/21 12:32:38] [debug] [filter_kube] API Server (ns=online-tst, pod=services.xx-prerender-cache-redis-ha-server-2) http_do=0, HTTP Status: 404 [2019/01/21 12:32:38] [debug] [filter_kube] API Server response {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"services.xx-prerender-cache-redis-ha-server-2\" not found","reason":"NotFound","details":{"name":"services.xx-prerender-cache-redis-ha-server-2","kind":"pods"},"code":404}

Suddenly it can not find the pods anymore

Configuration: Kubernetes 1.8.4 Docker version 17.05.0-ce, build 89658be OS: Redhat Linux 4.17.11 x86_64 x86_64 x86_64 GNU/Linux

`
[PARSER] Decode_Field_As escaped log Format json Name json Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Key time

[PARSER]
    Decode_Field_As escaped log
    Format json
    Name docker
    Time_Format %Y-%m-%dT%H:%M:%S.%L
    Time_Keep true
    Time_Key time

[PARSER]
    Format regex
    Name mongo
    Regex ^(?<time>[^ ]*)\s+(?<severity>\w)\s+(?<context>[^ ]+)\s+\[(?<connection>[^\]]+)]\s+(?<message>.*)$
    Time_Format %Y-%m-%dT%H:%M:%S.%L
    Time_Keep true
    Time_Key time
[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    debug
    HTTP_Server    On
    HTTP_Listen    0.0.0.0
    HTTP_PORT    2020
    Parsers_File parsers_custom.conf

[INPUT]
    Buffer_Chunk_Size 1MB
    Buffer_Max_Size 25MB
    DB /var/log/containers/fluent-bit.db
    Exclude_Path *kube-system*.log
    Mem_Buf_Limit 25MB
    Name tail
    Parser docker
    Path /var/log/containers/*.log
    Refresh_Interval 5
    Skip_Long_Lines On
    Tag kube.services.*

[INPUT]
    Buffer_Chunk_Size 1MB
    Buffer_Max_Size 25MB
    DB /var/log/containers/fluent-bit-nginx.db
    Mem_Buf_Limit 25MB
    Name tail
    Parser docker
    Path /var/log/containers/*nginx*.log
    Refresh_Interval 5
    Skip_Long_Lines On
    Tag kube.ingress.*

[FILTER]
    K8S-Logging.Parser On
    K8S-Logging.exclude True
    Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
    Kube_URL https://${KUBERNETES_SERVICE_HOST}:443
    Match kube.*
    Merge_Log On
    Merge_Log_Key k8s
    Name kubernetes
    tls.verify On
[OUTPUT]
    Host xxx.xxx.xxx.xxx
    Include_Tag_Key true
    Logstash_DateFormat %G.%V
    Logstash_Format On
    Logstash_Prefix k8s-services-tst
    Match kube.services.*
    Name es
    Port 9200
    Retry_Limit False
[OUTPUT]
    Host xxx.xxx.xxx.xxx
    Include_Tag_Key true
    Logstash_DateFormat %G.%V
    Logstash_Format On
    Logstash_Prefix k8s-ingress-tst
    Match kube.ingress.*
    Name es
    Port 9200
    Retry_Limit False

`

Am I missing some changes in the kubernetes filter? Can’t find anything. It already cost me a day of work.

Already tried: Changing Kube_URL to IP address Changing Kube_URL to DNS name Removed port Changed port to 6443

Changing back to 0.14.9 and it all works again…😒