fluent-bit: Kube Plugin SSL Verification seems to be broken

Bug Report

Describe the bug

Fluent bit logs upstream certificate check error:

[2019/10/04 22:26:39] [error] [filter_kube] upstream connection error
[2019/10/04 22:26:39] [error] [io_tls] flb_io_tls.c:348 X509 - Certificate verification failed, e.g. CRL, CA or signature check

but the upstream certificate is fine:

curl https://kubernetes.default.svc:443 --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Returns no certificate error, and the certificate definitely has the correct Alternative Names:

CN=kube-apiserver X509v3 Subject Alternative Name: IP Address:10.253.252.1, IP Address:10.21.13.5, IP Address:10.21.13.15, IP Address:10.21.13.25, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:at-host-02, DNS:at-host-12, DNS:at-host-22

This error occurs in banzaicloud/logging-operator, I therefore cannot set the ssl.verify = off (since there’s no option for it, raised here: https://github.com/banzaicloud/logging-operator/issues/172)

To Reproduce Start the provided configuration.

I assume that the issue might be the kubernetes url in the config, but all variations of the url are actually in the Subject Alternative Names. Possibly the SANs are not validated against?

I also found that: https://tls.mbed.org/kb/development/adding_new_subject_alternative_name_support

But it looks like DNS is supported by mbedtls, so still not clear.

Expected behavior

Fluentbit should successfully verify the certificate if a correct CA and Cert are provided, based on the error is also not clear what exactly failed (should be improved)

Screenshots

Your Environment

• Version used: fluent/fluent-bit:1.2.2
• Configuration:

[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    info
    Parsers_File parsers.conf
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020

[INPUT]
    Name             tail
    Path             /var/log/containers/*.log
    Parser           cri
    Tag              kubernetes.*
    Refresh_Interval 5
    Mem_Buf_Limit    5MB
    Skip_Long_Lines  On
    DB               /tail-db/tail-containers-state.db
    DB.Sync          Normal

[FILTER]
    Name                kubernetes
    Match               kubernetes.*
    Kube_Tag_Prefix     kubernetes.var.log.containers.
    Kube_URL            https://kubernetes.default.svc:443
    Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
    Merge_Log           On

[OUTPUT]
    Name          forward
    Match         *
    Host          10.21.240.33
    Port          24224
    
    Retry_Limit   False

• Environment name and version (e.g. Kubernetes? What version?): Kubernetes 1.15.3
• Operating System and version: Centos 7.6 / kernel 4.19.72
• Filters and plugins: see config

Additional context