splunk-connect-for-kubernetes: Metrics fails with RestClient::NotFound error="404 Not Found"

Hello everybody,

We’re running a Kubernetes cluster based on Microk8s nodes. Currently we’re implementing the integration to Splunk via splunk-connect-for-kubernetes.

We have the logging and objects up and running with success, but the metrics is failing.

We installed splunk-connect-for-kubernetes via Helm with the the following values (selected relevant parts only):

global:
  logLevel: info
  splunk:
    hec:
      # host is required and should be provided by user
      host: "xxx"
      # port to HEC, optional, default 8088
      port: 8088
      # token is required and should be provided by user
      token: "xxx"
      # protocol has two options: "http" and "https", default is "https"
      protocol: "https"
      # indexName tells which index to use, this is optional. If it's not present, will use "main".
      indexName:
      # insecureSSL is a boolean, it indicates should it allow inscure SSL connection (when protocol is "https"). Default is false.
      insecureSSL: true
      # The PEM-format CA certificate for this client.
      # NOTE: The content of the certificate itself should be used here, not the file path.
      #       The certificate will be stored as a secret in kubernetes.
      clientCert: |
        -----BEGIN CERTIFICATE-----
        xxx
        -----END CERTIFICATE-----
      # The private key for this client.
      # NOTE: The content of the key itself should be used here, not the file path.
      #       The key will be stored as a secret in kubernetes.
      clientKey: |
        -----BEGIN PRIVATE KEY-----
        xxx
        -----END PRIVATE KEY-----
      # The PEM-format CA certificate file.
      # NOTE: The content of the file itself should be used here, not the file path.
      #       The file will be stored as a secret in kubernetes.
      caFile: |
        -----BEGIN CERTIFICATE-----
        xxx
        -----END CERTIFICATE-----
      # For object and metrics
      indexRouting:
  kubernetes:
    # The cluster name used to tag logs. Default is cluster_name
    clusterName: "xxx"
    # Add privileged access to containers for openshift compatability
    openshift: false
  prometheus_enabled:

...

splunk-kubernetes-metrics:
  enabled: true
  ...
  # = Kubernetes Connection Configs =
  kubernetes:
    # The hostname or IP address that kubelet will use to connect to. If not supplied, status.hostIP of the node is used to fetch metrics from the Kubelet API (via the $KUBERNETES_NODE_IP environment variable).
    # Default is "#{ENV['KUBERNETES_NODE_IP']}"
    kubeletAddress:
    # The port that kubelet is listening on. Default is 10250
    kubeletPort:
    # The port that is used to get the metrics using apiserver proxy using ssl for the metrics aggregator
    kubeletPortAggregator:
    # This option is used to get the metrics from summary api on each kubelet using ssl
    useRestClientSSL: true
    # if insecureSSL is set to true, insecure HTTPS API call is allowed, default false
    insecureSSL: true
    # Path to the CA file.
    caFile:
    # Path to the file contains the API token. By default it reads from the file "token" in the `secret_dir`.
    bearerTokenFile:
    # Path of the location where pod's service account's credentials are stored. Usually you don't need to care about this config, the default value should work in most cases.
    secretDir:
    # The cluster name used to tag cluster metrics from the aggregator. Default is cluster_name
    clusterName:
    # Add privileged access to containers for openshift compatability`

What happened: Log from metrics pod:

2020-11-27 08:32:31 +0000 [info]: adding source type="kubernetes_metrics"
2020-11-27 08:32:32 +0000 [info]: #0 Use URL https://192.168.0.2:10250/stats/summary for creating client to query kubelet summary api
2020-11-27 08:32:32 +0000 [info]: #0 Use URL https://192.168.0.2:10250/stats for creating client to query kubelet stats api
2020-11-27 08:32:32 +0000 [info]: #0 Use URL https://192.168.0.2:10250/metrics/cadvisor for creating client to query cadvisor metrics api
2020-11-27 08:32:32 +0000 [warn]: parameter 'cluster_name' in <source>
  @type kubernetes_metrics
  tag "kube.*"
  node_name "xxx"
  use_rest_client_ssl true
  insecure_ssl true
  cluster_name xxx
  interval 15s
</source> is not used.
2020-11-27 08:32:32 +0000 [info]: #0 starting fluentd worker pid=10 ppid=1 worker=0
2020-11-27 08:32:32 +0000 [info]: #0 fluentd worker is now running worker=0
2020-11-27 08:32:48 +0000 [error]: #0 Unexpected error raised. Stopping the timer. title=:stats_metric_scraper error_class=RestClient::NotFound error="404 Not Found"
  2020-11-27 08:32:48 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response'
  2020-11-27 08:32:48 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!'
  2020-11-27 08:32:48 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result'
  2020-11-27 08:32:48 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit'
  2020-11-27 08:32:48 +0000 [error]: #0 /usr/share/ruby/net/http.rb:910:in `start'
  2020-11-27 08:32:48 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit'
  2020-11-27 08:32:48 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute'
  2020-11-27 08:32:48 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:233:in `_follow_redirection'
  2020-11-27 08:32:48 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:145:in `follow_redirection'
  2020-11-27 08:32:48 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:121:in `return!'
  2020-11-27 08:32:48 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result'
  2020-11-27 08:32:48 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit'
  2020-11-27 08:32:48 +0000 [error]: #0 /usr/share/ruby/net/http.rb:910:in `start'
  2020-11-27 08:32:48 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit'
  2020-11-27 08:32:48 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute'
  2020-11-27 08:32:48 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute'
  2020-11-27 08:32:48 +0000 [error]: #0 /opt/app-root/src/gem/fluent-plugin-kubernetes-metrics-1.1.4/lib/fluent/plugin/in_kubernetes_metrics.rb:647:in `scrape_stats_metrics'
  2020-11-27 08:32:48 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.9.1/lib/fluent/plugin_helper/timer.rb:80:in `on_timer'
  2020-11-27 08:32:48 +0000 [error]: #0 /usr/share/gems/gems/cool.io-1.6.0/lib/cool.io/loop.rb:88:in `run_once'
  2020-11-27 08:32:48 +0000 [error]: #0 /usr/share/gems/gems/cool.io-1.6.0/lib/cool.io/loop.rb:88:in `run'
  2020-11-27 08:32:48 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.9.1/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
  2020-11-27 08:32:48 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.9.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

Notice the: 2020-11-27 08:32:48 +0000 [error]: #0 Unexpected error raised. Stopping the timer. title=:stats_metric_scraper error_class=RestClient::NotFound error=“404 Not Found”

What you expected to happen: I expected metrics to be sent to Splunk…

How to reproduce it (as minimally and precisely as possible): As described, I think.

Anything else we need to know?: Unknown right now.

Environment:

• Kubernetes version (use kubectl version):

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.3-34+9e8b6537ec8b6f", GitCommit:"9e8b6537ec8b6f45c8e04a481350def07f5f761d", GitTreeState:"clean", BuildDate:"2020-11-05T18:58:59Z", GoVersion:"go1.15.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.3-34+9e8b6537ec8b6f", GitCommit:"9e8b6537ec8b6f45c8e04a481350def07f5f761d", GitTreeState:"clean", BuildDate:"2020-11-05T19:00:24Z", GoVersion:"go1.15.3", Compiler:"gc", Platform:"linux/amd64"}

• Ruby version (use ruby --version): Not installed

• OS (e.g: cat /etc/os-release):

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

• Splunk version:

Splunk
Version:
8.1.0
Build:
f57c09e87251

• Splunk Connect for Kubernetes helm chart version:

$ helm show chart splunk/splunk-connect-for-kubernetes
apiVersion: v1
appVersion: 1.4.3
dependencies:
- condition: splunk-kubernetes-logging.enabled
  name: splunk-kubernetes-logging
  repository: https://splunk.github.io/splunk-connect-for-kubernetes/
  version: 1.4.3
- condition: splunk-kubernetes-objects.enabled
  name: splunk-kubernetes-objects
  repository: https://splunk.github.io/splunk-connect-for-kubernetes/
  version: 1.4.3
- condition: splunk-kubernetes-metrics.enabled
  name: splunk-kubernetes-metrics
  repository: https://splunk.github.io/splunk-connect-for-kubernetes/
  version: 1.4.3
description: Collect logs, objects (metadata), and metrics with Splunk.
home: https://github.com/splunk/splunk-connect-for-kubernetes/tree/main/helm-chart
keywords:
- splunk
- monitoring
- logging
- metrics
- metadata
- objects
maintainers:
- email: DataEdge@splunk.com
  name: Splunk Inc.
name: splunk-connect-for-kubernetes
sources:
- https://github.com/splunk/splunk-connect-for-kubernetes/tree/main/helm-chart
- https://github.com/splunk/fluent-plugin-splunk-hec
- https://github.com/splunk/fluent-plugin-kubernetes-metrics
- https://github.com/splunk/fluent-plugin-k8s-metrics-agg
- https://github.com/splunk/fluent-plugin-kubernetes-objects
version: 1.4.3

• Others: N/A