datadog-agent: Failure to connect to kubelet

Hi,

I am getting logs in datadog, however the agent logs on K8 have the following errors:

[ TRACE ] trace-agent exited with code 0, disabling [ AGENT ] 2018-10-17 08:18:24 UTC | WARN | (datadog_agent.go:149 in LogMessage) | (base.py:212) | DEPRECATION NOTICE: device_name is deprecated, please use a device: tag in the tags list instead [ AGENT ] 2018-10-17 08:18:26 UTC | ERROR | (kubeutil.go:50 in GetKubeletConnectionInfo) | connection to kubelet failed: temporary failure in kubeutil, will retry later: try delay not elapsed yet [ AGENT ] 2018-10-17 08:18:26 UTC | ERROR | (runner.go:289 in work) | Error running check kubelet: [{“message”: “Unable to detect the kubelet URL automatically.”, “traceback”: “Traceback (most recent call last):\n File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/checks/base.py", line 352, in run\n self.check(copy.deepcopy(self.instances[0]))\n File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/kubelet/kubelet.py", line 107, in check\n raise CheckException("Unable to detect the kubelet URL automatically.")\nCheckException: Unable to detect the kubelet URL automatically.\n”}] [ AGENT ] 2018-10-17 08:18:28 UTC | ERROR | (autoconfig.go:604 in collect) | Unable to collect configurations from provider Kubernetes: temporary failure in kubeutil, will retry later: try delay not elapsed yet

image: repository: datadog/agent
tag: 6.4.2 pullPolicy: IfNotPresent

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 4
Comments: 16 (3 by maintainers)

Most upvoted comments

@Aiqbal1234 So I managed to get around this as when I updated my cluster from 1.10 --> 1.11 the kubelet checks started failing with the same message (EKS platform V2, k8s 1.11). I don’t think that the version of the agent is entirely the problem.

What I did to resolve it was:

Deployed the requisite kube-state-metrics
Pull all of my datadog daemonset yamls. Even though previous deployments of the same yamls had worked, I suspect that using latest of the agent had some impact collecting metrics from the cluster after I upgraded. I grabbed the RBAC-related details from links on this page and then created a manifest using the example.
Did a similar process for the Cluster Agent

fontanese on Mar 13, 2019

Also seeing this now. Weird thing is that it started about 6 days ago, on a new AKS cluster which had been running for about 2 weeks and has produced data ingested in Datadog (mainly logs).

2020-09-12T20:26:26.839013998Z 2020-09-12 20:26:26 UTC | CORE | ERROR | (pkg/collector/runner/runner.go:292 in work) | Error running check kubelet: [{"message": "Unable to detect the kubelet URL automatically.", "traceback": "Traceback (most recent call last):\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py\", line 827, in run\n    self.check(instance)\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/kubelet/kubelet.py\", line 297, in check\n    raise CheckException(\"Unable to detect the kubelet URL automatically.\")\ndatadog_checks.base.errors.CheckException: Unable to detect the kubelet URL automatically.\n"}]
2020-09-12T20:26:34.630932898Z 2020-09-12 20:26:34 UTC | CORE | ERROR | (pkg/autodiscovery/config_poller.go:123 in collect) | Unable to collect configurations from provider kubernetes: temporary failure in kubeutil, will retry later: try delay not elapsed yet
2020-09-12T20:26:41.839154350Z 2020-09-12 20:26:41 UTC | CORE | ERROR | (pkg/collector/python/kubeutil.go:38 in getConnections) | connection to kubelet failed: temporary failure in kubeutil, will retry later: try delay not elapsed yet
2020-09-12T20:26:41.839184152Z 2020-09-12 20:26:41 UTC | CORE | ERROR | (pkg/collector/runner/runner.go:292 in work) | Error running check kubelet: [{"message": "Unable to detect the kubelet URL automatically.", "traceback": "Traceback (most recent call last):\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py\", line 827, in run\n    self.check(instance)\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/kubelet/kubelet.py\", line 297, in check\n    raise CheckException(\"Unable to detect the kubelet URL automatically.\")\ndatadog_checks.base.errors.CheckException: Unable to detect the kubelet URL automatically.\n"}]
2020-09-12T20:26:44.630902490Z 2020-09-12 20:26:44 UTC | CORE | ERROR | (pkg/autodiscovery/config_poller.go:123 in collect) | Unable to collect configurations from provider kubernetes: temporary failure in kubeutil, will retry later: try delay not elapsed yet
2020-09-12T20:26:54.630977789Z 2020-09-12 20:26:54 UTC | CORE | ERROR | (pkg/autodiscovery/config_poller.go:123 in collect) | Unable to collect configurations from provider kubernetes: temporary failure in kubeutil, will retry later: try delay not elapsed yet
2020-09-12T20:26:56.838370937Z 2020-09-12 20:26:56 UTC | CORE | ERROR | (pkg/collector/python/kubeutil.go:38 in getConnections) | connection to kubelet failed: temporary failure in kubeutil, will retry later: try delay not elapsed yet
2020-09-12T20:26:56.838969278Z 2020-09-12 20:26:56 UTC | CORE | ERROR | (pkg/collector/runner/runner.go:292 in work) | Error running check kubelet: [{"message": "Unable to detect the kubelet URL automatically.", "traceback": "Traceback (most recent call last):\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py\", line 827, in run\n    self.check(instance)\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/kubelet/kubelet.py\", line 297, in check\n    raise CheckException(\"Unable to detect the kubelet URL automatically.\")\ndatadog_checks.base.errors.CheckException: Unable to detect the kubelet URL automatically.\n"}]
2020-09-12T20:27:04.632353674Z 2020-09-12 20:27:04 UTC | CORE | ERROR | (pkg/autodiscovery/config_poller.go:123 in collect) | Unable to collect configurations from provider kubernetes: temporary failure in kubeutil, will retry later: try delay not elapsed yet
2020-09-12T20:27:11.838674496Z 2020-09-12 20:27:11 UTC | CORE | ERROR | (pkg/collector/python/kubeutil.go:38 in getConnections) | connection to kubelet failed: temporary failure in kubeutil, will retry later: try delay not elapsed yet

andrei-dascalu on Sep 12, 2020

2020-06-23 09:47:14 UTC | CORE | ERROR | (pkg/collector/runner/runner.go:292 in work) | Error running check kubelet: [{“message”: “Unable to detect the kubelet URL automatically.”, “traceback”: “Traceback (most recent call last):\n File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py", line 820, in run\n self.check(instance)\n File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/kubelet/kubelet.py", line 291, in check\n raise CheckException("Unable to detect the kubelet URL automatically.")\ndatadog_checks.base.errors.CheckException: Unable to detect the kubelet URL automatically.\n”}]

SychevIgor on Jun 23, 2020