VictoriaMetrics: vmagent: too many connections to APISERVER on AKS

Describe the bug

I have average <1500 endpoints on my prod environment to scrape as below screenshot. image

When I am using netstat command inside vmagent pod to know how many connection “ESTABLISHED” to scrape I am getting right value except endpoint kubeapiserver. In the below output I excluded apiserver endpoint. ` ~ $ netstat -atn | grep ESTABLISHED | grep -v 172.18.0.1:443 | wc -l

1491 `

Now If I am going to know “ESTABLISHED” with KubeAPIserver then I am getting below output. ` ~ $ netstat -atn | grep ESTABLISHED | grep 172.18.0.1:443 | wc -l

1150 `

So, even I tried with “promscrape.disableKeepAlive: false” I am getting the same. Now my question is why vmagent is establishing those very high number of connections with kubeapiserver. I don’t think there is any sense.

My cluster was built with publicKubeApiserver config and we are getting prod issue SNAT port exhaustion on AKS. That’s product application is not able to server traffic to client as those ports were consumed by vmagent while connecting public KubeAPIserver.

Even I compared with other service like prometheus they are not creating that number of connections with KubeAPIserver.

To Reproduce

I am using with below config for vmagent to enable monitoring of the podmonitor or vmscapeservice. I am using VMOperator to deploy with below config.


apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
  name: vmagent-1
  namespace: monitoring-system
spec:
  podScrapeNamespaceSelector: {}
  podScrapeSelector: {}
  serviceScrapeNamespaceSelector: {}
  serviceScrapeSelector: {}
  extraArgs:
    remoteWrite.forcePromProto: "true"
    envflag.enable: "true"
    envflag.prefix: "VM_"
    sortLabels: "true"
    promscrape.noStaleMarkers: "true"
    promscrape.maxScrapeSize: 200MB
    promscrape.kubernetesSDCheckInterval: 30s
    promscrape.cluster.replicationFactor: "2"
  containers:
    - name: config-reloader
      image: prometheus-operator/prometheus-config-reloader:v0.71.2
      resources:
        limits:
          cpu: "1"
          memory: "2Gi"
  securityContext:
    fsGroup: 65534
    runAsGroup: 65534
    runAsNonRoot: true
    runAsUser: 65534
  image:
    repository: victoriametrics/vmagent
    tag: "v1.93.12"

Version

v1.93.12

Logs

There is no error on vmagent pods. I can’t paste logs info logs in public domain due to its having pods and NS info.

Screenshots

No response

Used command-line flags

No response

Additional information

No response

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 19 (7 by maintainers)

Commits related to this issue

Most upvoted comments

@Mahagon Here is vmagent version build from this commit, it includes support of using HTTP2 when connecting to kube api server: victoriametrics/vmagent:heads-vmagent-http-client-0-g62f8b28ce2 In order to use HTTP2 client you will need to add the following command-line flag to vmagent: promscrape.kubernetes.useHTTP2Client=true.

image Looks really good 😃 (placed the arrow a little to far to the left 😄)

Fyi, upcoming version of operator will create less urlWatchers and use less connections. It must reduce load on Kubernetes API server.

Related issue: https://github.com/VictoriaMetrics/operator/issues/1291

@Mahagon Thank you for quick response! I’ll proceed with adding HTTP/2 support than and will ask you for testing once it will be ready (later this week).

Awesome!

Will test it in our dev environment for a day and report back tomorrow 😃

Edit: guess i can report back faster:

it seems that there is no impact on the amount of the connections. I disabled vmagent for testing for a short while, and validated that it’s still having a big impact on the connection count.

image

Hello! We’re considering adding HTTP2 support for this discovery method, but before doing this it would be great to verify if amount of opened connections can be lowered by re-using http.Client which vmagent currently creates for each unique scrape job configuration.

This commit updated logic of creating http.Client used by discovery in Kubernetes. It would be great if you could verify the effect on amount of opened connections with this change.

Here is a docker image build from this commit for testing: victoriametrics/vmagent:heads-vmagent-http-client-0-g1d89f7e986 @Mahagon Could you check if this will help in your case?

@Haleygo do you have any opinion on this? It looks related to https://github.com/VictoriaMetrics/operator/pull/267

@hagen1778 No, it’s not related. since https://github.com/VictoriaMetrics/operator/pull/267 is about reduce pressure for apiserver in operator when generating scrape configs.