VictoriaMetrics: vmagent: too many connections to APISERVER on AKS
Describe the bug
I have average <1500 endpoints on my prod environment to scrape as below screenshot.
When I am using netstat command inside vmagent pod to know how many connection “ESTABLISHED” to scrape I am getting right value except endpoint kubeapiserver. In the below output I excluded apiserver endpoint. ` ~ $ netstat -atn | grep ESTABLISHED | grep -v 172.18.0.1:443 | wc -l
1491 `
Now If I am going to know “ESTABLISHED” with KubeAPIserver then I am getting below output. ` ~ $ netstat -atn | grep ESTABLISHED | grep 172.18.0.1:443 | wc -l
1150 `
So, even I tried with “promscrape.disableKeepAlive: false” I am getting the same. Now my question is why vmagent is establishing those very high number of connections with kubeapiserver. I don’t think there is any sense.
My cluster was built with publicKubeApiserver config and we are getting prod issue SNAT port exhaustion on AKS. That’s product application is not able to server traffic to client as those ports were consumed by vmagent while connecting public KubeAPIserver.
Even I compared with other service like prometheus they are not creating that number of connections with KubeAPIserver.
To Reproduce
I am using with below config for vmagent to enable monitoring of the podmonitor or vmscapeservice. I am using VMOperator to deploy with below config.
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: vmagent-1
namespace: monitoring-system
spec:
podScrapeNamespaceSelector: {}
podScrapeSelector: {}
serviceScrapeNamespaceSelector: {}
serviceScrapeSelector: {}
extraArgs:
remoteWrite.forcePromProto: "true"
envflag.enable: "true"
envflag.prefix: "VM_"
sortLabels: "true"
promscrape.noStaleMarkers: "true"
promscrape.maxScrapeSize: 200MB
promscrape.kubernetesSDCheckInterval: 30s
promscrape.cluster.replicationFactor: "2"
containers:
- name: config-reloader
image: prometheus-operator/prometheus-config-reloader:v0.71.2
resources:
limits:
cpu: "1"
memory: "2Gi"
securityContext:
fsGroup: 65534
runAsGroup: 65534
runAsNonRoot: true
runAsUser: 65534
image:
repository: victoriametrics/vmagent
tag: "v1.93.12"
Version
v1.93.12
Logs
There is no error on vmagent pods. I can’t paste logs info logs in public domain due to its having pods and NS info.
Screenshots
No response
Used command-line flags
No response
Additional information
No response
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 19 (7 by maintainers)
Commits related to this issue
- lib/promscrape/discovery/kubernetes: re-use http.Client for different group watchers Creating a separate http.Client for every groupWatcher results into maintaining a separate connections pool per cl... — committed to VictoriaMetrics/VictoriaMetrics by zekker6 2 years ago
- lib/promscrape/discovery/kubernetes: re-use http.Client for different group watchers Creating a separate http.Client for every groupWatcher results into maintaining a separate connections pool per cl... — committed to VictoriaMetrics/VictoriaMetrics by zekker6 2 years ago
- lib/promscrape/discovery/kubernetes: re-use http.Client for different group watchers Creating a separate http.Client for every groupWatcher results into maintaining a separate connections pool per cl... — committed to VictoriaMetrics/VictoriaMetrics by zekker6 2 years ago
- lib/promscrape/discovery/kubernetes: re-use http.Client for different group watchers Creating a separate http.Client for every groupWatcher results into maintaining a separate connections pool per cl... — committed to VictoriaMetrics/VictoriaMetrics by zekker6 2 years ago
- vmagent: add support of HTTP2 client for Kubernetes SD (#7114) ### Describe Your Changes Currently, vmagent always uses a separate `http.Client` for every group watcher in Kubernetes SD. With a h... — committed to VictoriaMetrics/VictoriaMetrics by zekker6 2 years ago
- vmagent: add support of HTTP2 client for Kubernetes SD (#7114) ### Describe Your Changes Currently, vmagent always uses a separate `http.Client` for every group watcher in Kubernetes SD. With a high... — committed to VictoriaMetrics/VictoriaMetrics by zekker6 2 years ago
- lib/promauth: follow-up for the commit eefae854500e742b4788c16cedd883e20f335a3d - Avoid a data race when multiple goroutines access and update roundTripper.trBase inside roundTripper.getTransport(). ... — committed to VictoriaMetrics/VictoriaMetrics by valyala 2 years ago
- lib/promauth: follow-up for the commit eefae854500e742b4788c16cedd883e20f335a3d - Avoid a data race when multiple goroutines access and update roundTripper.trBase inside roundTripper.getTransport(). ... — committed to VictoriaMetrics/VictoriaMetrics by valyala 2 years ago
- lib/promauth: follow-up for the commit eefae854500e742b4788c16cedd883e20f335a3d - Avoid a data race when multiple goroutines access and update roundTripper.trBase inside roundTripper.getTransport(). ... — committed to G-Research/VictoriaMetrics by valyala 2 years ago
- vmagent: add support of HTTP2 client for Kubernetes SD (#7114) ### Describe Your Changes Currently, vmagent always uses a separate `http.Client` for every group watcher in Kubernetes SD. With a h... — committed to truepele/VictoriaMetrics by zekker6 2 years ago
- lib/promauth: follow-up for the commit eefae854500e742b4788c16cedd883e20f335a3d - Avoid a data race when multiple goroutines access and update roundTripper.trBase inside roundTripper.getTransport(). ... — committed to truepele/VictoriaMetrics by valyala 2 years ago
- vmagent: add support of HTTP2 client for Kubernetes SD (#7114) ### Describe Your Changes Currently, vmagent always uses a separate `http.Client` for every group watcher in Kubernetes SD. With a h... — committed to truepele/VictoriaMetrics by zekker6 2 years ago
- lib/promauth: follow-up for the commit eefae854500e742b4788c16cedd883e20f335a3d - Avoid a data race when multiple goroutines access and update roundTripper.trBase inside roundTripper.getTransport(). ... — committed to truepele/VictoriaMetrics by valyala 2 years ago
@Mahagon Here is vmagent version build from this commit, it includes support of using HTTP2 when connecting to kube api server: victoriametrics/vmagent:heads-vmagent-http-client-0-g62f8b28ce2 In order to use HTTP2 client you will need to add the following command-line flag to vmagent:
promscrape.kubernetes.useHTTP2Client=true.Fyi, upcoming version of operator will create less urlWatchers and use less connections. It must reduce load on Kubernetes API server.
Related issue: https://github.com/VictoriaMetrics/operator/issues/1291
@Mahagon Thank you for quick response! I’ll proceed with adding HTTP/2 support than and will ask you for testing once it will be ready (later this week).
Awesome!
Will test it in our dev environment for a day and report back tomorrow 😃
Edit: guess i can report back faster:
it seems that there is no impact on the amount of the connections. I disabled vmagent for testing for a short while, and validated that it’s still having a big impact on the connection count.
Hello! We’re considering adding HTTP2 support for this discovery method, but before doing this it would be great to verify if amount of opened connections can be lowered by re-using
http.Clientwhich vmagent currently creates for each unique scrape job configuration.This commit updated logic of creating
http.Clientused by discovery in Kubernetes. It would be great if you could verify the effect on amount of opened connections with this change.Here is a docker image build from this commit for testing: victoriametrics/vmagent:heads-vmagent-http-client-0-g1d89f7e986 @Mahagon Could you check if this will help in your case?
@hagen1778 No, it’s not related. since https://github.com/VictoriaMetrics/operator/pull/267 is about reduce pressure for apiserver in operator when generating scrape configs.