keda: KEDA metric server is leaking tcp connections
Report
we noticed following errors E0716 16:51:54.766808 1 provider.go:124] keda_metrics_adapter/provider “msg”=“error getting metric for scaler” “error”=“Get "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=redacted\”: dial tcp 169.254.169.254:80: connect: cannot assign requested address".
after entering the network namespaces of the metrics adaptor we found keda-mteric0-adaptor had opened 28k connections which is the limit and for new connections we were getting the error mentioned above.
We are using keda 2.7.1 in our cluster
Expected Behavior
keda should not leak tcp connections
Actual Behavior
keda is creating new tcp connections
Steps to Reproduce the Problem
- Install keda 2.7.1
Logs from KEDA operator
E0716 16:51:54.766808 1 provider.go:124] keda_metrics_adapter/provider "msg"="error getting metric for scaler" "error"="Get \"http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=redacted\": dial tcp 169.254.169.254:80: connect: cannot assign requested address"
KEDA Version
2.7.1
Kubernetes Version
1.21.10
Platform
azure
Scaler Details
azure_storage azure service bus
Anything else?
No response
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 3
- Comments: 28 (11 by maintainers)
Hi, we have just installed the latest version
2.10.1and I have suspicion that the leak is still there.here is the last error log (it was fixed that time):
here is the consumed memory (there is 1 hours shift):
the blue line in the graph is the memory request.
I will try to check later today whether we can perform one more test which will last bit longer.
Regards, R.
mostly imds @amirschw nsenter -t <pid of keda process> -n /bin/bash ss -s ss -t | grep -i estab you will mostly find connections between keda and imds endpoint