pomerium: [Pomerium on GKE] Accessing Pomerium Metrics Endpoint Causing Pod Memory Spike
What happened?
Currently, I have a use case where I’m using Pomerium as an Ingress Controller setup in a GKE Cluster. This ingress controller has more than 500+ Ingress manifests configured. I want to set Cloud Monitoring can consume Pomerium Custom Metrics using Google Managed Prometheus, so we decided to set up our managed collection with the following manifests:
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
name: pomerium
namespace: pomerium
spec:
endpoints:
- port: metrics
interval: 30s
metricRelabeling:
- action: keep
regex: "http_client_requests_total"
sourceLabels: [__name__]
- action: keep
regex: "http_server_requests_total"
sourceLabels: [__name__]
- action: keep
regex: "grpc_client_requests_total"
sourceLabels: [__name__]
- action: keep
regex: "grpc_server_requests_total"
sourceLabels: [__name__]
selector:
matchLabels:
app.kubernetes.io/name: pomerium
But turns out it causing all of the Pomerium Controller pod memory to increase almost touching pod limit specs (4GB) in our case. I’m observing by sampling the metrics payload size and found that the payload size is having a size for more than 100MB for each call. Even if we increase the interval collection time, the memory spike still happened.
root@xxxxx-6zk5478998-wrr5q:/# curl http://pomerium-metrics.pomerium.svc.cluster.local:9090/metrics > sample.txt
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 122M 0 122M 0 0 16.0M 0 --:--:-- 0:00:07 --:--:-- 36.1M
root@xxxxx-6zk5478998-wrr5q:/# ls -lh --block-size=MB sample.txt
-rw-r--r-- 1 root root 128MB Nov 24 07:48 sample.txt
This is the current pod condition after the curl process is done (monitored using k9s)
What did you expect to happen?
I’m looking for assistance/advice on how to optimize the exported metrics so that every time the collector hits the metrics endpoint it will not cause the pod memory spike, it’s a big deal because it makes our HPA setup not work well. From my observation so far from the Pomerium documentation, there are no available options to customize the export metrics.
How’d it happen?
- Create 500+ Ingress configuration
- Create a service that hits internal FQDN Pomerium Metrics Endpoint (http://pomerium-metrics.pomerium.svc.cluster.local:9090/metrics)
- One of the Pomerium Controller pods will spike
What’s your environment like?
- Setup Pomerium as Ingress Controller
- Most of the Ingress Backend service is located in the different GKE clusters. it route using
ExternalService(https://www.pomerium.com/docs/k8s/ingress.html#external-services) - The number of
Ingressmanifest for Pomerium is more than 500+ objects - Pomerium configured with the following specs:
resources:
limits:
cpu: "5"
memory: 4Gi
requests:
cpu: 500m
memory: 3Gi
- Pomerium has HPA with the following setup:
spec:
maxReplicas: 18
metrics:
- resource:
name: memory
target:
averageUtilization: 80
type: Utilization
type: Resource
minReplicas: 6
What’s your config.yaml?
Since it’s setup as GKE Controller, this information might be useful: Pomerium CRD config
apiVersion: ingress.pomerium.io/v1
kind: Pomerium
metadata:
name: global
spec:
authenticate:
url: https://xxxxxxx
timeouts:
read: 60m
secrets: pomerium/bootstrap
storage:
postgres:
secret: pomerium/xxxxxxx-config
One of the Ingress config sample
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
ingress.pomerium.io/allow_public_unauthenticated_access: 'true'
ingress.pomerium.io/timeout: '60m'
ingress.pomerium.io/host_rewrite: 'xxxxxxxxxxxxx'
ingress.pomerium.io/set_request_headers: |
X-Custom-Header: asdf
X-Forwarded-Host: xxxxxxxxxxxxx
name: xxxxxxxxxxxxx
spec:
ingressClassName: pomerium
rules:
- host: xxxxxxxxxx
http:
paths:
- backend:
service:
name: qwertyqwertyqwerty
port:
name: http
path: /xxxxxxxxxxxxx/anotherpath
pathType: Prefix
- backend:
service:
name: abcabcabcabc
port:
name: http
path: /
pathType: Prefix
What did you see in the logs?
no specific logs related to metrics appear
Additional context
let me know any other information that might help us solve this issue, Thanks!
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 18 (6 by maintainers)
Let’s get the memory fix merged and see if it fixes the memory usage for you first.
The suggestion for label filtering also seems reasonable to me let’s address it separately though - I’ll resume the discussion in a dedicated ticket.