prometheus-operator: Cannot deprecate tsdb.retention, configure retention.time / retention.bytes, or use extraArgs
What did you do? I’d like to use these new flags and not use the deprecated one:
kubectl -n monitoring exec prometheus-prometheus-operator-prometheus-0 -c prometheus -- \
sh -c 'prometheus --version && prometheus --help 2>&1| grep retention -A4'
prometheus, version 2.7.1 (branch: HEAD, revision: 62e591f928ddf6b3468308b7ac1de1c63aa7fcf3)
build user: root@f9f82868fc43
build date: 20190131-11:16:59
go version: go1.11.5
--storage.tsdb.retention=15d
[DEPRECATED] How long to retain samples in
storage. This flag has been deprecated, use
"storage.tsdb.retention.time" instead
--storage.tsdb.retention.time=15d
How long to retain samples in storage.
Overrides "storage.tsdb.retention" if this flag
is set to anything other than default.
--storage.tsdb.retention.size=0
[EXPERIMENTAL] Maximum number of bytes that can
be stored for blocks. Units supported: KB, MB,
GB, TB, PB. This flag is experimental and can
be changed in future releases.
What did you expect to see?
The PrometheusSpec would have:
- config options for
retentionTimeandretentionSize - a schemaless key:value map for extra/arbitrary command-line args
The retentionTime field would be marked as experimental and subject to change, similar to what was done with ThanosSpec.
What did you see instead? Under which circumstances?
The Prometheus Spec API (2e5d20d) does not include configurations from storage.tsdb.retention.time and storage.tsdb.retention.size.
Prometheus operator configures Prometheus to use the deprecated flag:
kubectl -n monitoring exec prometheus-prometheus-operator-prometheus-0 -c prometheus -- \
ps | gsed 's, --,\n--,g'
PID USER TIME COMMAND
1 1000 1:30 /bin/prometheus
--web.console.templates=/etc/prometheus/consoles
--web.console.libraries=/etc/prometheus/console_libraries
--config.file=/etc/prometheus/config_out/prometheus.env.yaml
--storage.tsdb.path=/prometheus
--storage.tsdb.retention=10d
--web.enable-lifecycle
--storage.tsdb.no-lockfile
--web.external-url=http://ingestion-prometheus.beatportci.com/
--web.route-prefix=/
195 1000 0:00 ps
I also cannot determine how to ask the operator to add arbitrary arguments to the Prometheus command-line using the PrometheusSpec. If I could append arbitrary args, this could be worked around.
Environment
- Prometheus Operator version:
helm list prometheus-operator
...
prometheus-operator ... DEPLOYED prometheus-operator-4.1.1 monitoring
kubectl -n monitoring describe deployments prometheus-operator-operator | grep Image
Image: quay.io/coreos/prometheus-operator:v0.29.0
- Kubernetes version information:
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-04T04:48:55Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.7-gke.6", GitCommit:"144b681e3870074abfdaaf176e48b1b13360d7e5", GitTreeState:"clean", BuildDate:"2019-02-09T00:08:19Z", GoVersion:"go1.10.7b4", Compiler:"gc", Platform:"linux/amd64"}
- Kubernetes cluster kind:
1.11.7-gke.6
-
Manifests: https://github.com/helm/charts/tree/2811cbe/stable/prometheus-operator
-
Prometheus Operator Logs:
N/A
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 4
- Comments: 19 (9 by maintainers)
Roughly calculating with that number 1560 samples ingested per second, and up to 6 hours of data stored on disk due to the above reasons (I won’t discuss those further here as it’s an intentional design decision on the Prometheus project not here):
1560 samples per second * 60 seconds * 6 hours = 561 600 samples disk
A sample with compression is anywhere between 1.3-2 bytes per sample (let’s calculate conservatively with 2):
561 600 samples * 2 bytes = 1 123 200
That’s roughly ~1.2Gb just samples + index. I’d go with a PVC that is at least 2Gb large, 5Gb to be on the safe side.
Hi, I too am interested in using the new flags. Do you know when it will be ready in a release? Also, any advice on a workaround for this meanwhile?