prometheus-operator: Cannot deprecate tsdb.retention, configure retention.time / retention.bytes, or use extraArgs

What did you do? I’d like to use these new flags and not use the deprecated one:

kubectl -n monitoring exec prometheus-prometheus-operator-prometheus-0 -c prometheus -- \
  sh -c 'prometheus --version && prometheus --help 2>&1| grep retention -A4'

prometheus, version 2.7.1 (branch: HEAD, revision: 62e591f928ddf6b3468308b7ac1de1c63aa7fcf3)
  build user:       root@f9f82868fc43
  build date:       20190131-11:16:59
  go version:       go1.11.5
      --storage.tsdb.retention=15d
                                 [DEPRECATED] How long to retain samples in
                                 storage. This flag has been deprecated, use
                                 "storage.tsdb.retention.time" instead
      --storage.tsdb.retention.time=15d
                                 How long to retain samples in storage.
                                 Overrides "storage.tsdb.retention" if this flag
                                 is set to anything other than default.
      --storage.tsdb.retention.size=0
                                 [EXPERIMENTAL] Maximum number of bytes that can
                                 be stored for blocks. Units supported: KB, MB,
                                 GB, TB, PB. This flag is experimental and can
                                 be changed in future releases.

What did you expect to see? The PrometheusSpec would have:

  • config options for retentionTime and retentionSize
  • a schemaless key:value map for extra/arbitrary command-line args

The retentionTime field would be marked as experimental and subject to change, similar to what was done with ThanosSpec.

What did you see instead? Under which circumstances? The Prometheus Spec API (2e5d20d) does not include configurations from storage.tsdb.retention.time and storage.tsdb.retention.size.

Prometheus operator configures Prometheus to use the deprecated flag:

kubectl -n monitoring exec prometheus-prometheus-operator-prometheus-0 -c prometheus -- \
  ps | gsed 's, --,\n--,g'

PID   USER     TIME  COMMAND
    1 1000      1:30 /bin/prometheus
--web.console.templates=/etc/prometheus/consoles
--web.console.libraries=/etc/prometheus/console_libraries
--config.file=/etc/prometheus/config_out/prometheus.env.yaml
--storage.tsdb.path=/prometheus
--storage.tsdb.retention=10d
--web.enable-lifecycle
--storage.tsdb.no-lockfile
--web.external-url=http://ingestion-prometheus.beatportci.com/
--web.route-prefix=/
  195 1000      0:00 ps

I also cannot determine how to ask the operator to add arbitrary arguments to the Prometheus command-line using the PrometheusSpec. If I could append arbitrary args, this could be worked around.

Environment

  • Prometheus Operator version:
helm list prometheus-operator
...
prometheus-operator ... DEPLOYED	prometheus-operator-4.1.1	monitoring

kubectl -n monitoring describe deployments prometheus-operator-operator | grep Image
    Image:      quay.io/coreos/prometheus-operator:v0.29.0
  • Kubernetes version information:
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-04T04:48:55Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.7-gke.6", GitCommit:"144b681e3870074abfdaaf176e48b1b13360d7e5", GitTreeState:"clean", BuildDate:"2019-02-09T00:08:19Z", GoVersion:"go1.10.7b4", Compiler:"gc", Platform:"linux/amd64"}
  • Kubernetes cluster kind:
1.11.7-gke.6
N/A

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 4
  • Comments: 19 (9 by maintainers)

Most upvoted comments

Roughly calculating with that number 1560 samples ingested per second, and up to 6 hours of data stored on disk due to the above reasons (I won’t discuss those further here as it’s an intentional design decision on the Prometheus project not here):

1560 samples per second * 60 seconds * 6 hours = 561 600 samples disk

A sample with compression is anywhere between 1.3-2 bytes per sample (let’s calculate conservatively with 2):

561 600 samples * 2 bytes = 1 123 200

That’s roughly ~1.2Gb just samples + index. I’d go with a PVC that is at least 2Gb large, 5Gb to be on the safe side.

Hi, I too am interested in using the new flags. Do you know when it will be ready in a release? Also, any advice on a workaround for this meanwhile?