prometheus-operator: enable-admin-api is not possible

What did you do?

I tried to delete some stale data from Prometheus like curl -XPOST https://prometheus.internal.loveos.io/api/v1/admin/tsdb/clean_tombstones but for that I have to enable admin-api first, which is not possible at the moment

What did you expect to see?

some positive http response

What did you see instead? Under which circumstances?

{"status":"error","errorType":"unavailable","error":"Admin APIs disabled"}

Environment

Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-09T21:51:06Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.6", GitCommit:"6260bb08c46c31eea6cb538b34a9ceb3e406689c", GitTreeState:"clean", BuildDate:"2017-12-21T06:23:29Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes cluster kind:

kops
Manifests:

insert manifests relevant to the issue

Prometheus Operator Logs:

insert Prometheus Operator logs relevant to the issue here

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 5
Comments: 20 (12 by maintainers)

Most upvoted comments

What of the case where you really want to remove some metrics? I’ve run into a situation where a team just migrated their metrics from statsd to prometheus, without actually using labels. This created 22k unique metric names and now the autocomplete api and the expression browser are very unresponsive. It seems that the suggested sidecar approach would not enable me to do the required delete_series call to fix this.

wleese on May 22, 2018

@brancz I’m sure this is a problem for users of the api on the public Prometheus server. On your part, there is a default value that turns off the API. Is not it?

I now needed to delete several metrics. Unfortunately, this is not possible at the moment. I would like to still be able to configure Prometheus as the developers intended, and not to rest on your vision of the configuration situation. The operator is not a firewall. Everyone should think for themselves what and when he includes those or other options.

mobilesfinks on Jun 21, 2018

@wleese that makes sense. The problem I want to avoid is enabling users to create public Prometheus servers with unauthenticated/unauthorized admin APIs.

brancz on May 24, 2018

People may want to be able cleaning up some data manually, as described here https://prometheus.io/docs/prometheus/latest/querying/api/#delete-series

Could you just introduce optional parameter for enabling “TSDB Admin APIs”? If it is not enabled by default it should not be security concern, right? I could make and test a patch.

pkg/prometheus/statefulset.go
@@ -385,6 +385,10 @@ func makeStatefulSetSpec(p monitoringv1.Prometheus, c *Config, ruleConfigMaps []
                promArgs = append(promArgs, "-web.external-url="+p.Spec.ExternalURL)
        }

+       if p.Spec.EnableAdminApi == "true" {
+               promArgs = append(promArgs, "-web.enable-admin-api")
+       }
+
        webRoutePrefix := "/"
        if p.Spec.RoutePrefix != "" {
                webRoutePrefix = p.Spec.RoutePrefix

dmitrievav on Apr 13, 2018

As it is yet to be implemented, I’ll put up a PR adding the flag and warning users about potential impact.

audip on Jan 15, 2019

There are too many possibilities how people could protect their Prometheus server, we should just add the field and put a warning in the documentation, that it should always be protected.

brancz on Jun 26, 2018

We can add it as an experimental field, what we’re trying to do here is make a user to the right thing, not necessarily restrict it in total.

We’re all part of the upstream team and even implemented those APIs, we are well aware of their existance and dangers.

brancz on Jun 21, 2018