prometheus-operator: Bug Report: crd-apigroup flag doesn't work in the v0.26.0

What did you do? Use --crd-apigroup=test-monitoring.coreos.com with a custom API group

What did you expect to see? Expect it to watch the new API group instead of the default one

What did you see instead? Under which circumstances?

Environment

Prometheus Operator version: v0.26.0
Kubernetes version information:

Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:05:37Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes cluster kind:

insert how you created your cluster: kops, bootkube, tectonic-installer, etc.
Prometheus Operator Logs: In version 0.26.0

ts=2018-12-18T17:34:59.987778434Z caller=main.go:165 msg="Starting Prometheus Operator version '0.26.0'."
level=info ts=2018-12-18T17:35:00.011685443Z caller=operator.go:209 component=alertmanageroperator msg="connection established" cluster-version=v1.10.3
level=info ts=2018-12-18T17:35:00.015433094Z caller=operator.go:377 component=prometheusoperator msg="connection established" cluster-version=v1.10.3
level=info ts=2018-12-18T17:35:00.112100171Z caller=operator.go:631 component=alertmanageroperator msg="CRD updated" crd=Alertmanager
level=info ts=2018-12-18T17:35:00.136427173Z caller=operator.go:1427 component=prometheusoperator msg="CRD updated" crd=Prometheus
level=info ts=2018-12-18T17:35:00.151121508Z caller=operator.go:1427 component=prometheusoperator msg="CRD updated" crd=ServiceMonitor
level=info ts=2018-12-18T17:35:00.163643757Z caller=operator.go:1427 component=prometheusoperator msg="CRD updated" crd=PrometheusRule
ts=2018-12-18T17:35:03.113982058Z caller=main.go:253 msg="Unhandled error received. Exiting..." err="waiting for Alertmanager crd failed: timed out waiting for Custom Resource: failed to list CRD: alertmanagers.
monitoring.coreos.com is forbidden: User \"system:serviceaccount:test-monitoring:test-monitoring-prometheus-operator\" cannot list alertmanagers.monitoring.coreos.com at the cluster scope"

In the above log it should have been alertmanagers.test-monitoring.coreos.com instead of alertmanagers.monitoring.coreos.com.

In version v0.25.0 it seems to work

ts=2018-12-18T17:40:05.626316116Z caller=main.go:165 msg="Starting Prometheus Operator version '0.25.0'."
level=info ts=2018-12-18T17:40:05.675101571Z caller=operator.go:371 component=prometheusoperator msg="connection established" cluster-version=v1.10.3
level=info ts=2018-12-18T17:40:05.678287433Z caller=operator.go:209 component=alertmanageroperator msg="connection established" cluster-version=v1.10.3
level=info ts=2018-12-18T17:40:05.81445132Z caller=operator.go:627 component=alertmanageroperator msg="CRD updated" crd=Alertmanager
level=info ts=2018-12-18T17:40:05.871398564Z caller=operator.go:1417 component=prometheusoperator msg="CRD updated" crd=Prometheus
level=info ts=2018-12-18T17:40:05.883944736Z caller=operator.go:1417 component=prometheusoperator msg="CRD updated" crd=ServiceMonitor
level=info ts=2018-12-18T17:40:05.897371716Z caller=operator.go:1417 component=prometheusoperator msg="CRD updated" crd=PrometheusRule
level=info ts=2018-12-18T17:40:08.826665682Z caller=operator.go:225 component=alertmanageroperator msg="CRD API endpoints ready"
level=info ts=2018-12-18T17:40:09.027151092Z caller=operator.go:180 component=alertmanageroperator msg="successfully synced all caches"
level=info ts=2018-12-18T17:40:09.027376673Z caller=operator.go:448 component=alertmanageroperator msg="sync alertmanager" key=test-monitoring/test-monitoring-prometheus-alertmanager
level=info ts=2018-12-18T17:40:09.054331718Z caller=operator.go:448 component=alertmanageroperator msg="sync alertmanager" key=test-monitoring/test-monitoring-prometheus-alertmanager
level=info ts=2018-12-18T17:40:09.065334661Z caller=operator.go:448 component=alertmanageroperator msg="sync alertmanager" key=test-monitoring/test-monitoring-prometheus-alertmanager
level=info ts=2018-12-18T17:40:14.929313306Z caller=operator.go:387 component=prometheusoperator msg="CRD API endpoints ready"
level=info ts=2018-12-18T17:40:15.630402049Z caller=operator.go:322 component=prometheusoperator msg="successfully synced all caches"
level=info ts=2018-12-18T17:40:15.630626995Z caller=operator.go:956 component=prometheusoperator msg="sync prometheus" key=test-monitoring/test-monitoring-prometheus-prometheus
level=info ts=2018-12-18T17:40:15.705755289Z caller=operator.go:956 component=prometheusoperator msg="sync prometheus" key=test-monitoring/test-monitoring-prometheus-prometheus
level=info ts=2018-12-18T17:40:15.755423806Z caller=operator.go:956 component=prometheusoperator msg="sync prometheus" key=test-monitoring/test-monitoring-prometheus-prometheus

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 20 (13 by maintainers)

Commits related to this issue

Remove dead custom API group code * As stated in #2238, the use of the `--crd-apigroup` when moving to kube generators, this change just removes the flag to avoid confusion to anyone using this flag. — committed to anandsinghkunwar/prometheus-operator by anandsinghkunwar 5 years ago
Remove dead custom API group code * As stated in #2238, the use of the `--crd-apigroup` when moving to kube generators, this change just removes the flag to avoid confusion to anyone using this flag. — committed to anandsinghkunwar/prometheus-operator by anandsinghkunwar 5 years ago

Most upvoted comments

Unfortunately because we kind of have to use the generators I don’t see a good way of doing this, other than integrating functionality like this into the generators.

Yah, I figure it out in the hard way 😦

I’m curious how you do this generally with Kubernetes today, as Kubernetes itself doesn’t allow you to do things like this.

First, we have multiple clusters Second, we are running prometheus 0.25 just because it does have crdgroup flag.

We will probably run multiple prometheuses and will try to divide them by labels and using the same CRDs. I will update here on the results, and will open a PR that will remove this flag and document how to remove the flag for those who are still using it

YonatanKiron on Apr 17, 2019