prometheus-operator: Unable to update the alert rules

What did you do? I have added a new alert rule using custom-values.yaml and after that when tried to update the alert expression, its not reflecting in the prometheus-kube-prometheus-rulefiles config map.

custom-values.yaml - old settings

prometheus:
    # default rules are in templates/general.rules.yaml
    prometheusRules:
      test2a.rule.yaml: |-
        groups:
          - alert: httbinDeploymentMissing
            expr: kube_deployment_status_replicas{deployment="httpbin"} < 1
            for: 5m
            labels:
              severity: critical
            annotations:
              description: Prometheus could not find the Simple Deployment
              summary: Testing alerts.

custom-values.yaml - new settings

prometheus:
    # default rules are in templates/general.rules.yaml
    prometheusRules:
      test2a.rule.yaml: |-
        groups:
          - alert: httbinDeploymentMissing
            expr: kube_deployment_status_replicas{deployment="httpbin"} < 2
            for: 5m
            labels:
              severity: critical
            annotations:
              description: Prometheus could not find the Simple Deployment
              summary: Testing alerts.

This is the command I tried to upgrade the chart helm upgrade kube-prometheus coreos/kube-prometheus -f custom-values.yaml

What did you expect to see? In Prometheus dasboard, alert rule should be updated with an expression as expr: kube_deployment_status_replicas{deployment=“httpbin”} < 2

What did you see instead? Under which circumstances?

Environment

Kubernetes version information:

insert output of kubectl version here

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:21:50Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:13:31Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes cluster kind: kops insert how you created your cluster: kops, bootkube, tectonic-installer, etc.
Manifests:

insert manifests relevant to the issue

Prometheus Operator Logs:

insert Prometheus Operator logs relevant to the issue here

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 4
Comments: 18 (8 by maintainers)

Commits related to this issue

helm: Use CRDs for rules for operator 0.20.0+ Changed rules configmapst push -f to be PrometheusRule instead Deprecated `additionalRulesConfigMapLabels` in favor of `additionalRulesLabels` Fixes #15... — committed to vsliouniaev/prometheus-operator by vsliouniaev 6 years ago
helm: Use CRDs for rules for operator 0.20.0+ (#1581) * helm: Use CRDs for rules for operator 0.20.0+ Changed rules configmapst push -f to be PrometheusRule instead Deprecated `additionalRulesCon... — committed to prometheus-operator/prometheus-operator by vsliouniaev 6 years ago

Most upvoted comments

First is check what is your Rule Selector, by:

Get current prometheus CRD (with default namespace is monitoring) kubectl describe prometheus $(kubectl get -n monitoring -o=name) -n monitoring
Find Rule Selector, like:

Rule Selector:
    Match Labels:
      App:      prometheus-operator
      Release:  monitoring

In your PrometheusRule set labels as above:

...
kind: PrometheusRule
metadata:
  labels:
    app: prometheus-operator
    release: monitoring
...

shockfish on Sep 24, 2020

hmm ok, so ideally the helm charts would just adopt the CRDs (they are community maintained and sometimes lag behind a little). Until then you can modify them by just running

kubectl -n <namespace> edit prometheusrule <rule-object-name>

And you can find rules in a namespace by

kubectl -n <namespace> get prometheusrule

brancz on Jul 12, 2018

Oh, so I have to get every prometheusrule to yaml and work with that resource. Copy that, thank you, @brancz , now everything seems to work as expected.

Blasterdick on Jul 11, 2018