prometheus-operator: prometheusrules: Malformed alert rule halts prometheus-operator rules processing

What did you do?

Created a prometheusrule with a malformed alert rule.

What did you expect to see?

Prometheus ignore the bad rule, provide the name of the failed prometheusrule and continue it’s operation.

What did you see instead? Under which circumstances?

Alert rules processing stopped.

Environment

prom operator running with:

-manage-crds=false

• Prometheus Operator version:

  v0.23.2

• Kubernetes version information:

      Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.9+coreos.1", GitCommit:"cd373fe93e046b0a0bc7e4045af1bf4171cea395", GitTreeState:"clean", BuildDate:"2018-03-13T21:28:21Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
       ````
  
  

• Kubernetes cluster kind:

  tectonic-installer

• Manifests:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  clusterName: ""
  creationTimestamp: 2018-08-27T22:16:22Z
  deletionGracePeriodSeconds: null
  deletionTimestamp: null
  initializers: null
  labels:
    app: prometheus
    chart: prometheus-3.0.7
    heritage: Tiller
    prometheus: prometheus
    release: prometheus
    role: alert-rules
  name: prometheus-prometheus-rules
  namespace: foo
  resourceVersion: "134728661"
spec: |-
  ALERT high_load IF (1 - (avg by (instance, cpu, environment) (irate(node_cpu{mode="idle"}[1m])))) > 0.5 FOR 5m ANNOTATIONS { summary = "Instance {{ $labels.instance }} under high load", description = "{{ $labels.instance }} in {{$labels.environment}} is under high load.", }
  ALERT error_rate IF (1 - (avg by (instance, cpu, environment) (irate(controller_request_error_count[1m])))) > 200 FOR 5m ANNOTATIONS { summary = "Instance {{ $labels.instance }} high error rate count", description = "{{ $labels.instance }} in {{$labels.environment}} is have high error rate count.", }
  ALERT latency IF (avg by (instance, cpu, environment) (irate(http_request_duration_microseconds[1m]))) > 500 FOR 5m ANNOTATIONS { summary = "Instance {{ $labels.instance }} high error rate count", description = "{{ $labels.instance }} in {{$labels.environment}} is have high error rate count.", }

In the case above:

• Prometheus operator logs:

Failed to list *v1.PrometheusRule: json: cannot unmarshal string into Go struct field PrometheusRule.spec of type v1.PrometheusRuleSpec

Another case:

A valid alert spec causes validation error:

    groups:
    - name: general.rules
      rules:
      - alert: noop
        expr: 1
        for: 1m
        labels:
          severity: noop
        annotations:
          summary: 'noop alert'
          description: 'This alert is to enable validation that alerts can be triggered.'

The above rule makes prometheus operator to fail:

• Prometheus Operator Logs:

E0823 21:39:59.146619       1 reflector.go:205] github.com/coreos/prometheus-operator/pkg/prometheus/operator.go:345: Failed to list *v1.PrometheusRule: json: cannot unmarshal number into Go struct field Rule.expr of type string

If -manage-crds=false is set to true, since the prometheus operator tries to pick up config maps and those might also have malformed alert rules. the operator crashes with error.

ts=2018-07-11T14:21:11.558466227Z caller=main.go:175 msg="Unhandled error received. Exiting..." err="creating CRDs failed: waiting for PrometheusRule crd failed: timed out waiting for Custom Resource: failed to list CRD: json: cannot unmarshal number into Go value of type string"