prometheus-operator: Prometheus breaks: relabel configuration for replace action requires 'target_label' value

What did you do? Deployed SNMP with prometheus operator by using additionalscrapeconfigs

What did you expect to see? Prometheus in running state and see snmp targets under targets section.

What did you see instead? Under which circumstances?

$ ns get pods prometheus-k8s-0
NAME               READY   STATUS             RESTARTS   AGE
prometheus-k8s-0   2/3     CrashLoopBackOff   10         29m

Environment development

• Prometheus Operator version:

  v0.25.0

• Kubernetes version information:

  v1.9.2+coreos.0

• Kubernetes cluster kind:

  Kubespray

• Manifests:

prometheus-prometheus.yaml

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitoring
      port: web
  baseImage: quay.io/prometheus/prometheus
  externalLabels:
    cluster: test-10
  nodeSelector:
    prometheus-node: "1"
  replicas: 1
  resources:
    requests:
      memory: 2Gi
  retention: 60d
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  storage:
    volumeClaimTemplate:
      spec:
        resources:
          requests:
            storage: 10Gi
        selector:
          matchLabels:
            lvm-volume: database-prometheus
  version: v2.5.0
  additionalScrapeConfigs:
    name: snmp-endpoints
    key: endpoints.yaml

endpoints.yaml

- job_name: "snmp-exporter"
  scrape_timeout: 300s
  scrape_interval: 300s
  static_configs:
    - targets: [ 'cisco1.test8.com', 'cisco2.test8.com', 'cisco3.test8.com', 'cisco4.test8.com']
  metrics_path: /snmp
  params:
    module: [zec]
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: snmp-exporter.monitoring.svc.cluster.local:9116

• Prometheus Operator Logs:

caller=main.go:625 err="error loading config from \"/etc/prometheus/config_out/prometheus.env.yaml\": couldn't load configuration (--config.file=\"/etc/prometheus/config_out/prometheus.env.yaml\"): parsing YAML file /etc/prometheus/config_out/prometheus.env.yaml: relabel configuration for replace action requires 'target_label' value"