k8ssandra-operator: Prometheus scrape errors: invalid metric name or label names

What did you do?

I deployed a K8ssandraCluster with Prometheus metrics enabled (see below) using the k8ssandra-operator.

Three of the four targets of the cass-servicemonitors throw scrape errors like this one:

invalid metric name or label names: {__name__="mcac_hints_hints_created.10.244.3.158.7000_total", cassandra_datastax_com_cluster="k8ssandra", cassandra_datastax_com_datacenter="dc1", cluster="k8ssandra", container="cassandra", dc="dc1", endpoint="prometheus", exported_instance="10.244.4.231", instance="10.244.4.231:9103", job="k8ssandra-dc1-all-pods-service", mcac="org.apache.cassandra.metrics.hints_service.hints_created.10.244.3.158.7000", mcac_filtered="true", namespace="cassandra", pod="k8ssandra-dc1-default-sts-0", rack="default", service="k8ssandra-dc1-all-pods-service"}

Is there something I can do about it? Is something wrong with my K8ssandraCluster?

Did you expect to see some different?

No scrape errors.

Environment

• K8ssandra Operator version:

  Image:         docker.io/k8ssandra/k8ssandra-operator:v1.6.0
  Image ID:      docker.io/k8ssandra/k8ssandra-operator@sha256:f74057380f4c18ea45c0c76f12e7eb74730209f232703cdfec5739514a16a4c3
  

• Kubernetes version information:

  # kubectl version --output=yaml
  clientVersion:
    buildDate: "2023-03-15T13:37:18Z"
    compiler: gc
    gitCommit: ef70d260f3d036fc22b30538576bbf6b36329995
    gitTreeState: clean
    gitVersion: v1.24.12
    goVersion: go1.19.7
    major: "1"
    minor: "24"
    platform: linux/amd64
  kustomizeVersion: v4.5.4
  serverVersion:
    buildDate: "2023-03-15T13:30:13Z"
    compiler: gc
    gitCommit: ef70d260f3d036fc22b30538576bbf6b36329995
    gitTreeState: clean
    gitVersion: v1.24.12
    goVersion: go1.19.7
    major: "1"
    minor: "24"
    platform: linux/amd64
  

• Kubernetes cluster kind:

  bare metal, kubeadm

• Manifests:

  apiVersion: k8ssandra.io/v1alpha1
  kind: K8ssandraCluster
  metadata:
    name: k8ssandra
  spec:
    auth: false
    cassandra:
      config:
        jvmOptions:
          heap_initial_size: 4G
          heap_max_size: 4G
      datacenters:
        - metadata:
            name: dc1
          resources:
            requests:
              cpu: 500m
              memory: 5Gi
            limits:
              cpu: 2
              memory: 10Gi
          size: 4
          storageConfig:
            cassandraDataVolumeClaimSpec:
              storageClassName: local-path
              accessModes:
                - ReadWriteOnce
              resources:
                requests:
                  storage: 5Gi
          telemetry:
            prometheus:
              commonLabels:
                release: kube-prometheus-stack
              enabled: true
      serverVersion: "4.0.7"
    reaper:
      autoScheduling:
        enabled: true
      telemetry:
        prometheus:
          commonLabels:
            release: kube-prometheus-stack
          enabled: true