prometheus-operator: kube-prometheus: KubeClientErrors prom rule firing

Hello,

When I use v0.22.0 of kube-prometheus out-of-the-box (without making any modifications of my own), two instances of the KubeClientErrors prom rule are firing (more details below); here’s snippets from alertmanager (http://192.168.99.100:30903/#/alerts):

• Kubernetes API server client 'kubelet/10.0.2.15:10250' is experiencing 3% errors.
• Kubernetes API server client 'apiserver/192.168.99.100:8443' is experiencing 24% errors.

Any ideas as to what is causing this KubeClientErrors prom rule to fire? (Please let me know if by chance I should instead open an issue in https://github.com/kubernetes-monitoring/kubernetes-mixin - because that is where KubeClientErrors is defined.)

(By the way, with minikube I assume I should simply ignore how KubeCPUOvercommit & KubeMemOvercommit are firing? i.e. because I only have a single node with minikube, and the expr for these 2 rules includes (count(node:node_num_cpu:sum)-1).)

---

More details…

Startup minikube:

josh@MyMac$ minikube version
minikube version: v0.28.0
josh@MyMac$ minikube start --kubernetes-version v1.9.4 --bootstrapper=kubeadm --vm-driver=virtualbox --memory 4096 --extra-config=kubelet.authentication-token-webhook=true --extra-config=kubelet.authorization-mode=Webhook --extra-config=scheduler.address=0.0.0.0 --extra-config=controller-manager.address=0.0.0.0
josh@MyMac$ minikube ip
192.168.99.100

Get a couple files from this repo:

~/tmp/prom-op
josh@MyMac$ git clone https://github.com/coreos/prometheus-operator.git

~/tmp/prom-op/prometheus-operator
josh@MyMac$ git checkout tags/v0.22.0

~/tmp/prom-op/prometheus-operator/contrib/kube-prometheus
josh@MyMac$ mkdir ~/my-prom
josh@MyMac$ cp -p build.sh examples/minikube.jsonnet ~/my-prom/

Setup my own kube-prometheus directory, and run kube-prometheus (by the way, KubePodNotReady is also firing - due to the registry-creds pod not being in the Running status):

~/my-prom
josh@MyMac$ jb init
josh@MyMac$ jb install github.com/coreos/prometheus-operator/contrib/kube-prometheus/jsonnet/kube-prometheus/@v0.22.0
josh@MyMac$ ./build.sh minikube.jsonnet
josh@MyMac$ kubectl create -f manifests/
...
josh@MyMac$ kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                    READY     STATUS              RESTARTS   AGE       IP            NODE
kube-system   etcd-minikube                           1/1       Running             0          58m       10.0.2.15     minikube
kube-system   kube-addon-manager-minikube             1/1       Running             0          59m       10.0.2.15     minikube
kube-system   kube-apiserver-minikube                 1/1       Running             0          58m       10.0.2.15     minikube
kube-system   kube-controller-manager-minikube        1/1       Running             0          58m       10.0.2.15     minikube
kube-system   kube-dns-6f4fd4bdf-wwcrf                3/3       Running             0          59m       172.17.0.2    minikube
kube-system   kube-proxy-hkk6t                        1/1       Running             0          59m       10.0.2.15     minikube
kube-system   kube-scheduler-minikube                 1/1       Running             0          59m       10.0.2.15     minikube
kube-system   kubernetes-dashboard-77d8b98585-cq2xs   1/1       Running             0          59m       172.17.0.3    minikube
kube-system   registry-creds-lwfdh                    0/1       ContainerCreating   0          59m       <none>        minikube
kube-system   storage-provisioner                     1/1       Running             0          59m       10.0.2.15     minikube
monitoring    alertmanager-main-0                     2/2       Running             0          45m       172.17.0.7    minikube
monitoring    alertmanager-main-1                     2/2       Running             0          44m       172.17.0.9    minikube
monitoring    alertmanager-main-2                     2/2       Running             0          44m       172.17.0.10   minikube
monitoring    grafana-67dbf88558-zkwwn                1/1       Running             0          45m       172.17.0.5    minikube
monitoring    kube-state-metrics-69df7cf6bc-p2gpz     4/4       Running             0          45m       172.17.0.6    minikube
monitoring    node-exporter-qg4q5                     2/2       Running             0          45m       10.0.2.15     minikube
monitoring    prometheus-k8s-0                        3/3       Running             1          45m       172.17.0.8    minikube
monitoring    prometheus-k8s-1                        3/3       Running             1          44m       172.17.0.11   minikube
monitoring    prometheus-operator-6677fbc69-g69gw     1/1       Running             0          45m       172.17.0.4    minikube