rook: kubelet_volume* / persistent volume metrics missing

Is this a bug report or feature request? BUG

  • Bug Report

Bug Report /kind bug

What happened: I am trying to get kubelet_volume_* metrics working. According this google/cadvisor#1702 (comment) there should be support after 1.8 kube. However, I am now using kube 1.10.0 and still cannot see anything in /metrics? Should these be there or not?

curl -s https://localhost:10255/metrics -k|grep kubelet_volume|wc -L
0

pvc-d64b963b-4578-11e8-9cf2-d4ae526fcc30   100Gi      RWO            Delete           Bound     default/prom-test-prometheus-server         rook-block               2d

prom-test-prometheus-server         Bound     pvc-d64b963b-4578-11e8-9cf2-d4ae526fcc30   100Gi      RWO            rook-block     2d

What you expected to happen: I except that kubelet should expose prometheus volume metrics to /metrics path as well. Then it is easy to read with prometheus and to monitor the persistent volume usage status How to reproduce it (minimal and precise):

Create the default storage-class and a persistent-volume-claim but no persistent volume metrics will be published

Environment:

  • OS (e.g. from /etc/os-release): Debian GNU/Linux 9.4
  • Kernel (e.g. uname -a): 4.9.0-6-amd64
  • Cloud provider or hardware configuration: Bare-Metal
  • Rook version (use rook version inside of a Rook Pod): v0.7.0-87.g2d40c08
  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-12T14:26:04Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): Bare-metal
  • Ceph status (use ceph health in the Rook toolbox):
root@rook-tools:/# ceph -s
  cluster:
    id:     6af9b6be-2998-4b56-8d62-37fa609f5bde
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum rook-ceph-mon0,rook-ceph-mon2,rook-ceph-mon1
    mgr: rook-ceph-mgr0(active)
    osd: 9 osds: 9 up, 9 in
 
  data:
    pools:   2 pools, 200 pgs
    objects: 3040 objects, 10636 MB
    usage:   220 GB used, 16541 GB / 16762 GB avail
    pgs:     200 active+clean
 
  io:
    client:   42977 B/s wr, 0 op/s rd, 1 op/s wr

Please have a look at this issue currently it is not possible to monitor the pv usage. tia

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 11
  • Comments: 24 (15 by maintainers)

Most upvoted comments

@rootfs something to consider for ceph-csi, or is it already planned?

@travisn This is much required for ceph-csi for integration with Openshift and OCS, since these metrics will be required to show the PV and PVC usage. I have been looking around for these metrics for monitoring use-cases. It would be great if this could be prioritized.

@HaveFun83 The PR got merged to master so it will be available in the next release v1.1.

If there is a high need for this to be in v1.0, I would recommend joining the Community Meeting and bringing it up as a point (best to add an agenda item point, see Readme of repo).

@travisn @jbw976 What do you think on the spot about backporting this to v1.0 and possibly get this released in a patch release?

1.13 is in alpha now and it landed in the spec as well: https://github.com/container-storage-interface/spec/blob/master/spec.md#nodegetvolumestats So I guess it can be implemented soon? @kongslund

Sounds good to backport, it’s a low risk change.

I can see metrics fine from rook