rook: kubelet_volume* / persistent volume metrics missing
Is this a bug report or feature request? BUG
- Bug Report
Bug Report /kind bug
What happened: I am trying to get kubelet_volume_* metrics working. According this google/cadvisor#1702 (comment) there should be support after 1.8 kube. However, I am now using kube 1.10.0 and still cannot see anything in /metrics? Should these be there or not?
curl -s https://localhost:10255/metrics -k|grep kubelet_volume|wc -L
0
pvc-d64b963b-4578-11e8-9cf2-d4ae526fcc30 100Gi RWO Delete Bound default/prom-test-prometheus-server rook-block 2d
prom-test-prometheus-server Bound pvc-d64b963b-4578-11e8-9cf2-d4ae526fcc30 100Gi RWO rook-block 2d
What you expected to happen: I except that kubelet should expose prometheus volume metrics to /metrics path as well. Then it is easy to read with prometheus and to monitor the persistent volume usage status How to reproduce it (minimal and precise):
Create the default storage-class and a persistent-volume-claim but no persistent volume metrics will be published
Environment:
- OS (e.g. from /etc/os-release): Debian GNU/Linux 9.4
- Kernel (e.g.
uname -a): 4.9.0-6-amd64 - Cloud provider or hardware configuration: Bare-Metal
- Rook version (use
rook versioninside of a Rook Pod):v0.7.0-87.g2d40c08 - Kubernetes version (use
kubectl version):
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-12T14:26:04Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
- Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): Bare-metal
- Ceph status (use
ceph healthin the Rook toolbox):
root@rook-tools:/# ceph -s
cluster:
id: 6af9b6be-2998-4b56-8d62-37fa609f5bde
health: HEALTH_OK
services:
mon: 3 daemons, quorum rook-ceph-mon0,rook-ceph-mon2,rook-ceph-mon1
mgr: rook-ceph-mgr0(active)
osd: 9 osds: 9 up, 9 in
data:
pools: 2 pools, 200 pgs
objects: 3040 objects, 10636 MB
usage: 220 GB used, 16541 GB / 16762 GB avail
pgs: 200 active+clean
io:
client: 42977 B/s wr, 0 op/s rd, 1 op/s wr
Please have a look at this issue currently it is not possible to monitor the pv usage. tia
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 11
- Comments: 24 (15 by maintainers)
@travisn This is much required for ceph-csi for integration with Openshift and OCS, since these metrics will be required to show the PV and PVC usage. I have been looking around for these metrics for monitoring use-cases. It would be great if this could be prioritized.
@HaveFun83 The PR got merged to
masterso it will be available in the next releasev1.1.If there is a high need for this to be in v1.0, I would recommend joining the Community Meeting and bringing it up as a point (best to add an agenda item point, see Readme of repo).
@travisn @jbw976 What do you think on the spot about backporting this to v1.0 and possibly get this released in a patch release?
1.13 is in alpha now and it landed in the spec as well: https://github.com/container-storage-interface/spec/blob/master/spec.md#nodegetvolumestats So I guess it can be implemented soon? @kongslund
Sounds good to backport, it’s a low risk change.
I can see metrics fine from rook