ceph-ansible: `"use ceph-volume to create {{ osd_objectstore }} osds"` is very slow, because it calls `ceph volume lvm list` once per OSD on a host

Bug Report

What happened: Running ceph-ansible on a Luminous / ceph-volume lvm / bluestore cluster takes a very long time, most of which is in the "use ceph-volume to create {{ osd_objectstore }} osds" task. You can see below that it takes at least 2 hours per host - and this is not creating any OSDs, just going through the already-deployed ones on each host.

ceph-osd : use ceph-volume to create bluestore osds ------------------ 11228.03s
ceph-osd : use ceph-volume to create bluestore osds ------------------ 8466.27s
ceph-osd : use ceph-volume to create bluestore osds ------------------ 8390.08s

Looking at it, this is because the ceph-volume plugin runs ceph volume lvm list individually for each OSD, and that takes a while (between about 15 and 120s). It should, instead, call ceph volume lvm list once, and process the output itself.

Environment:

  • OS (e.g. from /etc/os-release): Ubuntu 16.04 and 18.04 (happens on both)
  • Docker version if applicable (e.g. docker version): n/a
  • Ansible version (e.g. ansible-playbook --version): 2.6
  • ceph-ansible version (e.g. git head or tag or stable branch): 3.2.3
  • Ceph version (e.g. ceph -v): Luminous

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 19 (16 by maintainers)

Most upvoted comments

@mcv21 Would you be able to test the ceph-volume fix provided by [1] ? The associated patch for Luminous is [2]. This will be part of the next 12.2.13 release.

[1] https://tracker.ceph.com/issues/37490 [2] https://github.com/ceph/ceph/pull/30094