ceph-csi: RBD Async: Failed to mirrored Cloned PVC created from snapshot (PVC from snapshot)

Failed to mirror PVC created from a snapshot

Steps to Reproduce

Create a PVC Create a snapshot of PVC Create a PVC from snapshot Create VolumeReplication to Enable Replication

74-00f8-11ec-89fe-0242ac110003 GRPC call: /replication.Controller/EnableVolumeReplication
I0819 14:18:34.837531       1 utils.go:178] ID: 2391 Req-ID: 0001-0009-rook-ceph-0000000000000005-33936174-00f8-11ec-89fe-0242ac110003 GRPC request: {"parameters":{"mirroringMode":"snapshot"},"secrets":"***stripped***","volume_id":"0001-0009-rook-ceph-0000000000000005-33936174-00f8-11ec-89fe-0242ac110003"}
I0819 14:18:34.856837       1 omap.go:86] ID: 2391 Req-ID: 0001-0009-rook-ceph-0000000000000005-33936174-00f8-11ec-89fe-0242ac110003 got omap values: (pool="replicapool-4", namespace="", name="csi.volume.33936174-00f8-11ec-89fe-0242ac110003"): map[csi.imageid:1972a93c67485 csi.imagename:csi-vol-33936174-00f8-11ec-89fe-0242ac110003 csi.volname:pvc-0bad8f8d-e00c-4d98-ab90-af8e5bac65be csi.volume.owner:default]
E0819 14:18:35.326354       1 replicationcontrollerserver.go:245] ID: 2391 Req-ID: 0001-0009-rook-ceph-0000000000000005-33936174-00f8-11ec-89fe-0242ac110003 failed to enable mirroring on "replicapool-4/csi-vol-33936174-00f8-11ec-89fe-0242ac110003" with error: rbd: ret=-22, Invalid argument
E0819 14:18:35.326454       1 utils.go:185] ID: 2391 Req-ID: 0001-0009-rook-ceph-0000000000000005-33936174-00f8-11ec-89fe-0242ac110003 GRPC error: rpc error: code = Internal desc = failed to enable mirroring on "replicapool-4/csi-vol-33936174-00f8-11ec-89fe-0242ac110003" with error: rbd: ret=-22, Invalid argument

rbd mirror image enable replicapool-4/csi-vol-33936174-00f8-11ec-89fe-0242ac110003 snapshot
2021-08-19T14:21:17.992+0000 7ffa219492c0 -1 librbd::api::Mirror: image_enable: mirroring is not enabled for the parent

sh-4.4# rbd info replicapool-4/csi-vol-33936174-00f8-11ec-89fe-0242ac110003
rbd image 'csi-vol-33936174-00f8-11ec-89fe-0242ac110003':
	size 1 GiB in 256 objects
	order 22 (4 MiB objects)
	snapshot_count: 0
	id: 1972a93c67485
	block_name_prefix: rbd_data.1972a93c67485
	format: 2
	features: layering, operations
	op_features: clone-child
	flags: 
	create_timestamp: Thu Aug 19 14:17:35 2021
	access_timestamp: Thu Aug 19 14:17:35 2021
	modify_timestamp: Thu Aug 19 14:17:35 2021
	parent: replicapool-4/csi-snap-13fc60cc-00f8-11ec-89fe-0242ac110003@csi-snap-13fc60cc-00f8-11ec-89fe-0242ac110003
	overlap: 1 GiB

Note:- if we flatten the image before mirroring than the image can be mirrored

About this issue

Original URL
State: open
Created 3 years ago
Comments: 17 (5 by maintainers)

Most upvoted comments

Therefore, it seems that RBD mirroring for the cloned image fails because ceph-csi deletes the temporary snapshot.

i don’t think deleting the snapshot is the problem but deleting the clone image created for volume snapshot is the problem.

@ushitora-anqou Currently it’s a limitation with cephcsi but we have a plan to work on it in a few releases. @Rakshith-R can you please add more details?

I am yet to do testing on this, but I’ll add more details soon.

Rakshith-R on Nov 7, 2023

Thank you for your answer. I understood.

Q3. Does this succeed if VolumeReplication is also created for the parent image? yes but it depends if the snapshot is not deleted and and its also replicated we can replicate cloned PVC as well.

IIRC, we also need to enable mirrorings of intermediate clone images (“csi-snap-XXX” or “csi-vol-XXX-temp”).

Yes that correct but its hard to maintain such chain as PVC and Volume Snapshot are independent one, we will try to document such drawbacks , @Rakshith-R is working on this one (we might get the support of it in next 1 or 2 release)

Madhu-1 on Oct 16, 2023

Thank you for your answer. I understood.

Q3. Does this succeed if VolumeReplication is also created for the parent image? yes but it depends if the snapshot is not deleted and and its also replicated we can replicate cloned PVC as well.

IIRC, we also need to enable mirrorings of intermediate clone images (“csi-snap-XXX” or “csi-vol-XXX-temp”).

satoru-takeuchi on Oct 16, 2023