cloudstack: KVM problem with volume snapshots kept only on the primary storage (non-managed storages)

ISSUE TYPE

• Bug Report

COMPONENT NAME

volume snapshots 

CLOUDSTACK VERSION

from 4.11.3.0 to main

SUMMARY

When snapshot.backup.to.secondary is set to false the snapshots are kept only on the primary storage. When you take more than one snapshot of a volume and delete this volume the snapshots are still active in the UI/DB but deleted from the storage. The problem with different primary storages is:

• for the current NFS implementation is that the snapshots are internal, and they are deleted when you delete the volume - active in the UI/DB
• for Ceph - the snapshots are deleted from the pool, but active in DB/UI
• probably with the new Linstor driver the problem will be the same
• for StorPool the snapshots are not deleted, but a random record is deleted from the snapshot_store_ref table and the reference to the snapshot’s path is lost
• with this PR - #5297 - the problem will be the same as the StorPool primary storage

STEPS TO REPRODUCE

- set `snapshot.backup.to.secondary` to `false`
- take a few snapshots of a volume
- delete the volume

The snapshots are visible in the UI, active in DB table `snapshots` and one of them is deleted in `snapshot_store_ref` DB table, because of this code in VolumeServiceImpl.deleteVolumeCallback:


                SnapshotDataStoreVO snapStoreVo = _snapshotStoreDao.findByVolume(vo.getId(), DataStoreRole.Primary);

                if (snapStoreVo != null) {
                    long storagePoolId = snapStoreVo.getDataStoreId();
                    StoragePoolVO storagePoolVO = storagePoolDao.findById(storagePoolId);

                    if (storagePoolVO.isManaged()) {
                        DataStore primaryDataStore = dataStoreMgr.getPrimaryDataStore(storagePoolId);
                        Map<String, String> mapCapabilities = primaryDataStore.getDriver().getCapabilities();

                        String value = mapCapabilities.get(DataStoreCapabilities.STORAGE_SYSTEM_SNAPSHOT.toString());
                        Boolean supportsStorageSystemSnapshots = new Boolean(value);

                        if (!supportsStorageSystemSnapshots) {
                            _snapshotStoreDao.remove(snapStoreVo.getId());
                        }
                    } else {
                        _snapshotStoreDao.remove(snapStoreVo.getId());
                    }
                }

EXPECTED RESULTS

Probably for all non-managed primary storages we should not delete the snapshots. For NFS this PR - #5297 will fix the problem with the internal snapshots. 

ACTUAL RESULTS

Snapshots are deleted from the storage, but active in the UI/DB as a garbage