one: Backup "nil:NilClass Error preparing disk files" error
Description
Backups from POWEROFF state can’t be done because of nil:NilClass Error preparing disk files. This error was present before modifications made here.
To Reproduce
Backup options for VM 84:
BACKUP_CONFIG=[
BACKUP_VOLATILE="NO",
FS_FREEZE="AGENT",
INCREMENTAL_BACKUP_ID="-1",
KEEP_LAST="7",
LAST_INCREMENT_ID="-1",
MODE="INCREMENT" ]
System datastore:
DATASTORE 0 INFORMATION
ID : 0
NAME : system
USER : oneadmin
GROUP : oneadmin
CLUSTERS : 0
TYPE : SYSTEM
DS_MAD : -
TM_MAD : shared
BASE PATH : /var/lib/one//datastores/0
DISK_TYPE : FILE
STATE : READY
DATASTORE CAPACITY
TOTAL: : 759G
FREE: : 491.4G
USED: : 267.6G
LIMIT: : -
PERMISSIONS
OWNER : um-
GROUP : u--
OTHER : ---
DATASTORE TEMPLATE
ALLOW_ORPHANS="FORMAT"
DISK_TYPE="FILE"
DS_MIGRATE="YES"
RESTRICTED_DIRS="/"
SAFE_DIRS="/var/tmp"
SHARED="YES"
TM_MAD="shared"
TYPE="SYSTEM_DS"
IMAGES
Images datastore:
DATASTORE 1 INFORMATION
ID : 1
NAME : images
USER : oneadmin
GROUP : oneadmin
CLUSTERS : 0
TYPE : IMAGE
DS_MAD : fs
TM_MAD : shared
BASE PATH : /var/lib/one//datastores/1
DISK_TYPE : FILE
STATE : READY
DATASTORE CAPACITY
TOTAL: : 759G
FREE: : 491.4G
USED: : 267.6G
LIMIT: : -
PERMISSIONS
OWNER : um-
GROUP : u--
OTHER : ---
DATASTORE TEMPLATE
ALLOW_ORPHANS="FORMAT"
CLONE_TARGET="SYSTEM"
CLONE_TARGET_SSH="SYSTEM"
DISK_TYPE="FILE"
DISK_TYPE_SSH="FILE"
DS_MAD="fs"
LN_TARGET="NONE"
LN_TARGET_SSH="SYSTEM"
RESTRICTED_DIRS="/"
SAFE_DIRS="/var/tmp"
TM_MAD="shared"
TM_MAD_SYSTEM="ssh"
TYPE="IMAGE_DS"
IMAGES
0
1
2
3
4
5
6
8
20
26
29
31
35
38
41
42
43
52
53
55
56
57
58
59
61
62
63
64
Gluster “Distributed-Disperse” external storage or local “soft-RAID 5”. It was tested on an OpenNebula multi-node cluster with external storage (Gluster) and single-node cluster with local storage (soft-RAID 5).
Trying to make a backup live with --reset:
onevm backup --reset -d 100 84
The following is a snippet from the /var/log/one/84.log after failed the backup action:
# [...]
# ------------------- # Convert and flatten # ------------------- /var/tmp/one/tm/lib/backup_qcow2.rb -d "0:" -
x /var/lib/one//datastores/0/84/backup/vm.xml -p /var/lib/one//datastores/0/84 Error: undefined method `each' for nil:NilClass Error preparing disk files: un
defined method `each' for nil:NilClass + '[' -d /var/lib/one//datastores/0/84/backup ']' + rm -rf /var/lib/one//datastores/0/66/backup + mkdir -p /var/lib/on
e//datastores/0/84/backup + echo [...]
Same error if --reset option is not specified.
Expected behavior
Backup without the nil:NilClass Error preparing disk files error.
Details
- Affected Component: [Storage]
- Hypervisor: [KVM]
- Version: [6.6.1]
- OS: Ubuntu 22.04
Additional context Add any other context about the problem here.
Progress Status
- Code committed
- Testing - QA
- Documentation (Release notes - resolved issues, compatibility, known issues)
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 23 (10 by maintainers)
Commits related to this issue
- B #6200 #6219: Improve qemu locks * Fix bug with empty bitmaps that may occur on power-off / incremental on the first backup * Add -U (force-share) option to convert operations to prevent locking ... — committed to OpenNebula/one by rsmontero a year ago
- B OpenNebula/one#6200 OpenNebula/one#6199: Resolved issues — committed to OpenNebula/docs by rsmontero a year ago
@rsmontero, under certain conditions, this is a consequences of an heavy load storage backend or a slow one. Doing a pause to let the storage flush and release the I/O, this will add some time to the backup operation, but it’s necessary sometimes. Sadly, the qemu-img bitmap command doesn’t have the –force-share capability. and the only way that i can imagine is doing this sleep, or may a more elaborated one, like here, proposed by @Franco-Sparrow.
Thanks in advance Team
First of all @Franco-Sparrow thanks for the thorough testing and details provided in the issue ❤️ 🙏
Code has been merge to 6.6 and master branches (see commit 78910010d35683864d3a479a2f0c6eb17c95229d) and will be delivered in the next maintenance release.