synology-docker: Updated binaries are incompatible with btrfs

Hi there, Thanks for the great script! I ran into this same issue on a DS1819+ with a brtfs filesystem, where after updating Docker through the script, running docker pull would produce the Failed to create btrfs snapshot: inappropriate ioctl for device. error.

I found that editing /var/packages/Docker/etc/dockerd.json to use the vfs storage device instead of brtfs worked. The Docker documentation suggests that the vfs storage driver, while not performant nor recommended for production use, is robust/stable, so I feel OK using it. In a brief test the overlay2 storage driver didn’t work for me, hence why I used vfs

In my case I also deleted all my existing images before re-pulling everything, since the storage driver has something to do with how the images are stored:

docker system prune
docker system prune -a

Final dockerd.json was as follows:

{
    "data-root" : "/var/packages/Docker/target/docker",
    "log-driver" : "json-file",
    "registry-mirrors" : [],
    "group": "administrators",
    "storage-driver" : "vfs"
}

After all that, the updated non-DSM Docker seems to be working well for me after re-pulling and re-configuring my images. That being said, the GUI for Docker in DSM has a few bugs: the days since reboot for each container is some very large value, and the logs are not viewable (presumably from json-file vs db).

That’s unfortunate. My NAS still uses EXT4, so I’m not able to reproduce your error. However, after some research, possibly changing the storage driver might help? I used Docker’s guide as a reference. Following step 6 of the guide, add "storage-driver": "btrfs" to /var/packages/Docker/etc/dockerd.json. Make a backup of the file first to revert back to the previous version in case of any errors. The full configuration could look something like this:

{
    "storage-driver": "btrfs",
    "data-root" : "/var/packages/Docker/target/docker",
    "log-driver" : "json-file",
    "registry-mirrors" : [],
    "group": "administrators"
}

You will have to restart the Docker service to pick up the changes. Revert back to the old version of dockerd.json if starting of the service fails.

synoservicectl --stop pkgctl-Docker
synoservicectl --start pkgctl-Docker

I’ve found the code responsible for this. In the Synology kernel, linux-4.4.x.tar.gz, the following BTRFS code is patched, and throws ENOTTY (= “Inappropriate ioctl for this device”).

I’m not familiar with BTRFS or kernel internals, but combined with patches in ./include/uapi/linux/btrfs.h, ./usr/include/linux/btrfs.h, the rest of ./fs/btrfs/, and /usr/local/include/synobtrfssnapusage/ in the toolkit image, my impression is that Synology patches btrfs in several ways, but particularly to restrict or modify btrfs snapshots, which are very heavily used by both Synology and Docker.

./fs/btrfs/ioctl.c

long btrfs_ioctl(struct file *file, unsigned int
		cmd, unsigned long arg)
{
	struct btrfs_root *root = BTRFS_I(file_inode(file))->root;
	void __user *argp = (void __user *)arg;

	switch (cmd) {
	case FS_IOC_GETFLAGS:
		return btrfs_ioctl_getflags(file, argp);
	case FS_IOC_SETFLAGS:
		return btrfs_ioctl_setflags(file, argp);
	case FS_IOC_GETVERSION:
		return btrfs_ioctl_getversion(file, argp);
	case FITRIM:
		return btrfs_ioctl_fitrim(file, argp);
	case BTRFS_IOC_SNAP_CREATE:
		return btrfs_ioctl_snap_create(file, argp, 0);
	case BTRFS_IOC_SNAP_CREATE_V2:
		return btrfs_ioctl_snap_create_v2(file, argp, 0);
	case BTRFS_IOC_SUBVOL_CREATE:
		return btrfs_ioctl_snap_create(file, argp, 1);
	case BTRFS_IOC_SUBVOL_CREATE_V2:
		return btrfs_ioctl_snap_create_v2(file, argp, 1);
	case BTRFS_IOC_SNAP_DESTROY:
		return btrfs_ioctl_snap_destroy(file, argp);
	case BTRFS_IOC_SUBVOL_GETFLAGS:
		return btrfs_ioctl_subvol_getflags(file, argp);
	case BTRFS_IOC_SUBVOL_SETFLAGS:
		return btrfs_ioctl_subvol_setflags(file, argp);
#ifdef MY_ABC_HERE
	case BTRFS_IOC_SUBVOL_GETINFO:
		return btrfs_ioctl_subvol_getinfo(file, argp);
#endif
	case BTRFS_IOC_DEFAULT_SUBVOL:
		return btrfs_ioctl_default_subvol(file, argp);
	case BTRFS_IOC_DEFRAG:
		return btrfs_ioctl_defrag(file, NULL);
	case BTRFS_IOC_DEFRAG_RANGE:
		return btrfs_ioctl_defrag(file, argp);
	case BTRFS_IOC_RESIZE:
		return btrfs_ioctl_resize(file, argp);
	case BTRFS_IOC_ADD_DEV:
		return btrfs_ioctl_add_dev(root, argp);
	case BTRFS_IOC_RM_DEV:
		return btrfs_ioctl_rm_dev(file, argp);
	case BTRFS_IOC_FS_INFO:
		return btrfs_ioctl_fs_info(root, argp);
	case BTRFS_IOC_DEV_INFO:
		return btrfs_ioctl_dev_info(root, argp);
	case BTRFS_IOC_BALANCE:
		return btrfs_ioctl_balance(file, NULL);
	case BTRFS_IOC_TRANS_START:
		return btrfs_ioctl_trans_start(file);
	case BTRFS_IOC_TRANS_END:
		return btrfs_ioctl_trans_end(file);
	case BTRFS_IOC_TREE_SEARCH:
		return btrfs_ioctl_tree_search(file, argp);
	case BTRFS_IOC_TREE_SEARCH_V2:
		return btrfs_ioctl_tree_search_v2(file, argp);
	case BTRFS_IOC_INO_LOOKUP:
		return btrfs_ioctl_ino_lookup(file, argp);
	case BTRFS_IOC_INO_PATHS:
		return btrfs_ioctl_ino_to_path(root, argp);
	case BTRFS_IOC_LOGICAL_INO:
		return btrfs_ioctl_logical_to_ino(root, argp);
	case BTRFS_IOC_SPACE_INFO:
		return btrfs_ioctl_space_info(root, argp);
#ifdef MY_ABC_HERE
	case BTRFS_IOC_SYNC_SYNO:
#endif
	case BTRFS_IOC_SYNC: {
		int ret;

#ifdef MY_ABC_HERE
		if (cmd == BTRFS_IOC_SYNC_SYNO) {
			goto skip_start_delalloc;
		}
#endif
		ret = btrfs_start_delalloc_roots(root->fs_info, 0, -1);
		if (ret)
			return ret;
#ifdef MY_ABC_HERE
skip_start_delalloc:
#endif
		ret = btrfs_sync_fs(file_inode(file)->i_sb, 1);

		wake_up_process(root->fs_info->transaction_kthread);
		return ret;
	}
	case BTRFS_IOC_START_SYNC:
		return btrfs_ioctl_start_sync(root, argp);
	case BTRFS_IOC_WAIT_SYNC:
		return btrfs_ioctl_wait_sync(root, argp);
	case BTRFS_IOC_SCRUB:
		return btrfs_ioctl_scrub(file, argp);
	case BTRFS_IOC_SCRUB_CANCEL:
		return btrfs_ioctl_scrub_cancel(root, argp);
	case BTRFS_IOC_SCRUB_PROGRESS:
		return btrfs_ioctl_scrub_progress(root, argp);
	case BTRFS_IOC_BALANCE_V2:
		return btrfs_ioctl_balance(file, argp);
	case BTRFS_IOC_BALANCE_CTL:
		return btrfs_ioctl_balance_ctl(root, arg);
	case BTRFS_IOC_BALANCE_PROGRESS:
		return btrfs_ioctl_balance_progress(root, argp);
	case BTRFS_IOC_SET_RECEIVED_SUBVOL:
		return btrfs_ioctl_set_received_subvol(file, argp);
#ifdef CONFIG_64BIT
	case BTRFS_IOC_SET_RECEIVED_SUBVOL_32:
		return btrfs_ioctl_set_received_subvol_32(file, argp);
#endif
	case BTRFS_IOC_SEND:
		return btrfs_ioctl_send(file, argp);
	case BTRFS_IOC_GET_DEV_STATS:
		return btrfs_ioctl_get_dev_stats(root, argp);
	case BTRFS_IOC_QUOTA_CTL:
		return btrfs_ioctl_quota_ctl(file, argp);
	case BTRFS_IOC_QGROUP_ASSIGN:
		return btrfs_ioctl_qgroup_assign(file, argp);
	case BTRFS_IOC_QGROUP_CREATE:
		return btrfs_ioctl_qgroup_create(file, argp);
	case BTRFS_IOC_QGROUP_LIMIT:
		return btrfs_ioctl_qgroup_limit(file, argp);
	case BTRFS_IOC_QUOTA_RESCAN:
		return btrfs_ioctl_quota_rescan(file, argp);
	case BTRFS_IOC_QUOTA_RESCAN_STATUS:
		return btrfs_ioctl_quota_rescan_status(file, argp);
	case BTRFS_IOC_QUOTA_RESCAN_WAIT:
		return btrfs_ioctl_quota_rescan_wait(file, argp);
#ifdef MY_ABC_HERE
	case BTRFS_IOC_QGROUP_QUERY:
		return btrfs_ioctl_qgroup_query(file, argp);
#endif
	case BTRFS_IOC_DEV_REPLACE:
		return btrfs_ioctl_dev_replace(root, argp);
	case BTRFS_IOC_GET_FSLABEL:
		return btrfs_ioctl_get_fslabel(file, argp);
	case BTRFS_IOC_SET_FSLABEL:
		return btrfs_ioctl_set_fslabel(file, argp);
	case BTRFS_IOC_FILE_EXTENT_SAME:
		return btrfs_ioctl_file_extent_same(file, argp);
	case BTRFS_IOC_GET_SUPPORTED_FEATURES:
		return btrfs_ioctl_get_supported_features(argp);
	case BTRFS_IOC_GET_FEATURES:
		return btrfs_ioctl_get_features(file, argp);
	case BTRFS_IOC_SET_FEATURES:
		return btrfs_ioctl_set_features(file, argp);
#ifdef MY_ABC_HERE
	case BTRFS_IOC_COMPR_CTL:
		return btrfs_ioctl_compr_ctl(file, argp);
#endif
#ifdef MY_ABC_HERE
	case BTRFS_IOC_SNAPSHOT_SIZE_QUERY:
		return btrfs_ioctl_snapshot_size_query(file, argp);
#endif
#ifdef MY_ABC_HERE
	case BTRFS_IOC_SYNO_CLONE_RANGE_V2:
		return btrfs_ioctl_syno_clone_range_v2(file, argp);
#endif
	}

	return -ENOTTY;
}

#ifdef CONFIG_COMPAT
long btrfs_compat_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
{
	switch (cmd) {
	case FS_IOC32_GETFLAGS:
		cmd = FS_IOC_GETFLAGS;
		break;
	case FS_IOC32_SETFLAGS:
		cmd = FS_IOC_SETFLAGS;
		break;
	case FS_IOC32_GETVERSION:
		cmd = FS_IOC_GETVERSION;
		break;
	default:
		return -ENOIOCTLCMD;
	}

	return btrfs_ioctl(file, cmd, (unsigned long) compat_ptr(arg));
}
#endif

./fs/btrfs/super.c

static long btrfs_control_ioctl(struct file *file, unsigned int cmd,
				unsigned long arg)
{
	struct btrfs_ioctl_vol_args *vol;
	struct btrfs_fs_devices *fs_devices;
	int ret = -ENOTTY;

	if (!capable(CAP_SYS_ADMIN))
		return -EPERM;

	vol = memdup_user((void __user *)arg, sizeof(*vol));
	if (IS_ERR(vol))
		return PTR_ERR(vol);

	switch (cmd) {
	case BTRFS_IOC_SCAN_DEV:
		ret = btrfs_scan_one_device(vol->name, FMODE_READ,
					    &btrfs_fs_type, &fs_devices);
		break;
	case BTRFS_IOC_DEVICES_READY:
		ret = btrfs_scan_one_device(vol->name, FMODE_READ,
					    &btrfs_fs_type, &fs_devices);
		if (ret)
			break;
		ret = !(fs_devices->num_devices == fs_devices->total_devices);
		break;
	case BTRFS_IOC_GET_SUPPORTED_FEATURES:
		ret = btrfs_ioctl_get_supported_features((void __user*)arg);
		break;
	}

	kfree(vol);
	return ret;
}

I have compiled btrfs-progs v4.4 through v4.16 on e2fs v1.43.9, and v4.16.1 through v5.15 on current e2fs. All have the same error.

I am still working on compiling btrfs-progs v4.0 through v4.3.1. They require kernel headers to compile, which is a bit of a pain. (No rule to make target 'kernel-shared/volumes.h', needed by 'btrfs.o'.) TBD.

I suspect that, because v4.0-v4.3.1 of btrfs-progs rely on kernel headers, they thereby import Synology’s btrfs related patches in the kernel, and that is what actually allows them to work. I haven’t yet tried copying a non-Synology-sourced btrfs-progs v4.0 (e.g. compiled against the standard linux 4.4.180 headers) to see if it works on Synology.

If a normal btrfs-progs v4.0 works, then the issue is some sort of change in btrfs-progs between v4.0 and v4.4. If it doesn’t, then the issue is the Synology patches to btrfs. I believe I could get the v4.0 btrfs-progs library compiled against the Synology kernel, but if the issue is Synology patches, I have no idea what versions of btrfs would be compatible even if v4.4+ could be coerced to use them for compilation.

For that matter, btrfs itself requires e2fs-progs, and Synology has patches to that, too. It’s possible that that has some sort of lingering effect, even on non-ext2/3/4 filesystem. No idea.

Testing against standard v4.0 btrfs-progs, and non-Synology-e2fs Synology-kerenel-btrfs compiled v4.0 progs, should give an answer to most of it. If it is actually because btrfs-progs v4.0-v4.3.1 accidentally includes the Synology btrfs kernel patches, rather than because of some intentional change by btrfs-progs, that’ll be a lot harder to test safely.

Impressive article @technorabilia! I was always a bit hesitant to bypass Docker’s package completely and to install Docker from scratch instead. But it’s great to know it can be done. If your suspicion is right, then Synology’s flawed kernel support for btrfs is the core issue here.

I’ll include a link to your article in the Known Issues, as it might help others too. Thanks for sharing your experience!