podman: Podman failed to destroy BTRFS snapshot on container delete

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

bug

Description

ROOTLESS Podman fails to delete BTRFS subvolumes when building an image or deleting a container. This causes a cascade of errors, such as container name re-use errors, as podman believes the container was removed when using podman ps -a however when attempting to re-run the podman run command the user will receive an name re-use error message.

Using SUDO this works just fine.

I would very much assume that this is a configuration issue on my part somewhere, as without privilege elevation using sudo I cannot delete the specified BTRFS by hand using btrfs su delete <path to subvolume> either.

Thanks in advance for your help.

Steps to reproduce the issue:

  1. Build a rootless image using the BTRFS driver.

  2. Get error message listed below.

Describe the results you received:

ERRO[4128] error deleting build container "de8886e87ab7c7e667426e84a096695f4d434fe8ed42149fb157e7b9a398b906": Failed to destroy btrfs snapshot /home/sbrady/.local/share/containers/storage/btrfs/subvolumes for 561da8272542ab2a71977655a51b5d20c20627fd3a917165d1fe89b0370f4f93: operation not permitted 
Error: Failed to destroy btrfs snapshot /home/sbrady/.local/share/containers/storage/btrfs/subvolumes for 561da8272542ab2a71977655a51b5d20c20627fd3a917165d1fe89b0370f4f93: operation not permitted

Describe the results you expected:

BTRFS subvolumes to be deleted on container deletion.

Additional information you deem important (e.g. issue happens only occasionally):

Consistent regardless of the image.

Output of podman version:

podman version 1.5.1

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.12.9
  podman version: 1.5.1
host:
  BuildahVersion: 1.10.1
  Conmon:
    package: conmon-2.0.0-2.1.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.0, commit: unknown'
  Distribution:
    distribution: '"opensuse-tumbleweed"'
    version: "20190904"
  MemFree: 1173884928
  MemTotal: 8254943232
  OCIRuntime:
    package: runc-1.0.0~rc8-1.4.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8
      spec: 1.0.1-dev
  SwapFree: 2141966336
  SwapTotal: 2147483648
  arch: amd64
  cpus: 4
  eventlogger: file
  hostname: rocinante
  kernel: 5.2.11-1-default
  os: linux
  rootless: true
  uptime: 2h 31m 17.3s (Approximately 0.08 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
store:
  ConfigFile: /home/sbrady/.config/containers/storage.conf
  ContainerStore:
    number: 18
  GraphDriverName: btrfs
  GraphOptions: null
  GraphRoot: /home/sbrady/.local/share/containers/storage
  GraphStatus:
    Build Version: 'Btrfs v5.2.1 '
    Library Version: "102"
  ImageStore:
    number: 16
  RunRoot: /var/run/user/1000/containers
  VolumePath: /home/sbrady/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.5.1-1.1.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.): Bare metal install on Intel i7 and spinning rust HDD.

Pastes of Storage.conf and libpod.conf

➜ la ~/.config/containers
total 40K
-rw-r--r-- 1 sbrady users 4.4K Sep  6 14:23 libpod.conf
-rw-r--r-- 1 sbrady users  205 Aug  9 13:30 mounts.conf
drwxr-xr-x 1 sbrady users   14 Aug  9 13:30 oci/
-rw-r--r-- 1 sbrady users  256 Aug  9 13:30 policy.json
-rw-r--r-- 1 sbrady users 1.1K Aug  9 13:30 registries.conf
drwxr-xr-x 1 sbrady users    0 Aug  9 13:30 registries.d/
-rw-r--r-- 1 sbrady users  12K Aug  9 13:30 seccomp.json
-rw-r--r-- 1 sbrady users 5.0K Sep  6 13:21 storage.conf
/home/sbrady/.local/share/containers
├── cache
│   └── blob-info-cache-v1.boltdb
└── storage
    ├── btrfs
    │   └── subvolumes
    ├── btrfs-containers
    │   ├── 1e96b0289a1d5ac651f5c521d325dd2911f73b2a6dffbcbfb2e982310ffbaf05
    │   ├── 30e2a4ea384aa36e4a9d5313a89a47efca248f89a6e134f71f0aa952b536b51c
    │   ├── 30f3976b8ab272bd229a770b0f0e9807ad8b00798178a6732909da3899308935
    │   ├── 3886c1a160034a4c7cae0c59b1a3ee93e0371b837bae4072336b2df16cd2a4cc
    │   ├── 3dd0a9edede5dd4fa3a5333665fcb69f45235e5456bb781863f39e41fe0047b7
    │   ├── 4f8e7486fbc706acaa2675f3cb8afa32a5e2b742810fea560c740ff7ddadfb86
    │   ├── 6e0b92b86a79fb1e8cc6d6c68f7c8e82d9eb3ef96b758c5d3b0a850d5ecdd30a
    │   ├── 7051dd05a2a3c0712b92a3d5277b17beedb221c29c7859f7279e778300cc0239
    │   ├── 812bfe158ec304e077e6d2e05eb7d3f9f01631e3bc7c6fc49c01d205117bba8f
    │   ├── 9e8c3f9cb6b6346f808a89e417d3b365f737cd849ae4e8c41b388f6740640be3
    │   ├── b10003959017ff33e909df830fb7acbd87c081bfcf53f997aba6c7afe5040ded
    │   ├── c20462222981918a27e00267ba8ba0118c1064b9d2728b529c1ff0c9d75cb238
    │   ├── c8bf5970068ed878adadacc10a8e04fd3471f16fa5f95750f501bc9c50cf596b
    │   ├── ca09086e22391e95198dd7aa5abc34f168cd64132488e74042c3aaa7860f162c
    │   ├── containers.json
    │   ├── containers.lock
    │   ├── d2878a44fb5ceed65a97f2a3c98f50cc7ec1f4ba02b672d8708278f9c9d1c2b9
    │   ├── de8886e87ab7c7e667426e84a096695f4d434fe8ed42149fb157e7b9a398b906
    │   ├── e0780ebf9f35bb28407986c191257acdb529d9aa198ca2e5132f06390ac3bf0c
    │   ├── e7315b2231cfaefb7991b403ffb18d60d26e25eb1950a090155b3c3be776ea19
    │   └── fea6040e4ffbb6fcf85570aee5475fb2a03853a540b8aa89be2d945c6af64290
    ├── btrfs-images
    │   ├── 0868e92e943cba2ce2ed3b5705d9dcd4adeec9da9088ab69b3c44af199072b3c
    │   ├── 0ed5811d6d9c68658a20eb354b1917bbf5af162c773eb4edf6168fae00ff09f6
    │   ├── 172b73ede26844c52a903bb4e905f18ecbdc1227a6e32b86eb66b60ce999224f
    │   ├── 18ffcb379eccf2d03f71066d45bf3d9c6078c8dc2de843eb41361f22aa8e8430
    │   ├── 23b52ed766eb03c4151be9e41a0eb2fdce003c910982665b7b92a468bca1c3b3
    │   ├── 337ab92e6b8b755823c3363b11948f13441bcb2811988617793642dd1c5c0ac9
    │   ├── 5e09dd17175ecdff0b478333a6d5f444ea2314e5f9114a5443d7fc9fec86b834
    │   ├── 69c54a0cfa733e6fdb478b5612127feec51153ce066cff4a32f1fdce84bb8af6
    │   ├── 6d038d18f5017765cfcbb2262ae3933429e5be0c64f5d70130de8c788791e1d7
    │   ├── a8c54eebc7056cd3dfccee64c28c12d7653fbe2b1fe61159f2c08fcfb15110b0
    │   ├── b14710b9d573f363bbbad56f0ff69e79b5f229b83daaecaf25d9856a84308df7
    │   ├── b151cdb91db489ee8ab7ba84839dd420164692b3031020c8ded00436421715b6
    │   ├── cac4ae0c405ea55bed5402512d66abadac6ea01c31e650ffd86af31560ad98bc
    │   ├── ced8a8fe165881fdef10a647838751d98e0a3d7aba06316f44bdf28f86e23d25
    │   ├── e6da4025fb017e4e79e7339c5953f4c1aa247c11fe7862920d53ef8879341cfa
    │   ├── fce289e99eb9bca977dae136fbe2a82b6b7d4c372474c9235adc1741675f587e
    │   ├── images.json
    │   └── images.lock
    ├── btrfs-layers
    │   ├── 06b85792928655f9c05298c2104b509396a9ae72fe22f739bb004a2ff87f99d0.tar-split.gz
    │   ├── 30fa407a2912badb23e133597472bec5d233e439a497d96751f5dcfb8617894e.tar-split.gz
    │   ├── 3f194c10a561e3b694e9602b088a4e516eb19ee468992bb5cc6845c717b06b49.tar-split.gz
    │   ├── 43316bbb040759a859d32501f20dd865db10b6ee62659fd09aead1a920097982.tar-split.gz
    │   ├── 46e841fc16afde233e84ec806cb528e3d0ae0c8ead03f074aa7699f67b8f1b4c.tar-split.gz
    │   ├── 51899d997ab4c1f790759deeeaaebb03d3742c34236e5a85499eb302cea6fb7b.tar-split.gz
    │   ├── 8297fa3a5e5f1eb097422189d1f7d9046dfcc658008fe395d8b27e93a5954691.tar-split.gz
    │   ├── 866d3ed87bbbc6beb540cc52cb79f8efaec0f3a8b7cf554f65c5df8fce524ec0.tar-split.gz
    │   ├── 9dae2a8870ddfae25701156bc993448175a4cf5bbc0f537e8563e286a5a37385.tar-split.gz
    │   ├── a17440e364015841337f4b80d2a42cf79936cba443ab941525785a7e35f0c0c6.tar-split.gz
    │   ├── a2f24cc38b696cc363f1270684da6d5ae40c5b086f6f6978462e7c7f551e4341.tar-split.gz
    │   ├── af0b15c8625bb1938f1d7b17081031f649fd14e6b233688eea3c5483994a66a3.tar-split.gz
    │   ├── b6b761c5afcb8b69e34a26df1ce16be5d16b3088c838060dcc2528cc3d4cdd5f.tar-split.gz
    │   ├── d548e5ae588bc66a2250ed8312145394cb9704ecdf50153fe608b93ec3c15a0f.tar-split.gz
    │   ├── layers.json
    │   └── layers.lock
    ├── cache
    │   └── blob-info-cache-v1.boltdb
    ├── libpod
    │   └── bolt_state.db
    ├── mounts
    ├── storage.lock
    ├── tmp
    └── volumes
        ├── brave-storage-volume
        └── test

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 29 (11 by maintainers)

Most upvoted comments

@t-msn - Would you be willing to submit your change (https://github.com/t-msn/storage/commit/41c2a90841cfc42f1373dac0de240773b405f536) as a Pull Request?

I think the second part needs an issue of its own. I know btrfs and overlayfs can co-exist in production environments with tens of thousands of snapshots. I’m not sure what the nature of this lack of support could be about. There are nuances that can be workload specific where one works better than the other, and even where overlayfs copy-up operation can be made more efficient using cloning (Btrfs has had reflinks since forever, and XFS enables them in the most recent xfsprogs at mkfs time).