podman: "Error: lock … is not a read-only lock" failure with `additionalimagestores` set

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Possibly related to #7309, if I have additionalstorage set in storage.conf I always get, e.g.:

$ sudo podman image pull --root /space/podman/images helloworld:latest 
Error: lock "/space/podman/images/overlay-images/images.lock" is not a read-only lock

(and the same with --storage-opt=overlay.mountopt=nodev also specified, as suggested in the above issue)

I’ve tried creating images.lock (and layers.lock) as empty files, and also copying over the (~64byte?) files from the default storage location.

If I temporarily edit containers.conf to swap the graphRoot and additionalstorage paths and then re-run the failing command (which is actually:

sudo podman container commit sys-build sys-build:latest

) and remove the --root option then the operation succeeds, but in the original default storage location rather than the original additional storage location.

What is the correct process to follow if I wish to maintain multiple storage locations to differentiate between (effectively) ephemeral images and images which should have longer life-spans?

Output of podman version:

Version:      3.1.0-rc2
API Version:  3.1.0-rc2
Go Version:   go1.16.2
Git Commit:   1b56ea2d9df82cbba2679f646c077881fefb49d6
Built:        Sat Mar 27 00:25:10 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.19.8
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: app-emulation/conmon-2.0.27
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: 65fad4bfcb250df0435ea668017e643e7f462155'
  cpus: 8
  distribution:
    distribution: gentoo
    version: unknown
  eventLogger: file
  hostname: dellr330
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.11.10-gentoo
  linkmode: dynamic
  memFree: 2663763968
  memTotal: 33390899200
  ociRuntime:
    name: crun
    package: app-emulation/crun-0.18
    path: /usr/bin/crun
    version: |-
      crun version 0.18
      commit: 808420efe3dc2b44d6db9f1a3fac8361dde42a95
      spec: 1.0.0
      +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 25769787392
  swapTotal: 25769787392
  uptime: 8h 29m 51.14s (Approximately 0.33 days)
registries:
  localhost:5000:
    Blocked: false
    Insecure: true
    Location: localhost:5000
    MirrorByDigestOnly: false
    Mirrors: []
    Prefix: localhost:5000
  search:
  - docker.io
  - docker.pkg.github.com
  - quay.io
  - public.ecr.aws
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /space/podman/images
    overlay.mountopt: nodev
  graphRoot: /space/podman/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 107
  runRoot: /var/run/podman
  volumePath: /space/podman/volumes
version:
  APIVersion: 3.1.0-rc2
  Built: 1616804710
  BuiltTime: Sat Mar 27 00:25:10 2021
  GitCommit: 1b56ea2d9df82cbba2679f646c077881fefb49d6
  GoVersion: go1.16.2
  OsArch: linux/amd64
  Version: 3.1.0-rc2

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes