podman: `overlay`-mode containerization breaks `apt`: `Invalid cross-device link`

This is sort of beating the horse dead, but I have only been working on this in my spare time, and I want to be sure. I got stumped on https://github.com/containers/podman/issues/13432 too…

but anyway now I’ve been able to work out a minimal reproducer the replicates your bug without having to download anything, which shows that the issue really is the many layers:

#!/bin/sh
#
# reproduce https://github.com/containers/podman/issues/13123
# 
# When:
#
# - using the non-fuse-overlayfs overlay storage driver
# - on a large number of container layers
# - in rootless mode
#
# then:
# 
# - rename()ing any directory in the container fails.
#
# Usage: rename-repro.sh [LAYERS]
#   LAYERS is the number of layers to create in the container.
# 

set -eu

LAYERS=${1:-50}

# The bug is in the native overlayfs backend, the one without fuse-overlayfs
export STORAGE_DRIVER=overlay STORAGE_OPTS=""

# verify configured driver is actually active: if you switch the driver you 
# have to erase the entire cache of containers/images/cached config. And sometimes even twice before it will stick?
if podman info --debug 2>&1 | grep -q 'delete libpod local files to resolve'; then
  echo "In order to switch graph drivers, you must erase your existing containers/images/cache."
  podman system reset
  if podman info --debug 2>&1 | grep -q 'delete libpod local files to resolve'; then
    echo "You chose to keep your existing containers. This reproduction script cannot work. Exiting."
    exit 1
  fi

  # sometimes the above *does not work*, so force it a second time over
  GRAPHROOT=$(podman info --format={{".Store.GraphRoot"}})
  rm -f "$GRAPHROOT"/libpod/* # XXX dangerous
  podman system reset -f # XXX dangerous
fi

cat <<EOF |
// rename.c: a direct interface to rename(2) without all that coreutils junk in the way
#include <stdio.h>
#include <errno.h>
int main(int argc, char* argv[]) {
  if(argc != 3) { errno = EINVAL; perror("argv"); return 1; }
  if(rename(argv[1], argv[2]) != 0) { perror("rename"); return 2; }
  return 0;
}
EOF
gcc -static -x c - -o rename

# Make a container with many layers
LAYERS=$(($LAYERS - 3)) # the test container has $LAYERS plus the base layer plus the COPY layer plus the final actual layer
(
echo "FROM alpine"
echo "COPY rename /usr/bin"
for i in `seq $LAYERS`; do
  echo 'RUN dd if=/dev/urandom of=$(mktemp) count=1'
done
) | podman build -f - -t layer .

podman --log-level=debug run -it --rm layer sh -c '
  mkdir A &&
  rename A B
'

I made a username with 3 characters in it (because https://github.com/containers/podman/issues/13432 showed me that 5 wouldn’t work at the moment).

sudo useradd -m tst
sudo -i -u tst

A shorter or longer name will change the precise number of layers where the problem appears. With USER=tst, it happens to appear at layer 50:

[tst@requiem ~]$ sh rename-repro.sh 50
[...]
STEP 49/49: RUN dd if=/dev/urandom of=$(mktemp) count=1
--> Using cache 95d8c74f6fc7a5b76f18c38d19b4e21bfc0d696ebdbe184cc56cb453b709c3cf
COMMIT layer
--> 95d8c74f6fc
Successfully tagged localhost/layer:latest
[...]
DEBU[0000] overlay: mount_data=lowerdir=l/6MRCIQO52TVXNM4HXTAAJR2ODI:l/IIOYYWVMYJPQJGJOICDQVFW2DC:l/WEWLGOFKIGV7GBF7U746D4DENB:l/RCPD3W3QL3A3QDJUX5ZSRYRZN6:l/KSG34SZDDECVWV5C6WWBINYWLM:l/DCBKO747JINKXJM36T2SLFK6RB:l/D2XXK3YKQ4JQSS5WD27JK36UG7:l/C353DSRGEGSYKHD6PSNQ3FLENX:l/YSVEGIGVDMDVNGBOLJTF5ET3SK:l/Y46ZPOYX3OCGOEWRUAVHE5ZX5X:l/X2M67CDXDF2LD5EHVIFOCC26X7:l/MMKDZQ4EGSHKEQEUGFGV5JDIFX:l/CCGJDIIT3BQKCCDC5CWB6QLYH3:l/3BUR3ZNLOGKNVU66VUX2E3CD57:l/57KVSXUB7UXCPCW4RKHFVBD4OQ:l/JCER4YLA4YHDGWRLKTSA422I3V:l/ZL3GS2HYKUVVB37EVPYGQSH423:l/PLVVYDKGCGXT3E4QBKJFMDTBS6:l/CWA3ZLDUBP3CVWVNANQ7I2NUYZ:l/P6FHVR22RGARI24JNVU7RNJ3PS:l/GGQMG7RTTKTNDHEDG4S2VQNMUY:l/XLMBB62DOG66SN5SQKPJH5CXX5:l/KSMDG6UUVBN4THICOUZRJ3IVZ2:l/3TAMAIP4ZWC7QGGLZTRL7JMDNX:l/VOWJFH4NWGEDOH7MAGOS7YMSKJ:l/TEOWIK6WH3UYW27623WLCJZMXB:l/NWIXLODDY2EUOLWDSAF23QS2J6:l/6UX4LIHS7Z5WTW6JLKAUCC4PGL:l/N32FCKQPZ76WBBIYF4Y3UG5TVC:l/XQBTXA4DTO63MP5CAKOIUWX6JK:l/ULM3RZONWX7Z5TEINKKWV3MZD5:l/W6BZA4OFDQWL24FKORN6TAHQAY:l/SCT75IHFFLEY7T6XJSAVDZY5J7:l/QDJTOMBN2T4Q3NK4ACBSSDXBHN:l/62XELSXZS2ASDT3GXAKTU247NL:l/35RPXRWRSXBZANK4QQ6KTNS6NT:l/JGNCNKJK4GJ47IWO5OY4UL6VTU:l/6IFLG5WBVVQERLVOW2GXEWW47N:l/DHZXSMH55QMV2DSEXGY5LMTVLU:l/KWGMTXCTHBZFH3IMAV4Y2IKLDK:l/3W6RRVHVZYHFHAVRHXQCZCS3B5:l/46EQXSIMMZE6K2U3V2NPAT7R2F:l/A3E35CXGESKFTUVPH2MFYUJ6J2:l/NCAIGBJHZ35YTYUEEQ4IE2C7OW:l/522MEICOJZP5XUB3G6YG6TEF5O:l/BSH4DGY2GHU2SXF67JZVCMAID5:l/ZMJE73AU5FB7GBS634AQ4QTN44:l/NMEBUD3IYBIU2LXH5WJWIDWCPS:l/M5U2EV7UM3DYUAW54JBZTW4VMN,upperdir=c52fffd0e9c9c8ae1a0b877da3244db2877d44ce7f6ae531a7bfd70f39599664/diff,workdir=c52fffd0e9c9c8ae1a0b877da3244db2877d44ce7f6ae531a7bfd70f39599664/work 
[...]
rename: Invalid cross-device link

Whereas layer 49 doesn’t have the problem:

[tst@requiem ~]$ sh rename-repro.sh.txt 49
[...]
STEP 48/48: RUN dd if=/dev/urandom of=$(mktemp) count=1
--> Using cache c5d8bdb24f47e1e1a92daf8139f766e7cfd4b62d03758c5d6f2fb131178c2f76
COMMIT layer
--> c5d8bdb24f4
Successfully tagged localhost/layer:latest
[...]
DEBU[0000] overlay: mount_data=,lowerdir=/home/tst/.local/share/containers/storage/overlay/l/IIOYYWVMYJPQJGJOICDQVFW2DC:/home/tst/.local/share/containers/storage/overlay/l/WEWLGOFKIGV7GBF7U746D4DENB:/home/tst/.local/share/containers/storage/overlay/l/RCPD3W3QL3A3QDJUX5ZSRYRZN6:/home/tst/.local/share/containers/storage/overlay/l/KSG34SZDDECVWV5C6WWBINYWLM:/home/tst/.local/share/containers/storage/overlay/l/DCBKO747JINKXJM36T2SLFK6RB:/home/tst/.local/share/containers/storage/overlay/l/D2XXK3YKQ4JQSS5WD27JK36UG7:/home/tst/.local/share/containers/storage/overlay/l/C353DSRGEGSYKHD6PSNQ3FLENX:/home/tst/.local/share/containers/storage/overlay/l/YSVEGIGVDMDVNGBOLJTF5ET3SK:/home/tst/.local/share/containers/storage/overlay/l/Y46ZPOYX3OCGOEWRUAVHE5ZX5X:/home/tst/.local/share/containers/storage/overlay/l/X2M67CDXDF2LD5EHVIFOCC26X7:/home/tst/.local/share/containers/storage/overlay/l/MMKDZQ4EGSHKEQEUGFGV5JDIFX:/home/tst/.local/share/containers/storage/overlay/l/CCGJDIIT3BQKCCDC5CWB6QLYH3:/home/tst/.local/share/containers/storage/overlay/l/3BUR3ZNLOGKNVU66VUX2E3CD57:/home/tst/.local/share/containers/storage/overlay/l/57KVSXUB7UXCPCW4RKHFVBD4OQ:/home/tst/.local/share/containers/storage/overlay/l/JCER4YLA4YHDGWRLKTSA422I3V:/home/tst/.local/share/containers/storage/overlay/l/ZL3GS2HYKUVVB37EVPYGQSH423:/home/tst/.local/share/containers/storage/overlay/l/PLVVYDKGCGXT3E4QBKJFMDTBS6:/home/tst/.local/share/containers/storage/overlay/l/CWA3ZLDUBP3CVWVNANQ7I2NUYZ:/home/tst/.local/share/containers/storage/overlay/l/P6FHVR22RGARI24JNVU7RNJ3PS:/home/tst/.local/share/containers/storage/overlay/l/GGQMG7RTTKTNDHEDG4S2VQNMUY:/home/tst/.local/share/containers/storage/overlay/l/XLMBB62DOG66SN5SQKPJH5CXX5:/home/tst/.local/share/containers/storage/overlay/l/KSMDG6UUVBN4THICOUZRJ3IVZ2:/home/tst/.local/share/containers/storage/overlay/l/3TAMAIP4ZWC7QGGLZTRL7JMDNX:/home/tst/.local/share/containers/storage/overlay/l/VOWJFH4NWGEDOH7MAGOS7YMSKJ:/home/tst/.local/share/containers/storage/overlay/l/TEOWIK6WH3UYW27623WLCJZMXB:/home/tst/.local/share/containers/storage/overlay/l/NWIXLODDY2EUOLWDSAF23QS2J6:/home/tst/.local/share/containers/storage/overlay/l/6UX4LIHS7Z5WTW6JLKAUCC4PGL:/home/tst/.local/share/containers/storage/overlay/l/N32FCKQPZ76WBBIYF4Y3UG5TVC:/home/tst/.local/share/containers/storage/overlay/l/XQBTXA4DTO63MP5CAKOIUWX6JK:/home/tst/.local/share/containers/storage/overlay/l/ULM3RZONWX7Z5TEINKKWV3MZD5:/home/tst/.local/share/containers/storage/overlay/l/W6BZA4OFDQWL24FKORN6TAHQAY:/home/tst/.local/share/containers/storage/overlay/l/SCT75IHFFLEY7T6XJSAVDZY5J7:/home/tst/.local/share/containers/storage/overlay/l/QDJTOMBN2T4Q3NK4ACBSSDXBHN:/home/tst/.local/share/containers/storage/overlay/l/62XELSXZS2ASDT3GXAKTU247NL:/home/tst/.local/share/containers/storage/overlay/l/35RPXRWRSXBZANK4QQ6KTNS6NT:/home/tst/.local/share/containers/storage/overlay/l/JGNCNKJK4GJ47IWO5OY4UL6VTU:/home/tst/.local/share/containers/storage/overlay/l/6IFLG5WBVVQERLVOW2GXEWW47N:/home/tst/.local/share/containers/storage/overlay/l/DHZXSMH55QMV2DSEXGY5LMTVLU:/home/tst/.local/share/containers/storage/overlay/l/KWGMTXCTHBZFH3IMAV4Y2IKLDK:/home/tst/.local/share/containers/storage/overlay/l/3W6RRVHVZYHFHAVRHXQCZCS3B5:/home/tst/.local/share/containers/storage/overlay/l/46EQXSIMMZE6K2U3V2NPAT7R2F:/home/tst/.local/share/containers/storage/overlay/l/A3E35CXGESKFTUVPH2MFYUJ6J2:/home/tst/.local/share/containers/storage/overlay/l/NCAIGBJHZ35YTYUEEQ4IE2C7OW:/home/tst/.local/share/containers/storage/overlay/l/522MEICOJZP5XUB3G6YG6TEF5O:/home/tst/.local/share/containers/storage/overlay/l/BSH4DGY2GHU2SXF67JZVCMAID5:/home/tst/.local/share/containers/storage/overlay/l/ZMJE73AU5FB7GBS634AQ4QTN44:/home/tst/.local/share/containers/storage/overlay/l/NMEBUD3IYBIU2LXH5WJWIDWCPS:/home/tst/.local/share/containers/storage/overlay/l/M5U2EV7UM3DYUAW54JBZTW4VMN,upperdir=/home/tst/.local/share/containers/storage/overlay/f8cfd6d74fb3fac0ab994ca75e0eb43bed27ae5950cbad36c4c34b915ae1dab4/diff,workdir=/home/tst/.local/share/containers/storage/overlay/f8cfd6d74fb3fac0ab994ca75e0eb43bed27ae5950cbad36c4c34b915ae1dab4/work,userxattr,volatile 
[...]

To see that #13375 fixed it, build that version:

(git clone https://github.com/containers/podman/ && cd podman && git checkout 7877b02aacf3e8d3d37f6283c6b8aa81688fd120 && make binaries)

Then test using the fixed version:

[tst@requiem ~]$ (PATH=`pwd`/podman/bin:$PATH; sh rename-repro.sh.txt 50)
[...]
STEP 49/49: RUN dd if=/dev/urandom of=$(mktemp) count=1
--> Using cache 95d8c74f6fc7a5b76f18c38d19b4e21bfc0d696ebdbe184cc56cb453b709c3cf
COMMIT layer
--> 95d8c74f6fc
Successfully tagged localhost/layer:latest
[...]
DEBU[0000] overlay: mount_data=lowerdir=l/6MRCIQO52TVXNM4HXTAAJR2ODI:l/IIOYYWVMYJPQJGJOICDQVFW2DC:l/WEWLGOFKIGV7GBF7U746D4DENB:l/RCPD3W3QL3A3QDJUX5ZSRYRZN6:l/KSG34SZDDECVWV5C6WWBINYWLM:l/DCBKO747JINKXJM36T2SLFK6RB:l/D2XXK3YKQ4JQSS5WD27JK36UG7:l/C353DSRGEGSYKHD6PSNQ3FLENX:l/YSVEGIGVDMDVNGBOLJTF5ET3SK:l/Y46ZPOYX3OCGOEWRUAVHE5ZX5X:l/X2M67CDXDF2LD5EHVIFOCC26X7:l/MMKDZQ4EGSHKEQEUGFGV5JDIFX:l/CCGJDIIT3BQKCCDC5CWB6QLYH3:l/3BUR3ZNLOGKNVU66VUX2E3CD57:l/57KVSXUB7UXCPCW4RKHFVBD4OQ:l/JCER4YLA4YHDGWRLKTSA422I3V:l/ZL3GS2HYKUVVB37EVPYGQSH423:l/PLVVYDKGCGXT3E4QBKJFMDTBS6:l/CWA3ZLDUBP3CVWVNANQ7I2NUYZ:l/P6FHVR22RGARI24JNVU7RNJ3PS:l/GGQMG7RTTKTNDHEDG4S2VQNMUY:l/XLMBB62DOG66SN5SQKPJH5CXX5:l/KSMDG6UUVBN4THICOUZRJ3IVZ2:l/3TAMAIP4ZWC7QGGLZTRL7JMDNX:l/VOWJFH4NWGEDOH7MAGOS7YMSKJ:l/TEOWIK6WH3UYW27623WLCJZMXB:l/NWIXLODDY2EUOLWDSAF23QS2J6:l/6UX4LIHS7Z5WTW6JLKAUCC4PGL:l/N32FCKQPZ76WBBIYF4Y3UG5TVC:l/XQBTXA4DTO63MP5CAKOIUWX6JK:l/ULM3RZONWX7Z5TEINKKWV3MZD5:l/W6BZA4OFDQWL24FKORN6TAHQAY:l/SCT75IHFFLEY7T6XJSAVDZY5J7:l/QDJTOMBN2T4Q3NK4ACBSSDXBHN:l/62XELSXZS2ASDT3GXAKTU247NL:l/35RPXRWRSXBZANK4QQ6KTNS6NT:l/JGNCNKJK4GJ47IWO5OY4UL6VTU:l/6IFLG5WBVVQERLVOW2GXEWW47N:l/DHZXSMH55QMV2DSEXGY5LMTVLU:l/KWGMTXCTHBZFH3IMAV4Y2IKLDK:l/3W6RRVHVZYHFHAVRHXQCZCS3B5:l/46EQXSIMMZE6K2U3V2NPAT7R2F:l/A3E35CXGESKFTUVPH2MFYUJ6J2:l/NCAIGBJHZ35YTYUEEQ4IE2C7OW:l/522MEICOJZP5XUB3G6YG6TEF5O:l/BSH4DGY2GHU2SXF67JZVCMAID5:l/ZMJE73AU5FB7GBS634AQ4QTN44:l/NMEBUD3IYBIU2LXH5WJWIDWCPS:l/M5U2EV7UM3DYUAW54JBZTW4VMN,upperdir=00082b6883d833c20bc737a9b186fdd0fc31feb083dc5be034e049f8cf20675c/diff,workdir=00082b6883d833c20bc737a9b186fdd0fc31feb083dc5be034e049f8cf20675c/work,,userxattr,volatile 
[...]
....

and it finishes without error.

I’m afraid there’s no Dockerfile. It was originally a ubuntu 18.04, which was modified during 1.5 years by installing various packages and executing a podman commit ….

Thanks for this clue! I sat down with this again last night and was able to trace it out. There’s an oversight in storage.drivers.overlay that is only triggered when using

1. the native overlayfs driver
2. in rootless mode
3. on a container image with many, many layers

Here’s how I traced this out.

Symptoms

I took note that my lower_layer created by my repro.sh (https://github.com/containers/podman/issues/13123#issuecomment-1047609910) and @Hi-Angel’s behaved differently: mine can run apt and make and rename folders (though it can’t rename a preexisting folder, one created in a lower overlay layer):

$ podman run --cap-add=SYS_PTRACE --rm -it lower_layer:latest 
root@809f172109a9:/# Z=$(mktemp -d); strace -e rename mv -v $Z $Z.bak
renamed '/tmp/tmp.Xq1VfMzyyq' -> '/tmp/tmp.Xq1VfMzyyq.bak'
+++ exited with 0 +++

but his fails on any apt attempt because (as noted above https://github.com/containers/podman/issues/13123#issuecomment-1028789380) rename() fails – even on folders made in the top layer:

$ podman run --cap-add=SYS_PTRACE --rm -it docker.io/kkharlamov/bugreport-enomem
╭─root@22b8358b18a0  / ‹node-›  ‹›
╰─$ Z=$(mktemp -d); strace -e rename mv -v $Z $Z.bak
rename("/tmp/tmp.YcUhoQVKZC", "/tmp/tmp.YcUhoQVKZC.bak") = -1 EXDEV (Invalid cross-device link)
created directory '/tmp/tmp.YcUhoQVKZC.bak'
removed directory '/tmp/tmp.YcUhoQVKZC'
+++ exited with 0 +++

I went searching for why.

Inspecting

I read up on user_namespaces(7) and I figured out with both containers running I could inspect “under their floorboards” by with by nsentering their kernel namespace:

$ lsns | grep podman
4026532871 user        9  2950 kousu podman
4026532872 mnt         5  2950 kousu podman
$ nsenter -U -m -t 2950 mount | grep "overlay on"   # -U == user, -m == mnt; requesting any others requires sudo because podman hasn't allocated those namespaces
overlay on /home/kousu/.local/share/containers/storage/overlay/d7c83476c8f942bef2f2a09af7299ea477539598d3615efe12b2aed98e04918c/merged type overlay (rw,relatime,lowerdir=/home/kousu/.local/share/containers/storage/overlay/l/PNCQVBDSULJ5RTIBTT5TWSZWB4:/home/kousu/.local/share/containers/storage/overlay/l/DWIB6VA26G7ADZPM2MWNXAER3R:/home/kousu/.local/share/containers/storage/overlay/l/N3UC4P6K6D5X2XB3KRM5C7Z4I2:/home/kousu/.local/share/containers/storage/overlay/l/VIPESUNTHVVSNWQ257T7QFCKIH:/home/kousu/.local/share/containers/storage/overlay/l/EFSU256NSEESIK66WLNKSB2N4A,upperdir=/home/kousu/.local/share/containers/storage/overlay/d7c83476c8f942bef2f2a09af7299ea477539598d3615efe12b2aed98e04918c/diff,workdir=/home/kousu/.local/share/containers/storage/overlay/d7c83476c8f942bef2f2a09af7299ea477539598d3615efe12b2aed98e04918c/work,index=off,metacopy=off,volatile,userxattr)
overlay on /home/kousu/.local/share/containers/storage/overlay/bea2e46a25207d77bf18bdc451d0a411ba0eda8a555f7c1a87d6514af43f94f5/merged type overlay (rw,relatime,lowerdir=l/V5MW4GSYR4LZWGJAEKI6VPQ5LK:l/PEXMFA6AB7IBPVULJDKF3UQHB6:l/E6TX2PDKNWUW5ZHPTQB4Z5S2WL:l/J3DH3KTLVQRVUYY5YCAZTNOCS3:l/WY72SHAFPAK4TEMGMFVMQVFSRN:l/EXBFDZ2H6TCANQXJC236E67UAU:l/XNONCHGOQAITDC2NRI2KDMLB7D:l/3QAHF6EAVPQIRQM2MZ2ICGBHRU:l/JPBLRBC6DBCTUGTGEKEUFD4FC7:l/G2QJVBN2XCDSGSLERY6LQTWP6U:l/WI33PZSLKMMFZ4K5WVPUHKL5B7:l/ABTJNOSJUPZU7QGCR5NRKLUOYG:l/6DCMRXXRSHMASV2UTC7NLDJUFH:l/42325RQYDGTWPOKNVNARUJPBR6:l/JQ7X3WNRAILJTIBASCCZH3YNEM:l/7XABMNHUJ4J2TCDUCACGLCVFZZ:l/ENEJ7W33SYHH7OVSO3NCS4M7CF:l/ORUBRX3UHCRNW25PTDMWYXGTSX:l/JCVLDMLQ5MVWB42VE2RAZDVQ4Q:l/FROMRMZFP7A6BZENO6JGITQXJD:l/IJKU2CGWWALCSF3ZJCNTLF4AQX:l/DQRXSB3GO7IJVFAPPWPPTJ5U53:l/36HVBAUQS3I7IL3YQGNZUEWTYN:l/ZSUTN5USUDHHDI3YPUPJHWSFBR:l/YQFKQE7M67ZGCX4D4TOJVMQO4J:l/2T6EYTLM3NCA7RZYVA67G2EKUA:l/4B53RLAY7572DOOPAPNAOQVVGS:l/GZM3TSIFEWFA2JC37LN5B7RUEF:l/XRUC27XIYZAGJVHYUCE7NQU7IQ:l/VWQF67YF7MWUWYM3HZJCRTY557:l/FJSYW3MGLECQXDLHNE7BWVRAH7:l/3GUNL5VSULGHAIQ7RFTNVU55BT:l/QUEYVQMQSTYOQEUDCBVMXH425Z:l/6X35PRBMCNP5JO4ANKBQEC533N:l/E4VBMJX4SPYO763RSN7S7YY3MA:l/I2EVLTFQQVKMRH6R5MI65BQZHQ:l/FP7SHTGIXEIUVUJLOWBLCBAVA6:l/Y5G5DRGQWU3OMI34RD2HCCD4AY:l/O2FTWOQXZ6XSF2ISLVJOJY56JR:l/FB5APONHPOTVLDMLRAF5JAHY2K:l/ALFC3CRGOYXBPVWPYCR6I3AY6Y:l/3GW7H35CMKQGZV33SO7NANKTAK:l/66FC6XTOTCYW6EKCQOCK52JERS:l/65VWKK22S3SF7ECUBT7NGPS3SG:l/3TD7BNOPW3MGNRFAJXUMPEVAUP:l/FOLA5JDRVJZOQLKA3SCCZ3INXM:l/H4CCKACX44KBFADT3R2565NBI4:l/66MR4A4SB34XLVB4DL5UOHDMK6:l/FP23GP3LTTANVAM4BTBF4PRUD3:l/26BI2XBIWFEWGNKX54W76KCJTE:l/5FDXS7LR7PVC5PR5C75RXA5ECW:l/C5AZGUWPJDVHZFKFEWLHYCIS35,upperdir=bea2e46a25207d77bf18bdc451d0a411ba0eda8a555f7c1a87d6514af43f94f5/diff,workdir=bea2e46a25207d77bf18bdc451d0a411ba0eda8a555f7c1a87d6514af43f94f5/work,index=off,metacopy=off)

What immediately stuck out:

1. I can see both containers’ mounts here, so podman must be sharing a single namespace between all containers.

   Good to know though not directly relevant.

2. I can tell the first line is the smaller container (lower_layer) because it is shorter – it has less layers to mount than docker.io/kkharlamov/bugreport-enomem.

   I also confirmed this by mktemping some files in each container and checking I could see the matching names via nsenter -U -m -t 2950 ls /home/kousu/.local/share/containers/storage/overlay/*/merged/tmp

3. The smaller container was mounted with absolute paths, but the smaller one seems to be using relative paths.

4. The smaller container has volatile,userxattr, but the larger container doesn’t.

This last point confused me, so I used strace -f -o container.trace podman run --rm -it docker.io/kkharlamov/bugreport-enomem to look closer, and found that indeed there was a chdir() that went with mount():

51758 chdir("/home/kousu/.local/share/containers/storage/overlay") = 0
51758 mount("overlay", "dab78adc3bc2642353e6f5716300f99f0b2ee95461248ab3d3e6bc56ae5c24bd/merged", "overlay", 0, "lowerdir=l/V5MW4GSYR4LZWGJAEKI6V"...) = 0

Root Cause

I read in https://www.kernel.org/doc/html/latest/filesystems/overlayfs.html that

The the “-o userxattr” mount option forces overlayfs to use the “user.overlay.” xattr namespace instead of “trusted.overlay.”. This is useful for unprivileged mounting of overlayfs.

I skimmed the rest of those docs and I still don’t really understand it all, except that indeed, overlayfs has to handle rename() intricately, and that rootless mode needs userxattr, so losing it must be the cause.

But I wondered why volatile was also missing. And that was the string I pulled on to unravel the mystery.

Tracing the Code

[kousu@requiem podman]$ git grep volatile
RELEASE_NOTES.md:- Containers created with the `--rm` option now automatically use the `volatile` storage flag when available for their root filesystems, causing them not to write changes to disk as often as they will be removed at completion anyways. This should result in improved performance.

so volatile is added because I passed --rm; but that didn’t seem to have anything to do with the bug. And sometimes in my experience that sort of counterintuitive connection is exactly what you need to highlight the relevant clues. I kept looking through git grep volatile and well, long story short, that word doesn’t show up that often, and I narrowed down where both flags get added:

https://github.com/containers/podman/blob/86a057e6be434159e3e60add5a7a7d649e0b4ad5/vendor/github.com/containers/storage/drivers/overlay/overlay.go#L1471-L1484

and, 👏 to the podman team, just below there’s this comment which precisely fits the symptoms above:

https://github.com/containers/podman/blob/86a057e6be434159e3e60add5a7a7d649e0b4ad5/vendor/github.com/containers/storage/drivers/overlay/overlay.go#L1492-L1495

1. A long mount command – due to many layers
2. Relative paths

and the code that implements that comment overwrites opts

https://github.com/containers/podman/blob/86a057e6be434159e3e60add5a7a7d649e0b4ad5/vendor/github.com/containers/storage/drivers/overlay/overlay.go#L1521-L1529

which fits the final symptom

1. Forgetting about volatile and userxattr

Ta-dah

So, that was the bug. https://github.com/containers/podman/pull/13375 should fix it, and you should be able to use your container without fuse-overlayfs, @Hi-Angel. Thanks for helping me solve it! “modified during 1.5 years by installing various packages and executing a podman commit” gave me exactly the insight I needed.