buildah: Buildah version 1.29.1 broken fuse-overlayfs in gitlab runner

Description

After our buildah image was upgraded using the v1.29.1 version we noticed all our pipelines that were using buildah build started to fail on our gitlab runners. Seems like an issue with the fuse-overlayfs package We’re using the quay.io/buildah/stable:latest image, after downgrading to v1.29.0 buildah worked again.

Steps to reproduce the issue:

Upgrade buildah-stable to v1.29.1
Run buildah build in gitlab runner

Describe the results you received: Buildah build failing to unmount and mount

Command used: $ buildah build -q -f .docker/Dockerfile -t $CI_REGISTRY_IMAGE:$BUILD_TAG -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA . Output: Line 1 time="2023-04-06T08:20:26Z" level=error msg="Unmounting /var/lib/containers/storage/overlay/de00ab76e27e966fa7c0c0b79a5ad1247cdf765946bc05ada5b6d99be3a42be5/merged: invalid argument" Line 2 Error: mounting new container: mounting build container "9c4bcdfd7a1a64ae4bb1d399c930ae9a29e52d240c1405e00bf7638cde951901": creating overlay mount to /var/lib/containers/storage/overlay/de00ab76e27e966fa7c0c0b79a5ad1247cdf765946bc05ada5b6d99be3a42be5/merged, mount_data="lowerdir=/var/lib/containers/storage/overlay/l/X3DKLX3WMGVRBGZJYZ4MSULUEY:/var/lib/containers/storage/overlay/l/VLKLEB5WCVLZFQUO2W3ANMY7ZY:/var/lib/containers/storage/overlay/l/YWHPWE4M7KVTL6WXCXO6LBJW52:/var/lib/containers/storage/overlay/l/RJM72NK2LQWGBJU2CB2BWSTGBT:/var/lib/containers/storage/overlay/l/6TW2TMTSHXLBWS4KEJYUFY73CA:/var/lib/containers/storage/overlay/l/MTOGBS4AKRVUFTNZYIFGJLFZCX:/var/lib/containers/storage/overlay/l/ZZXGS22J43ZTSNW3SH2S2L4WRU,upperdir=/var/lib/containers/storage/overlay/de00ab76e27e966fa7c0c0b79a5ad1247cdf765946bc05ada5b6d99be3a42be5/diff,workdir=/var/lib/containers/storage/overlay/de00ab76e27e966fa7c0c0b79a5ad1247cdf765946bc05ada5b6d99be3a42be5/work,nodev,fsync=0,volatile": invalid argument

Describe the results you expected: A working buildah build command

Screenshot 2023-04-06 at 11 11 07

About this issue

Original URL
State: closed
Created a year ago
Reactions: 9
Comments: 24 (4 by maintainers)

Most upvoted comments

Same issue for me, and I rolled back to v1.29.0 instead of the latest version.

Tiscs on Apr 6, 2023

Oh! hang on a sec…you’re right! That’s a bug in the build script. Those labels are completely wrong when the image uses RPMs. I’ll open an issue on that and get on about fixing it. Thanks for pointing out the mismatch.

cevich on Apr 11, 2023

@cevich some CI questions in here for the quay container images, in case you didn’t see this.

TomSweeneyRedHat on Apr 7, 2023

@rhatdan PTAL Should we roll https://github.com/containers/buildah/pull/4699 back?

TomSweeneyRedHat on Apr 7, 2023

I have a Gitlab runner with kernel version 5.15 and I’m also seeing this issue.

hkrutzer on Apr 7, 2023

@elacheche native overlay is easily supported on rootless setups after kernel 5.13 and above ( its was added in 5.11 but I think there were some bugs in 5.11 ) therefore folks running old kernels have no option but to fallback to use fuse-overlays for rootless builds OTOH for users running newer kernels buildah will automatically use native overlay on rootless setups.

This is also a CI/CD bug.

Indeed CI/CD has a issue if its modifying older tags 😃

flouthoc on Apr 7, 2023

same issue here. I wrote a minimal working example at https://gitlab.com/Blaimi/buildah-bughunt.

Blaimi on Apr 6, 2023