buildah: Unable to implicitly find registry credentials for pushing an image while running `buildah` in an unprivileged container

Hello!

I was able to successfully run buildah in an unprivileged container in one of my GKE clusters (with some black magic for mounting /dev/fuse inside a pod’s container, but I’ll skip that here). The cluster’s CRI is containerd.

Running buildah from quay.io/buildah/stable image, which is packed with buildah 1.20.1 at the moment. Unconfined seccomp and AppArmor policies. Just a default containerd’s unprivileged mode, no CAP_SYS_ADMIN and all that stuff.

buildah was able to build images successfully, but only as root:root. Running it as 1000:1000 (build user in the official image) was met with uid/guid remapping errors, appearing e.g. as apt’s E: setegid 65534 failed - setegid (22: Invalid argument). No problem, allowPrivilegeEscalation isn’t forced, so basically no difference if I run as root:root in an unprivileged container here.

What puzzled me was that buildah push wasn’t able to find registry creds after successful buildah login. I think logs will explain the situation better:

buildah bud --debug --format=docker --tag "$SOME_IMAGE" .
level=debug msg="running [buildah-in-a-user-namespace --debug bud --format=docker --tag <image tag here> .] with environment...
< blah-blah everything is ok here>

echo "$VERY_SECRET_REGISTRY_PASSWORD" | buildah login --debug --username $REGISTRY_USER --password-stdin $REGISTRY_URI
level=debug msg="Credentials not found"
level=debug msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io"
level=debug msg="Loading registries configuration \"/etc/containers/registries.conf\""
level=debug msg="Loading registries configuration \"/etc/containers/registries.conf.d/000-shortnames.conf\""
level=debug msg="GET https://registry-1.docker.io/v2/"
level=debug msg="Ping https://registry-1.docker.io/v2/ status 401"
level=debug msg="GET https://auth.docker.io/token?account=rcny&service=registry.docker.io"
level=debug msg="GET https://registry-1.docker.io/v2/"
Login Succeeded!

buildah push --debug --format=v2s2 "$SOME_IMAGE"
level=debug msg="running [buildah-in-a-user-namespace push --debug --format=v2s2 <image tag here>] with environment...
< skipping this buildah-in-a-user-namespace output with the list of env vars and their values >
level=debug msg="Destination argument not specified, assuming the same as the source: docker.io/blah/blah-blah:blah"
level=debug msg="[graphdriver] trying provided driver \"overlay\""
level=debug msg="overlay: imagestore=/var/lib/shared"
level=debug msg="overlay: mount_program=/usr/bin/fuse-overlayfs"
level=debug msg="backingFs=overlayfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false"
level=debug msg="Assuming docker:// as the transport method for DESTINATION: docker://docker.io/blah/blah-blah:blah"
level=debug msg="Loading registries configuration \"/etc/containers/registries.conf\""
level=debug msg="Loading registries configuration \"/etc/containers/registries.conf.d/000-shortnames.conf\""
level=debug msg="registry \"docker.io\" is not listed in registries configuration \"/etc/containers/registries.conf\", assuming it's not blocked"
level=debug msg="parsed reference into \"[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]docker.io/blah/blah-blah:blah\""
level=debug msg="pushing image to reference \"docker://blah/blah-blah:blah\" is allowed by policy"
------------>   level=debug msg="Credentials not found"   <------------
level=debug msg="Using registries.d directory /etc/containers/registries.d for sigstore configuration"
level=debug msg=" Using \"default-docker\" configuration"
level=debug msg="  Using file:///var/lib/containers/sigstore"
level=debug msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io"
level=debug msg="Using blob info cache at /var/lib/containers/cache/blob-info-cache-v1.boltdb"
level=debug msg="IsRunningImageAllowed for image containers-storage:[overlay@/var/lib/containers/storage]docker.io/blah/blah-blah:blah@e2cb8da31b83d73c030cf7dd1aef18388c93c07907e6bd070a83eb88a765c4e2"
level=debug msg=" Using transport \"containers-storage\" policy section \"\""
level=debug msg=" Requirement 0: allowed"
level=debug msg="Overall: allowed"
Getting image source signatures
level=debug msg="Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json]"
level=debug msg="... will first try using the original manifest unmodified"
level=debug msg="Checking /v2/blah/blah-blah/blobs/sha256:e02b468c64ed395a3f6d5aa747387a3890a5562d006bfda488bec4456467b129"
level=debug msg="GET https://registry-1.docker.io/v2/"
level=debug msg="Checking /v2/blah/blah-blah/blobs/sha256:b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116"
level=debug msg="Ping https://registry-1.docker.io/v2/ status 401"
level=debug msg="GET https://auth.docker.io/token?scope=repository%3Ablah%2Fblah-blah%3Apull%2Cpush&service=registry.docker.io"
level=debug msg="GET https://auth.docker.io/token?scope=repository%3Ablah%2Fblah-blah%3Apull%2Cpush&service=registry.docker.io"
level=debug msg="HEAD https://registry-1.docker.io/v2/blah/blah-blah/blobs/sha256:e02b468c64ed395a3f6d5aa747387a3890a5562d006bfda488bec4456467b129"
level=debug msg="HEAD https://registry-1.docker.io/v2/blah/blah-blah/blobs/sha256:b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116"
level=debug msg="... not present"
level=debug msg="exporting filesystem layer \"69d6618a6488254628a59f93ffd0b4dd26f5c0ab60820c935a36ab1fb9ce7bc1\" without compression for blob \"sha256:e02b468c64ed395a3f6d5aa747387a3890a5562d006bfda488bec4456467b129\""
Copying blob sha256:e02b468c64ed395a3f6d5aa747387a3890a5562d006bfda488bec4456467b129
level=debug msg="No compression detected"
level=debug msg="Compressing blob on the fly"
level=debug msg="Uploading /v2/blah/blah-blah/blobs/uploads/"
level=debug msg="POST https://registry-1.docker.io/v2/blah/blah-blah/blobs/uploads/"
level=debug msg="... not present"
level=debug msg="Trying to reuse cached location sha256:540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba compressed with gzip in docker.io/library/alpine"
level=debug msg="Checking /v2/library/alpine/blobs/sha256:540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba"
level=debug msg="GET https://auth.docker.io/token?scope=repository%3Ablah%2Fblah-blah%3Apull%2Cpush&scope=repository%3Alibrary%2Falpine%3Apull&service=registry.docker.io"
level=debug msg="Error initiating layer upload, response http.Response{Status:\"401 Unauthorized\", StatusCode:401, Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Content-Length\":[]string{\"236\"}, \"Content-Type\":[]string{\"application/json\"}, \"Date\":[]string{\"Sat, 08 May 2021 03:27:51 GMT\"}, \"Docker-Distribution-Api-Version\":[]string{\"registry/2.0\"}, \"Strict-Transport-Security\":[]string{\"max-age=31536000\"}, \"Www-Authenticate\":[]string{\"Bearer realm=\\\"https://auth.docker.io/token\\\",service=\\\"registry.docker.io\\\",scope=\\\"repository:blah/blah-blah:pull,push\\\",error=\\\"insufficient_scope\\\"\"}}, Body:(*http.bodyEOFSignal)(0xc0003ae7c0), ContentLength:236, TransferEncoding:[]string(nil), Close:true, Uncompressed:false, Trailer:http.Header(nil), Request:(*http.Request)(0xc000328b00), TLS:(*tls.ConnectionState)(0xc0003120b0)}"
level=debug msg="HEAD https://registry-1.docker.io/v2/library/alpine/blobs/sha256:540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba"
level=debug msg="... already exists"
level=debug msg="Trying to mount /v2/blah/blah-blah/blobs/uploads/?from=library%2Falpine&mount=sha256%3A540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba"
level=debug msg="POST https://registry-1.docker.io/v2/blah/blah-blah/blobs/uploads/?from=library%2Falpine&mount=sha256%3A540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba"
level=debug msg="Error mounting, response http.Response{Status:\"401 Unauthorized\", StatusCode:401, Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Content-Length\":[]string{\"309\"}, \"Content-Type\":[]string{\"application/json\"}, \"Date\":[]string{\"Sat, 08 May 2021 03:27:52 GMT\"}, \"Docker-Distribution-Api-Version\":[]string{\"registry/2.0\"}, \"Strict-Transport-Security\":[]string{\"max-age=31536000\"}, \"Www-Authenticate\":[]string{\"Bearer realm=\\\"https://auth.docker.io/token\\\",service=\\\"registry.docker.io\\\",scope=\\\"repository:blah/blah-blah:pull,push repository:library/alpine:pull\\\",error=\\\"insufficient_scope\\\"\"}}, Body:(*http.bodyEOFSignal)(0xc0003af6c0), ContentLength:309, TransferEncoding:[]string(nil), Close:true, Uncompressed:false, Trailer:http.Header(nil), Request:(*http.Request)(0xc0003d0700), TLS:(*tls.ConnectionState)(0xc0000f13f0)}"
level=debug msg="... Mount failed: Error mounting sha256:540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba from docker.io/library/alpine to docker.io/blah/blah-blah: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n"
level=debug msg="exporting filesystem layer \"b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116\" without compression for blob \"sha256:b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116\""
Copying blob sha256:b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116
level=debug msg="No compression detected"
level=debug msg="Compressing blob on the fly"
level=debug msg="Uploading /v2/blah/blah-blah/blobs/uploads/"
level=debug msg="POST https://registry-1.docker.io/v2/blah/blah-blah/blobs/uploads/"
level=debug msg="Error initiating layer upload, response http.Response{Status:\"401 Unauthorized\", StatusCode:401, Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Content-Length\":[]string{\"236\"}, \"Content-Type\":[]string{\"application/json\"}, \"Date\":[]string{\"Sat, 08 May 2021 03:27:52 GMT\"}, \"Docker-Distribution-Api-Version\":[]string{\"registry/2.0\"}, \"Strict-Transport-Security\":[]string{\"max-age=31536000\"}, \"Www-Authenticate\":[]string{\"Bearer realm=\\\"https://auth.docker.io/token\\\",service=\\\"registry.docker.io\\\",scope=\\\"repository:blah/blah-blah:pull,push\\\",error=\\\"insufficient_scope\\\"\"}}, Body:(*http.bodyEOFSignal)(0xc0003ae980), ContentLength:236, TransferEncoding:[]string(nil), Close:true, Uncompressed:false, Trailer:http.Header(nil), Request:(*http.Request)(0xc000328b00), TLS:(*tls.ConnectionState)(0xc000312160)}"
level=debug msg="parsed reference into \"[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]docker.io/blah/blah-blah:blah\""
error copying layers and metadata from "containers-storage:[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]docker.io/blah/blah-blah:blah" to "docker://blah/blah-blah:blah": Error writing blob: Error initiating layer upload to /v2/blah/blah-blah/blobs/uploads/ in registry-1.docker.io: errors:
denied: requested access to the resource is denied
unauthorized: authentication required
level=debug msg="shutting down the store"
level=error msg="exit status 125"

Tested that against quay.io, the result was the same.

Explicitly setting REGISTRY_AUTH_FILE has solved the problem: Credentials not found message from the buildah push output above has changed to Returning credentials from /tmp/registry_auth.json (the path from the environment variable’s value) and everything went ok afterwards. Is this the workaround or the correct solution?