earthly: bug: docker authentication expires after a few hours causing 401 errors while building

I have an Earthfile that extends a private image. After a few hours of using earthly, I start getting 401 errors. I’m still able to pull the image with docker pull, and running docker login doesn’t have any effect.

I am able to get it working again by restarting the earthly container manually, or by changing the arguments I run with.

$ earthly -i +test
     buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)
     quay.io/<org>/<image>:<tag> | --> Load metadata linux/amd64
     quay.io/<org>/<image>:<tag> | WARN: (Load metadata linux/amd64) unexpected status code [manifests ci]: 401 UNAUTHORIZED
Error: unexpected status code [manifests ci]: 401 UNAUTHORIZED

$ docker login quay.io                                              
Authenticating with existing credentials...
Login Succeeded

$ docker pull quay.io/<org>/<image>:<tag>
<tag>: Pulling from <org>/<image>
Digest: sha256:fcacc41495f6a7655032299d0ea83de0a9e577b64eaaaab8e4bf53a8f81bddb3
Status: Image is up to date for quay.io/<org>/<image>:<tag>
quay.io/<org>/<image>:<tag>

$ earthly -i +test
     buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)
     quay.io/<org>/<image>:<tag> | --> Load metadata linux/amd64
     quay.io/<org>/<image>:<tag> | WARN: (Load metadata linux/amd64) unexpected status code [manifests ci]: 401 UNAUTHORIZED
Error: unexpected status code [manifests ci]: 401 UNAUTHORIZED

$ earthly +test
                   buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)
                   buildkitd | Settings do not match. Restarting buildkit daemon with updated settings...
                   buildkitd | ...Done
 quay.io/<org>/<image>:<tag> | --> Load metadata linux/amd64
                       +base | --> FROM quay.io/<org>/<image>:<tag>
                     context | --> local context .
                       +base | [██████████] resolve quay.io/<org>/<image>:<tag>@sha256:fcacc41495f6a7655032299d0ea83de0a9e577b64eaaaab8e4bf53a8f81bddb3 ... 100%
                     context | [██████████] transferring .: ... 100%

This documentation suggests that Earthly should be able to inherited from the host. This page doesn’t contain anything special for quay.

Is it possible I’m missing something? Or is this unintended behavior?

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 17 (1 by maintainers)

Most upvoted comments

Finally upstream has updated. I’ll be sure to ping back once we pull in upstream buildkit with this change… and this long adventure will be over.

dchw on Apr 26, 2021

With the release (just now) of https://github.com/earthly/earthly/releases/tag/v0.5.11, yes.

dchw on May 11, 2021

Interesting. I let my setup mellow overnight, and it still seemed to work this morning. I was; however building with --no-cache each time to ensure it went through everything. I’ll let it sit more this morning and try without.

I’ve also been watching the auth hash: cat /home/dchw/.docker/config.json | jq -r .auths.\"quay.io\".auth | base64 -d, to see if it updates at all over this span, and it has not. Narrowing it down…

dchw on Mar 25, 2021