kaniko: USER command should not require user or group IDs to exist
It is common for a Dockerfile to contain a USER directive of the form USER 1000:1000. docker build does not require a user and group with those ids to exist, nor does it create them. However, Kaniko errors.
Interestingly, while creating a minimal dockerfile to demonstrate this I discovered that Kaniko errors in two different ways:
Working example using Docker:
$ cat usertest1
FROM alpine
USER 1000:1000
CMD ["whoami"]
$ docker build --file usertest1 -t usertest:1 .
Sending build context to Docker daemon 23.46MB
Step 1/3 : FROM alpine
---> 196d12cf6ab1
Step 2/3 : USER 1000:1000
---> Using cache
---> 4794c572f720
Step 3/3 : CMD ["whoami"]
---> Using cache
---> 6ab640ab2641
Successfully built 6ab640ab2641
Successfully tagged usertest:1
$ docker run --rm usertest:1
whoami: unknown uid 1000
Kaniko failure mode 1
$ cat usertest1
FROM alpine
USER 1000:1000
CMD ["whoami"]
$ docker run --rm -it -v $PWD:/workspace gcr.io/kaniko-project/executor:latest --dockerfile=usertest1 --no-push --context=dir:///workspace
INFO[0000] Downloading base image alpine
2018/12/04 13:25:52 No matching credentials were found, falling back on anonymous
INFO[0001] Taking snapshot of full filesystem...
INFO[0001] Skipping paths under /dev, as it is a whitelisted directory
INFO[0001] Skipping paths under /kaniko, as it is a whitelisted directory
INFO[0001] Skipping paths under /proc, as it is a whitelisted directory
INFO[0001] Skipping paths under /sys, as it is a whitelisted directory
INFO[0001] Skipping paths under /workspace, as it is a whitelisted directory
INFO[0001] USER 1000:1000
INFO[0001] cmd: USER
error building image: error building stage: open /etc/passwd: no such file or directory
Kaniko failure mode 2: Note the addition of the RUN directive and the different error message from Kaniko
$ cat usertest2
FROM alpine
RUN test 0
USER 1000:1000
CMD ["whoami"]
$ docker run --rm -it -v $PWD:/workspace gcr.io/kaniko-project/executor:latest --dockerfile=usertest2 --no-push --context=dir:///workspace
INFO[0000] Downloading base image alpine
2018/12/04 13:32:00 No matching credentials were found, falling back on anonymous
INFO[0001] Unpacking rootfs as cmd RUN test 0 requires it.
INFO[0001] Taking snapshot of full filesystem...
INFO[0001] Skipping paths under /dev, as it is a whitelisted directory
INFO[0001] Skipping paths under /kaniko, as it is a whitelisted directory
INFO[0001] Skipping paths under /proc, as it is a whitelisted directory
INFO[0001] Skipping paths under /sys, as it is a whitelisted directory
INFO[0001] Skipping paths under /workspace, as it is a whitelisted directory
INFO[0001] RUN test 0
INFO[0001] cmd: /bin/sh
INFO[0001] args: [-c test 0]
INFO[0001] Taking snapshot of full filesystem...
INFO[0001] Skipping paths under /dev, as it is a whitelisted directory
INFO[0001] Skipping paths under /kaniko, as it is a whitelisted directory
INFO[0001] Skipping paths under /proc, as it is a whitelisted directory
INFO[0001] Skipping paths under /sys, as it is a whitelisted directory
INFO[0001] Skipping paths under /workspace, as it is a whitelisted directory
INFO[0001] No files were changed, appending empty layer to config. No layer added to image.
INFO[0001] USER 1000:1000
INFO[0001] cmd: USER
error building image: error building stage: user: unknown userid 1000
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 15
- Comments: 30 (3 by maintainers)
based on what this issue is closed? this is critical issue for compatibility concern
It looks like this is also an issue when using
--cache=true: When reaching aUSERstatement on a cached layer that is not available locally, it fails:Happens for us as well and stops us from using Kaniko.
Any update on this?
For me, a USER command in a cached layer will produce the already mentioned error message. Is there some kind of workaround for this?
Anyone been able to work around this?
Using gcr.io/kaniko-project/executor:debug allowed me to workaround this issue
When using executor:latest
When using executor:debug
Suffering from this as well on
gcr.io/kaniko-project/executor:debug-v0.9.0:With the latest version:
I think it’s just this line https://github.com/GoogleContainerTools/kaniko/blob/b0b36ed85a9a85ffb1b2075622be3c4cd7111a82/pkg/util/command_util.go#L309 that calls os.user.LookupId which results in trying to read from /etc/passwd and similarly https://github.com/GoogleContainerTools/kaniko/blob/b0b36ed85a9a85ffb1b2075622be3c4cd7111a82/pkg/util/command_util.go#L309 does this for the group id.
Since all the code does is gets a User object and then returns the UID (which would match the userStr that we looked up in the first place) I feel like the correct behavior would be something like:
user.Lookup, and if it is not found then return an error does that seem like it would give consistent behavior?edit: playing around with this a little bit one issue seems to be that environment variables like HOME won’t be set automatically if you use a numeric userid but that can be worked around if needed.
Nope.
If anybody reading this is interested in digging into this and submitting a fix I’ll be happy to review it.
@hrobertson, i tried reproducing error cases Failure 1 and failure 2. I don’t see this error anymore for gcr.io/kaniko-project/executor:latest
Using
gcr.io/kaniko-project/executor:debugallowed me to get past this issue when running both locally and on Cloud Build. Shasum:5a6b3193506212ea7703214bf3b21cd898e5e2ec14d6b988cc9d11a9b34f32d4same as @mamoit building first without USER & cache enabled and then using USER will trigger this
I have done some further investigation and the behaviour is even more erroneous than I first thought!
Here I build three images containing the kaniko executor binary. One pulls the binary into a scratch base, the next into an alpine base, and the next creates a user within the image.
I build the three separate images:
And here I build the usertest3 image using kaniko inside each of those images.
kaniko binary in scratch
kaniko binary in alpine
As you can see, kaniko is erroneously looking for the /etc/passwd file in the environment in which kaniko is running, not the filesystem of the image it’s building! Note it shouldn’t even look for a passwd file at all when IDs are specified rather than names.
kaniko binary in alpine with an existing user: Note this user does not exist in the image kaniko is building, but in the environment in which kaniko itself is running.
Until the first RUN directive causes
Unpacking rootfs, kaniko is using the filesystem of the environment that Kaniko is in rather than that of the base image. As you can see below, using kaniko in thekaniko:alpine-with-userimage to build theusertest2dockerfile (see the initial post), again fails to find the user. Contrast this with the above output building theusertest3dockerfile.