server: Failed to fetch anonymous token when trying to pull Triton Docker image in CI

Description Every few CI runs I see this exact error when I try to pull the Triton Docker image. It causes my CI to fail, which is really frustrating!

#5 [internal] load metadata for nvcr.io/nvidia/tritonserver:21.09-py3
#5 sha256:427d8fa572bc82e08f6dcef7f8d657525a6eca13d4e4fe03fea3258a157c0558
#5 ERROR: failed to authorize: rpc error: code = Unknown desc = failed to fetch anonymous token: unexpected status: 401 Unauthorized
#3 [internal] load metadata for nvcr.io/nvidia/tritonserver:21.09-py3-min
#3 sha256:ab4baa3f48727d25fa6024bc84cf599d16c85f5a907845f4bb3b82991ea5d04b
#3 ERROR: failed to authorize: rpc error: code = Unknown desc = failed to fetch anonymous token: unexpected status: 401 Unauthorized

Triton Information 21.09

Are you using the Triton container or did you build it yourself?

I am trying to build Triton myself, which usually works well up until a few weeks ago when I started seeing this issue.

To Reproduce Try to pull the docker image a few times in a row. My CI runs every 10 minutes or so.

Expected behavior The docker image pull does not give me a 401.

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 2
Comments: 19 (6 by maintainers)

Most upvoted comments

Thank you for the update, Bryan! Appreciate that you were able to find a workaround.

I’ll leave this ticket open to work on until Hiren closes it, as this NGC issue is still impacting users. (CC: @cnegron-nv)

dyastremsky on Jun 2, 2022

Any update on this? I’m having the same issue while building locally with Docker.

...
[100%] Building FIL backend in Docker
docker build -t triton_fil_builder --build-arg BUILD_TYPE=Release --build-arg TRITON_VERSION=21.12 --build-arg TRITON_ENABLE_GPU=OFF --build-arg TRITON_ENABLE_STATS=OFF --build-arg TRITON_COMMON_REPO_TAG=r21.12 --build-arg TRITON_CORE_REPO_TAG=r21.12 --build-arg TRITON_BACKEND_REPO_TAG=r21.12 -f /tmp/tritonbuild/fil/ops/Dockerfile /tmp/tritonbuild/fil
#1 [internal] load build definition from Dockerfile
#1 sha256:903741c3d11eaa42596196898b4c439fa35df96d5e184cbef79273cdf713d95d
#1 transferring dockerfile: 3.43kB done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 sha256:f44d8a4831d856cd6119d80f473da935c07af1032648b9c9c7e2ce3b8f57d7df
#2 transferring context: 103B done
#2 DONE 0.0s

#3 [internal] load metadata for nvcr.io/nvidia/tritonserver:21.12-py3
#3 sha256:775ad3373fdaaaf160dca050597919008aec39667c13f611de897ebecf31b551
#3 ERROR: failed to authorize: rpc error: code = Unknown desc = failed to fetch anonymous token: unexpected status: 401 Unauthorized
------
 > [internal] load metadata for nvcr.io/nvidia/tritonserver:21.12-py3:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: failed to authorize: rpc error: code = Unknown desc = failed to fetch anonymous token: unexpected status: 401 Unauthorized

mattdibi on Feb 9, 2022