testcontainers-java: Cannot pull images when logged in to Docker Desktop (Status 500: unauthorized: incorrect username or password)

Test Containers 1.16.3 on Docker Engine 20.10.12 with Docker Desktop using WSL2 Backend on Windows 21H1.

When I am logged in to my account in Docker Desktop, Test Containers can no longer pull images from Docker Hub. Once I log out of my account, Test Containers is able to pull images as expected.

com.github.dockerjava.api.async.ResultCallbackTemplate - Error during callback
com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"message":"Head \"https://registry-1.docker.io/v2/testcontainers/ryuk/manifests/0.3.3\": unauthorized: incorrect username or password"}
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:247)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:269)
	at java.base/java.lang.Thread.run(Thread.java:833)

I think my Docker installation is fine, as I can pull images from the terminal when I am logged in to Docker Desktop. I don’t want to log out every time I use Test Containers. How can I fix this issue?

Logs (on INFO): https://pastebin.com/fyExxi76

Logs (on DEBUG): https://pastebin.com/b4H283fL

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 3
  • Comments: 37 (13 by maintainers)

Commits related to this issue

Most upvoted comments

Based on the idea behind the PR I tried the following workaround with the current release and it works for me on Windows:

docker login index.docker.io

Would anyone affected like to give it a try as well?

That worked for me on M1 Macbook Pro. It added an {"auths":{"index.docker.io":{}}} entry in my ~/.docker/config.json file. Thereafter the workaround persists across, restarts, etc.

Problem still exists with Test Containers 1.16.3 and Docker Desktop with versions: 4.2.0, 4.3.2, 4.6.1 and 4.7.0 Actually, it happens regardless if I’m logged in or not.

12:21:49.321 [main] INFO  org.testcontainers.utility.ImageNameSubstitutor - Image name substitution will be performed by: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor')
12:21:49.368 [main] INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy - Loaded org.testcontainers.dockerclient.NpipeSocketClientProviderStrategy from ~/.testcontainers.properties, will try it first
12:21:49.854 [main] INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with local Npipe socket (npipe:////./pipe/docker_engine)
12:21:49.854 [main] INFO  org.testcontainers.DockerClientFactory - Docker host IP address is localhost
12:21:49.888 [main] INFO  org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 20.10.14
  API Version: 1.41
  Operating System: Docker Desktop
  Total Memory: 51188 MB
12:21:55.569 [docker-java-stream--301690517] ERROR com.github.dockerjava.api.async.ResultCallbackTemplate - Error during callback
com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"message":"Head \"https://registry-1.docker.io/v2/testcontainers/ryuk/manifests/0.3.3\": unauthorized: incorrect username or password"}

Image pulled manually via docker pull testcontainers/ryuk:0.3.3 works just fine. It’s just a workaround though.

The fix that worked for me on an M1 Mac was: remove credsStore:desktop from ~/.docker/config.json, run docker login in terminal

@dermoritz So you say, login with exactly docker login index.docker.io did not help?

I cannot confirm that, the only working mitigation for me is manually pulling the image testcontainers/ryuk:0.3.3

I’m running Docker Desktop 4.7.0 (77141) on Windows 21H2. As a new user, I was completely unable to find any workaround from the selection above. I tried logging out, logging in via the UI, logging in via the command line, logging into registry.hub.docker.com, nothing worked.

I then went to Docker Desktop and clicked “Reset to factory defaults” and it worked perfectly first time (not logged in). I tried deleting the images and it re-download them perfectly.

However, when I then logged in using the command line and removed the images, it failed again. Once I reset Docker Desktop again, everything worked again.

Hope that helps.

@awojcik64 How I perceive the error so far with regards to pulling public images with Testcontainers:

  • Docker Desktop did the login through the GUI: not working ❌
  • Docker login through CLI and username/pw (docker login): working ✅
  • Docker logged out: working ✅

For further triangulation, may I ask you @SiwyDym if you are sure that on macOS, you recently did a login through the Docker Desktop GUI and it still worked? Also, please make sure the images are not cache before trying out Testcontainers.

So as a current workaround, I’d suggest doing the docker login from CLI instead of the GUI and everything should work fine.

I have exactly the same issue with Test Container 1.16.2. This issue only occurs if the Docker image has to be pulled. As long as I have a local image everything is fine.

I’m running into this problem in our CI environment.

We’re running testcontainers via maven and this is the command line we use to run maven: mvn verify -DskipUnitTests -Dregistry.url=${CI_REGISTRY} -Dregistry.username=${CI_REGISTRY_USER} -Dregistry.password=${CI_REGISTRY_PASSWORD}

We need to use a private registry to get the containers that are being tested, but for some reason, this seems to break testcontainers ability to fetch public containers:

119153 [main] WARN  🐳 [testcontainers/ryuk:0.3.4]  - Retrying pull for image: testcontainers/ryuk:0.3.4 (2s remaining)
124727 [docker-java-stream--909396508] ERROR com.github.dockerjava.api.async.ResultCallbackTemplate  - Error during callback
com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"message":"Get https://registry-1.docker.io/v2/testcontainers/ryuk/manifests/0.3.4: unauthorized: incorrect username or password"}
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:247)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:269)
	at java.base/java.lang.Thread.run(Thread.java:833)

Since the container that we’re running the Maven tests from within doesn’t have the docker executable, I can’t run docker login in our CI scripts.

Does anybody have anything I could try out to get this working?

This is the exact situation I have as well. Has anyone found a workaround or solution?

Would someone affected by this issue like to try out this PR from Jitpack? #5263

https://www.testcontainers.org/jitpack_dependencies/

This worked for me running the test from the Micronaut Data-JDBC guide. I used the following dependencies from jitpack:

    testImplementation "com.github.testcontainers.testcontainers-java:testcontainers:9baedef"
    testImplementation "com.github.testcontainers.testcontainers-java:mysql:9baedef"
    testImplementation "com.github.testcontainers.testcontainers-java:junit-jupiter:9baedef"

I am also having this exact issue. I am setting up a new development environment on a M1 Max MacBook Pro, with latest Docker (darwin/arm64). I have tried all the suggestions in the previous comments; i.e. logging out of docker, logging in via CLI (with and without registry.hub.docker.com), reset Docker to factory defaults, delete .docker/config.json.

The only thing that ever works for me is to manually pull the image testcontainers/ryuk:0.3.3

Docker Desktop 4.7.0 (77141) Docker version 20.10.14, build a224086 macOS 12.3.1 (21E258)

Thanks a lot for sharing your experiences @jodastephen, this really helps in triaging this issue. I am currently in the process of debugging and triaging as well.

What seems to help is to, in addition to performing a docker logout, also remove the Docker config file:

rm USER_HOME\.docker\config.json

I currently suspect this is caused by changes to the Docker Hub registry, but the permutation in which this is currently failing are unfortunately not 100% clear to me yet.

@SiwyDym I see. Can you instead confirm that it works if your Docker daemon is unautheticated? (after docker logout)

@angelomelonas We never tested TC with Act, but essentially, this will lead to Docker-in-Docker scenario, so the issue you are facing here, is very likely independent from the issue that is discussed here. I would suggest opening a dedicated issue or discussion for using TC with Act and Docker authentication.

I have exactly the same issue with Test Container 1.16.2. This issue only occurs if the Docker image has to be pulled. As long as I have a local image everything is fine.

sorry this isnt a valid solution, somebody must run this in a pipeline for example, the image must be pulled

@StFS Can you please ask about this in Slack, or open a dedicated discussion/issue? This issue is about Docker Desktop.

@kiview I just tried the command line fix on macOS 12.3.1 (M1 Pro) running Docker Desktop 4.8.1 and confirmed it works here too.

Thank you!

Edit: This was also using Testcontainers 1.16.2

Another interesting aspect to try out is if doing a docker login registry.hub.docker.com mitigates the situation.

I can confirm that issue occurs when I login by Docker Desktop GUI 😐 Docker Desktop: 4.7.0 (77141) OS: 12.1 (21C52)

I don’t know how I did that, but now containers are pulled correctly even if I am logged out of everything. I was only juggling logging in using GUI and command line. (tested on Docker Desktop 4.7.0)