testcontainers-java: Could not connect to Ryuk - Linux, Docker 20.10.0, 1.15.0-rc2

I found out that my issue must be related to the VPN i am connected to. When i turn if off everything works fine. When i am connected i get those connection problems. Will try to figure out whats going on with my networking.

Thanks so far.

Same issue here (1.15.1 and TCP connectivity), resolved by stopping VPN (Cisco AnyConnect) then restarting Docker for Windows then (re)starting VPN. Could not reproduce the issue after that though. Did not try ‘Allow local LAN access’, will do next time.

FYI 1.15.1 is released and fixes “No such image” issue (#3574). Please only comment if you still see the issue with 1.15.1 and it is related to the TCP connectivity, not the image pulling. Thanks.

Experiencing the same out of the blue when executing in EC2, this was still working this morning. The stack trace has this:

Caused by:
    org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=postgres:11.8-alpine, imagePullPolicy=DefaultPullPolicy())
    at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1278)
    at org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:612)
    at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:317)
    ... 114 more
        Caused by:
        com.github.dockerjava.api.exception.NotFoundException: Status 404: {"message":"No such image: testcontainers/ryuk:0.3.0"}

But looking at the Docker Hub that image does exists there.

Encountered the same issue (1.15.2, ubuntu 20.04 LTS with ufw enabled). It turns out that the Ryuk container is started on the default bridge network, while my other containers are started in my custom bridge network. Then the ufw firewall doesn’t allow communication between these private networks, which I could resolve by adding rules. A nicer solution for me would be if I could specify in which network the Ryuk container is started, but I can’t find that anywhere, such that it also starts in my custom bridge network.

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.3.1.RELEASE)

2020-12-15 10:33:24.017  INFO [ocpi-command,,,] 8572 --- [           main] o.t.d.DockerClientProviderStrategy       : Loaded org.testcontainers.dockerclient.NpipeSocketClientProviderStrategy from ~/.testcontainers.properties, will try it first
2020-12-15 10:33:24.587  INFO [ocpi-command,,,] 8572 --- [           main] o.t.d.DockerClientProviderStrategy       : Found Docker environment with local Npipe socket (npipe:////./pipe/docker_engine)
2020-12-15 10:33:24.588  INFO [ocpi-command,,,] 8572 --- [           main] org.testcontainers.DockerClientFactory   : Docker host IP address is localhost
2020-12-15 10:33:24.623  INFO [ocpi-command,,,] 8572 --- [           main] org.testcontainers.DockerClientFactory   : Connected to docker: 
  Server Version: 20.10.0
  API Version: 1.41
  Operating System: Docker Desktop
  Total Memory: 1989 MB
2020-12-15 10:33:24.627  INFO [ocpi-command,,,] 8572 --- [           main] o.t.utility.ImageNameSubstitutor         : Image name substitution will be performed by: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor')
2020-12-15 10:33:24.876  INFO [ocpi-command,,,] 8572 --- [           main] o.t.utility.RegistryAuthLocator          : Credential helper/store (docker-credential-desktop) does not have credentials for index.docker.io
2020-12-15 10:33:28.928  WARN [ocpi-command,,,] 8572 --- [containers-ryuk] o.testcontainers.utility.ResourceReaper  : Can not connect to Ryuk at localhost:55005

java.net.ConnectException: Connection refused: connect
	at java.base/java.net.PlainSocketImpl.connect0(Native Method)
	at java.base/java.net.PlainSocketImpl.socketConnect(PlainSocketImpl.java:101)
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
	at java.base/java.net.Socket.connect(Socket.java:591)
	at java.base/java.net.Socket.connect(Socket.java:540)
	at java.base/java.net.Socket.<init>(Socket.java:436)
	at java.base/java.net.Socket.<init>(Socket.java:213)
	at org.testcontainers.utility.ResourceReaper.lambda$null$1(ResourceReaper.java:160)
	at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
	at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:158)
	at java.base/java.lang.Thread.run(Thread.java:834)

Then it’s the same stacktrace a bunch of more times, like 10 times, and at the end:

2020-12-15 10:33:56.921 ERROR [ocpi-command,,,] 8572 --- [           main] o.testcontainers.utility.ResourceReaper  : Timed out waiting for Ryuk container to start. Ryuk's logs:
2020/12/15 09:33:26 Pinging Docker...
2020/12/15 09:33:26 Docker daemon is available!
2020/12/15 09:33:26 Starting on port 8080...
2020/12/15 09:33:26 Started!

Test fails with application context failed to load.

I’m trying to reproduce this TCP/connection to Ryuk issue, but haven’t managed to so far 😬

Using a fresh VM with Ubuntu 20.04 (LTS) x64 and Docker version 20.10.0, build 7287ab3

I’m testing with 1.15-0rc2, 1.15.0 and master/HEAD and so far not able to reproduce. Will continue to try.

Filed an issue for “No such image: testcontainers/ryuk:0.3.0” (Status 404) at #3574. Sorry for the spam.

Without having debugged the issue, yet, it seems like docker-java doesn’t transparently pull the missing image on status code 404.

It doesn’t, yes.

This should be the default behaviour.

Why? 404 is “not found” and 4xx type of an http code, non-retriable. I don’t think registries should return 404 for images that exist.

Although the original issue report is talking about some other, TCP related issue, so please consider reporting 404 separately, this is some other issue.

Also got com.github.dockerjava.api.exception.NotFoundException: Status 404: {"message":"No such image: testcontainers/ryuk:0.3.0"

Looks like related to docker version: Working version:

13:02:19.695 [main] DEBUG org.testcontainers.utility.TestcontainersConfiguration - Testcontainers configuration overrides will be loaded from file:/root/.testcontainers.properties
13:02:19.718 [main] WARN org.testcontainers.utility.TestcontainersConfiguration - Attempted to read Testcontainers configuration file at file:/root/.testcontainers.properties but the file was not found. Exception message: FileNotFoundException: /root/.testcontainers.properties (No such file or directory)
13:02:19.725 [main] INFO org.testcontainers.utility.ImageNameSubstitutor - Image name substitution will be performed by: DefaultImageNameSubstitutor (ConfigurationFileImageNameSubstitutor)
13:02:19.735 [main] DEBUG org.testcontainers.utility.ImageNameSubstitutor - Did not find a substitute image for rabbitmq:3.8.9 (using image substitutor: DefaultImageNameSubstitutor (ConfigurationFileImageNameSubstitutor))
13:02:19.764 [main] INFO org.testcontainers.dockerclient.DockerMachineClientProviderStrategy - docker-machine executable was not found on PATH ([/usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin, /usr/lib/jvm/java-1.8-openjdk/jre/bin, /usr/lib/jvm/java-1.8-openjdk/bin])
13:02:19.764 [main] DEBUG org.testcontainers.dockerclient.RootlessDockerClientProviderStrategy - $XDG_RUNTIME_DIR is not set.
13:02:19.765 [main] DEBUG org.testcontainers.dockerclient.RootlessDockerClientProviderStrategy - '/root/.docker/run' does not exist.
13:02:19.807 [main] DEBUG org.testcontainers.dockerclient.RootlessDockerClientProviderStrategy - '/run/user/0' does not exist.
13:02:20.330 [ducttape-0] DEBUG org.testcontainers.dockerclient.DockerClientProviderStrategy - Pinging docker daemon...
13:02:20.352 [ducttape-0] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: 
13:02:20.658 [main] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with local Unix socket (unix:///var/run/docker.sock)
13:02:20.659 [main] DEBUG org.testcontainers.dockerclient.DockerClientProviderStrategy - Transport type: 'okhttp', Docker host: 'unix:///var/run/docker.sock'
13:02:20.659 [main] DEBUG org.testcontainers.dockerclient.DockerClientProviderStrategy - Checking Docker OS type for local Unix socket (unix:///var/run/docker.sock)
13:02:20.668 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: bridge
13:02:20.684 [main] INFO org.testcontainers.DockerClientFactory - Docker host IP address is 172.17.0.1
13:02:20.686 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: 
13:02:20.700 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: 
13:02:20.718 [main] INFO org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 19.03.14
  API Version: 1.40
  Operating System: Ubuntu 16.04.4 LTS
  Total Memory: 31907 MB
13:02:20.718 [main] DEBUG org.testcontainers.DockerClientFactory - Ryuk is enabled
13:02:20.721 [main] DEBUG org.testcontainers.utility.ImageNameSubstitutor - Did not find a substitute image for testcontainers/ryuk:0.3.0 (using image substitutor: DefaultImageNameSubstitutor (ConfigurationFileImageNameSubstitutor))
13:02:20.724 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: ListImagesCmdImpl[imageNameFilter=testcontainers/ryuk:0.3.0,showAll=false,filters=org.testcontainers.shaded.com.github.dockerjava.core.util.FiltersBuilder@0]
13:02:20.803 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - Looking up auth config for image: testcontainers/ryuk:0.3.0 at registry: index.docker.io
13:02:20.804 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: /root/.docker/config.json (does not exist) and commandPathPrefix: 
13:02:20.804 [main] INFO org.testcontainers.utility.RegistryAuthLocator - Failure when attempting to lookup auth config. Please ignore if you don't have images in an authenticated registry. Details: (dockerImageName: testcontainers/ryuk:0.3.0, configFile: /root/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /root/.docker/config.json (No such file or directory)
13:02:20.805 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - No matching Auth Configs - falling back to defaultAuthConfig [null]
13:02:20.805 [main] DEBUG org.testcontainers.dockerclient.AuthDelegatingDockerClientConfig - Effective auth config [null]
13:02:21.019 [docker-java-stream--1668592714] INFO org.testcontainers.DockerClientFactory - Starting to pull image
13:02:21.041 [docker-java-stream--1668592714] DEBUG com.github.dockerjava.api.command.PullImageResultCallback - ResponseItem(stream=null, status=Pulling from testcontainers/ryuk, progressDetail=null, progress=null, id=0.3.0, from=null, time=null, errorDetail=null, error=null, aux=null)

Non working version:

19:03:19.418 [main] DEBUG org.testcontainers.utility.TestcontainersConfiguration - Testcontainers configuration overrides will be loaded from file:/root/.testcontainers.properties
19:03:19.441 [main] WARN org.testcontainers.utility.TestcontainersConfiguration - Attempted to read Testcontainers configuration file at file:/root/.testcontainers.properties but the file was not found. Exception message: FileNotFoundException: /root/.testcontainers.properties (No such file or directory)
19:03:19.448 [main] INFO org.testcontainers.utility.ImageNameSubstitutor - Image name substitution will be performed by: DefaultImageNameSubstitutor (ConfigurationFileImageNameSubstitutor)
19:03:19.459 [main] DEBUG org.testcontainers.utility.ImageNameSubstitutor - Did not find a substitute image for rabbitmq:3.8.9 (using image substitutor: DefaultImageNameSubstitutor (ConfigurationFileImageNameSubstitutor))
19:03:19.489 [main] INFO org.testcontainers.dockerclient.DockerMachineClientProviderStrategy - docker-machine executable was not found on PATH ([/usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin, /usr/lib/jvm/java-1.8-openjdk/jre/bin, /usr/lib/jvm/java-1.8-openjdk/bin])
19:03:19.490 [main] DEBUG org.testcontainers.dockerclient.RootlessDockerClientProviderStrategy - $XDG_RUNTIME_DIR is not set.
19:03:19.490 [main] DEBUG org.testcontainers.dockerclient.RootlessDockerClientProviderStrategy - '/root/.docker/run' does not exist.
19:03:19.533 [main] DEBUG org.testcontainers.dockerclient.RootlessDockerClientProviderStrategy - '/run/user/0' does not exist.
19:03:20.117 [ducttape-0] DEBUG org.testcontainers.dockerclient.DockerClientProviderStrategy - Pinging docker daemon...
19:03:20.140 [ducttape-0] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: 
19:03:20.458 [main] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with local Unix socket (unix:///var/run/docker.sock)
19:03:20.458 [main] DEBUG org.testcontainers.dockerclient.DockerClientProviderStrategy - Transport type: 'okhttp', Docker host: 'unix:///var/run/docker.sock'
19:03:20.458 [main] DEBUG org.testcontainers.dockerclient.DockerClientProviderStrategy - Checking Docker OS type for local Unix socket (unix:///var/run/docker.sock)
19:03:20.468 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: bridge
19:03:20.486 [main] INFO org.testcontainers.DockerClientFactory - Docker host IP address is 172.17.0.1
19:03:20.488 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: 
19:03:20.505 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: 
19:03:20.528 [main] INFO org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 20.10.0
  API Version: 1.41
  Operating System: Ubuntu 16.04.4 LTS
  Total Memory: 31907 MB
19:03:20.528 [main] DEBUG org.testcontainers.DockerClientFactory - Ryuk is enabled
19:03:20.531 [main] DEBUG org.testcontainers.utility.ImageNameSubstitutor - Did not find a substitute image for testcontainers/ryuk:0.3.0 (using image substitutor: DefaultImageNameSubstitutor (ConfigurationFileImageNameSubstitutor))
19:03:20.534 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: ListImagesCmdImpl[imageNameFilter=testcontainers/ryuk:0.3.0,showAll=false,filters=org.testcontainers.shaded.com.github.dockerjava.core.util.FiltersBuilder@0]
19:03:20.585 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - Looking up auth config for image: testcontainers/ryuk:0.3.0 at registry: index.docker.io
19:03:20.585 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: /root/.docker/config.json (does not exist) and commandPathPrefix: 
19:03:20.586 [main] INFO org.testcontainers.utility.RegistryAuthLocator - Failure when attempting to lookup auth config. Please ignore if you don't have images in an authenticated registry. Details: (dockerImageName: testcontainers/ryuk:0.3.0, configFile: /root/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /root/.docker/config.json (No such file or directory)
19:03:20.586 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - No matching Auth Configs - falling back to defaultAuthConfig [null]
19:03:20.587 [main] DEBUG org.testcontainers.dockerclient.AuthDelegatingDockerClientConfig - Effective auth config [null]
19:03:20.593 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: org.testcontainers.shaded.com.github.dockerjava.core.command.CreateContainerCmdImpl@4ff8d125[name=testcontainers-ryuk-478be443-d6a0-46f7-93af-512bc9f73bfa,hostName=<null>,domainName=<null>,user=<null>,attachStdin=<null>,attachStdout=<null>,attachStderr=<null>,portSpecs=<null>,tty=<null>,stdinOpen=<null>,stdInOnce=<null>,env=<null>,cmd=<null>,healthcheck=<null>,argsEscaped=<null>,entrypoint=<null>,image=testcontainers/ryuk:0.3.0,volumes=com.github.dockerjava.api.model.Volumes@19c65cdc,workingDir=<null>,macAddress=<null>,onBuild=<null>,networkDisabled=<null>,exposedPorts=com.github.dockerjava.api.model.ExposedPorts@74bf1791,stopSignal=<null>,stopTimeout=<null>,hostConfig=HostConfig(binds=[/var/run/docker.sock:/var/run/docker.sock:rw], blkioWeight=null, blkioWeightDevice=null, blkioDeviceReadBps=null, blkioDeviceWriteBps=null, blkioDeviceReadIOps=null, blkioDeviceWriteIOps=null, memorySwappiness=null, nanoCPUs=null, capAdd=null, capDrop=null, containerIDFile=null, cpuPeriod=null, cpuRealtimePeriod=null, cpuRealtimeRuntime=null, cpuShares=null, cpuQuota=null, cpusetCpus=null, cpusetMems=null, devices=null, deviceCgroupRules=null, deviceRequests=null, diskQuota=null, dns=null, dnsOptions=null, dnsSearch=null, extraHosts=null, groupAdd=null, ipcMode=null, cgroup=null, links=[], logConfig=LogConfig(type=null, config=null), lxcConf=null, memory=null, memorySwap=null, memoryReservation=null, kernelMemory=null, networkMode=null, oomKillDisable=null, init=null, autoRemove=true, oomScoreAdj=null, portBindings=null, privileged=false, publishAllPorts=true, readonlyRootfs=null, restartPolicy=null, ulimits=null, cpuCount=null, cpuPercent=null, ioMaximumIOps=null, ioMaximumBandwidth=null, volumesFrom=null, mounts=null, pidMode=null, isolation=null, securityOpts=null, storageOpt=null, cgroupParent=null, volumeDriver=null, shmSize=null, pidsLimit=null, runtime=null, tmpFs=null, utSMode=null, usernsMode=null, sysctls=null, consoleSize=null),labels={org.testcontainers=true},shell=<null>,networkingConfig=<null>,ipv4Address=<null>,ipv6Address=<null>,aliases=<null>,authConfig=<null>]
19:03:20.695 [main] DEBUG org.testcontainers.utility.ImageNameSubstitutor - Did not find a substitute image for rabbitmq:3.8.9 (using image substitutor: DefaultImageNameSubstitutor (ConfigurationFileImageNameSubstitutor))

My mistake, was too hasty to draw conclusions between the two issues.