testcontainers-java: Codeship builds fail: `can not connect to Ryuk`

I’ve created a pull request that should fix the issue. See #843.

Reopened as requested, bug seems to be still present.

@guss See the docs on custom properties.

As a side note for @kiview and @bsideup, it would be nice to have ryuk.container.timeout (or any other missing properties) added to that page for a complete reference.

Thanks for the ping - I forgot to follow up on this.

We were still experiencing the problem with 1.9.0-rc1, and have set ryuk.container.timeout=120 property since. This appears to have fixed it, but we’ve also had some updated on this infrastructure side and have upgrade to 1.9.1 since.

I’ll try to remove ryuk.container.timeout=120 some time this week and see how it behaves, just to try and isolate the problem.

We’re experiencing the same when the load on the build server (Jenkins) is (very) high.

One thing that eased the pain was upgrading docker where we used to use an early version of 2017. The docker service wasn’t responsive during peak testing (eight parallel test runs) which lead me to that direction.

Still, we had to reduce the number of parallel builds because of testcontainer timeouts. I haven’t tried to increase ryuk.container.timeout because IMO thirty seconds should be enough - also in a high load scenario.

Sorry about the weird link, Github is sometimes too enthusiastic about referencing issues.

I’ve tried with setting the timeout to 120, and indeed the build takes 2 more minutes but the Ryuk connection still fails:

[DEBUG] PullResponseItem[stream=<null>,status=Status: Downloaded newer image for quay.io/testcontainers/ryuk:0.2.2,progressDetail=<null>,progress=<null>,id=<null>,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] Looking up auth config for image: quay.io/testcontainers/ryuk:0.2.2
[DEBUG] RegistryAuthLocator has configFile: /home/circleci/.docker/config.json (exists) and commandPathPrefix: 
[DEBUG] registryName [quay.io] for dockerImageName [quay.io/testcontainers/ryuk:0.2.2]
[DEBUG] no matching Auth Configs - falling back to defaultAuthConfig [null]
[DEBUG] Effective auth config [null]
[DEBUG] Cmd: com.github.dockerjava.core.command.CreateContainerCmdImpl@6ba0ee4a[name=testcontainers-ryuk-0124f49d-4059-439a-93b6-cb8f82b9ac4e,hostName=<null>,domainName=<null>,user=<null>,attachStdin=<null>,attachStdout=<null>,attachStderr=<null>,portSpecs=<null>,tty=<null>,stdinOpen=<null>,stdInOnce=<null>,env=<null>,cmd=<null>,entrypoint=<null>,image=quay.io/testcontainers/ryuk:0.2.2,volumes=com.github.dockerjava.api.model.Volumes@27db45f,workingDir=<null>,macAddress=<null>,networkDisabled=<null>,exposedPorts=com.github.dockerjava.api.model.ExposedPorts@6ec3d8e4,stopSignal=<null>,hostConfig=com.github.dockerjava.api.model.HostConfig@1aeff8ca[binds=com.github.dockerjava.api.model.Binds@6068ebb2,blkioWeight=<null>,blkioWeightDevice=<null>,blkioDeviceReadBps=<null>,blkioDeviceReadIOps=<null>,blkioDeviceWriteBps=<null>,blkioDeviceWriteIOps=<null>,memorySwappiness=<null>,capAdd=<null>,capDrop=<null>,containerIDFile=<null>,cpuPeriod=<null>,cpuShares=<null>,cpuQuota=<null>,cpusetCpus=<null>,cpusetMems=<null>,devices=<null>,diskQuota=<null>,dns=<null>,dnsSearch=<null>,extraHosts=<null>,links=<null>,logConfig=<null>,lxcConf=<null>,memory=<null>,memorySwap=<null>,memoryReservation=<null>,kernelMemory=<null>,networkMode=<null>,oomKillDisable=<null>,autoRemove=true,oomScoreAdj=<null>,portBindings=<null>,privileged=<null>,publishAllPorts=true,readonlyRootfs=<null>,restartPolicy=<null>,ulimits=<null>,volumesFrom=<null>,pidMode=<null>,securityOpts=<null>,cgroupParent=<null>,volumeDriver=<null>,shmSize=<null>,pidsLimit=<null>,runtime=<null>,tmpFs=<null>],labels={org.testcontainers=true},networkingConfig=<null>,ipv4Address=<null>,ipv6Address=<null>,aliases=<null>,authConfig=<null>,execution=com.github.dockerjava.core.exec.CreateContainerCmdExec@5a466dd]
[DEBUG] Cmd: ee8c93190996a894ee8eb46d2aa2435b81b6c902cec44fdc831c019453cbd73a,com.github.dockerjava.core.exec.StartContainerCmdExec@4aac81ca
[DEBUG] Cmd: ee8c93190996a894ee8eb46d2aa2435b81b6c902cec44fdc831c019453cbd73a,false,com.github.dockerjava.core.exec.InspectContainerCmdExec@269c7104
[DEBUG] GET: OkHttpWebTarget(okHttpClient=org.testcontainers.shaded.okhttp3.OkHttpClient@6de84336, baseUrl=https://35.185.49.118:2376/, path=[/containers/ee8c93190996a894ee8eb46d2aa2435b81b6c902cec44fdc831c019453cbd73a/json], queryParams={})
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:23 min
[INFO] Finished at: 2018-10-30T11:38:48Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.jooq:jooq-codegen-maven:3.11.5:generate (default) on project model: Error running jOOQ code generation tool: Can not connect to Ryuk -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.jooq:jooq-codegen-maven:3.11.5:generate (default) on project model: Error running jOOQ code generation tool
...
Caused by: java.lang.IllegalStateException: Can not connect to Ryuk
    at org.testcontainers.utility.ResourceReaper.start (ResourceReaper.java:148)
    at org.testcontainers.utility.ResourceReaper.start (ResourceReaper.java:65)
    at org.testcontainers.DockerClientFactory.client (DockerClientFactory.java:124)
    at org.testcontainers.containers.GenericContainer.<init> (GenericContainer.java:162)
    at org.testcontainers.containers.JdbcDatabaseContainer.<init> (JdbcDatabaseContainer.java:40)
    at org.testcontainers.containers.MariaDBContainer.<init> (MariaDBContainer.java:25)
...

I’m having the same issue with version 1.8.1 Codeship pro, add_docker=true, ryuk.container.timeout = 120