testcontainers-java: Problems with docker in docker when exposing host's port to the container
We are basically creating Junit tests that test a Varnish configuration by using a rest client(rest assured) and a fake backend(wiremock) The tests themselves play both roles, as a client and fake backend, and because of that Varnish is run inside a container exposing to it the host’s port, in order to enable varnish -> wiremock communication.
This works perfectly without any problem in Mac but in Linux (where we have our Jenkins) fails with the following stacktrace
19:14:50 [MShield] Running shell script
19:14:50 + docker run --privileged=true -u jenkins -v /var/lib/jenkins/workspace/MShield:/usr/src/mymaven -v /var/lib/jenkins/.m2:/home/jenkins/.m2 -v /var/run/docker.sock:/var/run/docker.sock -e TESTCONTAINERS_RYUK_DISABLED=true -w /usr/src/mymaven marfeeljava:latest mvn test
19:14:52 [INFO] Scanning for projects...
19:14:52 [WARNING]
19:14:52 [WARNING] Some problems were encountered while building the effective model for com.marfeel:MShieldTesting:jar:1.0-SNAPSHOT
19:14:52 [WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing. @ line 21, column 21
19:14:52 [WARNING]
19:14:52 [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
19:14:52 [WARNING]
19:14:52 [WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
19:14:52 [WARNING]
19:14:52 [INFO] ------------------------------------------------------------------------
19:14:52 [INFO] Reactor Build Order:
19:14:52 [INFO]
19:14:52 [INFO] MarfeelMShield
19:14:52 [INFO] MShieldTesting
19:14:52 [INFO]
19:14:52 [INFO] ------------------------------------------------------------------------
19:14:52 [INFO] Building MarfeelMShield 1.0-SNAPSHOT
19:14:52 [INFO] ------------------------------------------------------------------------
19:14:52 [INFO]
19:14:52 [INFO] ------------------------------------------------------------------------
19:14:52 [INFO] Building MShieldTesting 1.0-SNAPSHOT
19:14:52 [INFO] ------------------------------------------------------------------------
19:14:52 [INFO]
19:14:52 [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ MShieldTesting ---
19:14:52 [INFO] Using 'UTF-8' encoding to copy filtered resources.
19:14:52 [INFO] Copying 3 resources
19:14:52 [INFO]
19:14:52 [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ MShieldTesting ---
19:14:52 [INFO] No sources to compile
19:14:52 [INFO]
19:14:52 [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ MShieldTesting ---
19:14:52 [INFO] Using 'UTF-8' encoding to copy filtered resources.
19:14:52 [INFO] Copying 12 resources
19:14:52 [INFO]
19:14:52 [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ MShieldTesting ---
19:14:52 [INFO] Changes detected - recompiling the module!
19:14:52 [INFO] Compiling 2 source files to /usr/src/mymaven/MShieldTesting/target/test-classes
19:14:53 [WARNING] /usr/src/mymaven/MShieldTesting/src/test/java/com/marfeel/mshield/MShieldVarnishITTest.java: /usr/src/mymaven/MShieldTesting/src/test/java/com/marfeel/mshield/MShieldVarnishITTest.java uses unchecked or unsafe operations.
19:14:53 [WARNING] /usr/src/mymaven/MShieldTesting/src/test/java/com/marfeel/mshield/MShieldVarnishITTest.java: Recompile with -Xlint:unchecked for details.
19:14:53 [INFO]
19:14:53 [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ MShieldTesting ---
19:14:53 [INFO] Surefire report directory: /usr/src/mymaven/MShieldTesting/target/surefire-reports
19:14:53
19:14:53 -------------------------------------------------------
19:14:53 T E S T S
19:14:53 -------------------------------------------------------
19:14:53 Running com.marfeel.mshield.MShieldNginxITTest
19:14:53 18:14:53.656 [main] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy - Will use 'okhttp' transport
19:14:54 18:14:53.990 [main] INFO org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy - Found docker client settings from environment
19:14:54 18:14:54.011 [main] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with Environment variables, system properties and defaults. Resolved:
19:14:54 dockerHost=unix:///var/run/docker.sock
19:14:54 apiVersion='{UNKNOWN_VERSION}'
19:14:54 registryUrl='https://index.docker.io/v1/'
19:14:54 registryUsername='jenkins'
19:14:54 registryPassword='null'
19:14:54 registryEmail='null'
19:14:54 dockerConfig='DefaultDockerClientConfig[dockerHost=unix:///var/run/docker.sock,registryUsername=jenkins,registryPassword=<null>,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/home/jenkins/.docker,sslConfig=<null>,apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'
19:14:54
19:14:54 18:14:54.246 [main] WARN org.testcontainers.utility.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: alpine:3.5, configFile: /home/jenkins/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /home/jenkins/.docker/config.json (No such file or directory)
19:14:55 18:14:55.168 [main] INFO org.testcontainers.DockerClientFactory - Docker host IP address is 172.17.0.1
19:14:55 18:14:55.271 [main] INFO org.testcontainers.DockerClientFactory - Connected to docker:
19:14:55 Server Version: 18.06.1-ce
19:14:55 API Version: 1.38
19:14:55 Operating System: Ubuntu 16.04.5 LTS
19:14:55 Total Memory: 128822 MB
19:14:55 [37m[1m?? Checking the system...[0m[0m
19:14:55 [32m? Docker version should be at least 1.6.0[0m
19:14:55 18:14:55.386 [main] WARN org.testcontainers.utility.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: alpine:3.5, configFile: /home/jenkins/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /home/jenkins/.docker/config.json (No such file or directory)
19:14:56 [32m? Docker environment should have more than 2GB free disk space[0m
19:14:56 18:14:56.408 [main] INFO docker[quay.io/testcontainers/sshd@sha256:18aa92...] - Pulling docker image: quay.io/testcontainers/sshd@sha256:18aa929c653284189fc9cefa45b731021857b6047a0a1757e909f958f258f088. Please be patient; this may take some time but only needs to be done once.
19:14:56 18:14:56.410 [main] WARN org.testcontainers.utility.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: quay.io/testcontainers/sshd:latest, configFile: /home/jenkins/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /home/jenkins/.docker/config.json (No such file or directory)
19:14:57 18:14:57.659 [main] INFO docker[quay.io/testcontainers/sshd@sha256:18aa92...] - Creating container for image: quay.io/testcontainers/sshd@sha256:18aa929c653284189fc9cefa45b731021857b6047a0a1757e909f958f258f088
19:14:57 18:14:57.660 [main] WARN org.testcontainers.utility.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: quay.io/testcontainers/sshd@sha256:18aa929c653284189fc9cefa45b731021857b6047a0a1757e909f958f258f088, configFile: /home/jenkins/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /home/jenkins/.docker/config.json (No such file or directory)
19:14:57 18:14:57.747 [main] INFO docker[quay.io/testcontainers/sshd@sha256:18aa92...] - Starting container with ID: 5d92dd595dcbdf27685f0e68882543fce3c5095323e01df5364eb7db60d54cb8
19:14:58 18:14:58.141 [main] INFO docker[quay.io/testcontainers/sshd@sha256:18aa92...] - Container quay.io/testcontainers/sshd@sha256:18aa929c653284189fc9cefa45b731021857b6047a0a1757e909f958f258f088 is starting: 5d92dd595dcbdf27685f0e68882543fce3c5095323e01df5364eb7db60d54cb8
19:16:05 18:15:58.232 [main] ERROR docker[quay.io/testcontainers/sshd@sha256:18aa92...] - Could not start container
19:16:05 org.testcontainers.containers.ContainerLaunchException: Timed out waiting for container port to open (172.17.0.1 ports: [32774] should be listening)
19:16:05 at org.testcontainers.containers.wait.strategy.HostPortWaitStrategy.waitUntilReady(HostPortWaitStrategy.java:47)
19:16:05 at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:35)
19:16:05 at org.testcontainers.containers.wait.HostPortWaitStrategy.waitUntilReady(HostPortWaitStrategy.java:23)
19:16:05 at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:35)
19:16:05 at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:591)
19:16:05 at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:263)
19:16:05 at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:216)
19:16:05 at org.testcontainers.containers.GenericContainer$$Lambda$53/515184459.call(Unknown Source)
19:16:05 at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:76)
19:16:05 at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:214)
19:16:05 at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:203)
19:16:05 at org.testcontainers.containers.PortForwardingContainer.createSSHSession(PortForwardingContainer.java:39)
19:16:05 at org.testcontainers.containers.PortForwardingContainer.getSshConnection(PortForwardingContainer.java:24)
19:16:05 at org.testcontainers.containers.PortForwardingContainer.exposeHostPort(PortForwardingContainer.java:60)
19:16:05 at org.testcontainers.Testcontainers.exposeHostPorts(Testcontainers.java:11)
19:16:05 at com.marfeel.mshield.MShieldNginxITTest.<clinit>(MShieldNginxITTest.java:21)
19:16:05 at sun.misc.Unsafe.ensureClassInitialized(Native Method)
19:16:05 at sun.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(UnsafeFieldAccessorFactory.java:43)
19:16:05 at sun.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:142)
19:16:05 at java.lang.reflect.Field.acquireFieldAccessor(Field.java:1088)
19:16:05 at java.lang.reflect.Field.getFieldAccessor(Field.java:1069)
19:16:05 at java.lang.reflect.Field.get(Field.java:393)
19:16:05 at org.junit.runners.model.FrameworkField.get(FrameworkField.java:73)
19:16:05 at org.junit.runners.model.TestClass.getAnnotatedFieldValues(TestClass.java:230)
19:16:05 at org.junit.runners.ParentRunner.classRules(ParentRunner.java:255)
19:16:05 at org.junit.runners.ParentRunner.withClassRules(ParentRunner.java:244)
19:16:05 at org.junit.runners.ParentRunner.classBlock(ParentRunner.java:194)
19:16:05 at org.junit.runners.ParentRunner.run(ParentRunner.java:362)
19:16:05 at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
19:16:05 at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
19:16:05 at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
19:16:05 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
19:16:05 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
19:16:05 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
19:16:05 at java.lang.reflect.Method.invoke(Method.java:497)
19:16:05 at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
19:16:05 at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
19:16:05 at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
19:16:05 at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
19:16:05 at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
19:16:05 18:15:58.233 [main] ERROR docker[quay.io/testcontainers/sshd@sha256:18aa92...] - Container log output (if any) will follow:
19:16:05 18:15:58.238 [tc-okhttp-stream-452121674] INFO docker[quay.io/testcontainers/sshd@sha256:18aa92...] - STDERR: chpasswd: password for 'root' changed
19:16:05 Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 64.836 sec <<< FAILURE!
19:16:05 com.marfeel.mshield.MShieldNginxITTest Time elapsed: 64.834 sec <<< ERROR!
19:16:05 java.lang.ExceptionInInitializerError
19:16:05 at sun.misc.Unsafe.ensureClassInitialized(Native Method)
19:16:05 at sun.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(UnsafeFieldAccessorFactory.java:43)
19:16:05 at sun.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:142)
19:16:05 at java.lang.reflect.Field.acquireFieldAccessor(Field.java:1088)
19:16:05 at java.lang.reflect.Field.getFieldAccessor(Field.java:1069)
19:16:05 at java.lang.reflect.Field.get(Field.java:393)
19:16:05 at org.junit.runners.model.FrameworkField.get(FrameworkField.java:73)
19:16:05 at org.junit.runners.model.TestClass.getAnnotatedFieldValues(TestClass.java:230)
19:16:05 at org.junit.runners.ParentRunner.classRules(ParentRunner.java:255)
19:16:05 at org.junit.runners.ParentRunner.withClassRules(ParentRunner.java:244)
19:16:05 at org.junit.runners.ParentRunner.classBlock(ParentRunner.java:194)
19:16:05 at org.junit.runners.ParentRunner.run(ParentRunner.java:362)
19:16:05 at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
19:16:05 at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
19:16:05 at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
19:16:05 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
19:16:05 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
19:16:05 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
19:16:05 at java.lang.reflect.Method.invoke(Method.java:497)
19:16:05 at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
19:16:05 at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
19:16:05 at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
19:16:05 at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
19:16:05 at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
19:16:05 Caused by: org.testcontainers.containers.ContainerLaunchException: Container startup failed
19:16:05 at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:221)
19:16:05 at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:203)
19:16:05 at org.testcontainers.containers.PortForwardingContainer.createSSHSession(PortForwardingContainer.java:39)
19:16:05 at org.testcontainers.containers.PortForwardingContainer.getSshConnection(PortForwardingContainer.java:24)
19:16:05 at org.testcontainers.containers.PortForwardingContainer.exposeHostPort(PortForwardingContainer.java:60)
19:16:05 at org.testcontainers.Testcontainers.exposeHostPorts(Testcontainers.java:11)
19:16:05 at com.marfeel.mshield.MShieldNginxITTest.<clinit>(MShieldNginxITTest.java:21)
19:16:05 ... 24 more
19:16:05 Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
19:16:05 at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:83)
19:16:05 at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:214)
19:16:05 ... 30 more
19:16:05 Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
19:16:05 at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:286)
19:16:05 at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:216)
19:16:05 at org.testcontainers.containers.GenericContainer$$Lambda$53/515184459.call(Unknown Source)
19:16:05 at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:76)
19:16:05 ... 31 more
19:16:05 Caused by: org.testcontainers.containers.ContainerLaunchException: Timed out waiting for container port to open (172.17.0.1 ports: [32774] should be listening)
19:16:05 at org.testcontainers.containers.wait.strategy.HostPortWaitStrategy.waitUntilReady(HostPortWaitStrategy.java:47)
19:16:05 at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:35)
19:16:05 at org.testcontainers.containers.wait.HostPortWaitStrategy.waitUntilReady(HostPortWaitStrategy.java:23)
19:16:05 at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:35)
19:16:05 at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:591)
19:16:05 at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:263)
19:16:05 ... 34 more
19:16:05
More info
Host’s docker.sock permissions are as follow
srw-rw---- 1 root docker 0 Sep 12 10:22 /var/run/docker.sock
The user that runs the docker is jenkins, who belongs to the docker group, and because of that we are using a custom maven:3.2.5-jdk-8 named marfeeljava, based on the following Dockerfile
FROM maven:3.2.5-jdk-8
RUN groupadd -g 999 docker && \
useradd -d /home/jenkins -m -r -u 1004 -g docker jenkins
The command run is the following and as said it works perfectly in my Mac 😐
docker run --privileged=true -u jenkins -v "$PWD":/usr/src/mymaven -v "$HOME/.m2":/home/jenkins/.m2 -v /var/run/docker.sock:/var/run/docker.sock -e TESTCONTAINERS_RYUK_DISABLED=true -w /usr/src/mymaven marfeeljava:latest mvn test
The sshd container is up and running, but it seems that testcontainers is unable to use it
The following code snippet shows how we initialize all the needed stuff
static {
Testcontainers.exposeHostPorts(BACKEND_PORT);
}
@ClassRule
public static WireMockClassRule wireMockRule = new WireMockClassRule(BACKEND_PORT);
@Rule
public WireMockClassRule instanceRule = wireMockRule;
private static ToStringConsumer OUTPUT_CONSUMER = new ToStringConsumer();
private static String CONTAINER_ADDRESS;
private static Integer CONTAINER_PORT;
@ClassRule
public static GenericContainer varnish = new GenericContainer(
new ImageFromDockerfile("mshield", false)
.withFileFromClasspath("Dockerfile", "Dockerfile")
.withFileFromClasspath("start.sh", "start.sh")
.withFileFromClasspath("default.vcl", "default.vcl")
.withFileFromClasspath("testVha.vcl", "testVha.vcl")
.withFileFromClasspath("testBackendFetch.vcl", "testBackendFetch.vcl")
.withFileFromClasspath("vha_40.vcl", "vha_40.vcl")
.withFileFromClasspath("docker-entrypoint.sh", "docker-entrypoint.sh"))
.withExposedPorts(6081).withEnv("hostPort",String.valueOf(BACKEND_PORT));
@Before
public void setUp() throws Exception {
//varnish.followOutput(OUTPUT_CONSUMER, OutputFrame.OutputType.STDOUT);
CONTAINER_ADDRESS = varnish.getContainerIpAddress();
CONTAINER_PORT = varnish.getMappedPort(6081);
}
Any hint will be highly appreciated
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 27 (10 by maintainers)
Hi! I also encounter this exception, so I tried to follow your advice @bsideup and run
ExposedHostTest
, which fails. After further debugging, it seems to have got something to do with the pid, which cannot be parsed (see https://github.com/docker-java/docker-java/issues/1259).Also, please check the logs of Ryuk, maybe it will show something