testcontainers-java: Intermittent failure in CI environment
I have a simple Redis integration test:
public class RedisResourceTest {
private static final String DOCKER_IMAGE = "redis:3.2.9";
private static final int TIMEOUT = 120;
@ClassRule
public static GenericContainer redis = new GenericContainer(DOCKER_IMAGE).withExposedPorts(6379)
.withStartupTimeout(Duration.ofSeconds(TIMEOUT));
@Test
public void testResourceCanSendAndRecieve() throws ResourceInitializationException {
String ipAddredd = redis.getContainerIpAddress()),
int redisPort = redis.getMappedPort(6379)))};
//TEST code
}
}
This passes every time running locally, however, when running in Jenkins (BlueOcean) it intermittently fails with the exception:
java.lang.AssertionError: A port exposed by a docker container should be accessible: 'Connection refused (Connection refused)' does not equal expected 'hello'
at org.rnorth.visibleassertions.VisibleAssertions.fail(VisibleAssertions.java:440)
at org.rnorth.visibleassertions.VisibleAssertions.assertEquals(VisibleAssertions.java:174)
at org.testcontainers.DockerClientFactory.checkExposedPort(DockerClientFactory.java:190)
at org.testcontainers.DockerClientFactory.lambda$client$2(DockerClientFactory.java:125)
at org.testcontainers.DockerClientFactory.runInsideDocker(DockerClientFactory.java:226)
at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:114)
at org.testcontainers.containers.GenericContainer.<init>(GenericContainer.java:116)
at io.committed.RedisResourceTest.<clinit>(RedisResourceTest.java:27)
at sun.misc.Unsafe.ensureClassInitialized(Native Method)
at sun.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(UnsafeFieldAccessorFactory.java:43)
at sun.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:156)
at java.lang.reflect.Field.acquireFieldAccessor(Field.java:1088)
at java.lang.reflect.Field.getFieldAccessor(Field.java:1069)
at java.lang.reflect.Field.get(Field.java:393)
at org.junit.runners.model.FrameworkField.get(FrameworkField.java:73)
at org.junit.runners.model.TestClass.getAnnotatedFieldValues(TestClass.java:230)
at org.junit.runners.ParentRunner.classRules(ParentRunner.java:255)
at org.junit.runners.ParentRunner.withClassRules(ParentRunner.java:244)
at org.junit.runners.ParentRunner.classBlock(ParentRunner.java:194)
at org.junit.runners.ParentRunner.run(ParentRunner.java:362)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
I’ve tried larger timeout values, but that doesn’t seem to help. The docker sock port is exposed to the machine, the logs report lots of activity and the fact it sometimes passes must imply it’s not simply that it can’t interact with Docker.
I realize that this must be something to do with the environment, maybe as the build is already run in a Docker Container, so I don’t expect you to have a fix for this, however, my question is, how do I start to debug such an issue? Maybe there is a way to see the logs from the container?
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 32 (25 by maintainers)
Commits related to this issue
- Make dockerHostIp detection less flaky, Fixes #479 — committed to testcontainers/testcontainers-java by bsideup 6 years ago
- Make sure the Docker Host IP address is retrieved correctly when running inside docker (#541) * fix: introduce 'withFollowStream' and 'awaitCompletion' methods to await the completion of one-off comm... — committed to testcontainers/testcontainers-java by jochenhebbrecht 6 years ago
Yes, I’ll make a PR. I’ll keep you updated.
@agibalov 1.4.3 is old, the latest is 1.6.0, please update
I created a shell script on the Jenkins Cloud Slave which executed, in for loop for like 1.000.000 times, the ‘ip route’ command and it always returned a valid value. So I don’t think it’s related to ECS-native networking.
I found out the LogToStringContainerCallback class doesn’t receive the output of the Docker container. I’m afraid there’s something wrong with the capturing of the stdout messages (probably in combination with ECS)