testcontainers-go: [Bug]: Unable to connect to HostAccessPorts on container startup

Testcontainers version

0.33.0

Using the latest Testcontainers version?

Yes

Host OS

Linux

Host arch

x86_64

Go version

1.23.1

Docker version

Client: Docker Engine - Community Version: 27.3.1 API version: 1.47 Go version: go1.22.7 Git commit: ce12230 Built: Fri Sep 20 11:41:00 2024 OS/Arch: linux/amd64 Context: default

Server: Docker Engine - Community Engine: Version: 27.3.1 API version: 1.47 (minimum version 1.24) Go version: go1.22.7 Git commit: 41ca978 Built: Fri Sep 20 11:41:00 2024 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.7.22 GitCommit: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c runc: Version: 1.1.14 GitCommit: v1.1.14-0-g2c9f560 docker-init: Version: 0.19.0 GitCommit: de40ad0

Docker info

Client: Docker Engine - Community Version: 27.3.1 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.17.1 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.29.7 Path: /usr/libexec/docker/cli-plugins/docker-compose

Server: Containers: 14 Running: 9 Paused: 0 Stopped: 5 Images: 34 Server Version: 27.3.1 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: systemd Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c runc version: v1.1.14-0-g2c9f560 init version: de40ad0 Security Options: apparmor seccomp Profile: builtin cgroupns Kernel Version: 6.8.0-40-generic Operating System: Ubuntu 22.04.5 LTS OSType: linux Architecture: x86_64 CPUs: 32 Total Memory: 62.52GiB Name: jhome ID: 43fdd48e-011e-40da-aff1-b76bc378d203 Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

What happened?

I’m attempting to utilize testcontainers-go to test my Caddy configuration as a gateway to my API server but I’m running into problems with how testcontainers-go exposes host ports and I believe this issue to be a bug.

Setup

In my tests, I’ve set up a httptest.Server to act as my API server, listening on a random port on the host. I then set up Caddy in a testcontainer and expose the API server port to the container via HostAccessPorts. My Caddy configuration defines the API server with a health check which Caddy checks on startup.

caddyfile_test.go
package caddy_test

import (
	"bytes"
	"context"
	"fmt"
	"io"
	"net"
	"net/http"
	"net/http/httptest"
	"testing"

	"github.com/stretchr/testify/assert"
	"github.com/stretchr/testify/require"
	"github.com/testcontainers/testcontainers-go"
	"github.com/testcontainers/testcontainers-go/wait"
)

const caddyFileContent = `
listen :80

reverse_proxy /api/* {
	to {$API_SERVER}

	health_uri /health
	health_status 200
	health_interval 10s
}
`

func TestCaddyfile(t *testing.T) {
	ctx := context.Background()

	apiServerListener, err := net.Listen("tcp", "0.0.0.0:0")
	assert.NoError(t, err)

	apiServerPort := apiServerListener.Addr().(*net.TCPAddr).Port
	apiServer := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintln(w, "Hello, World!")
	}))
	apiServer.Listener.Close()
	apiServer.Listener = apiServerListener
	apiServer.Start()

	caddyContainer, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
		ContainerRequest: testcontainers.ContainerRequest{
			Image:        "caddy:2.8.4",
			ExposedPorts: []string{"80/tcp"},
			WaitingFor:   wait.ForLog("server running"),
			Env: map[string]string{
				"API_SERVER": fmt.Sprintf("http://%s:%d", testcontainers.HostInternal, apiServerPort),
			},
			Files: []testcontainers.ContainerFile{
				{
					Reader:            bytes.NewReader([]byte(caddyFileContent)),
					ContainerFilePath: "/etc/caddy/Caddyfile",
				},
			},
			HostAccessPorts: []int{apiServerPort},
		},
		Started: true,
	})
	require.NoError(t, err)
	defer caddyContainer.Terminate(ctx)

	caddyURL, err := caddyContainer.PortEndpoint(ctx, "80/tcp", "http")
	require.NoError(t, err)

	resp, err := http.Get(caddyURL + "/api/test")
	require.NoError(t, err)
	defer resp.Body.Close()

	body, err := io.ReadAll(resp.Body)
	require.NoError(t, err)

	assert.Equal(t, http.StatusOK, resp.StatusCode)
	assert.Equal(t, "Hello, World!\n", string(body))

	lr, err := caddyContainer.Logs(ctx)
	assert.NoError(t, err)
	lb, err := io.ReadAll(lr)
	assert.NoError(t, err)
	fmt.Printf("== Caddy Logs ==\n%s================\n\n", string(lb))
}
Test Output
== Caddy Logs ==
{"level":"info","ts":1727952070.1965187,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1727952070.1969736,"msg":"adapted config to JSON","adapter":"caddyfile"}
{"level":"warn","ts":1727952070.1969776,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":1}
{"level":"info","ts":1727952070.1972885,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"warn","ts":1727952070.1973321,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}
{"level":"info","ts":1727952070.1973393,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1727952070.1973433,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1727952070.1973994,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000736a80"}
{"level":"info","ts":1727952070.1974878,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1727952070.197532,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
{"level":"info","ts":1727952070.1975832,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
+{"level":"info","ts":1727952070.1976013,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
{"level":"info","ts":1727952070.1976032,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["listen"]}
-{"level":"info","ts":1727952070.1976056,"logger":"http.handlers.reverse_proxy.health_checker.active","msg":"HTTP request failed","host":"host.testcontainers.internal:43017","error":"Get \"http://host.testcontainers.internal:43017/health\": dial tcp 172.17.0.3:43017: connect: connection refused"}
-{"level":"info","ts":1727952070.1976073,"logger":"http.handlers.reverse_proxy.health_checker.active","msg":"HTTP request failed","host":"host.testcontainers.internal:43017","error":"Get \"http://host.testcontainers.internal:43017/health\": dial tcp 172.17.0.3:43017: connect: connection refused"}
{"level":"info","ts":1727952070.1978004,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1727952070.1978037,"msg":"serving initial configuration"}
{"level":"info","ts":1727952070.197835,"logger":"tls.obtain","msg":"acquiring lock","identifier":"listen"}
{"level":"info","ts":1727952070.1985145,"logger":"tls.obtain","msg":"lock acquired","identifier":"listen"}
{"level":"info","ts":1727952070.1985347,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"listen"}
{"level":"info","ts":1727952070.1985307,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/data/caddy"}
{"level":"info","ts":1727952070.1986136,"logger":"tls","msg":"finished cleaning storage units"}
-{"level":"error","ts":1727952070.3384068,"logger":"http.log.error","msg":"no upstreams available","request":{"remote_ip":"172.17.0.1","remote_port":"54434","client_ip":"172.17.0.1","proto":"HTTP/1.1","method":"GET","host":"localhost:33500","uri":"/api/test","headers":{"User-Agent":["Go-http-client/1.1"],"Accept-Encoding":["gzip"]}},"duration":0.000040091,"status":503,"err_id":"qe8hu1acn","err_trace":"reverseproxy.(*Handler).proxyLoopIteration (reverseproxy.go:486)"}

================

--- FAIL: TestCaddyfile (1.10s)
    caddyfile_test.go:76: 
        	Error Trace:	/home/justin/workspace/test/caddyfile_test.go:76
        	Error:      	Not equal: 
        	            	expected: 200
        	            	actual  : 503
        	Test:       	TestCaddyfile
    caddyfile_test.go:77: 
        	Error Trace:	/home/justin/workspace/test/caddyfile_test.go:77
        	Error:      	Not equal: 
        	            	expected: "Hello, World!\n"
        	            	actual  : ""
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,2 +1 @@
        	            	-Hello, World!
        	            	 
        	Test:       	TestCaddyfile
FAIL
FAIL	github.com/hathvi/test	1.189s
FAIL

Problem

My problem with this set up is that Caddy logs a “connection refused” error for the health check even though the testcontainer is ready. I attempt to make a request to the Caddy server after startup but receive an HTTP 502 Bad Gateway error as the API server wasn’t initially reachable even though it’s running and accepting connections on the host. Caddy will continue to return an HTTP 502 until the next health check.

My Analysis

I can see that HostAccessPorts utilizes a separate container running an SSH server then sets up a PostReadies lifecycle hook on the Caddy container in order to then set up the forwarding in the SSH container. It appears to do the forwarding by firing off a go routine that connects to the SSH container with remote port forwarding, listening to the HostAccessPorts ports on the SSH container and tunneling this to the host.

PostReadies seems like a lot to late to setup the forwarding. I’m utilizing HostAccessPorts so I can talk to a server on the host from my testcontainer, so in order for my test container to be ready I would expect to be able to talk to that server before I do any of my testing. Logically I would assume I should be able to have a wait strategy that depends on that connection being made.

Test fix

I created a fork and updated exposeHostPorts to setup a lifecycle hook on PreCreates instead of PostReadies. This ensures the host port is accessible via the SSH container to the testcontainer from all lifecycle hooks and container command.

In theory this shouldn’t break anything even if someone sets up the listener for the host port in a later lifecycle hook as connections back on the host port are only established once you try connecting to the remote port.

testcontainers-go.patch
diff --git a/port_forwarding.go b/port_forwarding.go
index 88f14f2d..ad17fb10 100644
--- a/port_forwarding.go
+++ b/port_forwarding.go
@@ -150,8 +150,8 @@ func exposeHostPorts(ctx context.Context, req *ContainerRequest, ports ...int) (
        // after the container is ready, create the SSH tunnel
        // for each exposed port from the host.
        sshdConnectHook = ContainerLifecycleHooks{
-               PostReadies: []ContainerHook{
-                       func(ctx context.Context, c Container) error {
+               PreCreates: []ContainerRequestHook{
+                       func(ctx context.Context, req ContainerRequest) error {
                                return sshdContainer.exposeHostPort(ctx, req.HostAccessPorts...)
                        },
                },

Relevant log output

No response

Additional information

No response

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 22 (10 by maintainers)

Commits related to this issue

Most upvoted comments

I just came across this issue yesterday as I was exploring a similar problem with HostAccessPorts and the SSH tunnel not being set up in time - I too was depending on the host network being accessible during container startup.

The DNS solution works great and is certainly simpler, so thanks for that @stevenh !

My $.02 –

IMO, if the support for HostAccessPorts is to be kept around in spite of this, then I do think the fix provided by @hathvi is a logical one - I had implemented a similar custom container lifecycle hook to get things working the way we needed before coming across this issue.

If indeed there is no justification for continuing to support HostAccessPorts other than backwards compatibility, then it would be great to see the docs updated to point out the availability of host.docker.internal and perhaps even to illustrate its use with an example like the one here.

I was not able to get WSL2 running within VirtualBox last night, but I got things set up on a laptop this AM and can confirm your test works for me on Windows as well.

Based on the outputs you provided in your last post this should technically not be allowed unless something in the middle is doing something with the network packets such as NAT or proxying to the host. It looks like WSL2 uses NAT by default, but doesn’t allow this behavior itself.

server.go
package main

import (
	"fmt"
	"net"
	"net/http"
	"net/http/httptest"
	"os"
	"os/signal"
	"syscall"
)

func main() {
	l, err := net.Listen("tcp", "127.0.0.1:51403")
	if err != nil {
		panic(err)
	}

	s := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
		w.Write([]byte("Hello, world!"))
	}))
	s.Listener.Close()
	s.Listener = l
	s.Start()

	fmt.Printf("Listening: %s\n", s.Listener.Addr())

	sc := make(chan os.Signal, 1)
	signal.Notify(sc, syscall.SIGTERM)
	<-sc
}
PS C:\Users\justin\workspace\test> go run server.go
Listening: 127.0.0.1:51403
PS C:\Users\justin\workspace\test> wsl -d ubuntu

justin@jlap:/mnt/c/Users/justin/workspace/test$ ip route
default via 172.26.192.1 dev eth0 proto kernel
172.26.192.0/20 dev eth0 proto kernel scope link src 172.26.196.73

justin@jlap:/mnt/c/Users/justin/workspace/test$ curl -m 5 172.26.192.1:51403
curl: (28) Connection timed out after 5002 milliseconds

The WSL2 documentation even makes explicit mention of this https://learn.microsoft.com/en-us/windows/wsl/networking#connecting-via-remote-ip-addresses

When using remote IP addresses to connect to your applications, they will be treated as connections from the Local Area Network (LAN). This means that you will need to make sure your application can accept LAN connections.

For example, you may need to bind your application to 0.0.0.0 instead of 127.0.0.1. In the example of a Python app using Flask, this can be done with the command: app.run(host=‘0.0.0.0’). Keep security in mind when making these changes as this will allow connections from your LAN.

I tried digging more into how Docker is running under WSL2 but at first glance doesn’t seem as straight forward as I was hoping. I’ll try digging more into this later tonight.

I don’t contribute as much to open source as I should mainly due to anxiety socializing with others, but I’d be down to contributing the suggested changes if that seems like a good idea to you guys.

Are you two by chance on Mac or Windows? I think Windows, under Docker Desktop, routes traffic to the host a bit differently then Mac or Linux. I use Linux as my primary OS and am not super familiar with Windows. The test that @stevenh provided does not work for me, which I’ll describe why below.

I’d preface that my original problem with testcontainers was that it was non obvious how to access the host using testcontainers. I however was also not aware of the docker extra hosts config and my searching around testcontainers lead me to HostAccessPorts which I found to be setting up the listeners to late.

Docker by default uses a bridge network. It creates a new interface on the host named docker0 with it’s own subnet, in my case this is 172.17.0.1/16 with the interface address being 172.17.0.1. When you configure extra hosts to host-gateway it creates a DNS record in /etc/hosts to host.docker.internal 172.17.0.1.

When you create a new httptest.Server it by default binds to 127.0.0.1 on a random port of the machine you are running the app on, there’s no direct way of changing this behavior and you must swap out the httptest.Server.Listener to bind to a different interface. In order for the DNS solution to work the httptest.Server would need to bind to 172.17.0.1 ( the container gateway / bridge interface ).

HostAccessPorts requests the SSH server bind the remote port in the container to 127.0.0.1:PORT and tunnel traffic back to the client which in turn dials 127.0.0.1:PORT ( the httptest.Server ) on the client machine and copies those bytes from the tunnel to the server. If I were to swap out the httptest.Server.Listener to bind to 172.17.0.0.1 I’d have the same issue with HostAccessPorts but DNS would work.

My guess would be that you guys are likely on Windows and/or running Docker Desktop which likely does something similar to what HostAccessPorts is doing and/or Windows allows connecting to a port bound on 127.0.0.1 from any interface. I don’t have a Windows VM currently to do any testing.

I would say HostAccessPorts does solve a problem, especially since people are generally going to be using this for testing and I assume httptest.Server would be a somewhat common use case for people testing HTTP clients, however I don’t think this is a problem testcontainers necessarily needs to resolve as it feels out of scope, it’s just helpful that it does.

I would also say that if HostAccessPorts were kept it might be nice to extend its functionality to change which interface the local client dials, working more inline with how the ssh -R works. e.g., ssh -R [local-addr:]local-port:remote-addr:remote-port destination. Right now testcontainers doesn’t expose the ability to set local-addr or remote-addr and hard codes these to localhost ( here for the remote listener, and here for the local forwarding client ). I assume this would result in a breaking change however as you’d likely need something like HostAccessPorts []string instead of []int and support values like local-addr:port/tcp, port/tcp ( defaults to 127.0.0.1 ), local-addr:port ( defaults to tcp ), port ( defaults to 127.0.0.1 and tcp )