testcontainers-go: [Bug]: Unable to connect to HostAccessPorts on container startup
Testcontainers version
0.33.0
Using the latest Testcontainers version?
Yes
Host OS
Linux
Host arch
x86_64
Go version
1.23.1
Docker version
Client: Docker Engine - Community Version: 27.3.1 API version: 1.47 Go version: go1.22.7 Git commit: ce12230 Built: Fri Sep 20 11:41:00 2024 OS/Arch: linux/amd64 Context: default
Server: Docker Engine - Community Engine: Version: 27.3.1 API version: 1.47 (minimum version 1.24) Go version: go1.22.7 Git commit: 41ca978 Built: Fri Sep 20 11:41:00 2024 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.7.22 GitCommit: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c runc: Version: 1.1.14 GitCommit: v1.1.14-0-g2c9f560 docker-init: Version: 0.19.0 GitCommit: de40ad0
Docker info
Client: Docker Engine - Community Version: 27.3.1 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.17.1 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.29.7 Path: /usr/libexec/docker/cli-plugins/docker-compose
Server: Containers: 14 Running: 9 Paused: 0 Stopped: 5 Images: 34 Server Version: 27.3.1 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: systemd Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c runc version: v1.1.14-0-g2c9f560 init version: de40ad0 Security Options: apparmor seccomp Profile: builtin cgroupns Kernel Version: 6.8.0-40-generic Operating System: Ubuntu 22.04.5 LTS OSType: linux Architecture: x86_64 CPUs: 32 Total Memory: 62.52GiB Name: jhome ID: 43fdd48e-011e-40da-aff1-b76bc378d203 Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false
What happened?
I’m attempting to utilize testcontainers-go to test my Caddy configuration as a gateway to my API server but I’m running into problems with how testcontainers-go exposes host ports and I believe this issue to be a bug.
Setup
In my tests, I’ve set up a httptest.Server to act as my API server, listening on a random port on the host. I then set up Caddy in a testcontainer and expose the API server port to the container via HostAccessPorts. My Caddy configuration defines the API server with a health check which Caddy checks on startup.
caddyfile_test.go
package caddy_test
import (
"bytes"
"context"
"fmt"
"io"
"net"
"net/http"
"net/http/httptest"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/testcontainers/testcontainers-go"
"github.com/testcontainers/testcontainers-go/wait"
)
const caddyFileContent = `
listen :80
reverse_proxy /api/* {
to {$API_SERVER}
health_uri /health
health_status 200
health_interval 10s
}
`
func TestCaddyfile(t *testing.T) {
ctx := context.Background()
apiServerListener, err := net.Listen("tcp", "0.0.0.0:0")
assert.NoError(t, err)
apiServerPort := apiServerListener.Addr().(*net.TCPAddr).Port
apiServer := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "Hello, World!")
}))
apiServer.Listener.Close()
apiServer.Listener = apiServerListener
apiServer.Start()
caddyContainer, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
ContainerRequest: testcontainers.ContainerRequest{
Image: "caddy:2.8.4",
ExposedPorts: []string{"80/tcp"},
WaitingFor: wait.ForLog("server running"),
Env: map[string]string{
"API_SERVER": fmt.Sprintf("http://%s:%d", testcontainers.HostInternal, apiServerPort),
},
Files: []testcontainers.ContainerFile{
{
Reader: bytes.NewReader([]byte(caddyFileContent)),
ContainerFilePath: "/etc/caddy/Caddyfile",
},
},
HostAccessPorts: []int{apiServerPort},
},
Started: true,
})
require.NoError(t, err)
defer caddyContainer.Terminate(ctx)
caddyURL, err := caddyContainer.PortEndpoint(ctx, "80/tcp", "http")
require.NoError(t, err)
resp, err := http.Get(caddyURL + "/api/test")
require.NoError(t, err)
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
require.NoError(t, err)
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.Equal(t, "Hello, World!\n", string(body))
lr, err := caddyContainer.Logs(ctx)
assert.NoError(t, err)
lb, err := io.ReadAll(lr)
assert.NoError(t, err)
fmt.Printf("== Caddy Logs ==\n%s================\n\n", string(lb))
}
Test Output
== Caddy Logs ==
{"level":"info","ts":1727952070.1965187,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1727952070.1969736,"msg":"adapted config to JSON","adapter":"caddyfile"}
{"level":"warn","ts":1727952070.1969776,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":1}
{"level":"info","ts":1727952070.1972885,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"warn","ts":1727952070.1973321,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}
{"level":"info","ts":1727952070.1973393,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1727952070.1973433,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1727952070.1973994,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000736a80"}
{"level":"info","ts":1727952070.1974878,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1727952070.197532,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
{"level":"info","ts":1727952070.1975832,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
+{"level":"info","ts":1727952070.1976013,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
{"level":"info","ts":1727952070.1976032,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["listen"]}
-{"level":"info","ts":1727952070.1976056,"logger":"http.handlers.reverse_proxy.health_checker.active","msg":"HTTP request failed","host":"host.testcontainers.internal:43017","error":"Get \"http://host.testcontainers.internal:43017/health\": dial tcp 172.17.0.3:43017: connect: connection refused"}
-{"level":"info","ts":1727952070.1976073,"logger":"http.handlers.reverse_proxy.health_checker.active","msg":"HTTP request failed","host":"host.testcontainers.internal:43017","error":"Get \"http://host.testcontainers.internal:43017/health\": dial tcp 172.17.0.3:43017: connect: connection refused"}
{"level":"info","ts":1727952070.1978004,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1727952070.1978037,"msg":"serving initial configuration"}
{"level":"info","ts":1727952070.197835,"logger":"tls.obtain","msg":"acquiring lock","identifier":"listen"}
{"level":"info","ts":1727952070.1985145,"logger":"tls.obtain","msg":"lock acquired","identifier":"listen"}
{"level":"info","ts":1727952070.1985347,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"listen"}
{"level":"info","ts":1727952070.1985307,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/data/caddy"}
{"level":"info","ts":1727952070.1986136,"logger":"tls","msg":"finished cleaning storage units"}
-{"level":"error","ts":1727952070.3384068,"logger":"http.log.error","msg":"no upstreams available","request":{"remote_ip":"172.17.0.1","remote_port":"54434","client_ip":"172.17.0.1","proto":"HTTP/1.1","method":"GET","host":"localhost:33500","uri":"/api/test","headers":{"User-Agent":["Go-http-client/1.1"],"Accept-Encoding":["gzip"]}},"duration":0.000040091,"status":503,"err_id":"qe8hu1acn","err_trace":"reverseproxy.(*Handler).proxyLoopIteration (reverseproxy.go:486)"}
================
--- FAIL: TestCaddyfile (1.10s)
caddyfile_test.go:76:
Error Trace: /home/justin/workspace/test/caddyfile_test.go:76
Error: Not equal:
expected: 200
actual : 503
Test: TestCaddyfile
caddyfile_test.go:77:
Error Trace: /home/justin/workspace/test/caddyfile_test.go:77
Error: Not equal:
expected: "Hello, World!\n"
actual : ""
Diff:
--- Expected
+++ Actual
@@ -1,2 +1 @@
-Hello, World!
Test: TestCaddyfile
FAIL
FAIL github.com/hathvi/test 1.189s
FAIL
Problem
My problem with this set up is that Caddy logs a “connection refused” error for the health check even though the testcontainer is ready. I attempt to make a request to the Caddy server after startup but receive an HTTP 502 Bad Gateway error as the API server wasn’t initially reachable even though it’s running and accepting connections on the host. Caddy will continue to return an HTTP 502 until the next health check.
My Analysis
I can see that HostAccessPorts utilizes a separate container running an SSH server then sets up a PostReadies lifecycle hook on the Caddy container in order to then set up the forwarding in the SSH container. It appears to do the forwarding by firing off a go routine that connects to the SSH container with remote port forwarding, listening to the HostAccessPorts ports on the SSH container and tunneling this to the host.
PostReadies seems like a lot to late to setup the forwarding. I’m utilizing HostAccessPorts so I can talk to a server on the host from my testcontainer, so in order for my test container to be ready I would expect to be able to talk to that server before I do any of my testing. Logically I would assume I should be able to have a wait strategy that depends on that connection being made.
Test fix
I created a fork and updated exposeHostPorts to setup a lifecycle hook on PreCreates instead of PostReadies. This ensures the host port is accessible via the SSH container to the testcontainer from all lifecycle hooks and container command.
In theory this shouldn’t break anything even if someone sets up the listener for the host port in a later lifecycle hook as connections back on the host port are only established once you try connecting to the remote port.
testcontainers-go.patch
diff --git a/port_forwarding.go b/port_forwarding.go
index 88f14f2d..ad17fb10 100644
--- a/port_forwarding.go
+++ b/port_forwarding.go
@@ -150,8 +150,8 @@ func exposeHostPorts(ctx context.Context, req *ContainerRequest, ports ...int) (
// after the container is ready, create the SSH tunnel
// for each exposed port from the host.
sshdConnectHook = ContainerLifecycleHooks{
- PostReadies: []ContainerHook{
- func(ctx context.Context, c Container) error {
+ PreCreates: []ContainerRequestHook{
+ func(ctx context.Context, req ContainerRequest) error {
return sshdContainer.exposeHostPort(ctx, req.HostAccessPorts...)
},
},
Relevant log output
No response
Additional information
No response
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 22 (10 by maintainers)
Commits related to this issue
- setup ExposeHostPorts forwards on container start Fixes testcontainers/testcontainers-go#2811 Previously ExposedHostPorts would start an SSHD container prior to starting the testcontainer and inject... — committed to hathvi/testcontainers-go by hathvi 2 years ago
- setup ExposeHostPorts forwards on container start Fixes testcontainers/testcontainers-go#2811 Previously ExposedHostPorts would start an SSHD container prior to starting the testcontainer and inject... — committed to hathvi/testcontainers-go by hathvi 2 years ago
I just came across this issue yesterday as I was exploring a similar problem with
HostAccessPortsand the SSH tunnel not being set up in time - I too was depending on the host network being accessible during container startup.The DNS solution works great and is certainly simpler, so thanks for that @stevenh !
My $.02 –
IMO, if the support for
HostAccessPortsis to be kept around in spite of this, then I do think the fix provided by @hathvi is a logical one - I had implemented a similar custom container lifecycle hook to get things working the way we needed before coming across this issue.If indeed there is no justification for continuing to support
HostAccessPortsother than backwards compatibility, then it would be great to see the docs updated to point out the availability ofhost.docker.internaland perhaps even to illustrate its use with an example like the one here.I was not able to get WSL2 running within VirtualBox last night, but I got things set up on a laptop this AM and can confirm your test works for me on Windows as well.
Based on the outputs you provided in your last post this should technically not be allowed unless something in the middle is doing something with the network packets such as NAT or proxying to the host. It looks like WSL2 uses NAT by default, but doesn’t allow this behavior itself.
server.go
The WSL2 documentation even makes explicit mention of this https://learn.microsoft.com/en-us/windows/wsl/networking#connecting-via-remote-ip-addresses
I tried digging more into how Docker is running under WSL2 but at first glance doesn’t seem as straight forward as I was hoping. I’ll try digging more into this later tonight.
I don’t contribute as much to open source as I should mainly due to anxiety socializing with others, but I’d be down to contributing the suggested changes if that seems like a good idea to you guys.
Are you two by chance on Mac or Windows? I think Windows, under Docker Desktop, routes traffic to the host a bit differently then Mac or Linux. I use Linux as my primary OS and am not super familiar with Windows. The test that @stevenh provided does not work for me, which I’ll describe why below.
I’d preface that my original problem with testcontainers was that it was non obvious how to access the host using testcontainers. I however was also not aware of the docker extra hosts config and my searching around testcontainers lead me to
HostAccessPortswhich I found to be setting up the listeners to late.Docker by default uses a bridge network. It creates a new interface on the host named
docker0with it’s own subnet, in my case this is172.17.0.1/16with the interface address being172.17.0.1. When you configure extra hosts tohost-gatewayit creates a DNS record in /etc/hosts tohost.docker.internal 172.17.0.1.When you create a new
httptest.Serverit by default binds to127.0.0.1on a random port of the machine you are running the app on, there’s no direct way of changing this behavior and you must swap out thehttptest.Server.Listenerto bind to a different interface. In order for the DNS solution to work thehttptest.Serverwould need to bind to172.17.0.1( the container gateway / bridge interface ).HostAccessPortsrequests the SSH server bind the remote port in the container to127.0.0.1:PORTand tunnel traffic back to the client which in turn dials127.0.0.1:PORT( thehttptest.Server) on the client machine and copies those bytes from the tunnel to the server. If I were to swap out thehttptest.Server.Listenerto bind to172.17.0.0.1I’d have the same issue withHostAccessPortsbut DNS would work.My guess would be that you guys are likely on Windows and/or running Docker Desktop which likely does something similar to what
HostAccessPortsis doing and/or Windows allows connecting to a port bound on127.0.0.1from any interface. I don’t have a Windows VM currently to do any testing.I would say
HostAccessPortsdoes solve a problem, especially since people are generally going to be using this for testing and I assumehttptest.Serverwould be a somewhat common use case for people testing HTTP clients, however I don’t think this is a problem testcontainers necessarily needs to resolve as it feels out of scope, it’s just helpful that it does.I would also say that if
HostAccessPortswere kept it might be nice to extend its functionality to change which interface the local client dials, working more inline with how thessh -Rworks. e.g.,ssh -R [local-addr:]local-port:remote-addr:remote-port destination. Right now testcontainers doesn’t expose the ability to setlocal-addrorremote-addrand hard codes these tolocalhost( here for the remote listener, and here for the local forwarding client ). I assume this would result in a breaking change however as you’d likely need something likeHostAccessPorts []stringinstead of[]intand support values likelocal-addr:port/tcp,port/tcp( defaults to 127.0.0.1 ),local-addr:port( defaults to tcp ),port( defaults to 127.0.0.1 and tcp )