concourse: Fly and split DNS no longer working

Bug Report

Upgrading from Concourse/Fly 4.2.3 to 5.0.1 has introduced an issue where fly is no longer able to access our concourse instance, which is behind a VPN using split DNS.

Because of some of our security requirements, we can’t forward all DNS traffic through the VPN, and we would rather not add a private IP to a public DNS entry.

Steps to Reproduce

Run concourse in a private network
Run OpenVPN Access Server to allow connections into the private network
Attempt to use any fly command, specifically fly -t target login

Expected Results

login successful

Actual Results

could not reach the Concourse server called target:

Get https://concourse.internal-address/api/v1/info: dial tcp: lookup concourse.journera.net on <public dns ip>:53: no such host

is the targeted Concourse running? better go catch it lol

Additional Context

From my understanding, this is most likely because Go implemented their own DNS lookup system, which uses /etc/resolv.conf, instead of calling gethostbyname. Because of the split DNS, /etc/resolv.conf is not updated with the private DNS addresses, go is unable to lookup the address (it would work perfectly if it just used the cgo resolver).

The fact that it’s listing my DNS address instead of the VPN’s DNS address is telling, and is why I think it’s the go DNS resolver issue.

Version Info

Concourse version: 5.0.1
Deployment type (BOSH/Docker/binary): binary
Infrastructure/IaaS: AWS/Single server for web and worker/behind VPN
Did this used to work? yes, in 3.x and 4.x releases

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 24 (6 by maintainers)

Most upvoted comments

Updating thread with some specifics for posterity. ConcourseCI version 6.7.1 and macOS 11.1 are still affected by this same scenario.

aghassemlouei on Jan 5, 2021

I’m experiencing this same issue again with 5.7.2.

I’m on a Mac, targeting a Concourse instance behind a VPN with similar circumstances as @dkuntz2 originally described.

Rebuilding fly from source as mentioned above also fixes the problem for me.

Is this still a known issue?

jgriff on Dec 4, 2019

@archoversight ahh good call, sorry I didn’t notice your replies earlier. Will look into this asap! I wonder if we can work out a smoke test for this so it doesn’t regress again. 🤔

vito on Feb 2, 2021

I think the point of this ticket is to fix the official build. It is silly to force people to do this individually.

bacoboy on May 19, 2019

@eedwards-sk you can build your own copy of fly, you just need to run

CGO_ENABLED=1 go build github.com/concourse/concourse/fly

You should get a fly binary in $GOPATH/bin. If you want to build it somewhere else you can use the -o $path flag in the build command.

sleepdeprecation on May 19, 2019