puma-dev: DNS resolution not working on macOS with "bind: operation not permitted"

Puma dev’s DNS resolution doesn’t appear to work at all anymore on MacOS Catalina.

I’ve done puma-dev -cleanup and even uninstalled/reinstalled from homebrew, then sudo puma-dev -setup; puma-dev -install again.

dig @127.0.0.1 -p 9253 blah.test

; <<>> DiG 9.10.6 <<>> @127.0.0.1 -p 9253 blah.test
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

However everything looks good otherwise. Puma is running, resolver file in place, responds to HTTP but not DNS on port

$ ls -l /etc/resolver 
total 8
-rw-r--r--  1 thom  staff    55B Mar 24 10:22 test

$ cat /etc/resolver/test 
# Generated by puma-dev
nameserver 127.0.0.1
port 9253

$ ps ax|grep puma
39752   ??  S      0:00.04 /usr/local/Cellar/puma-dev/0.13/bin/puma-dev -launchd -dir ~/.puma-dev -d test -timeout 15m0s

$ cat ~/.puma-dev/blah
2999

$ curl -vH "Host: blah.test" localhost
*   Trying ::1...
* TCP_NODELAY set
* Connection failed
* connect to ::1 port 80 failed: Connection refused
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: blah.test
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 500 Internal Server Error
< Date: Wed, 25 Mar 2020 13:18:25 GMT
< Content-Length: 0
< 
* Connection #0 to host localhost left intact
* Closing connection 0

$ tail -n20 ~/Library/Logs/puma-dev.log
2020/03/24 10:22:09 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/thom/.puma-dev
* Domains: test
* DNS Server port: 9253
* HTTP Server port: inherited from launchd
* HTTPS Server port: inherited from launchd
! Puma dev listening on http and https
* Generated proxy connection for 'blah' to http://127.0.0.1:2999
2020/03/25 09:18:21 http: proxy error: dial tcp 127.0.0.1:2999: connect: connection refused

Summary: HTTP proxy works, but DNS does not.

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Comments: 27 (13 by maintainers)

Most upvoted comments

I just hit this as well, and wanted to report that the suggested firewall whitelisting worked, but only with the physical path to puma-dev (homebrew sets a symlink from /usr/local/bin/puma-dev to /usr/local/Cellar/puma-dev/0.14/bin/puma-dev). Just going to leave this here in case anyone else hits the same issue:

sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/local/Cellar/puma-dev/0.14/bin/puma-dev
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblockapp /usr/local/Cellar/puma-dev/0.14/bin/puma-dev

Ah. Excellent find @stevenkaras! Adding a bit of portability for new versions, since 0.15 is right around the corner…

PUMA_DEV_BIN_PATH="$(brew --prefix puma/puma/puma-dev)/bin/puma-dev" # /usr/local/Cellar/puma-dev/0.14/bin/puma-dev
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add "$PUMA_DEV_BIN_PATH"
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblockapp "$PUMA_DEV_BIN_PATH"

I suspect that every participant in this thread is having a slightly different issue.

@nonrational - fair point! I tried on another machine today and was not able to reproduce.

I tried restarting my machine today and the issue is resolved. 🤷 Thanks for you help!

I suspect it’s because puma-dev is not signed

I’ve had puma-dev working without issue for a long time. Today it stopped working 😦 (possibly related to me doing brew update and brew upgrade today)

I’ve tried uninstalling (/etc/resolver/ becomes empty) then reinstalling:

$ puma-dev -V
Version: 0.16.0 (go1.16.2)

$ sw_vers
ProductName:	macOS
ProductVersion:	11.3.1
BuildVersion:	20E241

$ sudo puma-dev -setup
Password:
* Configuring /etc/resolver to be owned by me
* Changing '/etc/resolver/test' to be owned by me

$ puma-dev -install   
2021/05/07 17:09:43 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Use '/usr/local/bin/puma-dev' as the location of puma-dev
* Installed puma-dev on ports: http 80, https 443

But then in ~/Library/Logs/puma-dev.log:

2021/05/07 17:09:43 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/me/.puma-dev
* Domains: test
* DNS Server port: 9253
* HTTP Server port: inherited from launchd
* HTTPS Server port: inherited from launchd
! Puma dev running...
! DNS Server failed: listen tcp 127.0.0.1:9253: bind: operation not permitted

Note that here it shows the DNS port not binding.

However a bit different if I run in the foreground:

$ puma-dev
2021/05/07 17:13:35 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/me/.puma-dev
* Domains: test
* DNS Server port: 9253
* HTTP Server port: 9280
* HTTPS Server port: 9283
! Puma dev running...
2021/05/07 17:13:35 ! HTTP Server failed: listen tcp 127.0.0.1:9280: bind: operation not permitted

Here the HTTP port isn’t binding.

I have no issue binding an HTTP port e.g. with python3 -m http.server 34999. Only puma-dev has an issue it seems.

I have tried repeating all this after running this:

PUMA_DEV_BIN_PATH="$(brew --prefix puma/puma/puma-dev)/bin/puma-dev"
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add "$PUMA_DEV_BIN_PATH"
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblockapp "$PUMA_DEV_BIN_PATH"

Didn’t help.

Same issue with custom ports:

puma-dev -dns-port 32000 -http-port 34999 -https-port 34998
2021/05/07 17:18:25 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/me/.puma-dev
* Domains: test
* DNS Server port: 32000
* HTTP Server port: 34999
* HTTPS Server port: 34998
! Puma dev running...
! HTTPS Server failed: listen tcp 127.0.0.1:34998: bind: operation not permitted
2021/05/07 17:18:25 ! HTTP Server failed: listen tcp 127.0.0.1:34999: bind: operation not permitted

Any advice or suggestions?

@nonrational I use zsh so at first I couldn’t get your sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblockapp $(which puma-dev) to work because I think it was getting confused by the symlink action (installed via brew, so lots of symlinks). I replaced the $(which puma-dev) with the actual location of the binary and suddently (after a few puma-dev restarts) everything is working again.

For those using homebrew, the binary is somewhere like this: /usr/local/Cellar/puma-dev/0.13/bin/puma-dev but YMMV based on homebrew install and version of puma-dev, etc.

ProductName:    Mac OS X
ProductVersion: 10.15.4
BuildVersion:   19E287

Finished latest security patches, still not having any issues with port bindings with firewall and gatekeeper enabled. ☹️

@swiknaba I’m upgrading my mojave machine w/ latest security patches now and will attempt to reproduce.

For anyone who can reproduce the issue and is comfortable at the command-line, I’d ask that you try to disable gatekeeper temporarily and see if that fixes the port binding issue. That, at least, will allow us to test the unsigned hypothesis.

sudo spctl --master-disable will disable gatekeeper (and bring back the previous “allow from anywhere” option in system preferences).

sudo spctl --master-enable will then re-enable gatekeeper, once you’re done testing.

For some reason when I try to run on OSX, puma-dev can’t seem to bind to any HTTP/S port…?

puma-dev -dns-port 32000 -http-port 34999 -https-port 34998
2020/03/27 09:57:38 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/thom/.puma-dev
* Domains: test
* DNS Server port: 32000
* HTTP Server port: 34999
* HTTPS Server port: 34998
! Puma dev listening on http and https
2020/03/27 09:57:38 Error listening: listen tcp 127.0.0.1:34999: bind: operation not permitted

This may be part of the root cause of the issue I’m facing. Although the HTTP bind doesn’t fail when running as a launch daemon, but then again in that case HTTP/S are being forwarded via launchd socket.

Confirmed that DNS port bind will fail silently. I have a WIP branch that will expose the error messages and bail.