coredns: plugin/route53: upstream no longer works when configured as self for recursion

I got around to trying out CoreDNS 1.4.0 today. I’m currently running 1.2.6 in production and my config works as I’d expect (well, with some already reported issues). I have a Route 53 zone gathered by the route53 plugin.

Corefile:

example.com {
  route53 example.com.:Z2CJ10OEFZZZZZ {
    upstream 127.0.0.1:53
    credentials default /configs/credentials
  }
}

. {
  forward . 127.0.0.1:253
}

Unbound is running on :253 for recursion. I have upstream pointed to use the port of coredns itself so that the CNAMEs being resolved will run through the full list of configured zones because a CNAME may point to an internal name (also proxied by coredns) or an external name (like in the case of an Amazon LB, which would be a recursive query).

Resolving a name that is an Amazon LB works on 1.2.6, but no longer works on 1.4.0.

Working query against 1.2.6:

# dig test.example.com @127.0.0.1 -p 53

; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> test.example.com @127.0.0.1 -p 53
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44400
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 4, ADDITIONAL: 0

;; QUESTION SECTION:
;test.example.com.		IN	A

;; ANSWER SECTION:
test.example.com.	4	IN	CNAME	test-2051344828.us-west-2.elb.amazonaws.com.
test-2051344828.us-west-2.elb.amazonaws.com. 4 IN A 1.2.3.4
test-2051344828.us-west-2.elb.amazonaws.com. 4 IN A 5.6.7.8

;; AUTHORITY SECTION:
example.com.		4	IN	NS	ns-1259.awsdns-29.org.
example.com.		4	IN	NS	ns-199.awsdns-24.com.
example.com.		4	IN	NS	ns-633.awsdns-15.net.
example.com.		4	IN	NS	ns-2014.awsdns-59.co.uk.

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Mon Mar 04 10:16:44 PST 2019
;; MSG SIZE  rcvd: 455

Same config, non-working query on 1.4.0:

# dig test.example.com @127.0.0.1 -p 353

; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> test.example.com @127.0.0.1 -p 353
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 27690
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 4c5c5aff50311445 (echoed)
;; QUESTION SECTION:
;test.example.com.		IN	A

;; Query time: 0 msec
;; SERVER: 127.0.0.1#353(127.0.0.1)
;; WHEN: Mon Mar 04 10:16:46 PST 2019
;; MSG SIZE  rcvd: 63

I’m not seeing anything show up in the container log aside from the initial loading messages. I’m guessing this is either related to EDNS or recent changes to plugins recursion availability.