go: crypto/x509: Go 1.15 will break default connections to AWS RDS

CN verification is still specified by RFC 6125 as a fallback if SAN isn’t present.

Breaking external compatibility to prove a point seems excessive.

https://tools.ietf.org/html/rfc6125#section-6.4.4

Amazon fixed this on their side (💙) so I’m going to close this.

Here are some details for anyone else encountering the same x509: certificate relies on legacy Common Name field error.

This error means that the certificate has the requested hostname or IP in the Common Name field, and not in the Subject Alternative Names extension. Verifying Common Name is unspecified, and has been deprecated for 20 years. Go 1.15 dropped support for it now, most browsers did it a long time ago.

You can fix this in a few different ways.

1. If the certificate is operated by a third-party, you should ask them to fix their certificates, like Amazon did. (If you are using AWS RDS there are instructions here.)
2. If you operate the certificate, you should make sure it has SANs, for example by using code like in generate_certificate.go or a tool like mkcert. Let’s Encrypt has an example of how to use SANs with openssl.
3. You can temporarily re-enable the legacy behavior by setting the GODEBUG=x509ignoreCN=0 environment variable. This will be removed in Go 1.16, let us know if and why you need it.
4. If you actually need to check the Common Name field, you can use the new VerifyConnection callback to customize the verification process, not setting VerifyOptions.DNSName and doing a manual check instead.

The ideal resolution here is for AWS to use the correct field in their certificates. Let’s wait to hear from them, and then we can decide whether to back off this change to Go 1.16.

Note that this change doesn’t make it impossible to (manually) use CN for verification, it simply aligns with the spec in not considering it a hostname or an IP for VerifyHostname or VerifyOptions.DNSName purposes.

CN has been deprecated for 20 years, the specs do not recommend supporting it anymore, the browsers successfully dropped it, the WebPKI CAs are prohibited from setting it to anything that is not also in the SANs, and its behavior is completely unspecified.

The last point is a security issue with Name Constraint validation (which we support), because CN is not constrained (because it’s not a hostname), so if we consider it a hostname we allow NCs to be bypassed. We have workarounds but they are complex unspecified hacks (see #24151).

I’m just gonna say - I caught this because I deployed the 1.15 beta to our staging environment shortly after it was released. This gave us and AWS months of lead time between detection and the deadline for the release.

Please try to run the betas/RC’s in your staging environments so that the entire community can benefit from the issues you find there, before they go live to production. Thanks!

AWS have added a note to https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL-certificate-rotation.html about this issue:

If you are using a Go version 1.15 application with a DB instance that was created or updated to the rds-ca-2019 certificate prior to July 28, 2020, you must update the certificate again. Run the modify-db-instance command shown in the AWS CLI section using rds-ca-2019 as the CA certificate identifier. In this case, it isn’t possible to update the certificate using the AWS Management Console. If you created your DB instance or updated its certificate after July 28, 2020, no action is required. For more information, see Go GitHub issue #39568

Confirmed that the modify-db-instance fix described above works for AWS Postgres RDS and Go 1.15rc1.

@FiloSottile If it’s decided that this is going to be rolled back then it would be best to do it before RC1 is released.

I don’t think this needs to be rolled back based only on the AWS RDS breakage.

I do think (hopefully incorrectly) that we’ll see more breakage as more people test this, but if we roll it back based on my hunch of what will come out of RC1, then we’re not actually learning what breaks and we’ll never get to land this, because the same argument will apply unchanged for Go 1.16rc1.

As an update, we intend on releasing the RC with this change. There will be documentation in the release notes on how to get the old behavior while people work through issues. We’re keeping this issue labeled as a release-blocker for Go 1.15, pending feedback from more people testing.

@psanford I prefer not to call out specific third-party services in the release notes, but hopefully a web search leads them here easily enough.

I dropped a worked out example of VerifyConnection for Common Name checking in https://github.com/golang/go/issues/40748#issuecomment-673612108, by the way.

Having to set InsecureSkipVerify in order to verify the common name isn’t great.

@alex certainly didn’t expect to see your name chiming in! Based on the output of openssl s_client -starttls postgres database:5432 < /dev/null | openssl x509 -noout -text it turns out GovCloud is indeed including SANs on new DBs:

        X509v3 extensions:
            X509v3 Authority Key Identifier:
                keyid:41:BB:F9:BA:FC:2A:4B:F0:3D:DE:60:EF:54:6F:87:6B:2D:72:52:5B
                DirName:/C=US/L=Seattle/ST=Washington/O=Amazon Web Services, Inc./OU=Amazon RDS/CN=Amazon RDS GovCloud Root CA
                serial:10:00

            X509v3 Subject Key Identifier:
                09:9A:13:D6:08:99:E5:DD:3E:7F:CA:A6:B5:F3:71:AD:79:E3:1F:16
            X509v3 Subject Alternative Name:
                DNS:dbname.us-gov-west-1.rds.amazonaws.com

Then I ran the same command against an older DB and sure enough, no SAN.

Solution The solution is to run the exact same command linked in https://github.com/golang/go/issues/39568#issuecomment-665863345, but replacing rds-ca-2019 with rds-ca-2017, even if you’re already using rds-ca-2017. There appears to be a similar date cutoff as described in the AWS link, where new DBs are fine but DBs older than a certain date need to be updated (the DB I had to update was created 07-22-2020).

Having to set InsecureSkipVerify in order to verify the common name isn’t great.

You are opting out of hostname verification because the certificate does not have hostnames (as the Common Name is not a hostname), I think the “hic sunt leones” is justified. If you want to check Common Name on top of hostname verification you don’t have to set InsecureSkipVerify.

We run a 3-D Secure Server. The certificates used by the scheme HTTPS endpoints are universally issued by internal CAs.

This change will also break our connections to several major card schemes, since they do not include SANs in certificates.

I’ll grant you, that we will be able to work around it. My point is, there are likely many areas where this causes issues. Since there’s actually an RFC that allows for a fallback, I would appriciate if this was allowed.

Though I applaud your attempt to enforce a 20 year old deprecation 😃

On a sidenote:

These CA’s universally also encode nearly everything as ASN.1 PrintableString. This causes us heaps of trouble, much more than this would.

There’s been several issues regarding this: https://github.com/golang/go/issues?q=is%3Aissue+is%3Aclosed+printablestring

For applications that have no choice but to work around this new limitation (no clout to change service certificates, interoperability with legacy systems, stuck in some circle of enterprise hell, etc), I wonder if it’s too easy to get it wrong? It just feels weird to simultaneously say “I’m skipping verification” but also “verify the connection like this”. Which one wins? You can’t tell unless you read the implementation.

Maybe there’s no other good option without breaking APIs, and I get that legacy behavior can’t be supported forever, but it gives an uneasy feeling.

@FiloSottile can we add note in the release notes pointing to the instructions from AWS on how to generate a new cert? I suspect there will be a bunch of RDS users who don’t realize that they need to do this and will be sad when they upgrade go 1.15 and things stop working.