configure-aws-credentials: OpenIDConnect provider's HTTPS certificate doesn't match configured thumbprint
Hey,
For the past hour or so I’ve been getting this error while using this action:
Error: OpenIDConnect provider's HTTPS certificate doesn't match configured thumbprint
I’ve checked the thumbprint and it’s correctly set, would this be an issue with GitHub, are there any known issues at the moment?
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 105
- Comments: 96 (9 by maintainers)
Commits related to this issue
- Update fingerprint following certificate rotation (#3) Certificate has been rotated, fingerprint has changed: https://github.com/aws-actions/configure-aws-credentials/issues/357 — committed to hectcastro/terraform-aws-github-actions-oidc by danial-k 2 years ago
- Fetch GitHub OIDC thumbprint directly aws-actions/configure-aws-credentials#357 — committed to br3ndonland/terraform-aws-github-actions-oidc by br3ndonland 2 years ago
- Fetch GitHub OIDC thumbprint directly aws-actions/configure-aws-credentials#357 — committed to br3ndonland/terraform-examples by br3ndonland 2 years ago
- Fetch GitHub OIDC thumbprint directly aws-actions/configure-aws-credentials#357 — committed to br3ndonland/terraform-examples by br3ndonland 2 years ago
For those using terraform to manage the OIDC provider in AWS:
Nice of them to give us plenty of warning on this.
Seems like GH updated their SSL certificate. You should go to IAM identity providers, find github connect and add new fingerprint (f879abce0008e4eb126e0097e46620f5aaae26ad):
PS: Huge thanks @micahhausler for the one-liner
I think this was just posted: https://github.blog/changelog/2023-06-27-github-actions-update-on-oidc-integration-with-aws/, looks like there are multiple thumbprints now, as we suspected. I will update our documentation.
Hello! @wellsiau-aws beat me to announcing this slightly, but I have some changes to share: the IAM team have released an update for thumbprint configuration for GitHub (and only GitHub at this time) that will permanently solve this issue!
As of July 6, 2023, it is no longer necessary to specify a specific thumbprint in your IAM Identity Provider configuration. AWS will secure communication with tokens.actions.githubusercontent.com using our library of trusted CAs rather than using a certificate thumbprint. You should see a message in your AWS account if you look at the configured identity provider confirming that this is the case.
The API still appears to require that some thumbprint is configured, since that will require other API and UI changes, but your thumbprint will be completely ignored and OIDC will still work. The message in the IAM console will persist to remind you of this. I successfully tested this out in my own AWS account by setting my configured thumbprint to
afafafafafafafafafafafafafafafafafafafafand was able to get authenticated.Note: this change will not affect GHES customers, since they have a different hostname and issuer from their IdP.
You should not need to make any changes to your existing configuration with this change. Please open up a separate issue in this repo if you’re continuing to have problems with the OIDC configuration, or reach out to AWS Support.
I found a tweet.
https://twitter.com/arkadiyt/status/1481418230082248714
I could obtain the new thumbprint as the AWS official doc https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html.
It seems the new thumbprint is
6938fd4d98bab03faadb97b34396831e3780aea1.Hi everyone, let’s try and keep the discussion here related to this action only. The way thing are today:
6938fd4d98bab03faadb97b34396831e3780aea1,1c58a3a8518e8759bf075b76b750d4f2df264fcd.The path forward to making this a better experience involves changing how IAM works, or how GitHub is managing their certificates, or both. I don’t have any information to share today other than the IAM team is aware of this sharp edge and we want to make it better in the future. If I have anything to announce there, I’ll share that as soon as possible.
We’ve updated the readme and the example CloudFormation template for this action’s documentation. I’d also note that if you’re using the CDK OpenIdConnectProvider construct, you will need to manually configure thumbprints as well. If anyone has any suggestions on how to make our documentation and setup process clearer, we’d be happy to take that feedback.
👋 To expand on Andy’s comment, I’d like to answer a few questions raised.
@danieljamesscott This was a routine rotation due to upcoming expiration. As such, you do not need to remove the previous thumbprint, but may do so as it is no longer in use.
@stevie- This issue was caused by the intermediate CA changing in our certificate’s trust chain, which we don’t expect to necessarily occur during every SSL rotation/renewal. We’re investigating here to determine how to avoid this changing when possible, and will ensure adequate communication in the future if it does need to change and user action is required.
@buffyg Thanks for your thorough analysis, we’ll be looking into options here. The AWS documentation implies there may be some options here to reduce the need for customers to update the thumbprint themselves, although our engineering team will need to investigate if they are feasible.
Again, I’d like to apologize for this issue and reiterate that we are taking steps to ensure this doesn’t happen again.
Error: OpenIDConnect provider’s HTTPS certificate doesn’t match configured thumbprint
I am using thumbprint : 6938fd4d98bab03faadb97b34396831e3780aea1
type: OpenID Connect provider: https://token.actions.githubusercontent.com thumbprint : 6938fd4d98bab03faadb97b34396831e3780aea1 Audience: sts.amazonaws.com
Github promised last year they wouldn’t break this again https://github.blog/changelog/2022-01-13-github-actions-update-on-oidc-based-deployments-to-aws/
This seems like a regression.
In the ideal world, I like to see GitHub as identity provider that built-in into AWS IAM, similar to Facebook / Google / Cognito:
This will eliminate the need to configure separate IdP in IAM.
ok - I have a reasonable enough solution that I am going to go with…
This code is subideal in that there are hard coded values in there but given the fact that there is no way to be handed a list of all known thumbprints with a single call this is the best I can come up with until someone makes a module (hint hint AWS). If you run this it will set the world to a known state and likely never change again until you are unlucky enough to get a new thumbprint - even this code will not succeed 100% of the time getting a complete list but you are much better off running this than not running it as it has a non-zero chance of showing the change over enough runs.
In my perfect world this list would be curated by those that know - Either AWS or GH in the form of a TF module or at the very least an HTTPS endpoint with the data in a nice format.
While it’s always annoying to have something like this break deployments, I will say it’s quite nice that I was able to:
We’ve all had to deal with support hell and black boxes when it comes to issues and resolutions before. The community, collaboration, and transparency here is refreshing!
👋🏾 Thanks all for spotting this and sorry about the issue. We’ve issued a change log here: https://github.blog/changelog/2022-01-13-github-actions-update-on-oidc-based-deployments-to-aws/.
This is something we’ll be looking into to ensure this doesn’t happen again.
FYI you can also run this command:
Just remove the
:and lowercase everything@altjx has to be updated in AWS IAM --> Identity Providers -->(token.actions.githubusercontent.com) Find Thumbprints on right side and add new one
6938fd4d98bab03faadb97b34396831e3780aea1or through cloudformation/terraform is you are using IaC. But, also, I am expecting someGithub Official thumbrpint announcement https://github.blog/changelog/2023-06-27-github-actions-update-on-oidc-integration-with-aws/
If you’re looking for a one-liner to find the latest thumbprint, here’s one I came up with based on the IAM docs:
How to get new thumbprint in advance? This will break once every year.
Is there any way to fully automate this?
GitHub already publishes all the information needed programatically and in compliance with OIDC. Namely PKI. their jwks_uri is signed by a globally trusted root CA. AWS should perform TLS verification like everyone else in the world and not come up with alternative schemes like pinning intermediate certificates.
Only thing AWS needs to do is verify the certificate chain as per the TLS spec. That they don’t do this is extremely weird.
I am using the AWS guide to verify the thumbprint and when I request the certificates from GitHub with:
I get two certificates, the first certificate is always
f879abce0008e4eb126e0097e46620f5aaae26adbut the second one randomly changes between6938fd4d98bab03faadb97b34396831e3780aea1and1c58a3a8518e8759bf075b76b750d4f2df264fcd.I understand there’s a new thumbprint, but how exactly do we apply this to the github action?
Can we provide the new thumbprint here, or what exactly must be done? Does this get applied on the AWS side? A bit confused. but still researching.
Thanks @suzuki-shunsuke I’ve also noticed there’s a PR open already, however the thumbprints are different!
https://github.com/aws-actions/configure-aws-credentials/pull/355/files
I had updated our OIDC provider with the new fingerprints but still ran into the same issue.
Don’t forget to update not only the OIDC provider in the account where the runners are hosted, but also in other accounts from which roles are being assumed.
Reiterating https://github.com/aws-actions/configure-aws-credentials/issues/357#issuecomment-1610207576, it’s not a question of
certificates[0].sha1_fingerprintorcertificates[*].sha1_fingerprintor any variation on that theme. The server presents different CA certificates, at random, presumably depending on which backend you reach.If you
terraform applythe above configuration in a loop, it will flap depending on which intermediate GitHub’s server presents when Terraform opens the TLS connection.A search for the new fingerprint turns up a result in the Azure CA index so presumably this behavior is inherited from some Azure infrastructure in use behind the scenes.
The example above that uses the
httpprovider to obtain the well-known configuration and then makes a TLS connection to thejwks_uriin the response is the most correct way to implement AWS’s instructions for fingerprinting an OIDC provider in Terraform. In theory you can filter the certificates onis_cabut in practice GitHub is only returning one at a time and CA certificates should come first so there’s no actual difference today.The issue here is that there are 2 different intermediate certificates in use, depending on the request, both issued from the same root. And because AWS does not trust the root and requires users to explicitly trust the intermediates directly presented by the server, you end up with tight coupling and these breakage incidents.
Reopening this because of reported issues. We publish some documentation on how to verify OIDC endpoint certificates so let’s follow that.
jwks_urithat GitHub gives us is https://token.actions.githubusercontent.com/.well-known/jwks, taken from https://token.actions.githubusercontent.com/.well-known/openid-configurationOn my local machine, I get
6938FD4D89BAB03FAADB97B34396831E3780AEA1. This is the same thumbprint that GitHub says to use for AWS OIDC. I think GitHub doesn’t intend to change this considering the churn that happened last time.’I’m not sure what’s going on, but the reports by @dabbeg and @ixti seem to indicate that GitHub is giving out different intermediates, probably depending on what global endpoint you’re hitting. We’ll reach out to some folks on the GitHub side to see if we can find if anything is going on.
In the meantime, our recommendation is to always manually verify the intermediate thumbprint you’re getting from the OIDC endpoint. You can do that with the one/some liners I’ve used here, or the IAM documentation I’ve linked above. Once you’ve verified your endpoint thumbprints, you should be safe to add them to your IdP configuration.
something strange is happening just now… run from 1hr ago (thumb generated via terraform data)
Duplicate of https://github.com/aws-actions/configure-aws-credentials/pull/355.
@jakejscott that command is giving a different fingerprint because you take the first cert of the chain, not the last as doc suggests.
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html
@jensenbox here’s the after. I removed one of the fingerprints, and overwrote the other with the thumbprint you see. Still works!
@zarenner I’ve also talked to AWS about making the underling trust ceremony corporate as opposed to per-customer. They already trust a number of IdPs, suggested they do the same with you, the mechanics can be between your engineering orgs.
I suggested using WebPKI issuance pins, that’s what AWS says, which is then garbled by Terraform (looking at the AWS docs, it starts out talking about using CA pins, but the Terraform resource docs instead suggest these should be server thumbprints, not CA, “A list of server certificate thumbprints for the OpenID Connect (OIDC) identity provider’s server certificate(s)”). The AWS doc, which gives the procedure for relying on root CAs, is https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html and the Terraform resource doc is https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_openid_connect_provider.
I’ve since been pointed to a write-up from @sleevi on considerations for OIDC FastFed, suggesting that subsetting WebPKI issuance has viability issues and that Private PKI should be used instead. Also should be pointed out that DigiCert has reduced the lifetime of their ICAs precisely to discourage pinning via ICA: https://knowledge.digicert.com/alerts/DigiCert-ICA-Update.html
Sleevi’s notes are here: https://docs.google.com/document/d/e/2PACX-1vQQ-FhNjW0ZhZVTnK1VG_87IBNZKBaJmweYZb1VBRdQCMAWXekqKfxdNl8wL6FWFDJ9pxXxpr66-GZp/pub
How are you suggesting to retrieving that pin from GitHub? Through TLS. And we’re back to square one. WebPKI is safe. Certificate Transparency log makes abuse obvious. Pinning is unnecessary complexity.
The developers of this action won’t be able to provide any insights or statuses for that ticket, but our stance is that we hope a permanent solution will get done before the next time this happens
There is little GitHub can do to avoid this breaking again and again, this is AWS fault: #669
GitHub, and any OIDC provider, should be free to deploy any TLS certificate chains they wanted without notifying their users.
@jensenbox push AWS to solve that “Internal P82323250” issue. No automation should be needed when a CA trust chain is in place.
Also @arianvp is right, all this terraform automations only work accidentally. But jwks_uri’s domain doesn’t necessarily match openid-configuration’s domain.
Note that technically AWS only pins the
jwks_uriand thejwks_uricould be a different domain than theopenid-configurationfile. Not that I think the domain ofjwks_uriwill change. But better be safe than sorry. If you want to be pedantically correct you should do this:@zarenner mentioned wanting to explore ways to avoid having the issuing CA change. To echo and emphasize @buffyg’s comment, the WebPKI doesn’t provide that guarantee, somewhat intentionally.
DigiCert has been explicit about this, and it’s a huge credit to them for implementing organizational changes to help surface these risks during routine situations (such as rotations), where folks are aware and can respond to consequences, because the alternative is these sorts of things get surfaced during non-routine situations, like responses to CA compromise or distrust.
A big part of this philosophy was seeing how long it took organizations to successfully transition off certificates from DigiNotar, WoSign/StartCom, and Symantec, three CAs that demonstrated significant security control failures. For an unfortunately large number of users, they had trouble rotating certificates precisely because they had end users who had pinned.
Beyond such cases, though, the WebPKI ecosystem has intentionally been moving to shorter (effective) lifetimes to root certificates. When you use a 10-year old root, for example, you also inherit 10 years of any mistakes it has made potentially causing harm to your organization. The old Verizon PKI demonstrated this best: Verizon did not even know everyone they had issued sub-CAs to, any of which could have maliciously issued certificates that anyone who trusted the Verizon roots would accept. DigiCert, when they acquired Verizon, spent an incredible amount of time investigating and revoking those certificates, because that’s a huge risk.
As one of the authors of RFC 7469, and as one who helped drive removing support from Chrome, I’ve definitely been trying to help educate folks about the dangers of pinning. Since we’re engineers here, the way I would describe Certificate pinning is a bit like reaching into an opaque C struct: you’re reaching into implementation details that aren’t ABI or API guaranteed.
That is, if you had a
foo.hthat hadthen it would be inherently unsafe in your libraries
bar.cto dobecause you’re reaching in to the internal implementation details and “violating” the API contract.
Pinning to the WebPKI is basically that: reaching into implementation details of the trust graph that, by design, aren’t guaranteed to be stable. If you need a stable API, then private PKI let’s you define those contracts as you need.
Hopefully this doesn’t come off as a lecture 😅 It’s just very much “here be dragons”, and any pinning to the Web PKI is going to (eventually) explode, because just like software itself needs to get regularly updated to respond to security risks, so does the WebPKI.
Note: while Google does publish a root bundle at https://pki.goog, that’s not really for encouraging pinning, and you’ll note it’s a broad selection of CAs. However, even with that broad selection, there have still been operational issues where Google ends up getting a certificate outside of that set, and interoperability issues can come up. So publishing a set of trust anchors isn’t much of a solution: it’s primarily there to ensure at least a minimum set or “everyone Google may decide to do business with” is included.
That’s a “Pull Request” and this is an “Issue”
Does anyone know if this was this replaced due to expiration of the old cert, or a security issue? I’d like to know whether to add the new cert in addition to the old one, or should I replace the old thumbprint.
Confirmed this also worked for me, I have both
6938fd4d98bab03faadb97b34396831e3780aea1and1c58a3a8518e8759bf075b76b750d4f2df264fcdas thumbprints nowTechnically GitHub did nothing wrong here. Nowhere does OIDC spec mandate pinning to intermediate TLS certificates. But it’s annoying that GitHub was aware of the problem and broke it anyway without heads up.
We should. because if they’re doing pinning they’re not following OIDC by the letter. OIDC actually says
jwks_urican change at any time. Pinning the intermediate cert ofjwks_uriis really bad and not compliant with the spec at all@adudek blame is on AWS, not GitHub: https://github.com/aws-actions/configure-aws-credentials/issues/357#issuecomment-1611608543
Not trusting TLS certificate chains issued by CAs and requiring users to hardcode fingerprints of certificates they don’t operate is poor design by AWS.
This topic is a big blame on GitHubs architects side as they should be aware of how their choice of infrastructure will impact a critical part of the authentication mechanism implemented by ANY OIDC CAPABLE system. I’d expect at least GitHub to know what a contract is and what to offer when you reach “well known” endpoint…
Thanks for this suggestion, but this only grabs the first returned Well Known intermediary. Per this GitHub blog post, you must trust them all (currently 2), or your connection could be unstable.
Suggestion:
Glad to see I am not the only one who suddenly got this error 😅 Is there not some way for us to automate the thumbprint? Kinda hectic that all your workflows just fall over and you have to go manually change the thumbprint 🤔
There is a viable mitigation here: rotating certificates includes the possibility of changing trust chains in the course of that. What Github could do is publish the trust bundles for their expected CAs, users retrieve that bundle, extract the thumbprints for all valid roots. This is a list, so you can provide multiple fingerprints so that you can tolerate getting different CAs per connection while a rotation is happening, since this is a distributed consistency problem that shouldn’t be taken for an atomic operation for clients.
Publishing trust bundles is in fact what Google does for the GCP IdP, but here it seems that the AWS resource definition requires not just the trust vector but enumeration of all the certificates used. Trust vector and name really should be sufficient, but Github could still give users relief by publishing the certs with their chains.
This is preferable to having to look at CT to look for all issuance against the host name, retrieving all valid roots and passing those, as CT is a detective rather than a preventive control against incorrect issuance. If you want to do pinning right, which you generally should for an IdP, you need to verify the root of trust for first use, can use that root to pull the bundle thereafter for updates.
You could also pull published trust bundles then look at what’s in CT to determine, but this really points to trust qualification being fairly wonky in the OIDC/OAuth world, which generally expect you either to trust a name via whatever WebPKI universe you accept, or you trust a specific key. AWS wants the latter, the sane middle ground is that you want a name and a specific trust vector, such that you can give trust bundles as opposed to individual HTTPS certificate.
Whatever should be done, there’s a bunch of complexity that really should be hidden via an IdP datasource provider.
@kamranNBS were you able to fix this? I am getting the same
The JWKS endpoint is discoverable/well-known, and that returns the signing certificates, which OIDC providers can used to verify material those certificates signed, such as GitHub’s ID tokens. But the certificates contained in there have a subject name of
vsts-vstsghrt-gh-vso-oauth.visualstudio.comand a shorter expiration. They are not what we’re after here.The certificates in question here are the CA certificates that GitHub uses to secure HTTPS traffic to
https://token.actions.githubusercontent.com. Providing thumbprints ensures that even if someone managed to hijack thegithubusercontent.comdomain, AWS will only transmit token material for identification if it can verify that the server is actually the one it expects, via the thumbprint.The natural place for this information would seem to be the meta API:
https://docs.github.com/en/rest/meta/meta
You can use this API today to pre-establish SSH trust with GitHub by adding the public keys returned there:
https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints
If you could discover the Actions OIDC server’s CA certificate fingerprints from the meta API, you could do the same here. Right now, it is impossible to automate this reliably as there is no machine-readable way to get this information deterministically.
I can only assume it is because of services like EKS that use self-signed certificates for their control plane and OIDC endpoints.
The implication of GitHub’s blog post is that you cannot use
data.tls_certificatebecause they may serve you 1 of those 2 different CA certificates at random. A simple test configuration to prove that:If you run this in a loop, sometimes you get this:
And other times:
I had hardcoded the value before in my TF - this is what I just wrote to make it more dynamic.
What is interesting is that it totally removed the one from Jan 2022:
Which seems to jive with what people were seeing above.
Either way I believe the correct fingerprint to be used is:
f879abce0008e4eb126e0097e46620f5aaae26addid the order of the certs change? the fingerprints of the top/bottom certs are as follows
I think the actual certificates are the same.
What are the security implications of having the wrong fingerprint? The Idp queries will still go to token.actions.githubusercontent.com ?
Also out of interest why are we querying https://token.actions.githubusercontent.com/.well-known/openid-configuration rather than just https://token.actions.githubusercontent.com to get the cert information?
It works well for me by the way (using both urls), thanks!
Is there recommended course of action for mitigation? Spent an hour thinking I broke something until I found this thread.
FYI, even after 12h+ since the change, I’m still getting two different cert chains from the same hostname. Simple test:
In ~30% of cases, the chain has 3 certificates, instead of 2:
vs
Fortunatelly, this is not a problem for OIDC, since Amazon is getting the same cert chain every time. At least from what I have seen. This problem may be ISP/country specific.