android: LetsEncrypt SSL Cert produces: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

Steps to Reproduce

  1. Have a certificate with 2 verification paths as explained here

  2. Register that certificate on a webserver

  3. Create an HttpWebRequest to with the webserver URL from step 2

         HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(address);
         request.Accept = "application/json";
         request.Method = "GET";
         using var response = await request.GetResponseAsync().ConfigureAwait(false); // throws exception
    

This only happens with LetsEncrypt certificates that were signed with the expired certificate DST Root CA X3. Our SSL certificate was issued in August 2021 with the dual signature. It is not an issue for Apple iOS or iPadOS

Chrome has an issue with the certificate on older devices, but not on recent devices Viewing the certificate in windows browsers showed the valid path Viewing the certificate on old emulators showed the invalid path and failed to be trusted On devices that chrome showed the certificate as valid, Xamarin Android app still failed to trust the certificate Certificate worked until September 29th when the DST Root CA X3 certificate expired

Work Around: Renewing the certificate with LetsEncrypt Acme after Sept 30th 2021 fixed the problem

Expected Behavior

SSL Works, web request succeeds

Actual Behavior

Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

Version Information

Microsoft Visual Studio Enterprise 2019 Version 16.11.2 VisualStudio.16.Release/16.11.2+31624.102 Microsoft .NET Framework Version 4.8.04084

Installed Version: Enterprise

Visual C++ 2019 00435-60000-00000-AA537 Microsoft Visual C++ 2019

ADL Tools Service Provider 1.0 This package contains services used by Data Lake tools

ASA Service Provider 1.0

ASP.NET and Web Tools 2019 16.11.75.64347 ASP.NET and Web Tools 2019

ASP.NET Web Frameworks and Tools 2019 16.11.75.64347 For additional information, visit https://www.asp.net/

Azure App Service Tools v3.0.0 16.11.75.64347 Azure App Service Tools v3.0.0

Azure Data Lake Node 1.0 This package contains the Data Lake integration nodes for Server Explorer.

Azure Data Lake Tools for Visual Studio 2.6.1000.0 Microsoft Azure Data Lake Tools for Visual Studio

Azure Functions and Web Jobs Tools 16.11.75.64347 Azure Functions and Web Jobs Tools

Azure Stream Analytics Tools for Visual Studio 2.6.1000.0 Microsoft Azure Stream Analytics Tools for Visual Studio

C# Tools 3.11.0-4.21403.6+ae1fff344d46976624e68ae17164e0607ab68b10 C# components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

Common Azure Tools 1.10 Provides common services for use by Azure Mobile Services and Microsoft Azure Tools.

Extensibility Message Bus 1.2.6 (master@34d6af2) Provides common messaging-based MEF services for loosely coupled Visual Studio extension components communication and integration.

Fabric.DiagnosticEvents 1.0 Fabric Diagnostic Events

IntelliCode Extension 1.0 IntelliCode Visual Studio Extension Detailed Info

Microsoft Azure HDInsight Azure Node 2.6.1000.0 HDInsight Node under Azure Node

Microsoft Azure Hive Query Language Service 2.6.1000.0 Language service for Hive query

Microsoft Azure Service Fabric Tools for Visual Studio 16.10 Microsoft Azure Service Fabric Tools for Visual Studio

Microsoft Azure Stream Analytics Language Service 2.6.1000.0 Language service for Azure Stream Analytics

Microsoft Azure Stream Analytics Node 1.0 Azure Stream Analytics Node under Azure Node

Microsoft Azure Tools for Visual Studio 2.9 Support for Azure Cloud Services projects

Microsoft Continuous Delivery Tools for Visual Studio 0.4 Simplifying the configuration of Azure DevOps pipelines from within the Visual Studio IDE.

Microsoft JVM Debugger 1.0 Provides support for connecting the Visual Studio debugger to JDWP compatible Java Virtual Machines

Microsoft Library Manager 2.1.113+g422d40002e.RR Install client-side libraries easily to any web project

Microsoft MI-Based Debugger 1.0 Provides support for connecting Visual Studio to MI compatible debuggers

Microsoft Visual C++ Wizards 1.0 Microsoft Visual C++ Wizards

Microsoft Visual Studio Tools for Containers 1.2 Develop, run, validate your ASP.NET Core applications in the target environment. F5 your application directly into a container with debugging, or CTRL + F5 to edit & refresh your app without having to rebuild the container.

Microsoft Visual Studio VC Package 1.0 Microsoft Visual Studio VC Package

Mono Debugging for Visual Studio 16.10.15 (552afdf) Support for debugging Mono processes with Visual Studio.

NuGet Package Manager 5.11.0 NuGet Package Manager in Visual Studio. For more information about NuGet, visit https://docs.nuget.org/

ProjectServicesPackage Extension 1.0 ProjectServicesPackage Visual Studio Extension Detailed Info

Razor (ASP.NET Core) 16.1.0.2122504+13c05c96ea6bdbe550bd88b0bf6cdddf8cde1725 Provides languages services for ASP.NET Core Razor.

Snapshot Debugging Extension 1.0 Snapshot Debugging Visual Studio Extension Detailed Info

SQL Server Data Tools 16.0.62107.28140 Microsoft SQL Server Data Tools

Test Adapter for Boost.Test 1.0 Enables Visual Studio’s testing tools with unit tests written for Boost.Test. The use terms and Third Party Notices are available in the extension installation directory.

Test Adapter for Google Test 1.0 Enables Visual Studio’s testing tools with unit tests written for Google Test. The use terms and Third Party Notices are available in the extension installation directory.

ToolWindowHostedEditor 1.0 Hosting json editor into a tool window

TypeScript Tools 16.0.30526.2002 TypeScript Tools for Microsoft Visual Studio

Visual Basic Tools 3.11.0-4.21403.6+ae1fff344d46976624e68ae17164e0607ab68b10 Visual Basic components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

Visual F# Tools 16.11.0-beta.21322.6+488cc578cafcd261d90d748d8aaa7b8b091232dc Microsoft Visual F# Tools

Visual Studio Code Debug Adapter Host Package 1.0 Interop layer for hosting Visual Studio Code debug adapters in Visual Studio

Visual Studio Container Tools Extensions 1.0 View, manage, and diagnose containers within Visual Studio.

Visual Studio Tools for CMake 1.0 Visual Studio Tools for CMake

Visual Studio Tools for Containers 1.0 Visual Studio Tools for Containers

Visual Studio Tools for Kubernetes 1.0 Visual Studio Tools for Kubernetes

VisualStudio.DeviceLog 1.0 Information about my package

VisualStudio.Foo 1.0 Information about my package

VisualStudio.Mac 1.0 Mac Extension for Visual Studio

Xamarin 16.11.000.174 (d16-11@e8f56f1) Visual Studio extension to enable development for Xamarin.iOS and Xamarin.Android.

Xamarin Designer 16.11.0.17 (remotes/origin/11e0001f0b17269345e80b58fb3adf1ba4efe2cd@11e0001f0) Visual Studio extension to enable Xamarin Designer tools in Visual Studio.

Xamarin Templates 16.10.5 (355b57a) Templates for building iOS, Android, and Windows apps with Xamarin and Xamarin.Forms.

Xamarin.Android SDK 11.4.0.5 (d16-11/7776c9f) Xamarin.Android Reference Assemblies and MSBuild support. Mono: c633fe9 Java.Interop: xamarin/java.interop/d16-11@48766c0 ProGuard: Guardsquare/proguard/v7.0.1@912d149 SQLite: xamarin/sqlite/3.35.4@85460d3 Xamarin.Android Tools: xamarin/xamarin-android-tools/d16-11@683f375

Xamarin.iOS and Xamarin.Mac SDK 14.20.0.25 (3b53e529b) Xamarin.iOS and Xamarin.Mac Reference Assemblies and MSBuild support.

Log File

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 19
  • Comments: 84 (5 by maintainers)

Most upvoted comments

There is a client-side workaround.

On the Android you can manualy disable the certificate “Digital Signature Trust Co. - DST Root CA X3”.

  • Go to “Settings > Security > Encryption & credentials” > Trusted credentials"
  • Scroll down and disable “Digital Signature Trust Co. - DST Root CA X3”

In my case it works, but as I mentioned it’s a workaround, not the solution.

Scott Helme has a fantastic write-up of the situation. Go and read that first, then I’ll describe how (I think) this applies to xamarin-android.

I’m going to copy the key diagram from that article (source):

image

The red chain is what used to happen: the IdenTrust DST Root CA X3 is an old root certificate which is trusted pretty much everywhere, including on Android devices from 2.3.6 onwards. This is what LetsEncrypt used to use as their root, and it meant that everyone trusted them. However, this IdenTrust DST Root CA X3 recently expired, which means that a bunch of devices won’t trust anything signed by it. LetsEncrypt needed to move to their own root certificate.

The blue chain is the ideal new one – the ISRG Root X1 is LetsEncrypt’s own root certificate, which is included on Android 7.1.1+. Android devices >= 7.1.1 will trust certificates which have been signed by this ISRG Root X1.

However, the problem is that old pre-7.1.1 Android devices don’t know about ISRG Root X1, and don’t trust this.

The workaround which LetsEncrypt is using is that old Android devices don’t check whether the root certificate has expired. They therefore by default serve a chain which includes LetsEncrypt’s root ISRG Root X1 certificate (which up-to-date devices trust), but also include a signature from that now-expired IdenTrust DST Root CA X3. This means that old Android devices trust the chain (as they trust the IdenTrust DST Root CA X3, and don’t check whether it’s expired), and newer devices also trust the chain (as they’re able to work out that even though the root of the chain has expired, they still trust that middle ISRG Root X1 certificate as a valid root in its own right, and therefore trust it).

This is the green path, the one which LetsEncrypt currently serves by default.

However, the BoringSSL library used by xamarin-android isn’t Android’s SSL library. It 1) Doesn’t trust the IdenTrust DST Root CA X3 because it’s expired, and 2) Isn’t smart enough to figure out that it does trust the ISRG Root X1 which is also in the chain. So if you serve the green chain in the image above, it doesn’t trust it. Gack.

The options therefore are:

  1. Don’t use BoringSSL and do use Android’s SSL library. This means that xamarin-android behaves the same as other Android apps, and trusts the expired root. This is done by using AndroidClientHandler as described previously. This should fix Android >= 2.3.6.
  2. Do use BoringSSL but remove the expired IdenTrust DST Root CA X3 from Android’s trust store (“Digital Signature Trust Co. - DST Root CA X3” in the settings). This tricks BoringSSL into stopping its chain at the ISRG Root X1, which it trusts (on Android 7.1.1+). However this will only work on Android devices which trust the ISRG Root X1, which is 7.1.1+.
  3. Do use BoringSSL, and change your server to serve a chain which roots in the ISRG Root X1, rather than the expired IdenTrust DST Root CA X3 (the blue chain in the image above), using --preferred-chain "ISRG Root X1". This means that BoringSSL ignores the IdenTrust DST Root CA X3 entirely, and roots to the ISRG Root X1. This again will only work on Android devices which trust the ISRG Root X1, i.e. 7.1.1+.
  4. Do the same as 3, but by manually editing fullchain.pem.
  5. Use another CA such as ZeroSSL, which uses a root which is trusted back to Android 2.2, and which won’t expire until 2038.

I got my Xamarin app connections working again without any server or code changes. On each Android user device go to: Settings->Biometrics & security->Other security settings->View security certificates Find item: Digital Signature Trust Co “DST root ca x3” Disable this item Test your App, mine worked after this, tested on a few handsets and emulator on Android 8,9 and 10.

I’ve seen talk on the forums to use --preferred-chain "ISRG Root X1" if it makes sense for your devices. I’m not going to test it (partly because I’m not sure where I should put this when using cert-manager), but maybe it helps someone else 😛

https://github.com/acmesh-official/acme.sh/wiki/Preferred-Chain

That method solved it for me. It’s an option for the certbot cli which is used to get/renew let’s encrypt certificates. We use a dockerized version of it to automatically renew our certs. The full command for renewal looks like this certbot renew --force-renewal --preferred-chain "ISRG Root X1". This will essentially result in a cert chain that is similar to the above “delete the old DST cert from the fullchain file” method.

I still can’t find a server-side solution without implementing AndroidClientHandler on the client. I find it somewhat disastrous on the part of Xamarin, no wonder a lot of people are switching to Flutter.

This works great. Question: Will this be inserted again by Certbot when it renews the cert?

I’m pretty sure that’s the case unless you change the config. --preferred-chain "ISRG Root X1" might be what you’re looking for, but I haven’t personally tested it.

sudo certbot renew --force-renewal --preferred-chain "ISRG Root X1" worked for me.

Looks like Electron came up with the proper fix to BoringSSL here: https://github.com/electron/electron/pull/31213, and BoringSSL had been patched here: https://bugs.chromium.org/p/boringssl/issues/detail?id=439. Although, to quote:

As for Mono, you’ll have to talk to them. They appear not to have updated their BoringSSL for five years, and have made their own changes at this point, not ones we support. They’ve effectively using their own forked TLS library at this point, not BoringSSL

I’m very concerned that we’ve heard nothing from the Xamarin maintainers on this. Other ecosystems at least put official guidance in place here.

You might have some other libraries like ffimageloading etc that you might use inside your app, so if you just use your own httpclienthandler that would solve only part of your connections problem. To resolve all in-app connectivity with a letsencrypt ssl certificate you’d need unfortunately make changes on server. If you have a nginx server follow this hint: https://github.com/electron/electron/issues/31212#issuecomment-931540391 My bet is it is viable for other linux servers too. This hack/fix is verified and working and we kept using letsencrypt. Basically you edit the fullchain.pem for the domain and just delete the last certificate block from it, then restart http server service.

It’s close to @Doug910 solution but on server and you know we couldn’t ask customers to deal with device settings, they’ll just uninstall the app that is not working and give you one star. 😃

I also faced the same issue after updating targetframework and xamarin form version. I got a hint from the solution making a customer HttpClientHandler for each platform and found that there is a way to config the HttpClient implementation for Xamarin.Android (Project Properties > Android Options > Advanced > HttpClient Implementation). I’ve changed this value from Default to Android, and everything worked well without any issue.

More details: https://docs.microsoft.com/en-us/xamarin/android/app-fundamentals/http-stack?tabs=windows

@angelru Note that if you still give a custom handler to your HttpClient constructor, this won’t use the native one despite the changes in your project properties. You need to use either HttpClient() to use your config or you specify the native handler by hand HttpClient(new AndroidClientHandler());

I don’t understand why I have to do this, how do I do it with the refit?

i’m using Refit too. i make dependency handler for Xamarin.Android.Net.AndroidClientHandler

IMyOwnNetService.cs

public interface IMyOwnNetService {
    HttpClientHandler GetHttpClientHandler();
}

MyOwnNetService.cs (android)

[assembly: Dependency(typeof(MyOwnNetService))]
public class MyOwnNetService: IMyOwnNetService {
    public HttpClientHandler GetHttpClientHandler() {
        return new Xamarin.Android.Net.AndroidClientHandler();
    }
}

MyOwnNetService.cs (ios)

[assembly: Dependency(typeof(MyOwnNetService))]
public class MyOwnNetService: IMyOwnNetService {
    public HttpClientHandler GetHttpClientHandler() {
        return new HttpClientHandler();
    }
}
var hc = new HttpClient(Xamarin.Forms.DependencyService.Get<IMyOwnNetService>().GetHttpClientHandler()) {
    BaseAddress = new Uri("YOUR BASE URL")  
};
var api=RestService.For<MyAPIService>(hc);

its working for me

Just met the same issue while our certificates on our API was renewed. Forcing again a renewal of the certificate does not fix the issue. This impacts ALL builds.

Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
  at /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/boringssl/ssl/handshake_client.c:1132
at Mono.Btls.MonoBtlsContext.ProcessHandshake () [0x00042] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/System/Mono.Btls/MonoBtlsContext.cs:220
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status, System.Boolean renegotiate) [0x000da] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/System/Mono.Net.Security/MobileAuthenticatedStream.cs:715
  at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus,bool)
  at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00000] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/System/Mono.Net.Security/AsyncProtocolRequest.cs:289
  at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x000fc] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/System/Mono.Net.Security/AsyncProtocolRequest.cs:223

Note : Using AndroidClientHandler solves the issue.

Workaround that worked for us was to edit the certs fullchain.pem from winacme and manually remove the last certification.

The one that says 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1 i:0 = Digital Signature Trust Co., CN = DST Root CA X3

@canton7 This is something we are actively working on and trying to validate. Even if we didn’t fork BoringSSL, we would still need to bump to the right version and make sure we didn’t regress / negatively impact anyone.

@robertgoodwin00 Xamarin is partially at fault I think – they’re the ones who bundled BoringSSL, apparently did some custom modifications (although I don’t know the extent or whether they impacted this), and haven’t updated it to fix this issue. I’ve no idea if they’re planning to fix this properly, and I’m somewhat worried that we haven’t heard anything at all from them.

Is there a way to use the Native httpclient in SignalR ?

This solution worked for me: I use SignalR, the server is on kubernetes with Cert-Manager add-on. I added the preferredChain option to the ClusterIssuer resource:

...
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    preferredChain: "ISRG Root X1" # ADD THIS LINE FOR NEW LET'S ENCRYPT CA CHAIN for SignalR!!
...

After deleting the old cert, the new one was accepted by SignalR 🎉

I just wanted to mention, doing the certbot renew --force-renewal --preferred-chain "ISRG Root X1" did indeed solve the issue for us.

It was an impactful hit to our GKE but it was worth it to break it for everyone for a few minutes while the cert was pulled and reissued with the chain reference to the DST cert removed.

Before

CONNECTED(00000003)
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
---
Certificate chain
 0 s:/CN=xxxxxxxxxxxxx.com
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
 2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3

After

CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = xxxxxxxxxxxxx.com
verify return:1
---
Certificate chain
 0 s:CN = xxxxxxxxxxxxx.com
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1

Other tidtibs folks might find useful:

Some versions of curl also can’t handle this (this was version curl 7.54.0)

* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: certificate has expired
* stopped the pause stream!
* Closing connection 0
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

And here’s a thread at Let’s Encrypt with lots of other information for those being impacted: https://community.letsencrypt.org/t/r3-intermediate-certificate-has-expired/160797

Yeah, we are now using ZeroSSL which was easier and cheaper to swap in without any changes to our ACME setup (digicert’s acme setup sounds annoying) and put us on a root that exists for those clients.

So SignalR doesn’t use any HttpClient? That’s too bad. I also found this which sounded too simple to be true: https://github.com/dotnet/runtime/issues/23666

This seems to work for me: #4688 (comment)

How is it different from this property?

<AndroidHttpClientHandlerType>Xamarin.Android.Net.AndroidClientHandler</AndroidHttpClientHandlerType>

I’m was also having the same issue but after some research trials and errors I found out the solution well it’s more like a workaround. The root certificate of Let’s encrypt is expired so just remove it from the yourcert-chain.prem.

  • Open the file in any text editor
  • Locate the last -----BEGIN CERTIFICATE----- at the end of the file
  • Delete it from -----BEGIN CERTIFICATE----- to -----END CERTIFICATE-----
  • Save and reload the nginx