keyvault-acmebot: Trouble with retriable failures that run into ACME rate limits using Cloudflare DNS

Describe the bug I’m using Cloudflare DNS, mostly for wildcard certificates but also a few regular ones. The implementation of #256 has greatly helped, since I often encountered failures with Cloudflare. Today, I got an email from Let’s Encrypt about the pending expiry of one of my certificates, so I digged a bit deeper into it. Essentially, it looks like the when DNS validation is pending (as far as I understand, it could not be completed fast enough), a retry is scheduled and an error is logged. This seems to quickly run into Let’s Encrypts rate limit, but I’m not sure I completely understand the problem yet. I try to give as much context as I can😀

This morning, I had 5 Slack notifications from Keyvault Acmebot, in this order:

1. Null Reference Error (Exception 1 at the end)
2. ACME rate limit (Exception 2 at the end)
3. Retriable error in CheckIsReady (Exception 3 at the end)
4. Notification about successful renewal (problably for another domain?)
5. Notification about a successful renewal (again another domain)

The monitoring from CheckIsReady looks like this. The errors I’m reporting here are the ones happening around 2021-04-05 00:00 UTC time.

The error logs have two different forms, either this: Or like that:

To Reproduce

Just initially configure a certificate, the error happens during automatically scheduled renewal.

Environment (please complete the following information):

• Certificate Type: Mostly wildcard certificates, with some sub domain certificates
• Certificate Deploy Target: Azure KeyVault

Additional context

I was able to manually renew the certificates via renew-certificates a few hours later today. At least I think, since the error message didn’t indicate which certificates failed😀 Checking my Slack channel for earlier logs, it looks like it’s always behaving like this - certificate renewal often fails, but is usually sometime successful before Let’s Encrypt sends out an email reminder.

Exception 1

Microsoft.Azure.WebJobs.Extensions.DurableTask.FunctionFailedException: The activity function 'Order' failed: "Object reference not set to an instance of an object.". See the function execution logs for additional details.
---> System.NullReferenceException: Object reference not set to an instance of an object.
  at ACMESharp.Protocol.AcmeProtocolClient.DecodeResponseErrorAsync(HttpResponseMessage resp, String message, String opName)
  at ACMESharp.Protocol.AcmeProtocolClient.SendAcmeAsync(Uri uri, HttpMethod method, Object message, HttpStatusCode[] expectedStatuses, Boolean skipNonce, Boolean skipSigning, Boolean includePublicKey, CancellationToken cancel, String opName)
  at ACMESharp.Protocol.AcmeProtocolClient.GetNonceAsync(CancellationToken cancel)
  at KeyVault.Acmebot.Internal.AcmeProtocolClientFactory.CreateClientAsync() in /home/runner/work/keyvault-acmebot/keyvault-acmebot/KeyVault.Acmebot/Internal/AcmeProtocolClientFactory.cs:line 56
  at KeyVault.Acmebot.Functions.SharedActivity.Order(IReadOnlyList`1 dnsNames) in /home/runner/work/keyvault-acmebot/keyvault-acmebot/KeyVault.Acmebot/Functions/SharedActivity.cs:line 112
  at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync(Object instance, Object[] arguments) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs:line 52
  at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker, ParameterHelper parameterHelper, CancellationTokenSource timeoutTokenSource, CancellationTokenSource functionCancellationTokenSource, Boolean throwOnTimeout, TimeSpan timerInterval, IFunctionInstance instance) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 555
  at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance, ParameterHelper parameterHelper, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 501
  at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 279
  --- End of inner exception stack trace ---
  at Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableOrchestrationContext.CallDurableTaskFunctionAsync[TResult](String functionName, FunctionType functionType, Boolean oneWay, String instanceId, String operation, RetryOptions retryOptions, Object input, Nullable`1 scheduledTimeUtc) in D:\a\r1\a\azure-functions-durable-extension\src\WebJobs.Extensions.DurableTask\ContextImplementations\DurableOrchestrationContext.cs:line 710
  at KeyVault.Acmebot.Functions.SharedOrchestrator.IssueCertificate(IDurableOrchestrationContext context) in /home/runner/work/keyvault-acmebot/keyvault-acmebot/KeyVault.Acmebot/Functions/SharedOrchestrator.cs:line 25
  at Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2.InvokeAsync(TReflected instance, Object[] arguments) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\VoidTaskMethodInvoker.cs:line 20
  at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync(Object instance, Object[] arguments) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs:line 52
  at Microsoft.Azure.WebJobs.Extensions.DurableTask.TaskOrchestrationShim.InvokeUserCodeAndHandleResults(RegisteredFunctionInfo orchestratorInfo, OrchestrationContext innerContext) in D:\a\r1\a\azure-functions-durable-extension\src\WebJobs.Extensions.DurableTask\Listener\TaskOrchestrationShim.cs:line 150

Exception 2

Microsoft.Azure.WebJobs.Extensions.DurableTask.FunctionFailedException: The activity function 'Order' failed: "Failed to deserialize exception from TaskActivity: {"$type":"ACMESharp.Protocol.AcmeProtocolException, ACMESharp","ProblemType":15,"ProblemTypeRaw":"urn:ietf:params:acme:error:rateLimited","ProblemDetail":"Rate limit for '/acme' reached","ProblemStatus":-1,"StackTrace":"   at ACMESharp.Protocol.AcmeProtocolClient.SendAcmeAsync(Uri uri, HttpMethod method, Object message, HttpStatusCode[] expectedStatuses, Boolean skipNonce, Boolean skipSigning, Boolean includePublicKey, CancellationToken cancel, String opName)\r\n   at ACMESharp.Protocol.AcmeProtocolClient.CreateOrderAsync(IEnumerable`1 identifiers, Nullable`1 notBefore, Nullable`1 notAfter, CancellationToken cancel)\r\n   at KeyVault.Acmebot.Functions.SharedActivity.Order(IReadOnlyList`1 dnsNames) in /home/runner/work/keyvault-acmebot/keyvault-acmebot/KeyVault.Acmebot/Functions/SharedActivity.cs:line 114\r\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync(Object instance, Object[] arguments) in C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionInvoker.cs:line 52\r\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker, ParameterHelper parameterHelper, CancellationTokenSource timeoutTokenSource, CancellationTokenSource functionCancellationTokenSource, Boolean throwOnTimeout, TimeSpan timerInterval, IFunctionInstance instance) in C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs:line 555\r\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance, ParameterHelper parameterHelper, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs:line 501\r\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs:line 279","Message":"Rate limit for '/acme' reached","Data":{"$type":"System.Collections.ListDictionaryInternal, System.Private.CoreLib"},"InnerException":null,"HelpLink":null,"Source":"ACMESharp","HResult":-2146233088}". See the function execution logs for additional details.
---> DurableTask.Core.Exceptions.TaskFailedExceptionDeserializationException: Failed to deserialize exception from TaskActivity: {"$type":"ACMESharp.Protocol.AcmeProtocolException, ACMESharp","ProblemType":15,"ProblemTypeRaw":"urn:ietf:params:acme:error:rateLimited","ProblemDetail":"Rate limit for '/acme' reached","ProblemStatus":-1,"StackTrace":"   at ACMESharp.Protocol.AcmeProtocolClient.SendAcmeAsync(Uri uri, HttpMethod method, Object message, HttpStatusCode[] expectedStatuses, Boolean skipNonce, Boolean skipSigning, Boolean includePublicKey, CancellationToken cancel, String opName)\r\n   at ACMESharp.Protocol.AcmeProtocolClient.CreateOrderAsync(IEnumerable`1 identifiers, Nullable`1 notBefore, Nullable`1 notAfter, CancellationToken cancel)\r\n   at KeyVault.Acmebot.Functions.SharedActivity.Order(IReadOnlyList`1 dnsNames) in /home/runner/work/keyvault-acmebot/keyvault-acmebot/KeyVault.Acmebot/Functions/SharedActivity.cs:line 114\r\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync(Object instance, Object[] arguments) in C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionInvoker.cs:line 52\r\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker, ParameterHelper parameterHelper, CancellationTokenSource timeoutTokenSource, CancellationTokenSource functionCancellationTokenSource, Boolean throwOnTimeout, TimeSpan timerInterval, IFunctionInstance instance) in C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs:line 555\r\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance, ParameterHelper parameterHelper, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs:line 501\r\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs:line 279","Message":"Rate limit for '/acme' reached","Data":{"$type":"System.Collections.ListDictionaryInternal, System.Private.CoreLib"},"InnerException":null,"HelpLink":null,"Source":"ACMESharp","HResult":-2146233088}
---> Newtonsoft.Json.JsonSerializationException: Unable to find a constructor to use for type ACMESharp.Protocol.AcmeProtocolException. A class should either have a default constructor, one constructor with arguments or a constructor marked with the JsonConstructor attribute. Path 'ProblemType', line 1, position 77.
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateNewObject(JsonReader reader, JsonObjectContract objectContract, JsonProperty containerMember, JsonProperty containerProperty, String id, Boolean& createdFromNonDefaultCreator)
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
  at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
  at DurableTask.Core.Serializing.JsonDataConverter.Deserialize(String data, Type objectType) in C:\source\durabletask\src\DurableTask.Core\Serializing\JsonDataConverter.cs:line 104
  at DurableTask.Core.Serializing.DataConverter.Deserialize[T](String data) in C:\source\durabletask\src\DurableTask.Core\Serializing\DataConverter.cs:line 54
  at DurableTask.Core.Common.Utils.RetrieveCause(String details, DataConverter converter) in C:\source\durabletask\src\DurableTask.Core\Common\Utils.cs:line 401
  --- End of inner exception stack trace ---
  --- End of inner exception stack trace ---
  at Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableOrchestrationContext.CallDurableTaskFunctionAsync[TResult](String functionName, FunctionType functionType, Boolean oneWay, String instanceId, String operation, RetryOptions retryOptions, Object input, Nullable`1 scheduledTimeUtc) in D:\a\r1\a\azure-functions-durable-extension\src\WebJobs.Extensions.DurableTask\ContextImplementations\DurableOrchestrationContext.cs:line 710
  at KeyVault.Acmebot.Functions.SharedOrchestrator.IssueCertificate(IDurableOrchestrationContext context) in /home/runner/work/keyvault-acmebot/keyvault-acmebot/KeyVault.Acmebot/Functions/SharedOrchestrator.cs:line 25
  at Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2.InvokeAsync(TReflected instance, Object[] arguments) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Execut…

Exception 3

Microsoft.Azure.WebJobs.Extensions.DurableTask.FunctionFailedException: The activity function 'CheckIsReady' failed: "ACME validation status is invalid, but retriable error. It will retry automatically.". See the function execution logs for additional details.
---> KeyVault.Acmebot.RetriableOrchestratorException: ACME validation status is invalid, but retriable error. It will retry automatically.
  at KeyVault.Acmebot.Functions.SharedActivity.CheckIsReady(ValueTuple`2 input) in /home/runner/work/keyvault-acmebot/keyvault-acmebot/KeyVault.Acmebot/Functions/SharedActivity.cs:line 309
  at Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2.InvokeAsync(TReflected instance, Object[] arguments) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\VoidTaskMethodInvoker.cs:line 20
  at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync(Object instance, Object[] arguments) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs:line 52
  at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker, ParameterHelper parameterHelper, CancellationTokenSource timeoutTokenSource, CancellationTokenSource functionCancellationTokenSource, Boolean throwOnTimeout, TimeSpan timerInterval, IFunctionInstance instance) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 555
  at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance, ParameterHelper parameterHelper, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 501
  at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 279
  --- End of inner exception stack trace ---
  at Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableOrchestrationContext.CallDurableTaskFunctionAsync[TResult](String functionName, FunctionType functionType, Boolean oneWay, String instanceId, String operation, RetryOptions retryOptions, Object input, Nullable`1 scheduledTimeUtc) in D:\a\r1\a\azure-functions-durable-extension\src\WebJobs.Extensions.DurableTask\ContextImplementations\DurableOrchestrationContext.cs:line 710
  at KeyVault.Acmebot.Functions.SharedOrchestrator.IssueCertificate(IDurableOrchestrationContext context) in /home/runner/work/keyvault-acmebot/keyvault-acmebot/KeyVault.Acmebot/Functions/SharedOrchestrator.cs:line 43
  at Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2.InvokeAsync(TReflected instance, Object[] arguments) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\VoidTaskMethodInvoker.cs:line 20
  at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync(Object instance, Object[] arguments) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs:line 52
  at Microsoft.Azure.WebJobs.Extensions.DurableTask.TaskOrchestrationShim.InvokeUserCodeAndHandleResults(RegisteredFunctionInfo orchestratorInfo, OrchestrationContext innerContext) in D:\a\r1\a\azure-functions-durable-extension\src\WebJobs.Extensions.DurableTask\Listener\TaskOrchestrationShim.cs:line 150