google-cloud-dotnet: Inserting lots of data to Google BigQuery randomly throws an SSL Exception

👋🏻 G’Day!

Problem

An SSL error is randomly thrown while importing a lot of data into Google BigQuery.

Details

I’ve got a (once off) maintenance task which is trying to insert about 10mil rows into a Google BigQuery table. Code is nothing too crazy - because this is a once off import.

Randomly (well, it feels random) an SSL error thrown which crashes my app.

Authentication failed because the remote party sent a TLS alert: ‘DecryptError’.

I’ve added in some retrying ability (using Polly for .NET) and we’re back on track.

I’m not really sure how to reproduce it, but I’ve provided some info here to maybe help. It’s happened after 30/40k have been pushed up. Other times, hundreds of thousands pushed up. With Polly, it retries and it works again/continues … until the next random failure … which Polly retries OK and we rinse/repeat.

Environment details

PS C:\Users\justi> dotnet --info
.NET SDK:
 Version:   7.0.304
 Commit:    7e794e2806

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.22621
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\7.0.304\

Host:
  Version:      7.0.7
  Architecture: x64
  Commit:       5b20af47d9

• Package name and version: Google.Cloud.BigQuery.V2 v3.2.0

Here’s some sample code I’ve got which is doing this:

// Configure the retry policy with Polly
var retryPolicy = Policy.Handle<Exception>()
    .WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)), (exception, timeSpan, retryCount, context) =>
    {
        // Log the exception to the console
        Console.WriteLine($"Retry #{retryCount} after {timeSpan.TotalSeconds} seconds due to: {exception}");
    });

try
{
    // Execute the retry policy for the HTTP request
    var pollyResult = await retryPolicy.ExecuteAsync(async () =>
    {
        var result = await client.InsertRowsAsync(destinationTable, rows, null, cancellationToken);

        return result;
    });


    _logger.LogDebug("Finished sending all rows to Google BigQuery. Status: {bigQueryInsertStatus}", pollyResult.Status);

    pollyResult.ThrowOnAnyError();

}
catch (Exception exception)
{
    // Log the final exception to the console
    Console.WriteLine($"Error occurred after retries: {exception}");

    throw;
}