azure-functions-host: 64bit v3 runtime fails with out memory exceptions where 32bit runtime does not

We have had a ~2 .netcorev2.2 functions app running for some time. It was running on 64 bit runtime. After upgrading to ~3 functions frequently fail with out of memory exceptions when using the 64bit runtime. The 32bit runtime does not fail.

The exceptions seem to always occur in json deserialization.

Investigative information

Timestamp: 12/31/2019, 9:50:18 AM (Local time)
Function App version: 3.0
Function App name:
Function name(s):
Region: UK South
Invocation ID:

INFORMATION9:50:07.056 AM
Executing 'api-Payments' (Reason='This function was programmatically called via the host APIs.', Id=7e6c2c73-89ac-4881-b4f3-b6de55f23eb7)

Applications Insight logging:

// All telemetry for Operation ID: 7cc2acc54214dc439fd9ac8d0c774da2
union *
| where timestamp > datetime("2019-12-30T17:50:18.720Z") and timestamp < datetime("2020-01-01T17:50:18.720Z")
| where operation_Id == "7cc2acc54214dc439fd9ac8d0c774da2"

Repro steps

Switch to 64bit runtime and exercise the API.

Expected behaviour

No exceptions

Actual behavior

Out of memory exceptions raised

Known workarounds

Use 32bit runtime

Related information

FSharp language
Source available privately
HttpTrigger
Suspect the issue is with the customized json.net deserialization.

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 6
Comments: 25 (7 by maintainers)

Most upvoted comments

We have found an infrastructure issue that is causing OutOfMemoryException, it is affecting functions. We have a repro of the issue and are actively working to find the root cause.

soninaren on Jan 7, 2020

A fix for this is currently rolling out. We’ll update and close the issue once deployment is completed.

fabiocav on Jan 22, 2020

For me, my scenario is a lot simpler and always threw OutOfMemoryException. I have a single TimerTrigger function which downloads a ~55Mb file, compresses it and writes to a blob. This has run successfully for over a year - I upgraded it to Azure Functions ~3 and selected 64-bit. When it fails, it is it is usually at about the 20Mb range on the line await response.Content.ReadAsStreamAsync().

Complete code, no databases or anything complex processing, just a single download from HTTP. Problem goes completely when 32-bit.

var response = await client.GetAsync(uri);
if (response.IsSuccessStatusCode)
{
    await using var uploadContent = new MemoryStream();
    await using var compressStream = new GZipStream(uploadContent, CompressionLevel.Optimal);

    var downloadContent = await response.Content.ReadAsStreamAsync();
    downloadContent.CopyTo(compressStream);
    uploadContent.Position = 0;

    var blob = container.GetBlockBlobReference(date);
    await blob.UploadFromStreamAsync(uploadContent);
}

Simplified version that just read into a memory stream also showed the same problem. Input file has was never more than 60Mb. When 64-bit, always OutOfMemoryException, when 32-bit, works every time.

var response = await Client.GetAsync(uri, HttpCompletionOption.ResponseHeadersRead);
await using var content = new MemoryStream(await response.Content.ReadAsByteArrayAsync());

yarseyah on Jan 7, 2020

Rollout for the fix is complete now. Closing the issue.

soninaren on Feb 6, 2020

I have the same issue as well:

There are bunch of function apps that were operating normally for over a year on v2 runtime and .NET Core 2.1 x64
Recently I switched them to v3 runtime + .NET Core 3.1 x64 and some of them started to fail with OutOfMemory exception - there were functions with different triggers and different load so for now I don’t see any common pattern which may cause such behavior
I still face this issue even with FUNCTIONS_V2_COMPATIBILITY_MODE = true in the App Settings
With “switching to 32 bit platform” workaround everything works ok

@fabiocav please let me know if you still need some extra info for the investigation.

myarotskaya on Jan 3, 2020