azure-functions-python-worker: MemoryErrors Began Recently, Breaking our Python Function App

Update - The updated memory config is available in East Asia now. It will be deployed to rest of the regions tomorrow.

Currently consumption sku doesn’t support > 1.5 GB workloads. App service plan function apps should though…

We expect to have an additional ~600MB available to the container.

Please notify us, @balag0, when the memory allowance has been increased to 1.5 GB and Azure Functions becomes usable for us again. Or why not make it 3 GB, like λ has? 😉

We expect to have an additional ~600MB available to the container.

I will try to add a wiki page to the host repo soon once I have more information. The limits will be configured such that the function app will be able to use all of 1.5 GB (minus whatever the functions runtime takes). We will account for the 400MB and adjust the overall limit accordingly if required.

The limit of 1.5 GB is just for the commit size which is the same limit as what we already have on Windows consumption sku. Is your concern that there is 0 swap space available? Even if we enabled swap space the overall limit of 1.5 GB will apply for that also.

On windows skus apps should have atleast 500MB of temp storage available to them as documented at https://github.com/projectkudu/kudu/wiki/Understanding-the-Azure-App-Service-file-system You can expect to see the same limits for Linux also.

Thanks for the detailed information @ericdrobinson (got the name right this time 😃 ). This was very helpful.

Turns out this was due to a bad configuration setting on our side which limited the amount of memory allocated to the container. Unfortunately it would be sometime next time before we can fix this.

Once the configuration is fixed, you should start seeing more available memory on function startup.

As for the memory usage being so high at startup, I see the following values with a different (may be simpler) function app Proc Mem: rss=33562624, vms=664834048 So it is possible additional memory usage you see is due to the nature of the function app itself.

Even with paged memory, the app still cannot exceed the 1.5 GB limit. So swap memory being 0 shouldn’t matter as long as you get 1.5 GB (less memory taken by Functions runtime process).

I’m back with more information.

I decided to take a look at what’s going on with the memory in the function environment. I added the psutil module and wrote a simple informational function:

def log_context_stats():
    """
    Retrieves and logs certain stats about the operating context for
    this process.
    """

    proc = psutil.Process(os.getpid())
    mem_info = proc.memory_info()

    stats = f"""Function Invocation Stats:
            Num CPUs: {os.cpu_count()}
            Virt Mem: {psutil.virtual_memory()}
            Swap Mem: {psutil.swap_memory()}
            Proc Mem: rss={mem_info.rss}, vms={mem_info.vms}
            """

    logging.info(stats)

I then called this function in two locations:

1. Immediately upon function invocation.
2. Immediately after catching the initial MemoryError.

Here is the output* from location 1:

Function Invocation Stats:
Num CPUs: 2
Virt Mem: svmem(
    total=1,039,429,632,
    available=240,713,728,
    percent=76.8,
    used=641,794,048,
    free=75,522,048,
    active=683,995,136,
    inactive=177,491,968,
    buffers=15,876,096,
    cached=306,237,440,
    shared=253,952,
    slab=67,407,872)
Swap Mem: sswap(total=0, used=0, free=0, percent=0, sin=0, sout=0)
Proc Mem:
    rss=144,080,896,
    vms=1,122,570,240

And here is the output* from location 2:

Function Invocation Stats:
Num CPUs: 2
Virt Mem: svmem(
    total=1,039,429,632, 
    available=39,194,624, 
    percent=96.2, 
    used=893,681,664, 
    free=79,187,968, 
    active=860,884,992, 
    inactive=22,732,800, 
    buffers=413,696, 
    cached=66,146,304, 
    shared=253,952, 
    slab=40,701,952) 
Swap Mem: sswap(total=0, used=0, free=0, percent=0, sin=0, sout=0) 
Proc Mem: 
    rss=304,037,888, 
    vms=1,570,119,680

(* Newlines and thousands separators added to numbers for readability.)

First, I should note that the swap_memory is showing 0 for everything. This might be due to the fact that the function is run in a Docker container and special settings are needed to access that information in these environments. Alternatively, it could be that someone turned off swap.

Second, what I found extremely startling is that a fresh run of the function showed only ~240MB of virtual_memory available. When the MemoryError was hit, the available memory had dropped by almost exactly 200MB (to ~39MB). Similarly, the process’ virtual memory starts at ~1.1GB and is already at ~1.57GB by the time the app runs out of memory.

It is perhaps valuable to recall at this point that the audio file being processed is just over the 100MiB line that causes the memory error to trigger when fully decompressed. I have no idea if the audio file is just large enough to run us out of available memory with array allocations/reallocations or if there is something otherwise special about the ~100MiB number.

Regardless, there is much about this state of affairs that is disconcerting.

• Why is there only ~240MB available when the function starts up fresh?
• Perhaps more telling/pertinent, is the process’ virtual memory. Why does it start at over 1GB?
• Is swap still enabled? Even if we ran out of RAM, I would expect the process to slow down as it shuffled memory over to disk (of which there should be plenty of space). I had previously assumed that it was swap memory that saved us with our massive files back in March.

While having no swap memory might help explain the situation we’re experiencing, it is pretty shocking to see such high memory usage numbers on a clean start of a relatively simple function app instance.

I hope this information is helpful in tracking down the source of the issue.