streamlit: file_uploader with large file results in server getting killed

Checklist

  • I have searched the existing issues for similar issues.
  • I added a very descriptive title to this issue.
  • I have provided sufficient information below to help reproduce this issue.

Summary

I am using st.file_uploader to upload ~3.5gb file into my streamlit dashboard. This however, results in the server getting killed. When I inspect using btop, Upon file upload, the server starts using an astronomic amount of RAM.

EDIT: I tried imposing a RAM limit by dockerizing the dashboard and settting a 4gb RAM limit, but it just gets killed in the container.

Reproducible Code Example

import streamlit as st

uploaded_file = st.file_uploader("Choose a video file", type=["mp4", "mov", "avi"])

Steps To Reproduce

streamlit run dashboard.py --maxUploadSize=5000

Expected Behavior

It’s probably better to stream it in chunks to disk when clicking an ‘upload file’ button.

Current Behavior

The server gets killed.

Is this a regression?

  • Yes, this used to work in a previous version.

Debug info

  • Streamlit version:
  • Python version:
  • Operating System:
  • Browser:

Additional Information

No response

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 18
  • Comments: 15 (1 by maintainers)

Most upvoted comments

I’m facing the very same issue here (server killed when uploading 3.5GB files) and I could gather more information about it.

The problem I see is double, and it may point at a memory leak, or a lack of cache clearance as mentioned in #2408:

  1. The uploaded files exist twice in the process memory, which means that to upload 3.5GB files you need at least 7GB of free memory
  2. One of the copies perists after the script ends running, which means that uploaded files accumulate in memory until the memory is full

To verify it, I created a script (below) that:

  • Tells the process memory usage when the script starts
  • Lets you upload a file
  • Tells the memory usage after capturing the uploaded file
  • Deletes the uploaded file variable and collects garbage
  • Tells the memory usage after clearing the file
import gc
import os

import psutil
import streamlit as st


def humansize(num, suffix="B"):
    for unit in ("", "Ki", "Mi", "Gi", "Ti", "Pi", "Ei", "Zi"):
        if abs(num) < 1024.0:
            return f"{num:3.1f}{unit}{suffix}"
        num /= 1024.0

    return f"{num:.1f}Yi{suffix}"


def process_memory():
    process = psutil.Process(os.getpid())
    mem_info = process.memory_info()
    return humansize(mem_info.rss)


gc.collect()

st.title("File Upload Test")
st.write(f"Debug - initial memory usage: {process_memory()}")
uploaded_file = st.file_uploader("Choose a file to upload")
if uploaded_file is not None:
    st.write(f"Uploaded file size: {humansize(uploaded_file.size)}")

st.write(f"Memory usage after upload: {process_memory()}")

del uploaded_file
gc.collect()

st.write(f"Final memory usage after deleting the file: {process_memory()}")

And what I can observe is that:

  1. Memory grows by twice the file size every time a file is uploaded
  2. Half of the memory increase persists from run to run, and it accumulates.

Here is a snapshot of what this looks like after uploading a 1GB file (process memory usage was at 47MiB at cold start):

image

And every time I re-upload the file the process memory increases by 1 additional GB.

Ok, after some more debugging I figured out a workaround to fix both problems: the duplicate memory usage and the persistence between executions: In order to prevent the objects from staying in memory you need to

  1. Clear the UploadedFileManager session files
  2. Clear the session state
import streamlit.runtime.scriptrunner as scriptrunner

# Clearing the session files removes the file copy, reducing memory footprint by half
ctx = scriptrunner.get_script_run_ctx()
ctx.uploaded_file_mgr.remove_session_files(ctx.session_id)

uploaded_file = st.file_uploader("Choose a file to upload", key='file-upload')

# deleting the file_uploader input value after capturing it prevents it from persisting between runs.
del st.session_state['file-upload']

Doing it like this:

  1. When the file is uploaded the process memory is roughly the size of the uploaded file and just a bit more
  2. When the script is ended the memory has gone down to what it was before uploading the file