streamlit: file_uploader with large file results in server getting killed
Checklist
- I have searched the existing issues for similar issues.
- I added a very descriptive title to this issue.
- I have provided sufficient information below to help reproduce this issue.
Summary
I am using st.file_uploader to upload ~3.5gb file into my streamlit dashboard. This however, results in the server getting killed. When I inspect using btop, Upon file upload, the server starts using an astronomic amount of RAM.
EDIT: I tried imposing a RAM limit by dockerizing the dashboard and settting a 4gb RAM limit, but it just gets killed in the container.
Reproducible Code Example
import streamlit as st
uploaded_file = st.file_uploader("Choose a video file", type=["mp4", "mov", "avi"])
Steps To Reproduce
streamlit run dashboard.py --maxUploadSize=5000
Expected Behavior
It’s probably better to stream it in chunks to disk when clicking an ‘upload file’ button.
Current Behavior
The server gets killed.
Is this a regression?
- Yes, this used to work in a previous version.
Debug info
- Streamlit version:
- Python version:
- Operating System:
- Browser:
Additional Information
No response
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 18
- Comments: 15 (1 by maintainers)
I’m facing the very same issue here (server killed when uploading 3.5GB files) and I could gather more information about it.
The problem I see is double, and it may point at a memory leak, or a lack of cache clearance as mentioned in #2408:
To verify it, I created a script (below) that:
And what I can observe is that:
Here is a snapshot of what this looks like after uploading a 1GB file (process memory usage was at 47MiB at cold start):
And every time I re-upload the file the process memory increases by 1 additional GB.
Ok, after some more debugging I figured out a workaround to fix both problems: the duplicate memory usage and the persistence between executions: In order to prevent the objects from staying in memory you need to
Doing it like this: