ruff: Ruff fails with memory allocation of X bytes failed
Hi guys,
for some time now, ruff has been failing with error “memory allocation of 7453010365139656813 bytes failed” if I run it for a specific file with option --force-exclude
. Without it, is works fine, but includes excluded files for which it runs. It is working fine when checking the whole project, but I also use it among file watchers in Pycharm where it fails. I am using a custom config with some file exclusions, so in order to avoid ruff reports for excluded files, I need --force-exclude
option. However, the file watcher has not been working properly for me, not from start, but for some time now, could be weeks or months. It only happens in one of my bigger projects, so it does not seem broken, maybe only demanding somehow. It does not bother me much, because it is so fast I can run it for the whole project instantly. 😃
Python: 3.8 Ruff: 0.0.292
Do you have any idea what could be causing this? Thanks
About this issue
- Original URL
- State: closed
- Created 8 months ago
- Reactions: 5
- Comments: 38 (18 by maintainers)
Thank you all for your help and reproductions! Your reproductions and error reports allowed me to reproduce the bug locally and I now have a fix ready for review.
I created a small script that deserializes the cache and prints the current offset. This can be useful to investigate where the file is corrupt.
The problematic cache file is
3276897697753128385
. The deserialization works fine up to theModuleImport
at offset 8144:The next entry import entry (index 17) starts at offset 8181 but the length is only one byte instead of 8 (I can tell because what starts after is the name
typing.pvi.device.TextRead
)I suspect that we run into some form of race and the file only gets written partially. This is possible because we use a
BufWriter
that chunks the write calls. However, POSIX gives no guarantees that multiplewrite
calls from the same process are atomic. It only guarantees that a single write call is atomic (to some extent).I think a first good starting point to fix this is to write the cache to a temporary file and then rename it when writing was successful. This doesn’t just help with potential races but also protects about corrupted cache files in case the ruff process dies in the middle of writing the cache (for whatever reason).
I noticed that yesterday too. It can happen if the caches are incompatible. I’m surprised you run into this because the ruff version is part of the cache key: Different ruff versions should never use the same cache. At least, that’s how it’s supposed to work.
Can you try running
ruff clean
. This should fix your problem momentarily and isn’t a long-term solution.Do I understand it correctly that you use a pre-built ruff version (you’re not building ruff yourself)?
I also ran into this problem and
ruff clean
fixed it.I was not switching between Ruff versions (all stables on ruff 0.1.2) but was running Ruff over and over. At some seemingly-random point I ran ruff and got the error:
The absolute path to the broken checkout is
/scratch/development/pvi
.I currently have this error, so I am reporting here in the hope it is useful.
Running with
--no-cache
works fine.I tried copying (
cp -R
) the project directory to see if I could reopen it in a new devcontainer and reproduce the error. In the new environment it worked fine, but it was using the version 0.2.1, whereas my existing environment was using 0.1.14. Updating my existing environment to 0.2.1 fixes the issue and rolling back to 0.1.14 breaks it again. Rolling back the new environment to 0.1.14 does not cause the issue. I also tried making another copy of the project and reopened in devcontainer with ruff pinned to 0.1.14, but this did not cause the problem either.The project is open source. I have made a branch from the checkout I currently see the issue in, including a zipped cache from the existing environment (sad) and the new environment (happy) in the root of the repo.
Something else that could might be relevant. In the current state of that branch, there is a formatting issue that black and ruff disagree on (
ruff format
disagrees with black,ruff --fix
andruff check
are both happy with what black does, I think this is expected). So, maybe it is something to do with checking black formatting compatibility?Given I can’t even reproduce the error by copying the entire failing project directory I am not sure if checking out the project will help, but maybe the cache will be useful. I will leave the broken checkout untouched in case there are any other diagnostics that could be done with it.
Thanks @rumbarum I suspect that we run into some sort of concurrency issue that corrupts the cache. I’ll play a bit with file watchers to see if I can corrupt my local cache.
@MichaReiser Currently,
.ruff_cache
is empty and no error. I run ruff without --no-cache and gather caching. When error happen again I will touch you.@rumbarum would you be able to share the repro file and the state of your cache directory so that we could try to reproduce locally?
I had encountered same error. ls 0.1.13 CACHEDIR.TAG
ruff 0.1.13 python 3.10.13 mac_os 14.1.2
My command,
ruff clean
not workingrm -rf 0.1.13
not working--no-cache
workedI am using ruff for globally not related for 1 project. Any request is welcomed.
I am also running into the same issue, but in my case
ruff clean
didn’t solve it. What solved in my case was running:rm -rf /path/to/project/.ruff_cache/0.1.*
since inside
.ruff_cache
I had all of this:0.1.11 0.1.13 0.1.7 0.1.8 CACHEDIR.TAG
Our team is working with a fairly large repo and we’re getting the same issue.
Again, running
ruff clean
fixes the issue.