ruff: Ruff fails with memory allocation of X bytes failed

Thank you all for your help and reproductions! Your reproductions and error reports allowed me to reproduce the bug locally and I now have a fix ready for review.

I created a small script that deserializes the cache and prints the current offset. This can be useful to investigate where the file is corrupt.

The problematic cache file is 3276897697753128385. The deserialization works fine up to the ModuleImport at offset 8144:

        module import entry 16@8144
[crates/ruff/tests/cache_issue.rs:199] self.read_len() = 21
          module: pvi.device.TextFormat
          range: 287..297

The next entry import entry (index 17) starts at offset 8181 but the length is only one byte instead of 8 (I can tell because what starts after is the name typing.pvi.device.TextRead)

@8144: 15 00 00 00 00 00 00 00 (length of name)
@8152: 70 76 69 2E 64 65 76 69 63 65 2E 54 65 78 74 46 6F 72 6D 61 74 (import name: pvi.device.TextFormat)
@8173: 1F 01 00 00 (range start) 
@8177: 29 01 00 00 (range end)
@8181: 13 74 79 70 69 6E 67 2E 70 76 69 2E 64 65 76 69 63 65 2E 54 65 78 74 52 65 61 64 (next module entry
@8181: typing.pvi.device.TextRead (the line above in text)

I suspect that we run into some form of race and the file only gets written partially. This is possible because we use a BufWriter that chunks the write calls. However, POSIX gives no guarantees that multiple write calls from the same process are atomic. It only guarantees that a single write call is atomic (to some extent).

I think a first good starting point to fix this is to write the cache to a temporary file and then rename it when writing was successful. This doesn’t just help with potential races but also protects about corrupted cache files in case the ruff process dies in the middle of writing the cache (for whatever reason).

I noticed that yesterday too. It can happen if the caches are incompatible. I’m surprised you run into this because the ruff version is part of the cache key: Different ruff versions should never use the same cache. At least, that’s how it’s supposed to work.

Can you try running ruff clean. This should fix your problem momentarily and isn’t a long-term solution.

Do I understand it correctly that you use a pre-built ruff version (you’re not building ruff yourself)?

I also ran into this problem and ruff clean fixed it.

I was not switching between Ruff versions (all stables on ruff 0.1.2) but was running Ruff over and over. At some seemingly-random point I ran ruff and got the error:

poetry run ruff . --fix
memory allocation of 7453001542924727087 bytes failed
memory allocation of 8675468274863120500 bytes failed
make: *** [ruff] Abort trap: 6

The absolute path to the broken checkout is /scratch/development/pvi.

I currently have this error, so I am reporting here in the hope it is useful.

❯ ruff check
memory allocation of 3343762647483511827 bytes failed
Aborted (core dumped)

Running with --no-cache works fine.

I tried copying (cp -R) the project directory to see if I could reopen it in a new devcontainer and reproduce the error. In the new environment it worked fine, but it was using the version 0.2.1, whereas my existing environment was using 0.1.14. Updating my existing environment to 0.2.1 fixes the issue and rolling back to 0.1.14 breaks it again. Rolling back the new environment to 0.1.14 does not cause the issue. I also tried making another copy of the project and reopened in devcontainer with ruff pinned to 0.1.14, but this did not cause the problem either.

The project is open source. I have made a branch from the checkout I currently see the issue in, including a zipped cache from the existing environment (sad) and the new environment (happy) in the root of the repo.

Something else that could might be relevant. In the current state of that branch, there is a formatting issue that black and ruff disagree on (ruff format disagrees with black, ruff --fix and ruff check are both happy with what black does, I think this is expected). So, maybe it is something to do with checking black formatting compatibility?

Given I can’t even reproduce the error by copying the entire failing project directory I am not sure if checking out the project will help, but maybe the cache will be useful. I will leave the broken checkout untouched in case there are any other diagnostics that could be done with it.

Thanks @rumbarum I suspect that we run into some sort of concurrency issue that corrupts the cache. I’ll play a bit with file watchers to see if I can corrupt my local cache.

@MichaReiser Currently, .ruff_cache is empty and no error. I run ruff without --no-cache and gather caching. When error happen again I will touch you.

@rumbarum would you be able to share the repro file and the state of your cache directory so that we could try to reproduce locally?

I had encountered same error. ls 0.1.13 CACHEDIR.TAG

ruff 0.1.13 python 3.10.13 mac_os 14.1.2

My command,

/opt/homebrew/bin/ruff --config ~/ruff.toml --fix  /path/to/repo/a-file.py

ruff clean not working rm -rf 0.1.13 not working --no-cache worked

I am using ruff for globally not related for 1 project. Any request is welcomed.

I am also running into the same issue, but in my case ruff clean didn’t solve it. What solved in my case was running: rm -rf /path/to/project/.ruff_cache/0.1.*

since inside .ruff_cache I had all of this: 0.1.11 0.1.13 0.1.7 0.1.8 CACHEDIR.TAG

Our team is working with a fairly large repo and we’re getting the same issue.

Again, running ruff clean fixes the issue.