poetry: lazy-wheel causes poetry to use all the RAM

[tool.poetry]
name = "repro"
version = "0.1.0"
description = ""
authors = []
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.10"
ruff = "^0.1.15"

[tool.poetry.group.jax.dependencies]
jaxlib = { extras = [
    "cuda12-pip",
], version = "^0.4.21+cuda12.cudnn89", source = "jax-releases" }
jax = { extras = ["cuda12-pip"], version = "^0.4.21" }

[[tool.poetry.source]]
name = "jax-releases"
url = "https://storage.googleapis.com/jax-releases/jax_cuda_releases.html"
priority = "explicit"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

if we do not disable lazy wheel then trying to poetry install this (with an empty cache) causes poetry to use all the RAM until the OS kills it

About this issue

Original URL
State: closed
Created 5 months ago
Comments: 18 (17 by maintainers)

Most upvoted comments

cachecontrol 0.14 is released with the fix, so I think the only fixes that really need making before release are

take that update
the recursion-breaking (anticipating further surprises, rather than because we know of another way to trigger this)

dimbleby on Feb 2, 2024

ok, I think the confusing server behaviour is actually a bug in cachecontrol

because of the need to download everything anyway for hashes, the wheel already is in the cache when making the range request
cachecontrol correctly spots that the range request has Cache-Control: no-cache and therefore does not try to respond to it from the cache
however it fails to spot the same thing when adding conditional headers https://github.com/psf/cachecontrol/blob/7e3edbf61fa8d6cd2db1a0eed3a2e1c2fbc67d18/cachecontrol/adapter.py#L74, https://github.com/psf/cachecontrol/blob/7e3edbf61fa8d6cd2db1a0eed3a2e1c2fbc67d18/cachecontrol/controller.py#L273

so this unconditionally loads the cached response, and sees a Last-Modified header. That causes cachecontrol to add an If-Modified-Since header, and that causes the server to return 304.

Then we end up using the cached response anyway, which was for the whole wheel and not for the range request at all

dimbleby on Jan 31, 2024

I think the issue is HTTPRepository._get_info_from_wheel() recursing into itself

the try to do it with lazy wheel fails
the try to do it without lazy wheel goes: huh, looks like ranges are supported, let’s recurse

dimbleby on Jan 31, 2024