uv: [perf] uv pip install resolution is slow when installing from VCS

I don’t know if it is useful but there is something called sparse checkout.

Yes, you could use sparse checkout here effectively to speed things up quite substantially. Normally you’d do in the CLI in the following way:

1. Clone the repo: git clone --filter=blob:none --depth 1 --sparse git@github.com:functime-org/functime.git
2. Reapply a sparse filter to keep all the pyproject.toml’s inside the cloned repo: git sparse-checkout set '**/pyproject.toml'

Note the first --sparse is needed to keep the repo with only top-level content. --filter=blob:none is still needed to keep blobs out (this is what pip does) and saves a decent amount of space. --depth=1 or shallow is still nice to keep as it saves a more space depending on the history size.

After that, you do git sparse-checkout to filter to keep only all the pyproject.toml’s in the repo (if any).

This recudes the clone size of functime down to Receiving objects: 100% (11/11), 15.56 KiB | 1.56 MiB/s, done. Then with the sparse checkout reduces the functime repo to virtually 3.4k since there’s a single pyproject.toml.

I’ve been using a similar technique personally on very large repos at work in CI/CD for quite some time.

If we have a Git dependency, the first step is that we need to clone the repo. Then we read the pyproject.toml if it exists. Perhaps in theory we could try to only clone pyproject.toml (we can’t know whether it exists in advance, but we could try), I don’t know if it’s even possible fetch a single file from Git though. Maybe if we know it’s on GitHub, we could try to add a fast path for it by downloading the file directly rather than using a Git client.

For that particular repo (functime.git @ main), it seems to be downloading 285 MB for a regular clone and 231 MB for filter=blob:none. That’s a surprisingly small difference.

Yeah, not sure how much performance impact this will have in general, but one advantage of enabling it is that it is well tested in pip as it has been enabled in for ~2.5 years now.

For that particular repo (functime.git @ main), it seems to be downloading 285 MB for a regular clone and 231 MB for filter=blob:none. That’s a surprisingly small difference.

(But I confirmed that we do read the metadata from pyproject.toml, and we don’t build the wheel, which is good.)

Mmm I think for me basically the entire time is spent cloning the repo. That’s a bummer. Maybe a datapoint for @ibraheemdev when it comes to seeing if we can make clones any faster.

Ahh I see! Let me take a look – we should be able to clone the repo and read the metadata without building the wheel in this case. (But we do need to clone it, we don’t do selective reads (e.g., just checkout the pyproject.toml) from Git.)

Ciao Charlie, thank you very much for the prompt reply.

And functime does ship per-platform wheels which suggests they’re compiling some native code.

Yes, we have some Rust plugins for Polars!

I think the problem might be that when you install from PyPI, they can serve you a wheel, which is a built artifact (since the uploader of the package uploaded wheels for it, for a bunch of platforms). But if you install from VCS, you’re required to build the package from source. And building from source can be really long and expensive – it completely depends on the package, we basically have to call out to their build method, which could involve compiling native code.

Indeed! I should’ve been more precise, sorry. What bugged me was the resolution time:

From VCS:

+Resolved 21 packages in 37.95s
   Built functime @ git+https://github.com/functime-org/functime.git@0608c78118b5defd42df73b812
   Built lightgbm==4.3.0
Downloaded 21 packages in 1m 48s
Installed 21 packages in 282ms

From PyPI:

+Resolved 21 packages in 417ms
   Built lightgbm==4.3.0
Downloaded 21 packages in 30.98s
Installed 21 packages in 284ms

You say that in the first case it’s 38s because it has to download and build the binary? Couldn’t uv try to fetch pyproject.toml to perform resolution first? I guess the overall time would not change, since build would need to happen anyway.

Feel free to close the issue.