tokenizers: Cargo.lock is not available
Since https://github.com/huggingface/tokenizers/pull/1131, all Cargo.lock files were removed. This makes development cleaner, but it becomes very hard to reproduce a specific release of tokenizers.
This helps with tools that are strict about its dependencies, like Nix. People have run into this problem with poetry2nix (https://github.com/nix-community/poetry2nix/issues/968) as well as packaging applications in nixpkgs (https://github.com/NixOS/nixpkgs/pull/227126).
The current solution in such cases is to generate a Cargo.lock ourselves and ship it with the package. The result is that tokenizers won’t operate the same as the developers have intended. It could cause hard to reproduce bug-reports.
Would it be possible to distribute Cargo.lock in future releases?
About this issue
- Original URL
- State: closed
- Created a year ago
- Reactions: 1
- Comments: 15 (3 by maintainers)
No, the lock files are now part of the release: https://github.com/huggingface/tokenizers/commit/284fbd8e215a34d3b8de849a1d0bca292116b11f
There hasn’t been a release since.
And yes I removed the
Cargo.lockfrom bothmainbecause they keep messing the CI (and benchmarks in safetensors). I’ll figure out a way to add them in the release branches.I think this is still relevant.