tokenizers: Cargo.lock is not available

Since https://github.com/huggingface/tokenizers/pull/1131, all Cargo.lock files were removed. This makes development cleaner, but it becomes very hard to reproduce a specific release of tokenizers.

This helps with tools that are strict about its dependencies, like Nix. People have run into this problem with poetry2nix (https://github.com/nix-community/poetry2nix/issues/968) as well as packaging applications in nixpkgs (https://github.com/NixOS/nixpkgs/pull/227126).

The current solution in such cases is to generate a Cargo.lock ourselves and ship it with the package. The result is that tokenizers won’t operate the same as the developers have intended. It could cause hard to reproduce bug-reports.

Would it be possible to distribute Cargo.lock in future releases?

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 1
  • Comments: 15 (3 by maintainers)

Most upvoted comments

There hasn’t been a release since.

And yes I removed the Cargo.lock from both main because they keep messing the CI (and benchmarks in safetensors). I’ll figure out a way to add them in the release branches.

I think this is still relevant.