tokenizers: Tokenizer throwing PanicException

thread ‘<unnamed>’ panicked at ‘called Result::unwrap() on an Err value: Internal’, /__w/tokenizers/tokenizers/tokenizers/src/models/unigram/trainer.rs:203:53 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace Preparing data… Training tokenizer… Traceback (most recent call last): File “tokenization_2.py”, line 26, in <module> tokenizer.train([args.file], trainer) pyo3_runtime.PanicException: called Result::unwrap() on an Err value: Internal

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15

Most upvoted comments

Hi @tanmaylaud,

Can you provide a script that triggered the error maybe? Or some more context? Without it it’s a bit hard to help. cheers