tokenizers: Tokenizer throwing PanicException
thread ‘<unnamed>’ panicked at ‘called
Result::unwrap()
on anErr
value: Internal’, /__w/tokenizers/tokenizers/tokenizers/src/models/unigram/trainer.rs:203:53 note: run withRUST_BACKTRACE=1
environment variable to display a backtrace Preparing data… Training tokenizer… Traceback (most recent call last): File “tokenization_2.py”, line 26, in <module> tokenizer.train([args.file], trainer) pyo3_runtime.PanicException: calledResult::unwrap()
on anErr
value: Internal
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 15
Hi @tanmaylaud,
Can you provide a script that triggered the error maybe? Or some more context? Without it it’s a bit hard to help. cheers