tt-metal: Llama 3 TG/6U (subdevices) tests do not run properly with Huggingface 3.3 weights
From @psethiTT 's WH 6U testing:
The huggingface weights for 3.3-70B-Instruct cause some output errors for this test on 6U. We also have noticed other errors in the mini stress test on 6U, which I believe were assertions on huggingface
It seems only the Meta weights work for llama subdevices, contradicting the documentation and our intent.
It would reduce a lot of friction for CI and test migration work if we can get these to work, or switch to Huggingface entirely. Meta weights have various issues that cause a lot of inconvenience, including:
- just being another source of weights
- inferior automation
- no real integration with
HF_HOMEor other huggingface cache systems - The
llama-stacktool doesn’t work with certain versions of Python and you also can’t even direct it where to download 😆
About this issue
- Original URL
- State: open
- Created a year ago
- Comments: 15 (13 by maintainers)
We should keep both tokenization encodings for now to still support both HF and Meta weights, but for this specific test I agree, we can hard code to take the HF encoding (add a comment there so people know) and update the expected outputs.
I didn’t run the evaluation tools on it, just briefly did an eye-check on the output. Let me see if we can easily change this and run this through Shield CI?