tt-metal: Llama 3 TG/6U (subdevices) tests do not run properly with Huggingface 3.3 weights

From @psethiTT 's WH 6U testing:

Image

The huggingface weights for 3.3-70B-Instruct cause some output errors for this test on 6U. We also have noticed other errors in the mini stress test on 6U, which I believe were assertions on huggingface

It seems only the Meta weights work for llama subdevices, contradicting the documentation and our intent.

It would reduce a lot of friction for CI and test migration work if we can get these to work, or switch to Huggingface entirely. Meta weights have various issues that cause a lot of inconvenience, including:

  • just being another source of weights
  • inferior automation
  • no real integration with HF_HOME or other huggingface cache systems
  • The llama-stack tool doesn’t work with certain versions of Python and you also can’t even direct it where to download 😆

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Comments: 15 (13 by maintainers)

Most upvoted comments

We should keep both tokenization encodings for now to still support both HF and Meta weights, but for this specific test I agree, we can hard code to take the HF encoding (add a comment there so people know) and update the expected outputs.

I didn’t run the evaluation tools on it, just briefly did an eye-check on the output. Let me see if we can easily change this and run this through Shield CI?