tt-metal: [Bounty $500-2500] Bring up mistralai/Ministral-8B-Instruct-2410 on Wormhole (N150/N300)

Bounty: Enable & Validate mistralai/Ministral-8B-Instruct-2410 on Tenstorrent Hardware

Description

This bounty involves enabling and validating mistralai/Ministral-8B-Instruct-2410 on Tenstorrent’s hardware. The goal is to ensure that mistralai/Ministral-8B-Instruct-2410 compiles, runs end-to-end (inference), meets minimal performance benchmarks, and includes sufficient documentation for community adoption.

Target Model & Difficulty Ratings

  • Model: mistralai/Ministral-8B-Instruct-2410
  • Model Hardware Target: Wormhole (N150/N300)
  • Theoretical Maximum Throughput: 23 t/s/u (tokens per second per user)

Difficulty Ratings (Performance Targets):

  • Easy ($500): Achieve ≥ 25% of theoretical maximum throughput (~6 t/s/u).
  • Medium ($1500): Achieve ≥ 50% of theoretical maximum throughput (~12 t/s/u).
  • Hard ($2500): Achieve ≥ 70% of theoretical maximum throughput (~16 t/s/u).

Success Criteria

1. Functional Bring-Up

  • mistralai/Ministral-8B-Instruct-2410 compiles and runs on Tenstorrent hardware with no blocking errors.

2. Performance Validation

  • Clearly document throughput and latency.
  • Achieve at least the baseline targets for the selected difficulty rating.

3. Accuracy Validation

  • Validate accuracy by running mistralai/Ministral-8B-Instruct-2410 both on Tenstorrent hardware and a CPU baseline.
  • Document comparative accuracy clearly, meeting or exceeding:
    • Top-1 accuracy greater than 80% and top-5 accuracy greater than 95% when compared to CPU baseline.
    • Reference current model benchmarks here: Tenstorrent Transformer Performance

4. Documentation

  • Provide clear instructions for building, installing dependencies, and running on Tenstorrent.
  • Note any relevant parameter tuning or known issues.

🔴 Strict (hard requirement, PRs may be rejected)

5. Code Reuse & Clean Structure

  • PRs that copy-paste entire codebases or duplicate existing functionality will not be accepted.
  • If the model is transformer-based and similar to Llama or Qwen, it must use the tt-transformers base in tt-metal/models/tt_transformers, reusing modules with minimal changes.
  • Code should follow a clean, modular structure for internal reuse and customer deliverables.
  • Contributors must avoid unnecessary duplication — bring-up is only successful if done in a maintainable, production-ready way.

Bring-up Checklist

  • Model functionality verified
  • Post commit CI passes
  • Model regression CI testing passes (if applicable)
  • Device performance regression CI testing passes (if applicable)
  • New/Existing tests provide coverage for changes
  • Demo script to show the inference output

Notes

  • Work may involve multiple pull requests across relevant repositories (e.g., TT-metal or others).
  • If additional patches or dependencies are required, please include a short guide for maintainers.

Helpful Links

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 34 (12 by maintainers)

Most upvoted comments

@Priyanshuthapliyal2005 Sorry for the delay - missed this thinking the tag was a part of the quote.

I would recommend committing your code and creating a new docker image. So you have the latest version of software

@Priyanshuthapliyal2005 the image name ghcr.io/tenstorrent/tt-metal/tt-metalium-ubuntu-22.04-release-amd64:latest-rc. latest-rc means it is whatever the latest version at the time of you pulling. Most likely the tokenizer API changed and and your work has to be updated in order to work with the current version of TTNN/Metalium.

I will need some help from your side. Can you:

  1. Run ls on your instance root and share the content? There should be a .whl file and I need to know the full name of it
  2. When did you created the instance?

More importantly, a recent PR migrated the Mistral example to use tt-transformers (our centralized transformer inference micro framework). Though I understand the complexity of upgrading version mid-development while there’s a potential leak. I think the best next step is to migrate your code to use and be under tt-transformers. The bug might already be fixed in newer versions and it will be a required step before the merge being accepted.

Edit: As of a few hours ago, the old implementation has been removed: #22688

Ok, I’ll update to tt-transformers and let you know

cc @marty1885 can you please comment on @Priyanshuthapliyal2005’s model loading process?

@Priyanshuthapliyal2005 I have updated the issue description. Please make sure to follow and look at point 5 in the success criteria.

Hi @cguerreroTT, sorry for missing the 2-week update. My end-semester exams are going on, so progress has been a bit slow. I’ve made some progress, but it’s not complete yet. I’ll share a proper update by 26 May 2025, 5 PM UTC. Thanks for understanding!

@Priyanshuthapliyal2005 just checking in since we haven’t received any updates in the past two weeks. Please provide a quick status update if you’re still working on this.

Sorry it took us a while due to super high demand. But you would have the access now. Please check your email.

Thank you, sir. I have received the email access.

hey @Priyanshuthapliyal2005!

Happy to assign you to this.

Be sure that you read over the BOUNTY TERMS, as you agree to all of them by picking up this issue.

If we haven’t received any communication from you in a two week period, we will re-assign the issue, and you will no longer be eligible for completion of this bounty.

If you have any questions feel free to ping me here on this issue! Look forward to seeing your submission.

Hi @Shubhamsaboo, I’d like to take on this task. Could you please assign it to me?

@Shubhamsaboo can you un-assign me this bounty since I switched mine to the TT-xla one?

hey @JushBJJ!

Happy to assign you to this.

Be sure that you read over the BOUNTY TERMS, as you agree to all of them by picking up this issue.

If we haven’t received any communication from you in a two week period, we will re-assign the issue, and you will no longer be eligible for completion of this bounty.

If you have any questions feel free to ping me here on this issue! Look forward to seeing your submission.

@Shubhamsaboo I would like to get assigned this