tt-metal: [Bounty $500-2500] Bring up microsoft/phi-2 on Wormhole (N150/N300)

Bounty: Enable & Validate microsoft/phi-2 on Tenstorrent Hardware

Description

This bounty involves enabling and validating microsoft/phi-2 on Tenstorrent’s hardware. The goal is to ensure that microsoft/phi-2 compiles, runs end-to-end (inference), meets minimal performance benchmarks, and includes sufficient documentation for community adoption.

Target Model & Difficulty Ratings

  • Model: microsoft/phi-2
  • Model Hardware Target: N150 / N300
  • Theoretical Maximum Throughput: 68 t/s/u (tokens per second per user)

Difficulty Ratings (Performance Targets):

  • Easy ($500): Achieve ≥ 25% of theoretical maximum throughput (~17 t/s/u).
  • Medium ($1500): Achieve ≥ 50% of theoretical maximum throughput (~34 t/s/u).
  • Hard ($2500): Achieve ≥ 70% of theoretical maximum throughput (~48 t/s/u).

Success Criteria

1. Functional Bring-Up

  • microsoft/phi-2 compiles and runs on N150/N300/LoudBox with no blocking errors.

2. Performance Validation

  • Clearly document throughput and latency.
  • Achieve at least the baseline targets for the selected difficulty rating.

3. Accuracy Validation

  • Validate accuracy by running microsoft/phi-2 both on Tenstorrent hardware and a CPU baseline.
  • Document comparative accuracy clearly, meeting or exceeding:
    • Top-1 accuracy greater than 80% and top-5 accuracy greater than 95% when compared to CPU baseline.
    • Reference current model benchmarks here: Tenstorrent Transformer Performance

4. Documentation

  • Provide clear instructions for building, installing dependencies, and running on Tenstorrent.
  • Note any relevant parameter tuning or known issues.

Bring-up Checklist

  • Model functionality verified
  • Post commit CI passes
  • Model regression CI testing passes (if applicable)
  • Device performance regression CI testing passes (if applicable)
  • New/Existing tests provide coverage for changes
  • Demo script to show the inference output

Notes

  • Work may involve multiple pull requests across relevant repositories (e.g., TT-metal or others).
  • If additional patches or dependencies are required, please include a short guide for maintainers.

Helpful Links

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 34 (13 by maintainers)

Most upvoted comments

What’s your good email?

benrawane2012@gmail.com

Hello again, just to keep you posted about the progress, I’ve small issue concern decoding the output/resizing the tiles

Image

@Shubhamsaboo I’m on it, I’ll submit next days ASAP, thanks

The expected deliverable will be a table like the reference table (accuracy plus performance) plus a video recording of of how to install and compile the model plus how to running the inference