tt-metal: LLMs end2end perf regression
Updated main post with information from the first few comments
Commit that introduced the issue
-> The performance issues were introduced on commit https://github.com/tenstorrent/tt-metal/commit/87378f73d1e1c55e2f9c3d4d58676bf1521b79fe , which updates the device pool logic.
Issue
5 to 6x end-to-end performance degradation on LLMs. Tested on Mixtral and Falcon40b. Waiting for Llama numbers.
This issue is affecting only end-to-end performance. Device performance (tested on Mixtral) remains the same.
I’ve generated tracy files for both the bad and good performance (good commit: c1c3f6b). They are available in gdrive here
On the tracy files there is a larger gap between a device execution/iteration and WorkerPop (see figures in comment below). Specifically, the good perf commit shows gaps of 85us ~ 2ms across all 8 devices, versus 10ms ~ 45ms on the bad one.
Reproduction
Requires flag: WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml
Other flags that I have setup in my env:
export TT_METAL_ASYNC_DEVICE_QUEUE=1
export WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml
export TTNN_CONFIG_OVERRIDES='{"enable_fast_runtime_mode": true}'
# Bad perf commit
git checkout 87378f73d1e1c55e2f9c3d4d58676bf1521b79fe
git cherry-pick 14dd9a6a4dafc6973a821bf5bdfe5cc6fe87918c # Small fix required
# Good perf commit is the one before: c1c3f6b5716c29baa4244485d2ec47a1c6250ce4
# Falcon40b test is the easiest to replicate
pytest models/demos/t3000/falcon40b/tests/test_perf_falcon.py::test_perf_bare_metal[falcon_40b-prefill_seq2048_bfp8-8chips]
For Mixtral reproduction you will need the weights in your machine. Follow the Mixtral README on how to setup (or copy weights from an available machine.
For example:
- sjc-labt-3002
/home/mtairum/mistral/Mixtral-8x7B-v0.1/repack_weights.pt/home/mtairum/mistral/Mixtral-8x7B-v0.1/tokenizer.model
Once you have the repack weights you just have to point the folder in the flags:
export MIXTRAL_CKPT_DIR=<repacked_output_dir>
export MIXTRAL_TOKENIZER_PATH=<path_to_tokenizer_dir>
export MIXTRAL_CACHE_PATH=<weights_cache_dir>
To run Mixtral perf test:
pytest -svv models/demos/t3000/mixtral8x7b/tests/test_mixtral_perf.py::test_mixtral_model_perf[wormhole_b0-True-2048-150-0.025]
Mixtral-only TODO after perf goes back to normal
Mixtral current perf test is measuring for 1L. The time for each layer is not linear, so we get a more accurate perf estimate when running 32L.
Once the perf is fixed, I’ll update the perf test with tighter expected values to avoid this issue in the future.
Current target should be 158.4ms e2e perf for 32L and 2048 KV cache length.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 20 (20 by maintainers)
Commits related to this issue
- #9479: Temporary update Mixtral perf times until perf regression is fixed — committed to tenstorrent/tt-metal by mtairum 2 years ago
- #9479: Temporary update Mixtral perf times until perf regression is fixed — committed to tenstorrent/tt-metal by mtairum 2 years ago
- #9479: Temporary update Mixtral perf times until perf regression is fixed — committed to tenstorrent/tt-metal by mtairum 2 years ago
- #9479: Temporary update Mixtral perf times until perf regression is fixed — committed to tenstorrent/tt-metal by mtairum 2 years ago
- #9479: Temporary update to llama perf times until perf is fixed. NOTE: devices are on synchronous mode. — committed to tenstorrent/tt-metal by avoraTT 2 years ago
- #9479: Temporary update to llama perf times until perf is fixed. NOTE: devices are on synchronous mode. — committed to tenstorrent/tt-metal by avoraTT 2 years ago
- #9479: Temporary update to llama perf times until perf is fixed. NOTE: devices are on synchronous mode. — committed to tenstorrent/tt-metal by avoraTT 2 years ago
- #9479: fix cpu core worker bug — committed to tenstorrent/tt-metal by aliuTT 2 years ago
- #9479: Temporary update Mixtral perf times until perf regression is fixed — committed to tenstorrent/tt-metal by mtairum 2 years ago
- #9479: Temporary update Mixtral perf times until perf regression is fixed — committed to tenstorrent/tt-metal by mtairum 2 years ago
- #9479: Temporary update to llama perf times until perf is fixed. NOTE: devices are on synchronous mode. — committed to tenstorrent/tt-metal by avoraTT 2 years ago
- #9479: fix cpu core worker bug — committed to tenstorrent/tt-metal by aliuTT 2 years ago
- #9479: fix cpu core worker bug — committed to tenstorrent/tt-metal by aliuTT 2 years ago
- #9479: Temporary update to llama perf times until perf is fixed. NOTE: devices are on synchronous mode. — committed to tenstorrent/tt-metal by avoraTT 2 years ago
- #9479: Update Mixtral perf estimates now that perf regression has been fixed — committed to tenstorrent/tt-metal by mtairum 2 years ago
- #9479: Update Mixtral perf estimates now that perf regression has been fixed — committed to tenstorrent/tt-metal by mtairum 2 years ago
- #9479: Update Mixtral perf estimates now that perf regression has been fixed — committed to tenstorrent/tt-metal by mtairum 2 years ago
- #9479: Temporary update to llama perf times until perf is fixed. NOTE: devices are on synchronous mode. — committed to tenstorrent/tt-metal by avoraTT 2 years ago
- #9479: Temporary update to llama perf times until perf is fixed. NOTE: devices are on synchronous mode. — committed to tenstorrent/tt-metal by avoraTT 2 years ago
- #9479: Update Mixtral perf estimates now that perf regression has been fixed — committed to tenstorrent/tt-metal by mtairum 2 years ago
Pushed to main, seems like issue is resolved.
Great work @aliuTT .
I did some testing and can confirm that the perf on main matches what we would expect. Mixtral e2e measured at 117ms.
Also tested on previous bad commit with the fix on and the perf is around ~150ms which is close to what it was as well. 👍
@mikevin920 @mtairum Found the bug, which was causing device to always set its work executor to worker 0. Try out
dc3c72fas a fix, on your repro cases for a few weeks ago and on main if possible. I was seeing the mixtral perf recovering to ~250ms e2e again with this change. I have a PR to push this to main.I was getting a hang on
sjc-snva-t3011andsjc-lab-t3005for mixtral tests. I’m able to repro a passing case onsjc-lab-t3004will continue looking.@mtairum Can you also try without PROFILER and TRACY enabled. They can perturb e2e measurements.
@mtairum Can you try your mixtral test with sync and async on both
c1c3f6b571and87378f73d1e. I’m debugging with Kevin and Ammar, we’re seeing without the DevicePool changes in87378f73d1e, async mode is causing a perf degradation for larger sequence lengths for LLAMA.Ah I wasn’t using eth dispatch, it wasn’t mentioned in the original repro steps. I’ll try this again