tt-metal: Device profiler on ttnn resnet50 is hanging on BH
On blackhole, we initially get kernel compile error, but with the following patch, it runs:
diff --git a/tt_metal/hw/inc/blackhole/dev_mem_map.h b/tt_metal/hw/inc/blackhole/dev_mem_map.h
index 075edd005c..b97e3c5601 100644
--- a/tt_metal/hw/inc/blackhole/dev_mem_map.h
+++ b/tt_metal/hw/inc/blackhole/dev_mem_map.h
@@ -48,7 +48,7 @@
/////////////
// Firmware/kernel code holes
-#define MEM_BRISC_FIRMWARE_SIZE (5 * 1024 + 128)
+#define MEM_BRISC_FIRMWARE_SIZE (5 * 1024 + 256)
// TODO: perhaps put NCRISC FW in the scratch area and free 1.5K after init (GS/WH)
#define MEM_NCRISC_FIRMWARE_SIZE 1536
#define MEM_TRISC0_FIRMWARE_SIZE 1536
But when trying to profile Resnet50, the run with profiler hangs. (Passes fine without profiler).
Branch: asarje/bh-rn50-20250123
Compile with profiler: ./build_metal.sh -p --debug
Run the model with profiler: python -m tracy -p -r -v -m pytest "\"tests/ttnn/integration_tests/resnet/test_ttnn_functional_resnet50.py::test_resnet_50[pretrained_weight_false-batch_size=16-act_dtype=DataType.BFLOAT8_B-weight_dtype=DataType.BFLOAT8_B-math_fidelity=MathFidelity.LoFi-device_params={'l1_small_size': 24576}]\""
The run will hang.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 45 (37 by maintainers)
Note the following in case you want to reproduce the hang and test our discovered solutions.
On IRD machines, Weka is slow so when generating large device profiler csvs things can slowdown dramatically. Point the generated folder to localdev with a symlink.
With above done, please run below on profiler builds (i.e.
build_metal.sh -p):It will result in the same hang on that branch if you do not have our discovered solutions are applied.
Confirmed that on BH-30 on commit https://github.com/tenstorrent/tt-metal/commit/681d3f7510fe9293ac08ad40da60061afc2b00f7 on main which has the fix, we don’t see a hang when we profile resnet50. This is without using any env vars or mods to the code.
No it does, this is just a better reproduction step for quicker testing
Aright thanks! I believe however that we’re going to use other workaround, #10 from the table, which protects CSR writes and adds SFPI compiler flag.
Confirmed with Mo, the timer at the end was not a hang, it was the profiler that kept on writing through another thread. So the above modifications does remove the hangs.
updated
Disabling L1 cache but keeping the other changes (disable/enable_gathering() and compiler option) still allowes the test to run fully (with the timer still going on at the end)
Pushed the original branch back.
Applied the compiler arg to 11e39069784738311c39c42ec9b3bcb11cfbf6f5 , mo/bh_model_test Does not seem to remove the hang, but this change along with disabling gathering tends to move the hang to downsampling instead of conv2
The command line argument:
-fno-rvtt-sfpu-replaywill disable sfpi’s use of the replay instruction. This needs to be inserted into the cflags computed inJitBuildCompute::JitBuildComputein the filett_metal/jit_build/build.cppI will be looking at this issue from the LLK side @ejouretTT