tt-metal: Device profiler on ttnn resnet50 is hanging on BH

On blackhole, we initially get kernel compile error, but with the following patch, it runs:

diff --git a/tt_metal/hw/inc/blackhole/dev_mem_map.h b/tt_metal/hw/inc/blackhole/dev_mem_map.h
index 075edd005c..b97e3c5601 100644
--- a/tt_metal/hw/inc/blackhole/dev_mem_map.h
+++ b/tt_metal/hw/inc/blackhole/dev_mem_map.h
@@ -48,7 +48,7 @@

 /////////////
 // Firmware/kernel code holes
-#define MEM_BRISC_FIRMWARE_SIZE (5 * 1024 + 128)
+#define MEM_BRISC_FIRMWARE_SIZE (5 * 1024 + 256)
 // TODO: perhaps put NCRISC FW in the scratch area and free 1.5K after init (GS/WH)
 #define MEM_NCRISC_FIRMWARE_SIZE 1536
 #define MEM_TRISC0_FIRMWARE_SIZE 1536

But when trying to profile Resnet50, the run with profiler hangs. (Passes fine without profiler).

Branch: asarje/bh-rn50-20250123 Compile with profiler: ./build_metal.sh -p --debug Run the model with profiler: python -m tracy -p -r -v -m pytest "\"tests/ttnn/integration_tests/resnet/test_ttnn_functional_resnet50.py::test_resnet_50[pretrained_weight_false-batch_size=16-act_dtype=DataType.BFLOAT8_B-weight_dtype=DataType.BFLOAT8_B-math_fidelity=MathFidelity.LoFi-device_params={'l1_small_size': 24576}]\""

The run will hang.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 45 (37 by maintainers)

Commits related to this issue

Most upvoted comments

Note the following in case you want to reproduce the hang and test our discovered solutions.

On IRD machines, Weka is slow so when generating large device profiler csvs things can slowdown dramatically. Point the generated folder to localdev with a symlink.

sudo mkdir /localdev/{username}/profiler_generated
sudo chmod 777 /localdev/{username}/profiler_generated
cd $TT_METAL_HOME
rm -rf generated
ln -s /localdev/{username}/profiler_generated generated

With above done, please run below on profiler builds (i.e. build_metal.sh -p):

TT_METAL_DEVICE_PROFILER=1 pytest "tests/ttnn/integration_tests/resnet/test_ttnn_functional_resnet50.py::test_resnet_50[pretrained_weight_false-batch_size=16-act_dtype=DataType.BFLOAT8_B-weight_dtype=DataType.BFLOAT8_B-math_fidelity=MathFidelity.LoFi-device_params={'l1_small_size': 24576}]"

It will result in the same hang on that branch if you do not have our discovered solutions are applied.

Confirmed that on BH-30 on commit https://github.com/tenstorrent/tt-metal/commit/681d3f7510fe9293ac08ad40da60061afc2b00f7 on main which has the fix, we don’t see a hang when we profile resnet50. This is without using any env vars or mods to the code.

No it does, this is just a better reproduction step for quicker testing

Aright thanks! I believe however that we’re going to use other workaround, #10 from the table, which protects CSR writes and adds SFPI compiler flag.

Confirmed with Mo, the timer at the end was not a hang, it was the profiler that kept on writing through another thread. So the above modifications does remove the hangs.

@amahmudTT Thank you, could you please provide the branch and the command that you’re running either here or in the table, Row2ColumnG (in the note body)? Also, can you please confirm the results that I put in the table, ColumnG? https://docs.google.com/spreadsheets/d/1i0bfPhjZcUI1ce_CTE44a-VMnvquQjY-kRPAb5oMXow/edit?gid=0#gid=0

updated

Disabling L1 cache but keeping the other changes (disable/enable_gathering() and compiler option) still allowes the test to run fully (with the timer still going on at the end)

Pushed the original branch back.

Applied the compiler arg to 11e39069784738311c39c42ec9b3bcb11cfbf6f5 , mo/bh_model_test Does not seem to remove the hang, but this change along with disabling gathering tends to move the hang to downsampling instead of conv2

   1.43 Kbps /138.5% =   0.00 Mbps | Tx: 137 KB | 64.14 MB | 2:21.62025-02-25 12:01:45.549 | DEBUG    | models.demos.ttnn_resnet.tt.ttnn_functional_resnet50:run_downsample_if_req:168 - Running downsample
   1.43 Kbps /138.5% =   0.00 Mbps | Tx: 255.56 KB | 64.22 MB | 13:40.2

Next step: try @ncvetkovicTT 's branch + command line arguments from @pgkeller to disable the use of the replay instruction for the SFPU.

The command line argument: -fno-rvtt-sfpu-replay will disable sfpi’s use of the replay instruction. This needs to be inserted into the cflags computed in JitBuildCompute::JitBuildCompute in the file tt_metal/jit_build/build.cpp

I will be looking at this issue from the LLK side @ejouretTT