tt-metal: [Bug Report] Adding `reconfig_data_format` improves performance compared to not adding it.
Describe the bug
I’m conducting tests before changing reconfig_data_format_srca(const uint32_t srca_new_operand) to reconfig_data_format_srca(const uint32_t srca_old_operand, const uint32_t srca_new_operand) to improve kernel performance.
I expect that reconfig(old, new) should always perform better than reconfig(new) because it checks the cb dtype of both old and new. However, I unexpectedly found that reconfig(new) is faster.
After further debugging, I discovered that the STALL called within reconfig(new) is faster compared to not calling reconfig at all.
for (uint32_t tile_index = 0; tile_index < block_size; ++tile_index) {
tile_regs_acquire();
// reconfig_data_format_srca<true>(tt::CBIndex::c_0, tt::CBIndex::c_0); // TRISC kernel duration around 182xx ~ 184xx,
// reconfig_data_format_srca<true>(tt::CBIndex::c_0); // TRISC kernel duration around 176xx ~ 178xx
TTI_STALLWAIT(p_stall::STALL_CFG, p_stall::UNPACK0); // TRISC kernel duration around 176xx ~ 178xx
copy_tile(tt::CBIndex::c_0, tile_index, 0);
tile_regs_commit();
tile_regs_wait();
// pack_reconfig_data_format(tt::CBIndex::c_1);
pack_reconfig_data_format(tt::CBIndex::c_1, tt::CBIndex::c_1);
pack_tile(0, tt::CBIndex::c_1);
tile_regs_release();
}
The test code is designed solely to verify the performance increase due to the reconfig modification. It assumes that all data is already in L1, and in the reader and writer, only push, wait, and pop are being executed without any NOC processing.
- All cb data types are
bfloat16.
// Reader
void kernel_main() {
constexpr uint32_t cb_in0 = 0;
cb_push_back(cb_in0, 256);
}
// Writer
void kernel_main() {
uint32_t num_tiles = 256;
uint32_t start_id = 0;
constexpr uint32_t cb_out1 = 1;
constexpr uint32_t block_size = 256;
uint32_t end_id = start_id + num_tiles;
for (uint32_t i = start_id; i < end_id; i += block_size) {
cb_wait_front(cb_out1, block_size);
cb_pop_front(cb_out1, block_size);
}
}
related kernels:
- https://github.com/tenstorrent/tt-metal/blob/hyungsuk/sram_perf_test/ttnn/cpp/ttnn/operations/examples/example/device/kernels/dataflow/reader_unary.cpp
- https://github.com/tenstorrent/tt-metal/blob/hyungsuk/sram_perf_test/ttnn/cpp/ttnn/operations/examples/example/device/kernels/compute/eltwise_sfpu.cpp
- https://github.com/tenstorrent/tt-metal/blob/hyungsuk/sram_perf_test/ttnn/cpp/ttnn/operations/examples/example/device/kernels/dataflow/writer_unary.cpp
To Reproduce Steps to reproduce the behavior:
- Go to https://github.com/tenstorrent/tt-metal/tree/hyungsuk/sram_perf_test
- run cmd: ./tt_metal/tools/profiler/profile_this.py -n example -c “pytest ./tests/ttnn/unit_tests/operations/test_examples.py”
- check
DEVICE TRISC0,1,2 KERNEL DURATION [ns]
Expected behavior
- The
reconfig(old, new)version should be faster thanreconfig(new). - If there are cases where I should use
reconfig(new)instead ofreconfig(old, new)for performance, I would like to know how to determine this.
Screenshots If applicable, add screenshots to help explain your problem.
Please complete the following environment information:
- OS: Ubuntu 22.04.4 LTS
- Version of software (eg. commit 01e12e61defd74edc9365fb092573fa6300af2fe)
Additional context Add any other context about the problem here.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 25 (21 by maintainers)
Thank you for the detailed answer.
Fortunately, it seems that Tracy is working correctly and the performance issue is also solvable. I’ll assume this problem will eventually be resolved and go ahead and try using the
reconfig(old, new)approach for now.The issue has been root-caused, and stems from a very non-trivial effect of STALLWAIT instructions:
Key takeaways:
Our best guess right now is that it seems to be some interaction between STALLWAIT instructions that we haven’t seen before. Simulation is our best bet at seeing what exactly is happening. Working on that now.
In my opinion, issues where there’s a gap between theory and practice should be addressed quickly. This is because when we try to optimize based on theoretical assumptions, but the actual behavior differs, it becomes quite confusing. Explaining this to the team or documenting it in a GitHub issue is more complicated than expected. This is because when the theory and actual behavior don’t align, it’s usually assumed to be a mistake by the person working on it, so it’s hard to get others to trust it. @amahmudTT
Not exactly sure when this issue will be actively looked upon, but I want to leave my opinion if it helps when we pick this issue up. The addition of one instruction might be placing things differently in the instruction cache, and since the code is inside a loop it may have a significant effect. It is worth testing with a nop in place of the STALLWAIT. The extra instruction might be alighing the instructions to different blocks and not making them replace each other.
I agree with your explanation.
If the stall takes less time and
reconfig(old, new)becomes slower thanreconfig(new)due to the added parameters, if statements, and additional function calls, that would be understandable.However, in the current situation, explicitly calling
stallmakes the kernel faster than not calling it. I believe this is a bug.@amahmudTT