tt-metal: [Bug Report] Adding `reconfig_data_format` improves performance compared to not adding it.

Describe the bug I’m conducting tests before changing reconfig_data_format_srca(const uint32_t srca_new_operand) to reconfig_data_format_srca(const uint32_t srca_old_operand, const uint32_t srca_new_operand) to improve kernel performance.

I expect that reconfig(old, new) should always perform better than reconfig(new) because it checks the cb dtype of both old and new. However, I unexpectedly found that reconfig(new) is faster.

After further debugging, I discovered that the STALL called within reconfig(new) is faster compared to not calling reconfig at all.

        for (uint32_t tile_index = 0; tile_index < block_size; ++tile_index) {
            tile_regs_acquire();
            // reconfig_data_format_srca<true>(tt::CBIndex::c_0, tt::CBIndex::c_0); // TRISC kernel duration around 182xx ~ 184xx, 
            // reconfig_data_format_srca<true>(tt::CBIndex::c_0); // TRISC kernel duration around 176xx ~ 178xx
            TTI_STALLWAIT(p_stall::STALL_CFG, p_stall::UNPACK0);  // TRISC kernel duration around 176xx ~ 178xx
            copy_tile(tt::CBIndex::c_0, tile_index, 0);
            tile_regs_commit();

            tile_regs_wait();
            // pack_reconfig_data_format(tt::CBIndex::c_1);
            pack_reconfig_data_format(tt::CBIndex::c_1, tt::CBIndex::c_1);
            pack_tile(0, tt::CBIndex::c_1);
            tile_regs_release();
       }

The test code is designed solely to verify the performance increase due to the reconfig modification. It assumes that all data is already in L1, and in the reader and writer, only push, wait, and pop are being executed without any NOC processing.

  • All cb data types are bfloat16.
// Reader
void kernel_main() {
    constexpr uint32_t cb_in0 = 0;
    cb_push_back(cb_in0, 256);
}

// Writer
void kernel_main() {
    uint32_t num_tiles = 256;
    uint32_t start_id = 0;

    constexpr uint32_t cb_out1 = 1;
    constexpr uint32_t block_size = 256;

    uint32_t end_id = start_id + num_tiles;
    for (uint32_t i = start_id; i < end_id; i += block_size) {
        cb_wait_front(cb_out1, block_size);
        cb_pop_front(cb_out1, block_size);
    }
}

related kernels:

To Reproduce Steps to reproduce the behavior:

  1. Go to https://github.com/tenstorrent/tt-metal/tree/hyungsuk/sram_perf_test
  2. run cmd: ./tt_metal/tools/profiler/profile_this.py -n example -c “pytest ./tests/ttnn/unit_tests/operations/test_examples.py”
  3. check DEVICE TRISC0,1,2 KERNEL DURATION [ns]

Expected behavior

  1. The reconfig(old, new) version should be faster than reconfig(new).
  2. If there are cases where I should use reconfig(new) instead of reconfig(old, new) for performance, I would like to know how to determine this.

Screenshots If applicable, add screenshots to help explain your problem.

Please complete the following environment information:

  • OS: Ubuntu 22.04.4 LTS
  • Version of software (eg. commit 01e12e61defd74edc9365fb092573fa6300af2fe)

Additional context Add any other context about the problem here.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 25 (21 by maintainers)

Most upvoted comments

Thank you for the detailed answer.
Fortunately, it seems that Tracy is working correctly and the performance issue is also solvable. I’ll assume this problem will eventually be resolved and go ahead and try using the reconfig(old, new) approach for now.

The issue has been root-caused, and stems from a very non-trivial effect of STALLWAIT instructions:

  1. Back to back STALLWAIT instructions are not possible by hardware and second STALLWAIT is being actually stalled at the instruction buffer, until the previous STALLWAIT is retired (the conditions listed in that stallwait are met)
  2. the first STALLWAIT in sequence of back to back stallwaits is the one that’s extra added: (STALL_CFG, UNPACK0). The second one is (STALL_UNPACK, TRISC_CFG)
  3. The fact that second STALLWAIT is stuck at the instruction buffer effectively gates the issue of UNPACK instruction that comes after second STALLWAIT so that’s why unpack is effectively slower - UNPACK instruction gets delayed
  4. Given that unpack is delayed, there’s a very clear inactivity window of the unpacker due to this delay, so packer gets to execute faster by experiencing less conflicts on L1 write port. The difference isn’t too big, but per tile there’s 4 clocks that get lost due to L1 bank conflict
  5. Having packer to execute 4 clock cycles faster per tile, gives you 4*256 = ~1000 cycles of overall difference which I was measuring using tracy

Key takeaways:

  • back to back STALLWAIT instructions should be avoided. They kill perf since second STALLWAIT really acts as an unconditional stall thread. In this case the first STALLWAIT can be avoided if using the reconfig with both old and new CB ids. Obviously in this case we are seeing that it actually overall speeds up the operation, so this leads us to the second point
  • copy_tile() is bottlenecked by packer. It is worth analyzing how to make packer faster, there’s lots of cycles between end of packing and updating semaphore. Shaving a few cycles from here would resolve this discrepancy we are seeing with this specific issue, and speed up copy_tile overall which will benefit us in achieving some target perf numbers that we are pushing for as well (with the models team). This is something that the team will be looking into with a high priority internally, due to the potential benefits to model perf.

Our best guess right now is that it seems to be some interaction between STALLWAIT instructions that we haven’t seen before. Simulation is our best bet at seeing what exactly is happening. Working on that now.

In my opinion, issues where there’s a gap between theory and practice should be addressed quickly. This is because when we try to optimize based on theoretical assumptions, but the actual behavior differs, it becomes quite confusing. Explaining this to the team or documenting it in a GitHub issue is more complicated than expected. This is because when the theory and actual behavior don’t align, it’s usually assumed to be a mistake by the person working on it, so it’s hard to get others to trust it. @amahmudTT

Not exactly sure when this issue will be actively looked upon, but I want to leave my opinion if it helps when we pick this issue up. The addition of one instruction might be placing things differently in the instruction cache, and since the code is inside a loop it may have a significant effect. It is worth testing with a nop in place of the STALLWAIT. The extra instruction might be alighing the instructions to different blocks and not making them replace each other.

I agree with your explanation.

If the stall takes less time and reconfig(old, new) becomes slower than reconfig(new) due to the added parameters, if statements, and additional function calls, that would be understandable.

However, in the current situation, explicitly calling stall makes the kernel faster than not calling it. I believe this is a bug.

@amahmudTT