tt-metal: Tilize block not working for less than 32 rows

To recreate

git checkout sjameel/transpose_tilize_bug

build

TT_METAL_DPRINT_ONE_FILE_PER_RISC=1 TT_METAL_DPRINT_CORES=0,0 pytest tests/tt_eager/python_api_testing/unit_testing/misc/test_transpose.py::test_tranpose_hw_rm_no_padding

It will print these dprint outputs in generated/dprint

Before tilize_block data:

0.535156 0.00390625 0.871094 0.597656 0.679688 0.410156 0.683594 0.601562 0.496094 0.878906 0.929688 0.445312 0.214844 0.582031 0.648438 0.542969 0.0664062 0.535156 0.890625 0.410156 0.984375 0.953125 0.0820312 0.445312 0.738281 0.0898438 0.972656 0.59375 0.472656 0.980469 0.335938 0.78125 

0.613281 0.8125 0.652344 0.328125 0.25 0.0820312 0.144531 0.996094 0.253906 0.578125 0.324219 0.121094 0.980469 0.515625 0.222656 0.800781 0.367188 0.253906 0.480469 0.269531 0.582031 0.503906 0.496094 0.527344 0.257812 0.0703125 0.109375 0.125 0.554688 0.542969 0.992188 0.101562 

0.699219 0.4375 0.804688 0.304688 0.5625 0.425781 0.246094 0.148438 0.136719 0.21875 0.246094 0.269531 0.0976562 0.234375 0.25 0.570312 0.773438 0.0390625 0.757812 0.449219 0.5 0.335938 0.691406 0.109375 0.960938 0.425781 0.597656 0.421875 0.046875 0.875 0.320312 0.574219 

0.949219 0.613281 0.632812 0.0859375 0.0546875 0.71875 0.464844 0.355469 0.289062 0.78125 0.578125 0.949219 0.894531 0.953125 0.359375 0.824219 0.109375 0.421875 0.316406 0.660156 0.0234375 0.726562 0.371094 0.261719 0.460938 0.0546875 0.472656 0.0820312 0.0820312 0.648438 0.03125 0.0625 

0.40625 0.394531 0.078125 0.222656 0.320312 0.589844 0.441406 0.15625 0.816406 0.53125 0.375 0.390625 0.601562 0.535156 0.957031 0.242188 0.867188 0.316406 0.894531 0.339844 0.847656 0.0976562 0.742188 0.292969 0.949219 0.871094 0.53125 0.988281 0.804688 0.746094 0.246094 0.0585938 

0.507812 0.738281 0.199219 0.5 0.96875 0.8125 0.582031 0.234375 0.714844 0.0351562 0.390625 0.824219 0.824219 0.496094 0.527344 0.296875 0.558594 0.988281 0.1875 0.0195312 0.597656 0.53125 0.25 0.382812 0.425781 0.996094 0.0820312 0.703125 0.984375 0.492188 0.464844 0.972656 

0.976562 0.175781 0.859375 0.152344 0.316406 0.0273438 0.582031 0.851562 0.863281 0.925781 0.117188 0.539062 0.75 0.21875 0.0585938 0.890625 0.9375 0.65625 0.554688 0.457031 0.0976562 0.359375 0.414062 0.304688 0.945312 0.441406 0.5625 0.460938 0.890625 0.707031 0.0390625 0 

0.476562 0.5625 0.167969 0.378906 0.402344 0.253906 0.417969 0.113281 0.09375 0.804688 0.625 0.550781 0.21875 0.476562 0.253906 0.953125 0.953125 0.625 0.484375 0.617188 0.589844 0.835938 0.589844 0.675781 0.800781 0.28125 0.242188 0.578125 0.15625 0.554688 0.824219 0.535156 

after tilize (human readable)

0.535156 0.00390625 0.871094 0.597656 0.679688 0.410156 0.683594 0.601562 0.496094 0.878906 0.929688 0.445312 0.214844 0.582031 0.648438 0.542969 0.0664062 0.535156 0.890625 0.410156 0.984375 0.953125 0.0820312 0.445312 0.738281 0.0898438 0.972656 0.59375 0.472656 0.980469 0.335938 0.78125

0.613281 0.8125 0.652344 0.328125 0.25 0.0820312 0.144531 0.996094 0.253906 0.578125 0.324219 0.121094 0.980469 0.515625 0.222656 0.800781 0.367188 0.253906 0.480469 0.269531 0.582031 0.503906 0.496094 0.527344 0.257812 0.0703125 0.109375 0.125 0.554688 0.542969 0.992188 0.101562

0.699219 0.4375 0.804688 0.304688 0.5625 0.425781 0.246094 0.148438 0.136719 0.21875 0.246094 0.269531 0.0976562 0.234375 0.25 0.570312 0.773438 0.0390625 0.757812 0.449219 0.5 0.335938 0.691406 0.109375 0.960938 0.425781 0.597656 0.421875 0.046875 0.875 0.320312 0.574219

0.949219 0.613281 0.632812 0.0859375 0.0546875 0.71875 0.464844 0.355469 0.289062 0.78125 0.578125 0.949219 0.894531 0.953125 0.359375 0.824219 0.109375 0.421875 0.316406 0.660156 0.0234375 0.726562 0.371094 0.261719 0.460938 0.0546875 0.472656 0.0820312 0.0820312 0.648438 0.03125 0.0625

0.40625 0.394531 0.078125 0.222656 0.320312 0.589844 0.441406 0.15625 0.816406 0.53125 0.375 0.390625 0.601562 0.535156 0.957031 0.242188 0.0664062 0.300781 0.101562 0.8125 0.394531 0.527344 0.320312 0.300781 0.03125 0.421875 0.902344 0.753906 0.902344 0.675781 0.00390625 0.00390625

0.507812 0.738281 0.199219 0.5 0.96875 0.8125 0.582031 0.234375 0.714844 0.0351562 0.390625 0.824219 0.824219 0.496094 0.527344 0.296875 0.628906 0.453125 0.808594 0.566406 0.472656 0.972656 0.503906 0.144531 0.0429688 0.839844 0.0390625 0.441406 0.800781 0.988281 0.421875 0.683594

0.976562 0.175781 0.859375 0.152344 0.316406 0.0273438 0.582031 0.851562 0.863281 0.925781 0.117188 0.539062 0.75 0.21875 0.0585938 0.890625 0.597656 0.472656 0.382812 0.117188 0.996094 0.9375 0.382812 0.660156 0.3125 0.078125 0.746094 0.574219 0.367188 0.816406 0.871094 0.179688

0.476562 0.5625 0.167969 0.378906 0.402344 0.253906 0.417969 0.113281 0.09375 0.804688 0.625 0.550781 0.21875 0.476562 0.253906 0.953125 0.652344 0.304688 0.515625 0.171875 0.0234375 0.410156 0.0859375 0.207031 0.875 0.792969 0.800781 0.976562 0.566406 0.761719 0.363281 0.0078125

Bottom 4 rows have incorrect values in the last 8 or so values.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 30 (29 by maintainers)

Commits related to this issue

Most upvoted comments

@ncvetkovicTT, @rtawfik01, thanks for the explanations.

I think that the issue where this was first mentioned is #12234, but I am not sure if this was really fixed or not, do you perhaps know @ttmtrajkovic? @miacim has closed this bug and I believe there was a problem with repro. I dont think these are related, although they are referring to the same bits

I am not in favour of a solution where these bits would be set all the time and draw higher power. Power may be critical for BH, we are not sure yet, which is why I would prefer to properly craft a reconfigure sequence that would be safe. An API function that changes these bits should have built-in semaphores/CSR waits (whatever should work) to account for safe change in state. The perf cost is understandable, but it should be limited as we shouldn’t be doing this back and forth all the time. This way, any reconfiguration of these bits will be safe, and perf cost would happen if we do it in the middle of the op.

@ncvetkovicTT, I’ve glanced through your workaround but I am not sure if its final or just something to enable OP team? I’d prefer to close this with proper safe mechanism so we don’t stumble upon the same problem again.

It was initially problem that I could not repro issue with printing dest, but then I figure it out, added tests that expose issue and fixed bug with printing dest register.

@abhullar-tt are you referring to coalesced writes? That is an interesting perspective, we haven’t tried disabling that and testing. @ncvetkovicTT there is a disable_gathering()enable_gathering() function pair in the code. We can try using it before/after kernel, just for sanity.

thanks @ncvetkovicTT , let us know if you can get an ETA on how long a proper fix might take. If it is long then we will go with the workaround. If it’s a couple days we can wait. Either is fine, it’s just knowing the ETA will help us take the right decision.

You’ll have the ETA on Monday ❤️

OMG this is amazing thanks @ncvetkovicTT

Hey, @sjameelTT @abhullar-tt, please find the workaround with proper syncs on your branch sjameel/transpose_tilize_bug. Once you guys test it out, I can merge to tt-llk-bh repo.

As you can see from my log, 100% of the test runs pass:

PASSED tests/tt_eager/python_api_testing/unit_testing/misc/test_transpose.py::test_tranpose_hw_rm_no_padding[499-32-8-1-1]
====================== 500 passed, 500 warnings in 7.95s =======================
                 Device | INFO     | Closing user mode device drivers

@ncvetkovicTT, How do you plan to do inter-thread synchronization with delays only, what does CSR delay wait on? Maybe BH has new types of waits, that can make a trisc from one thread wait on an event of another thread, but from what I know just delaying within a thread won’t work. Keep in mind that adding delays in thread can seem like the issue is gone because timing is changed and you’re no longer hitting a race.

Milos

@ncvetkovicTT, @rtawfik01, thanks for the explanations.

I think that the issue where this was first mentioned is #12234, but I am not sure if this was really fixed or not, do you perhaps know @ttmtrajkovic? @miacim has closed this bug and I believe there was a problem with repro. I dont think these are related, although they are referring to the same bits

I am not in favour of a solution where these bits would be set all the time and draw higher power. Power may be critical for BH, we are not sure yet, which is why I would prefer to properly craft a reconfigure sequence that would be safe. An API function that changes these bits should have built-in semaphores/CSR waits (whatever should work) to account for safe change in state. The perf cost is understandable, but it should be limited as we shouldn’t be doing this back and forth all the time. This way, any reconfiguration of these bits will be safe, and perf cost would happen if we do it in the middle of the op.

@ncvetkovicTT, I’ve glanced through your workaround but I am not sure if its final or just something to enable OP team? I’d prefer to close this with proper safe mechanism so we don’t stumble upon the same problem again.

@ttmtrajkovic just to recap, DEST_ACCESS_CFG_remap_addrs_RMW and DEST_ACCESS_CFG_remap_addrs_RMW are the registers needed to enable strided packer operations. These registers are both set to true currently with pack_untilize and reduce inits, which were the locations pack_untilize was used or fused, but those flags are not set to true in other kernel inits such as binary ops, transpose_wh, etc.

Those flags can only be toggled when both math & pack are fully idle, so that means in fused kernels with pack_untilize, each init function will need syncs for both math & pack to be idle in order to reconfig those bits. Enabling those bits caused issues with destination register debug prints, and also potential increase in power draw for the destination reg. If the debug print issue is fixed, the above bits can be set to true for all ops and it will not impact functionality. Otherwise if we don’t want these bits set as true for all ops, then the workaround here is to add CSR waits to make sure all transactions in-flight for both pack & math are complete.

Hey @abhullar-tt, @sjameelTT and @zzigler-tt, you can find another branch called ncvetkovic/14609_transpose_tilize_bug which proposes another fix for the issue.

@rtawfik01 remembered that there was already an issue behaving like this. The real cause of the bug are not the out-of-sync PACK/MATH as I initially thought, but the fact that DEST_ACCESS_CFG_remap_addrs_RMW and DEST_ACCESS_CFG_remap_addrs_RMW bits that are accessed in _llk_math_hw_configure_ cannot be toggled due to poor latching of the register. By default we pass false as untilize_en argument when performing unary_op_init_common, and then after tilize_block we want to set it to true when we call pack_untilize_dst_init_short. The value of these bits doesn’t really matter for our software, they don’t affect the LLKs or anything, but they do make our lives miserable by disallowing us to print the dest properly.

I think that the issue where this was first mentioned is #12234, but I am not sure if this was really fixed or not, do you perhaps know @ttmtrajkovic?

@nvelickovicTT FYI

TL;DR: I think that I won’t find a proper fix next week, so I say that we’ll have the solution after 18 Nov.

Hey @sjameelTT @ntarafdar, I tried out some potential solutions for other problems that I thought might help with this one, but to no avail.

To give you a brief overview, there seems to be issues with synchronizing the Tensix cores in the kernel that’s in use. I tried moving around some synchronization points in the underlying LLKs, but I have to do proper debugging. Instead of analyzing the LLKs visually and trying to figure out where the sync issue is, I’ll probably have to look at some waveforms which will take some time to set up.

Introducing a couple of NOPs in transpose_wh_init_short fixes the issue. The way I would further debug this is to have one dump with a working number of NOPs and another with a failing number of NOPs. I hope this is possible to extract, but more likely there will be a gradual increase of failed runs as I reduce the number of NOPs. Since I have to set everything up, there will be a day or two of overhead work. @nvelickovicTT do you agree on the ETA?

Hey @zzigler-tt @sjameelTT, I have pushed a workaround for this issue to the sjameel/transpose_tilize_bug. The issue is with MATH and PACK not being synced after tilize_blocka and before transpose_wh_init_short. This can be resolved by using the full transpose_wh_init for now, but I am conducting an investigation of how we can fix this properly and why transpose_wh_init_short modifies the content of the intermed0 circular buffer. Please take a look and conduct some more tests on it, or tell me what I can do to test it more. Cheers!

@sjameelTT we are currently looking into a related issue with tilize on BH. Most likely it is the same underlying cause as in your case, but in any case, we’ll know more when we fix the one currently investigated. I am hoping to figure it out by the end of the week (it’s a tricky bug that requires simulator to be used).