tt-metal: [$500 Bounty] DST_ACCUM_MODE not forwarded for some functions with default parameters - For Wormhole code

Some functions have default argument for the template parameter defining is_fp32_dest_acc_en. When we generated the define DST_ACCUM_MODE which was meant to be passed to all the functions taking in the template parameter is_fp32_dest_acc_en. We missed some of those which have default arguments for this parameter. This may be harmless in some cases but opens up opportunities for bugs. This issue deals with such problems in Wormhole code.

One example is

template <bool is_fp32_dest_acc_en = false>
inline void llk_pack_dest_section_done() {
    _llk_pack_dest_section_done_<DST_SYNC_MODE, is_fp32_dest_acc_en>();
}

We call the above function from /tt_metal/include/compute_kernel_api/reg_api.h

ALWI void tile_regs_release() {
    PACK(( llk_pack_dest_section_done()  ));
}

We need to remove the default arguments and pass the defined DST_ACCUM_MODE that is defined. In the above example having the right code would actually follow a different code path.

The files containing functions with default arguments :

  • ./tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_binary_api.h
  • ./tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_common_api.h
  • ./tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_reduce_api.h
  • ./tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_math_unary_datacopy_api.h
  • ./tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_pack_api.h
  • ./tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_sfpu/ckernel_sfpu_recip.h
  • ./tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_sfpu/llk_math_eltwise_unary_sfpu_recip.h
  • ./tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_unpack_AB_api.h
  • ./tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_unpack_AB_matmul_api.h
  • ./tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_unpack_A_api.h
  • ./tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_unpack_common_api.h
  • ./tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_unpack_reduce_api.h
  • ./tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_unpack_tilize_api.h
  • ./tt_metal/hw/ckernels/wormhole_b0/metal/llk_api/llk_unpack_untilize_api.h
  • ./tt_metal/third_party/tt_llk_wormhole_b0/common/inc/cpack_common.h
  • ./tt_metal/third_party/tt_llk_wormhole_b0/common/inc/cunpack_common.h
  • ./tt_metal/third_party/tt_llk_wormhole_b0/common/inc/sfpu/ckernel_sfpu_recip.h
  • ./tt_metal/third_party/tt_llk_wormhole_b0/llk_lib/llk_math_common.h
  • ./tt_metal/third_party/tt_llk_wormhole_b0/llk_lib/llk_math_eltwise_binary.h
  • ./tt_metal/third_party/tt_llk_wormhole_b0/llk_lib/llk_math_eltwise_unary_datacopy.h
  • ./tt_metal/third_party/tt_llk_wormhole_b0/llk_lib/llk_math_reduce.h
  • ./tt_metal/third_party/tt_llk_wormhole_b0/llk_lib/llk_pack.h
  • ./tt_metal/third_party/tt_llk_wormhole_b0/llk_lib/llk_pack_common.h
  • ./tt_metal/third_party/tt_llk_wormhole_b0/llk_lib/llk_unpack_A.h
  • ./tt_metal/third_party/tt_llk_wormhole_b0/llk_lib/llk_unpack_AB.h
  • ./tt_metal/third_party/tt_llk_wormhole_b0/llk_lib/llk_unpack_AB_matmul.h
  • ./tt_metal/third_party/tt_llk_wormhole_b0/llk_lib/llk_unpack_common.h
  • ./tt_metal/third_party/tt_llk_wormhole_b0/llk_lib/llk_unpack_reduce.h
  • ./tt_metal/third_party/tt_llk_wormhole_b0/llk_lib/llk_unpack_tilize.h
  • ./tt_metal/third_party/tt_llk_wormhole_b0/llk_lib/llk_unpack_untilize.h

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 23 (14 by maintainers)

Commits related to this issue

Most upvoted comments

@amahmudTT As far as I can tell, this bug is present only in ./tt-metal/tests/tt_metal/tt_metal/test_kernels/compute/3T tests. These tests look like legacy ones and don’t run in CI thats why bug was not catched during merge.

Bug fix just got merged. So I’d say we’re done here. @vdmrzv thank you for your contribution! 🚀

I went over the PR, it looks good to me. It also fixes two bugs. In one case the default value was always false and this

  -  llk_pack_dest_init<DstTileFaceLayout::RowMajor, false>();
   + llk_pack_dest_init<DST_ACCUM_MODE, false>();

Although these should not change behavior (in one case two different code paths have similar effect, and the second one the value is ignored in the final call) it is still helpful because we may use the variable in the future, and copy pasting might have polluted the code base more.

@vdmrzv thanks for finishing the task.

So I would say this work is complete @fvranicTT but I would also suggest we open up a new bounty to change all the bools to enums. @vmilicevicTT , that bounty is most lilkely to catch any bug (if there is any) that we might have missed.

If so, can you @vdmrzv go over all the ones which does not run in CI and fix them ? Thanks !

Fixed in attached PR

@amahmudTT It turns out compiler will catch it, i.e. it did catch it in my run.

@Shubhamsaboo This and Blackhole related part of bounty is done and merged to main branch.

I am happy to assign this issue to your @vdmrzv!

What they meant was that you should start from tt-llk repo and files mentioned in the ticket that originate from it, and then build on top of that change.