tt-metal: preserve_fp32_precision flag is accepted as true for OPs for which it is not valid

Background: The preserve_fp32_precision flag was introduced to enable full fp32 precision for SFPU-only OPs, meaning in cases that the data is unpacked directly into Dest using copy_tile(). It does so by setting the unpack_dst_format for CBs to fp32 instead of tf32, so that the unpacker puts fp32 datums from the CB directly into Dest unchanged (skipping SrcA/B regs which are limited to 19bits).

Issue: The preserve_fp32_precision flag is not valid for all OPs, but there is no warning/error produced when it is used incorrectly. For example reduce_tile utilizes the FPU, which works on the srcA and srcB registers, and setting preserve_fp32_precision=True will set the data type of srcA and srcB to fp32 instead of tf32 and lead to wrong results.

Also, the question was raised whether or not the flag is allowed in cases where a compute kernel may use one LLK for which the flag is supported but a second one for which it is not. One such example could be reloading fp32 partials into the Dest (flag supported) in order to perform further accumulation using matmul (flag not supported). In these cases, since the flag works by changing the unpack_dst_format for the buffer, if the buffer is only used for unpacking to Dest then the flag should be allowed. If the buffer is used to unpack to Dest but also unpack into SrcA/B, then the flag is not allowed.

To Reproduce: This was detected in this bug https://github.com/tenstorrent/tt-metal/issues/10337 and this example can be used.

Expected behavior: We need a mechanism to disallow this flag to be set to true for OPs for which it is not valid, and better documentation surrounding this flag.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 18 (18 by maintainers)

Commits related to this issue

Most upvoted comments

@rdjogoTT Seems like all work as intended. Salute to the nice work! I think you can close this issue after the PR is merged to the main branch.

@razorback3 I have implemented the discussed solution on my branch rd/preserve_fp32_precision. I tested it with my own typecast op and made all necessary changes on the TT side. I also looked into where the flag was used in Moreh files and made appropriate changes to accommodate for the new vector flag. However, I need your help with the Moreh program factory files to set the correct CBs to have the flag enabled. Can someone from the Moreh side help with this?

For example in ttnn/cpp/ttnn/operations/moreh/moreh_matmul/device/moreh_matmul_program_factory.cpp I have initialized a vector vector<PreserveFP32Target> preserve_fp32_precision(NUM_CIRCULAR_BUFFERS, PreserveFP32Target::Disabled); which represents all CBs having the flag disabled. To enable the flag you can individually program which CBs you want it for by setting preserve_fp32_precision[cb_id] = PreserveFP32Target::DEST.

Nice work! @dongjin-na will check the functionality on the given branch and give feedback soon 😀.

We need a mechanism to disallow this flag to be set to true for OPs for which it is not valid.

What does the above sentence mean among the two options below?

  1. If reduce_tile exists in the compute kernel, that compute kernel does not support the preserve_fp32 feature.
  2. In the compute kernel, if both add_tiles and reduce_tile are present, preserve_fp32 is disabled only for reduce_tile, which doesn’t support it, while add_tiles remains unaffected.

add_tiles also uses FPU and is not compatible with the flag, but if you had some other function for which preserve_fp32_precision is valid, that one should work fine, at least that is what we aim for.

We need a mechanism to disallow this flag to be set to true for OPs for which it is not valid.

What does the above sentence mean among the two options below?

  1. If reduce_tile exists in the compute kernel, that compute kernel does not support the preserve_fp32 feature.
  2. In the compute kernel, if both add_tiles and reduce_tile are present, preserve_fp32 is disabled only for reduce_tile, which doesn’t support it, while add_tiles remains unaffected.