tt-metal: Blackhole: ttnn.transformer.scaled_dot_product_attention_decode (paged) Llama failing

tests/tt_eager/python_api_testing/unit_testing/misc/test_scaled_dot_product_attention_decode.test_sdpa_decode_paged_attention

with parameters:

(1,32,1, 1024*64, 128,(8, 4), True),
(1,32,8, 1024*64, 128,(8, 4), True),

"block_size", (64,)

fails with,

2025-01-13 17:39:07.375 | DEBUG    | tests.tt_eager.python_api_testing.unit_testing.misc.test_scaled_dot_product_attention_decode:run_test_sdpa_decode_paged_attention:733 - Testing causal with sequence length: 142
2025-01-13 17:39:07.375 | INFO     | tests.tt_eager.python_api_testing.unit_testing.misc.test_scaled_dot_product_attention_decode:run_test_sdpa_decode_paged_attention:736 - Using chunk size: 128
2025-01-13 17:39:07.375 | INFO     | tests.tt_eager.python_api_testing.unit_testing.misc.test_scaled_dot_product_attention_decode:run_test_sdpa_decode_paged_attention:737 - Using padded layer length: 256
2025-01-13 17:39:07.375 | INFO     | tests.tt_eager.python_api_testing.unit_testing.misc.test_scaled_dot_product_attention_decode:run_test_sdpa_decode_paged_attention:738 - Using padded num heads: 32
2025-01-13 17:39:09.401 | DEBUG    | tests.tt_eager.python_api_testing.unit_testing.misc.test_scaled_dot_product_attention_decode:run_test_sdpa_decode_paged_attention:821 - python vs pytorch: Max ATOL Delta: 8.806135472075068e+36, Max RTOL Delta: inf, PCC: -0.01640503628702707, PCC check failed

and

2025-01-13 17:39:20.813 | DEBUG    | tests.tt_eager.python_api_testing.unit_testing.misc.test_scaled_dot_product_attention_decode:run_test_sdpa_decode_paged_attention:733 - Testing causal with sequence length: 0
2025-01-13 17:39:20.813 | INFO     | tests.tt_eager.python_api_testing.unit_testing.misc.test_scaled_dot_product_attention_decode:run_test_sdpa_decode_paged_attention:736 - Using chunk size: 32
2025-01-13 17:39:20.813 | INFO     | tests.tt_eager.python_api_testing.unit_testing.misc.test_scaled_dot_product_attention_decode:run_test_sdpa_decode_paged_attention:737 - Using padded layer length: 32
2025-01-13 17:39:20.813 | INFO     | tests.tt_eager.python_api_testing.unit_testing.misc.test_scaled_dot_product_attention_decode:run_test_sdpa_decode_paged_attention:738 - Using padded num heads: 32
2025-01-13 17:39:22.775 | DEBUG    | tests.tt_eager.python_api_testing.unit_testing.misc.test_scaled_dot_product_attention_decode:run_test_sdpa_decode_paged_attention:821 - python vs pytorch: Max ATOL Delta: 7.310753976817037e+37, Max RTOL Delta: inf, PCC: -0.01494582707426299, PCC check failed

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 24 (18 by maintainers)

Commits related to this issue

Most upvoted comments

Firmware update on the board. Some non-deterministic hangs were caused by hardware.

@mtairum can you try llama 8B end to end?

Hangs are resolved with a BH LLK fix on branch, mentioned here https://github.com/tenstorrent/tt-metal/issues/16013#issuecomment-2660365250. Lowering this to P1 since Llama8B shapes are unblocked and only the PCC issue on certain shapes remains

Failed PCC has tended to be an indicator of alignment issues on BH. @llongTT is the resident expert on this.

stack overflow with watcher is expected: https://github.com/tenstorrent/tt-metal/issues/15066