tt-metal: Blackhole: ttnn.transformer.scaled_dot_product_attention_decode (paged) Llama failing
tests/tt_eager/python_api_testing/unit_testing/misc/test_scaled_dot_product_attention_decode.test_sdpa_decode_paged_attention
with parameters:
(1,32,1, 1024*64, 128,(8, 4), True),
(1,32,8, 1024*64, 128,(8, 4), True),
"block_size", (64,)
fails with,
2025-01-13 17:39:07.375 | DEBUG | tests.tt_eager.python_api_testing.unit_testing.misc.test_scaled_dot_product_attention_decode:run_test_sdpa_decode_paged_attention:733 - Testing causal with sequence length: 142
2025-01-13 17:39:07.375 | INFO | tests.tt_eager.python_api_testing.unit_testing.misc.test_scaled_dot_product_attention_decode:run_test_sdpa_decode_paged_attention:736 - Using chunk size: 128
2025-01-13 17:39:07.375 | INFO | tests.tt_eager.python_api_testing.unit_testing.misc.test_scaled_dot_product_attention_decode:run_test_sdpa_decode_paged_attention:737 - Using padded layer length: 256
2025-01-13 17:39:07.375 | INFO | tests.tt_eager.python_api_testing.unit_testing.misc.test_scaled_dot_product_attention_decode:run_test_sdpa_decode_paged_attention:738 - Using padded num heads: 32
2025-01-13 17:39:09.401 | DEBUG | tests.tt_eager.python_api_testing.unit_testing.misc.test_scaled_dot_product_attention_decode:run_test_sdpa_decode_paged_attention:821 - python vs pytorch: Max ATOL Delta: 8.806135472075068e+36, Max RTOL Delta: inf, PCC: -0.01640503628702707, PCC check failed
and
2025-01-13 17:39:20.813 | DEBUG | tests.tt_eager.python_api_testing.unit_testing.misc.test_scaled_dot_product_attention_decode:run_test_sdpa_decode_paged_attention:733 - Testing causal with sequence length: 0
2025-01-13 17:39:20.813 | INFO | tests.tt_eager.python_api_testing.unit_testing.misc.test_scaled_dot_product_attention_decode:run_test_sdpa_decode_paged_attention:736 - Using chunk size: 32
2025-01-13 17:39:20.813 | INFO | tests.tt_eager.python_api_testing.unit_testing.misc.test_scaled_dot_product_attention_decode:run_test_sdpa_decode_paged_attention:737 - Using padded layer length: 32
2025-01-13 17:39:20.813 | INFO | tests.tt_eager.python_api_testing.unit_testing.misc.test_scaled_dot_product_attention_decode:run_test_sdpa_decode_paged_attention:738 - Using padded num heads: 32
2025-01-13 17:39:22.775 | DEBUG | tests.tt_eager.python_api_testing.unit_testing.misc.test_scaled_dot_product_attention_decode:run_test_sdpa_decode_paged_attention:821 - python vs pytorch: Max ATOL Delta: 7.310753976817037e+37, Max RTOL Delta: inf, PCC: -0.01494582707426299, PCC check failed
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 24 (18 by maintainers)
Firmware update on the board. Some non-deterministic hangs were caused by hardware.
@mtairum can you try llama 8B end to end?
Hangs are resolved with a BH LLK fix on branch, mentioned here https://github.com/tenstorrent/tt-metal/issues/16013#issuecomment-2660365250. Lowering this to P1 since Llama8B shapes are unblocked and only the PCC issue on certain shapes remains
Failed PCC has tended to be an indicator of alignment issues on BH. @llongTT is the resident expert on this.
stack overflow with watcher is expected: https://github.com/tenstorrent/tt-metal/issues/15066