tt-metal: Deterministic hang in to_torch depending on deallocate location
We’ve tracked down a deterministic hang with a very strange trigger based on when an intermediate tensor is deallocated.
To reproduce:
Checkout 4c917440c47577050a4374f01caa2465616d2c8a and run:
pytest models/demos/mistral7b/tests/test_mistral_perf.py::test_mistral_model_perf[32-15-0.16]
Hanging configuration
Here’s where things get interesting. This is the configuration (as seen in commit xxx) that causes the hang:
Decoder:
attn_norm = self.attention_norm(x) # <- attn_norm is the problematic tensor
# Attention module expects a list of inputs, attn masks (multi-device support)
r = self.attention.forward(
[attn_norm],
current_pos,
[attn_masks],
)
# ttnn.deallocate(attn_norm) <- attn_norm is deallocated inside attention.forward() in this version
Attention:
xqkv_fused = ttnn.linear(
attn_norm,
wqkv,
program_config=self.wqkv_program_config,
memory_config=self.model_config["XQKV_MM_OUTPUT_MEMCFG"],
compute_kernel_config=self.compute_kernel_config,
dtype=self.dtype,
)
ttnn.deallocate(attn_norm, force=False) # <- when deallocated here, the final to_torch on the model output hangs!
Working configuration
Now, if instead we comment out the deallocate call in the attention block and instead deallocate attn_norm in the decoder after the attention block returns, there is no hang, e.g. in this configuration:
Decoder:
attn_norm = self.attention_norm(x) # <- attn_norm is the problematic tensor
# Attention module expects a list of inputs, attn masks (multi-device support)
r = self.attention.forward(
[attn_norm],
current_pos,
[attn_masks],
)
ttnn.deallocate(attn_norm) # Deallocating here avoids the hang?!
Attention:
xqkv_fused = ttnn.linear(
attn_norm,
wqkv,
program_config=self.wqkv_program_config,
memory_config=self.model_config["XQKV_MM_OUTPUT_MEMCFG"],
compute_kernel_config=self.compute_kernel_config,
dtype=self.dtype,
)
# ttnn.deallocate(attn_norm, force=False) <- no longer deallocate here even though unused in the rest of this function
This is weird AF and I hope it helps someone debug the actual cause of this hang, which might also be presenting elsewhere in the stack but less deterministically.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 17 (14 by maintainers)
Commits related to this issue
- #7822: Reproducer for deterministic hang — committed to tenstorrent/tt-metal by yieldthought 3 years ago
- #7822: Reproduced on FD 2.0 based on today's main — committed to tenstorrent/tt-metal by yieldthought 3 years ago
- #7822: Trying to reproduce hang — committed to tenstorrent/tt-metal by eyonland 2 years ago
- #7822: Reproduced on FD 2.0 based on today's main — committed to tenstorrent/tt-metal by yieldthought 3 years ago
- #7822: update repro branch for current main — committed to tenstorrent/tt-metal by yieldthought 2 years ago
- #7822: Fix conditionals for bmm multi core reuse optimized for when to update rt args — committed to tenstorrent/tt-metal by tt-aho 2 years ago
- #7822: Fix conditionals for bmm multi core reuse optimized for when to update rt args — committed to tenstorrent/tt-metal by tt-aho 2 years ago
- #7822: Fix conditionals for bmm multi core reuse optimized for when to update rt args — committed to tenstorrent/tt-metal by tt-aho 2 years ago
@yieldthought @mtairum we have been asked to try this with fd2.0 in case with some weird interaction, it goes away