tt-metal: Deterministic hang in to_torch depending on deallocate location

We’ve tracked down a deterministic hang with a very strange trigger based on when an intermediate tensor is deallocated.

To reproduce:

Checkout 4c917440c47577050a4374f01caa2465616d2c8a and run: pytest models/demos/mistral7b/tests/test_mistral_perf.py::test_mistral_model_perf[32-15-0.16]

Hanging configuration

Here’s where things get interesting. This is the configuration (as seen in commit xxx) that causes the hang:

Decoder:

attn_norm = self.attention_norm(x) # <- attn_norm is the problematic tensor
# Attention module expects a list of inputs, attn masks (multi-device support)
r = self.attention.forward(
    [attn_norm],
    current_pos,
    [attn_masks],
)
# ttnn.deallocate(attn_norm) <- attn_norm is deallocated inside attention.forward() in this version

Attention:

xqkv_fused = ttnn.linear(
    attn_norm,
    wqkv,
    program_config=self.wqkv_program_config,
    memory_config=self.model_config["XQKV_MM_OUTPUT_MEMCFG"],
    compute_kernel_config=self.compute_kernel_config,
    dtype=self.dtype,
)
 
ttnn.deallocate(attn_norm, force=False) # <- when deallocated here, the final to_torch on the model output hangs!

Working configuration

Now, if instead we comment out the deallocate call in the attention block and instead deallocate attn_norm in the decoder after the attention block returns, there is no hang, e.g. in this configuration:

Decoder:

attn_norm = self.attention_norm(x) # <- attn_norm is the problematic tensor
# Attention module expects a list of inputs, attn masks (multi-device support)
r = self.attention.forward(
    [attn_norm],
    current_pos,
    [attn_masks],
)
ttnn.deallocate(attn_norm) # Deallocating here avoids the hang?!

Attention:

xqkv_fused = ttnn.linear(
    attn_norm,
    wqkv,
    program_config=self.wqkv_program_config,
    memory_config=self.model_config["XQKV_MM_OUTPUT_MEMCFG"],
    compute_kernel_config=self.compute_kernel_config,
    dtype=self.dtype,
)
 
# ttnn.deallocate(attn_norm, force=False) <- no longer deallocate here even though unused in the rest of this function

This is weird AF and I hope it helps someone debug the actual cause of this hang, which might also be presenting elsewhere in the stack but less deterministically.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 17 (14 by maintainers)

Commits related to this issue

Most upvoted comments

@yieldthought @mtairum we have been asked to try this with fd2.0 in case with some weird interaction, it goes away