tt-metal: [Bug Report] Small performance regression for binary_ng

Describe the bug

Fine-grained profiling comparing ttnn.add with ttnn.experimental.add shows 456μs vs 526μs. There seems to be a greater delay in between BINARY_OP zones when profiling various elementwise binary_ng operations such as add/mul.

To Reproduce

Run on-device profiling of ttnn.experimental.add and ttnn.add on a [1024, 8192] random int32 or float32 tensor with layout=ttnn.TILE_LAYOUT.

Expected behavior

Timings for binary_ng should be equal to or better than what exists currently.

Please complete the following environment information:

  • Version: 907fffd392cd4c8101313decfc4fea99a3fea2f8

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (9 by maintainers)

Most upvoted comments

I’m going to close this case as committing the fixes is tracked elsewhere.

This issue is fixed. @dchenTT moved most/all of the binary_ng cases to read with the reader kernel, rather than the original split to read A with the reader and B with the writer - that was the cause of the ~15% degradation. This is fixed in binary_ng already.

The other ~4x degradation when doing row broadcast was cause by loop ordering resulting in discarding and having to re-create the subtile-broadcast result. That’s also fixed by @dchenTT , not by changing the loop ordering but by using the subtile broadcast LLK to speed things up. The overall result is parity or slightly better between all broadcast cases and the non-broadcast case. This is fixed in a PR that’s still in development and will go in after binary_ng is on-by-default.

Two tensors of identical shape, e.g.

import ttnn
import torch

device = ttnn.open_device(device_id=0)
dtype = ttnn.int32 # or ttnn.float32

a_torch = torch.randint(-(2**10-1), 2**10-1, (1 * 1024, 8 * 1024))
b_torch = torch.randint(-(2**10-1), 2**10-1, (1 * 1024, 8 * 1024))

a: ttnn.Tensor = ttnn.from_torch(a_torch, dtype=dtype, layout=ttnn.TILE_LAYOUT, device=device)
b: ttnn.Tensor = ttnn.from_torch(b_torch, dtype=dtype, layout=ttnn.TILE_LAYOUT, device=device)
c = ttnn.add(a, b)

a: ttnn.Tensor = ttnn.from_torch(a_torch, dtype=dtype, layout=ttnn.TILE_LAYOUT, device=device)
b: ttnn.Tensor = ttnn.from_torch(b_torch, dtype=dtype, layout=ttnn.TILE_LAYOUT, device=device)
c = ttnn.experimental.add(a, b)

ttnn.DumpDeviceProfiler(device)
ttnn.close_device(device)