tt-metal: [Bug Report] Embedding backward result is incorrect

Describe the bug The result of embedding backward is incorrect

To Reproduce Steps to reproduce the behavior:

  1. checkout https://github.com/tenstorrent-metal/tt-metal/tree/hyungsuk/embedding_backward_bug branch
  2. run 'pytest ./tests/tt_eager/python_api_testing/unit_testing/backward_ops/test_backward_embedding.py `
  3. check golden_output_tensor_a and tt_output_tensor_a

Expected behavior A clear and concise description of what you expected to happen.

golden_output_tensor_a and tt_output_tensor_a should be same

Screenshots If applicable, add screenshots to help explain your problem.

Please complete the following environment information:

Additional context Add any other context about the problem here.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 29 (17 by maintainers)

Commits related to this issue

Most upvoted comments

@razorback3 We’re making progress here on the LLK #9817, it’s mostly done. After that it should only take a day or two to implement the embedding b/w op.

I have everything working for embedding_backwards op now. It should support arbitrary batch, seq_len, vocab size, and embedding dim, with some small non-blocking restrictions such as:

  • seq_len % 32 == 0
  • input gradient dtype (this is the output of the op) is same as output gradient dtype (this is the input to the op)

I had to work through a couple more bugs when adding support for arbitrary embedding dim, so I will push to get this op merged early next week. It needs some cleanup right now.

Work is on this branch: jedi Test (runs 3 shapes):

pytest tests/ttnn/unit_tests/operations/backward/test_backward_embedding.py

If you look at the code in that test, we actually test with arbitrary input tokens now:

input_index = torch.randint(0, num_embeddings, input_shape)

The LLK changes are merged in. I will take a look at finishing the embedding_bw op

Just a quick update, I started looking at the task today. The current implementation of the backward embedding op is very limited. I need to write the new proper op, I don’t think it can really be made as a composite op.

@davorchap @jvasilje

We missed checking this issue in today’s meeting because it is marked as “p1_fixing_unittest” rather than “p1_critical”.

May I ask how this is going? Is it pended?

Yes , this is P1. We will review tomorrow.