tt-metal: [Bug Report] Embedding backward result is incorrect
Describe the bug The result of embedding backward is incorrect
To Reproduce Steps to reproduce the behavior:
- checkout https://github.com/tenstorrent-metal/tt-metal/tree/hyungsuk/embedding_backward_bug branch
- run 'pytest ./tests/tt_eager/python_api_testing/unit_testing/backward_ops/test_backward_embedding.py `
- check golden_output_tensor_a and tt_output_tensor_a
Expected behavior A clear and concise description of what you expected to happen.
golden_output_tensor_a and tt_output_tensor_a should be same
Screenshots If applicable, add screenshots to help explain your problem.
Please complete the following environment information:
- OS: [e.g. Ubuntu 20.04]
- Version of software (eg. commit)
- branch: https://github.com/tenstorrent-metal/tt-metal/tree/hyungsuk/embedding_backward_bug
Additional context Add any other context about the problem here.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 29 (17 by maintainers)
Commits related to this issue
- #6232: embedding backward bug test — committed to tenstorrent/tt-metal by hschoi4448 3 years ago
- #6232: Embedding backward op — committed to tenstorrent/tt-metal by deleted user 2 years ago
- #6232: Add embedding backwards op for training - Remove old embedding_bw - Update embedding test — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #6232: Add embedding backwards op for training - Remove old embedding_bw - Update embedding test — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #6232: Add embedding backwards op for training - Remove old embedding_bw - Update embedding test — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #6232: Add embedding backwards op for training - Remove old embedding_bw - Update embedding test — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #6232: Add embedding backwards op for training - Remove old embedding_bw - Update embedding test — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #6232: Add embedding backwards op for training - Remove old embedding_bw - Update embedding test — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #6232: Add embedding backwards op for training - Remove old embedding_bw - Update embedding test - ttnn.bfloat16 input indices only works for small vocab sizes — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #6232: Add embedding backwards op for training - Remove old embedding_bw - Update embedding test - ttnn.bfloat16 input indices only works for small vocab sizes - output gradient (input to embedding_bw... — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #6232: Add embedding backwards op for training - Remove old embedding_bw - Update embedding test - ttnn.bfloat16 input indices only works for small vocab sizes - output gradient (input to embedding_bw... — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #6232: Add embedding backwards op for training - Remove old embedding_bw - Update embedding test - ttnn.bfloat16 input indices only works for small vocab sizes - output gradient (input to embedding_bw... — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #6232: Add embedding backwards op for training - Remove old embedding_bw - Update embedding test - ttnn.bfloat16 input indices only works for small vocab sizes - output gradient (input to embedding_bw... — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #6232: Add embedding backwards op for LLM training - Remove old embedding_bw op from MCW - Update embedding test to test arbitrary tokens within vocab size - Update embedding test to test various inpu... — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #6232: Add embedding backwards op for LLM training - Remove old embedding_bw op from MCW - Update embedding test to test arbitrary tokens within vocab size - Update embedding test to test various inpu... — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #6232: Add embedding backwards op for LLM training - Remove old embedding_bw op from MCW - Update embedding test to test arbitrary tokens within vocab size - Update embedding test to test various inpu... — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #6232: Add embedding backwards op for LLM training - Remove old embedding_bw op from MCW - Update embedding test to test arbitrary tokens within vocab size - Update embedding test to test various inpu... — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #6232: Add embedding backwards op for LLM training - Remove old embedding_bw op from MCW - Update embedding test to test arbitrary tokens within vocab size - Update embedding test to test various inpu... — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #6232: Add embedding backwards op for LLM training - Remove old embedding_bw op from MCW - Update embedding test to test arbitrary tokens within vocab size - Update embedding test to test various inpu... — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
@razorback3 We’re making progress here on the LLK #9817, it’s mostly done. After that it should only take a day or two to implement the embedding b/w op.
I have everything working for
embedding_backwardsop now. It should support arbitrary batch, seq_len, vocab size, and embedding dim, with some small non-blocking restrictions such as:seq_len% 32 == 0I had to work through a couple more bugs when adding support for arbitrary embedding dim, so I will push to get this op merged early next week. It needs some cleanup right now.
Work is on this branch:
jediTest (runs 3 shapes):If you look at the code in that test, we actually test with arbitrary input tokens now:
The LLK changes are merged in. I will take a look at finishing the
embedding_bwopJust a quick update, I started looking at the task today. The current implementation of the backward embedding op is very limited. I need to write the new proper op, I don’t think it can really be made as a composite op.
Yes , this is P1. We will review tomorrow.