tt-metal: [Bug Report] Latest update to moreh_adamw has issue
Describe the bug
After updating to latest tt-metal main branch, we see that training loss goes down way slower. After turning off program cache, training loss goes as expected. We expect that bug is located in latest update to moreh_adamw cache functions. One of the ideas, that we aren’t properly passing step now.
Expected behavior Speed of convergence should not be affected by program caching.
Please complete the following environment information:
- OS: Ubuntu 20.04
- Version of software 05ae26fa027ff80e45614a9d324acc0d36940718
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16 (16 by maintainers)
Commits related to this issue
- #14186: Fixed moreh_adam — committed to tenstorrent/tt-metal by dmakoviichuk-tt 2 years ago
- #14186: Fixed moreh_adam and moreh_adamw (#14243) * #14186: Fixed moreh_adam * #0: fixed adam too — committed to tenstorrent/tt-metal by dmakoviichuk-tt 2 years ago
- #14186: Fixed moreh_adam and moreh_adamw (#14243) * #14186: Fixed moreh_adam * #0: fixed adam too — committed to ct-clmsn/tt-metal by dmakoviichuk-tt 2 years ago
fyi this dude (@mrshaw01) told me to add everything instead of just setting it to zero (something something code convention) smh, fml
We’ve got it. o2buzzle will debug this issue. Thank you for the information!