tt-metal: [Bug Report] Latest update to moreh_adamw has issue

Describe the bug After updating to latest tt-metal main branch, we see that training loss goes down way slower. After turning off program cache, training loss goes as expected. We expect that bug is located in latest update to moreh_adamw cache functions. One of the ideas, that we aren’t properly passing step now.

Expected behavior Speed of convergence should not be affected by program caching.

Please complete the following environment information:

  • OS: Ubuntu 20.04
  • Version of software 05ae26fa027ff80e45614a9d324acc0d36940718

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (16 by maintainers)

Commits related to this issue

Most upvoted comments

fyi this dude (@mrshaw01) told me to add everything instead of just setting it to zero (something something code convention) smh, fml

We’ve got it. o2buzzle will debug this issue. Thank you for the information!