tt-metal: WH B0 CI Machines unable to run tests without crashing on 7.D FW

Post-commit WH B0 CI has been failing on the UMD branch. abhullar/umd

This branch requires at least 7.D firmware to run properly.

@davorchap @abhullar-tt and @TT-billteng have said this FW is good to run WH post-commit.

However, running post-commit causes a crash and the machine to reboot.

Machine info is below:

tt-admin@172.27.28.73 (no password, ask Raymond to add SSH keys)

NO SMI SCREENSHOT - machine is down currently

Will post more specific test info below.

About this issue

  • Original URL
  • State: closed
  • Created 10 months ago
  • Comments: 17 (1 by maintainers)

Most upvoted comments

Unable to get a stable CI post commit run on any WH machines today.

I have filed issues to cloud for two WH BMs that went out of service:

https://github.com/tenstorrent/cloud/issues/1269 https://github.com/tenstorrent/cloud/issues/1271

I will next try another machine with newer firmware to see if we can have CI run on that machine instead

@TT-billteng also observed the same behaviour on Saturday with DirtBox, with c++ gtests.