tt-metal: SD model fails from main on Wormhole N300

Describe the bug SD demo test fails from the main branch.

To Reproduce Steps to reproduce the behavior:

  1. Checkout to main
  2. Run the command: pytest --disable-warnings --input-path="models/demos/wormhole/stable_diffusion/demo/input_data.json" models/demos/wormhole/stable_diffusion/demo/demo.py::test_demo
  3. See the error: FAILED models/demos/wormhole/stable_diffusion/demo/demo.py::test_demo[image_size0-50-1-device_params0] - RuntimeError: TT_FATAL @ ../tt_eager/tt_dnn/op_library/sharded/sharded_op.cpp:40: this->grid_size.x <= device_grid.x && this->grid_..

Expected behavior The PyTest passes.

Please complete the following environment information:

  • OS: [e.g. Ubuntu 20.04]
  • Version of software (f587cb5)
  • N300

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (7 by maintainers)

Commits related to this issue

Most upvoted comments

I am still looking into it. I was able to root-cause the issue mentioned here fixed it but there are other layers which are failing as well. My approach is to insure to run the entire model followed by validating functional correctness and then performance.

@dvartaniansTT @mbahnasTT @vshenoyTT You will notice a hang on stable diffusion on N300 quite often, though you should observe it pass at some point.

Given time of the release, I think we will have to ship it as is and warn users not to run on N300.