tt-metal: SD model fails from main on Wormhole N300
Describe the bug SD demo test fails from the main branch.
To Reproduce Steps to reproduce the behavior:
- Checkout to main
- Run the command:
pytest --disable-warnings --input-path="models/demos/wormhole/stable_diffusion/demo/input_data.json" models/demos/wormhole/stable_diffusion/demo/demo.py::test_demo - See the error:
FAILED models/demos/wormhole/stable_diffusion/demo/demo.py::test_demo[image_size0-50-1-device_params0] - RuntimeError: TT_FATAL @ ../tt_eager/tt_dnn/op_library/sharded/sharded_op.cpp:40: this->grid_size.x <= device_grid.x && this->grid_..
Expected behavior The PyTest passes.
Please complete the following environment information:
- OS: [e.g. Ubuntu 20.04]
- Version of software (f587cb5)
- N300
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 15 (7 by maintainers)
Commits related to this issue
- #9945: Skip SD for nightly FD, device perf tests, and single-card demos as it hangs on di/dt — committed to tenstorrent/tt-metal by tt-rkim 2 years ago
- #9945: Skip SD for nightly FD, device perf tests, and single-card demos as it hangs on di/dt — committed to tenstorrent/tt-metal by tt-rkim 2 years ago
- #9945: Skip SD for nightly FD, device perf tests, and single-card demos as it hangs on di/dt — committed to tenstorrent/tt-metal by tt-rkim 2 years ago
- #9945: Skip SD for nightly FD, device perf tests, and single-card demos as it hangs on di/dt (#10179) — committed to tenstorrent/tt-metal by tt-rkim 2 years ago
- Revert "#9945: Skip SD for nightly FD, device perf tests, and single-card demos as it hangs on di/dt (#10179)" This reverts commit 6db20ff5ae988431538f91e3eb54b26d4529f020. — committed to tenstorrent/tt-metal by tt-rkim 2 years ago
- Revert "#9945: Skip SD for nightly FD, device perf tests, and single-card demos as it hangs on di/dt (#10179)" This reverts commit 6db20ff5ae988431538f91e3eb54b26d4529f020. — committed to tenstorrent/tt-metal by tt-rkim 2 years ago
- Revert "#9945: Skip SD for nightly FD, device perf tests, and single-card demos as it hangs on di/dt (#10179)" This reverts commit 6db20ff5ae988431538f91e3eb54b26d4529f020. — committed to tenstorrent/tt-metal by tt-rkim 2 years ago
- Revert "Revert "#9945: Skip SD for nightly FD, device perf tests, and single-card demos as it hangs on di/dt (#10179)"" Because it may still be messing us up here This reverts commit 70ac45d02042253... — committed to tenstorrent/tt-metal by tt-rkim 2 years ago
- #9945: fix n300 crash issue — committed to tenstorrent/tt-metal by shwetankTT 2 years ago
- #9945: fix n300 crash issue — committed to tenstorrent/tt-metal by shwetankTT 2 years ago
- #9945: Enable and fix device perf test — committed to tenstorrent/tt-metal by s-jovic 2 years ago
- #9945: Enable and fix device perf tests — committed to tenstorrent/tt-metal by s-jovic 2 years ago
- #9945: Enable and fix device perf tests — committed to tenstorrent/tt-metal by s-jovic 2 years ago
- #9945: Address comments — committed to tenstorrent/tt-metal by s-jovic 2 years ago
- #9945: Enable and fix SD device perf test (#17025) — committed to tenstorrent/tt-metal by s-jovic 2 years ago
- #9945: Enable and fix SD device perf test (#17025) — committed to tenstorrent/tt-metal by s-jovic 2 years ago
- #9945: Enable and fix SD device perf test (#17025) — committed to tenstorrent/tt-metal by s-jovic 2 years ago
- #9945: Enable and fix SD device perf test (#17025) — committed to tenstorrent/tt-metal by s-jovic 2 years ago
I am still looking into it. I was able to root-cause the issue mentioned here fixed it but there are other layers which are failing as well. My approach is to insure to run the entire model followed by validating functional correctness and then performance.
@dvartaniansTT @mbahnasTT @vshenoyTT You will notice a hang on stable diffusion on N300 quite often, though you should observe it pass at some point.
Given time of the release, I think we will have to ship it as is and warn users not to run on N300.