tt-metal: ttnn.full and ttnn.ones operations fail by assert valid_page_size on some dimensions
When testing ttnn ops with TILE layout and random shapes like [4, 7, 21, 133] most of the operations work correctly.
But ttnn.full and ttnn.ones operations fail with assertion like:
TT_FATAL @ ../tt_metal/impl/buffers/buffer.cpp:31: valid_page_size
info:
For valid non-interleaved buffers page size 2048 must equal buffer size 156408. For interleaved-buffers page size should be divisible by buffer size
which is followed by crash:
corrupted size vs. prev_size
Fatal Python error: Aborted
or sometimes:
malloc(): invalid size (unsorted)
Fatal Python error: Aborted
Note that similar operation ttnn.zeros works in same conditions!
Problem is observed on both GS and WH cards.
To Reproduce Steps to reproduce the behavior:
- Checkout branch
ngrujic/op_bug_unit_tests(soon to be merged intomain). - Run unit test
test_eltwise_full_and_ones.pyusing this command:
pytest tests/ttnn/python_api_testing/non_working_unit_tests/wormhole/test_eltwise_full_and_ones.py
Expected behavior There are few test cases presented in the unit test, which are failing with:
TT_FATAL @ ../tt_metal/impl/buffers/buffer.cpp:31: valid_page_size
info:
For valid non-interleaved buffers page size 2048 must equal buffer size 156408. For interleaved-buffers page size should be divisible by buffer size
backtrace:
--- tt::tt_metal::Buffer::Buffer(tt::tt_metal::Device*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::optional<tt::tt_metal::ShardSpecBuffer>, bool)
--- tt::tt_metal::tensor_impl::detail::allocate_interleaved_buffer_on_device(unsigned int, tt::tt_metal::Device*, tt::tt_metal::Shape const&, tt::tt_metal::DataType, tt::tt_metal::Layout, tt::tt_metal::MemoryConfig const&)
--- tt::tt_metal::tensor_impl::allocate_buffer_on_device(unsigned int, tt::tt_metal::Device*, tt::tt_metal::Shape const&, tt::tt_metal::DataType, tt::tt_metal::Layout, tt::tt_metal::MemoryConfig const&, std::optional<tt::tt_metal::ShardSpecBuffer>)
--- /home/ubuntu/tt-metal/build/lib/libtt_eager.so(+0x6dab76) [0x7f6b51f6ab76]
--- /home/ubuntu/tt-metal/build/lib/libtt_eager.so(_ZNSt8__detail9__variant17__gen_vtable_implILb1ENS0_12_Multi_arrayIPFSt10shared_ptrIN2tt8tt_metal6BufferEEOZNS5_11tensor_impl16to_device_bufferI8bfloat16EES7_RKSt7variantIJNS5_12OwnedStorageENS5_13DeviceStorageENS5_15BorrowedStorageENS5_22MultiDeviceHostStorageENS5_18MultiDeviceStorageEEEPNS5_6DeviceERKNS5_5ShapeENS5_8DataTypeENS5_6LayoutERKNS5_12MemoryConfigESt8optionalINS5_15ShardSpecBufferEESU_ISt17reference_wrapperINS5_12CommandQueueEEEEUlOT_E_SJ_EJEEESt5tupleIJSJ_EESt16integer_sequenceImJLm0EEEE14__visit_invokeES14_SJ_+0x22) [0x7f6b51f6b0c2]
--- /home/ubuntu/tt-metal/build/lib/libtt_eager.so(+0x6a8140) [0x7f6b51f38140]
--- std::_Function_handler<tt::tt_metal::Tensor (tt::tt_metal::Tensor const&, tt::tt_metal::Device*, tt::tt_metal::MemoryConfig const&, std::optional<std::reference_wrapper<tt::tt_metal::CommandQueue> >), tt::tt_metal::Tensor (*)(tt::tt_metal::Tensor const&, tt::tt_metal::Device*, tt::tt_metal::MemoryConfig const&, std::optional<std::reference_wrapper<tt::tt_metal::CommandQueue> >)>::_M_invoke(std::_Any_data const&, tt::tt_metal::Tensor const&, tt::tt_metal::Device*&&, tt::tt_metal::MemoryConfig const&, std::optional<std::reference_wrapper<tt::tt_metal::CommandQueue> >&&)
--- tt::tt_metal::tensor_impl::to_device_wrapper(tt::tt_metal::Tensor const&, tt::tt_metal::Device*, tt::tt_metal::MemoryConfig const&, std::optional<std::reference_wrapper<tt::tt_metal::CommandQueue> >)
--- /home/ubuntu/tt-metal/build/lib/libtt_eager.so(+0x764d86) [0x7f6b51ff4d86]
--- tt::tt_metal::Device::push_work(std::function<void ()>&&, bool)
--- tt::tt_metal::Tensor::to(tt::tt_metal::Device*, tt::tt_metal::MemoryConfig const&) const
--- /home/ubuntu/tt-metal/build/lib/libtt_eager.so(+0x562a17) [0x7f6b51df2a17]
--- tt::tt_metal::full(tt::tt_metal::Shape, float, tt::tt_metal::DataType, tt::tt_metal::Layout, tt::tt_metal::Device*, tt::tt_metal::MemoryConfig const&)
--- /home/ubuntu/tt-metal/tt_eager/tt_lib/_C.so(+0x1d13f9) [0x7f6b523683f9]
--- /home/ubuntu/tt-metal/tt_eager/tt_lib/_C.so(+0x1038e1) [0x7f6b5229a8e1]
--- python(PyCFunction_Call+0x59) [0x5d5499]
--- python(_PyEval_EvalFrameDefault+0x6eb0) [0x54d9f0]
--- python(_PyEval_EvalCodeWithName+0x26a) [0x54552a]
--- python(_PyFunction_Vectorcall+0x393) [0x5d5a23]
--- python(PyObject_Call+0x62) [0x5d4c12]
--- python(_PyEval_EvalFrameDefault+0x1f26) [0x548a66]
--- python(_PyEval_EvalCodeWithName+0x26a) [0x54552a]
--- python(_PyFunction_Vectorcall+0x393) [0x5d5a23]
--- python(PyObject_Call+0x62) [0x5d4c12]
--- python(_PyEval_EvalFrameDefault+0x1f26) [0x548a66]
--- python(_PyEval_EvalCodeWithName+0x26a) [0x54552a]
--- python(_PyFunction_Vectorcall+0x393) [0x5d5a23]
--- python() [0x57a4af]
--- python(_PyObject_MakeTpCall+0x296) [0x5d6066]
--- python(_PyEval_EvalFrameDefault+0x690a) [0x54d44a]
--- python(_PyEval_EvalCodeWithName+0x26a) [0x54552a]
--- python(_PyFunction_Vectorcall+0x393) [0x5d5a23]
--- python(PyObject_Call+0x62) [0x5d4c12]
--- python(_PyEval_EvalFrameDefault+0x1f26) [0x548a66]
--- python(_PyEval_EvalCodeWithName+0x26a) [0x54552a]
--- python(_PyFunction_Vectorcall+0x393) [0x5d5a23]
--- python(PyObject_Call+0x62) [0x5d4c12]
--- python(_PyEval_EvalFrameDefault+0x1f26) [0x548a66]
--- python(_PyEval_EvalCodeWithName+0x26a) [0x54552a]
--- python(_PyFunction_Vectorcall+0x393) [0x5d5a23]
--- python(PyObject_Call+0x62) [0x5d4c12]
--- python(_PyEval_EvalFrameDefault+0x1f26) [0x548a66]
--- python(_PyEval_EvalCodeWithName+0x26a) [0x54552a]
--- python(_PyFunction_Vectorcall+0x393) [0x5d5a23]
--- python() [0x57a4af]
--- python(_PyObject_MakeTpCall+0x296) [0x5d6066]
--- python(_PyEval_EvalFrameDefault+0x690a) [0x54d44a]
--- python(_PyEval_EvalCodeWithName+0x26a) [0x54552a]
--- python(_PyFunction_Vectorcall+0x393) [0x5d5a23]
--- python(PyObject_Call+0x62) [0x5d4c12]
--- python(_PyEval_EvalFrameDefault+0x1f26) [0x548a66]
--- python(_PyEval_EvalCodeWithName+0x26a) [0x54552a]
--- python(_PyFunction_Vectorcall+0x393) [0x5d5a23]
--- python(PyObject_Call+0x62) [0x5d4c12]
--- python(_PyEval_EvalFrameDefault+0x1f26) [0x548a66]
--- python(_PyEval_EvalCodeWithName+0x26a) [0x54552a]
--- python(_PyFunction_Vectorcall+0x393) [0x5d5a23]
--- python(PyObject_Call+0x62) [0x5d4c12]
--- python(_PyEval_EvalFrameDefault+0x1f26) [0x548a66]
--- python(_PyEval_EvalCodeWithName+0x26a) [0x54552a]
--- python(_PyFunction_Vectorcall+0x393) [0x5d5a23]
--- python(_PyEval_EvalFrameDefault+0x725) [0x547265]
--- python(_PyEval_EvalCodeWithName+0x26a) [0x54552a]
--- python(_PyFunction_Vectorcall+0x393) [0x5d5a23]
--- python(_PyEval_EvalFrameDefault+0x725) [0x547265]
--- python(_PyFunction_Vectorcall+0x1b6) [0x5d5846]
--- python(_PyEval_EvalFrameDefault+0x725) [0x547265]
--- python(_PyEval_EvalCodeWithName+0x26a) [0x54552a]
--- python(PyEval_EvalCode+0x27) [0x684327]
--- python() [0x673a41]
--- python() [0x673abb]
--- python() [0x673b61]
--- python(PyRun_SimpleFileExFlags+0x197) [0x6747e7]
--- python(Py_RunMain+0x212) [0x6b4072]
--- python(Py_BytesMain+0x2d) [0x6b43fd]
--- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f6b915b9083]
--- python(_start+0x2e) [0x5da67e]
which is followed by crash:
corrupted size vs. prev_size
Fatal Python error: Aborted
Current thread 0x00007f1d5c860740 (most recent call first):
File "/home/ubuntu/tt-metal/tests/tt_eager/python_api_testing/sweep_tests/common.py", line 50 in run_tt_lib_test
File "/home/ubuntu/tt-metal/tests/tt_eager/python_api_testing/sweep_tests/common.py", line 74 in _try_except_wrapper
File "/home/ubuntu/tt-metal/tests/tt_eager/python_api_testing/sweep_tests/common.py", line 93 in run_test_and_save_results
File "/home/ubuntu/tt-metal/tests/tt_eager/python_api_testing/sweep_tests/run_pytorch_test.py", line 241 in run_sweep_test
File "/home/ubuntu/tt-metal/tests/tt_eager/python_api_testing/sweep_tests/run_pytorch_test.py", line 301 in run_sweep_tests
File "/home/ubuntu/tt-metal/tests/ttnn/python_api_testing/sweep_tests/run_sweep_test.py", line 27 in test_run_sweep
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/python.py", line 195 in pytest_pyfunc_call
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/python.py", line 1789 in runtest
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 167 in pytest_runtest_call
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 260 in <lambda>
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 339 in from_call
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 259 in call_runtest_hook
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 220 in call_and_report
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 131 in runtestprotocol
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 112 in pytest_runtest_protocol
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 349 in pytest_runtestloop
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 324 in _main
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 270 in wrap_session
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/config/__init__.py", line 167 in main
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/config/__init__.py", line 190 in console_main
File "/home/ubuntu/tt-metal/python_env/bin/pytest", line 8 in <module>
Aborted (core dumped)
Note to observe this crash (and not only assertion fail), you must run sweep:
Running sweeps To get additional information and results for different combinations of input shapes, types, layouts and memory configs for which this operation was tested you can also run locally sweeps and check the results. To do this you should:
- Run non working sweep by using
pytest tests/ttnn/python_api_testing/sweep_tests/run_sweep_test.py --input-path tests/ttnn/python_api_testing/sweep_tests/test_configs/ci_sweep_tests_broken/wormhole/ttnn_eltwise_full_test.yaml --input-method cli --cli-input results_ttnn_full - After the run is completed all test sweeps results should be available inside specified output directory.
There are more sweeps which you can try by changing the above command to target files:
tests/ttnn/python_api_testing/sweep_tests/test_configs/ci_sweep_tests_broken/wormhole/ttnn_eltwise_full_test.yaml
tests/ttnn/python_api_testing/sweep_tests/test_configs/ci_sweep_tests_broken/wormhole/ttnn_eltwise_ones_test.yaml
tests/ttnn/python_api_testing/sweep_tests/test_configs/ci_sweep_tests_broken/grayskull/ttnn_eltwise_full_test.yaml
tests/ttnn/python_api_testing/sweep_tests/test_configs/ci_sweep_tests_broken/grayskull/ttnn_eltwise_ones_test.yaml
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 15 (15 by maintainers)
Commits related to this issue
- #8633: Add unit test for full and ones valid_page_size bug — committed to tenstorrent/tt-metal by nemanjagrujic 3 years ago
- #8633: Add unit test for full and ones valid_page_size bug — committed to tenstorrent/tt-metal by nemanjagrujic 3 years ago
- #8633: Add unit test for full and ones valid_page_size bug — committed to tenstorrent/tt-metal by nemanjagrujic 3 years ago
- #8633: Add unit test for full and ones valid_page_size bug — committed to tenstorrent/tt-metal by nemanjagrujic 3 years ago
- #8633: Fix for full and ones op — committed to tenstorrent/tt-metal by ruthreshx 2 years ago
- #8633: Add tt_fatal for full and ones op — committed to tenstorrent/tt-metal by ruthreshx 2 years ago
- #8633: Add tt_fatal for full and ones op — committed to tenstorrent/tt-metal by ruthreshx 2 years ago
- #8633: Add tt_fatal for full and ones op — committed to tenstorrent/tt-metal by ruthreshx 2 years ago
- #8633:Add TT_Fatal for full and ones op (#12921) #8633: Add tt_fatal for full and ones op — committed to tenstorrent/tt-metal by ruthreshx 2 years ago
- #8633: Update ttnn.full, ttnn.ones, ttnn.zeros documentation — committed to tenstorrent/tt-metal by VirdhatchaniKN 2 years ago
- #8633: Update ttnn.full, ttnn.ones, ttnn.zeros documentation — committed to tenstorrent/tt-metal by VirdhatchaniKN 2 years ago
- #8633: Update ttnn.full, ttnn.ones, ttnn.zeros documentation — committed to tenstorrent/tt-metal by VirdhatchaniKN 2 years ago
- #8633: Update ttnn.full, ttnn.ones, ttnn.zeros documentation — committed to tenstorrent/tt-metal by VirdhatchaniKN 2 years ago
- #8633: Update ttnn.full, ttnn.ones, ttnn.zeros documentation — committed to tenstorrent/tt-metal by VirdhatchaniKN 2 years ago
- #8633: Update ttnn.full, ttnn.ones, ttnn.zeros documentation (#13808) — committed to tenstorrent/tt-metal by VirdhatchaniKN 2 years ago
- #8633: Update ttnn.full, ttnn.ones, ttnn.zeros documentation (#13808) — committed to ct-clmsn/tt-metal by VirdhatchaniKN 2 years ago
@VirdhatchaniKN Have in mind that the limitation is not added to docs:
https://docs.tenstorrent.com/tt-metal/latest/ttnn/ttnn/api/ttnn.full.html
and, example given in docs does not work since we have this limitation.