tt-metal: ttnn.full and ttnn.ones operations fail by assert valid_page_size on some dimensions

When testing ttnn ops with TILE layout and random shapes like [4, 7, 21, 133] most of the operations work correctly.

But ttnn.full and ttnn.ones operations fail with assertion like:

TT_FATAL @ ../tt_metal/impl/buffers/buffer.cpp:31: valid_page_size
info:
For valid non-interleaved buffers page size 2048 must equal buffer size 156408. For interleaved-buffers page size should be divisible by buffer size

which is followed by crash:

corrupted size vs. prev_size
Fatal Python error: Aborted

or sometimes:

malloc(): invalid size (unsorted)
Fatal Python error: Aborted

Note that similar operation ttnn.zeros works in same conditions!

Problem is observed on both GS and WH cards.

To Reproduce Steps to reproduce the behavior:

  1. Checkout branch ngrujic/op_bug_unit_tests (soon to be merged into main).
  2. Run unit test test_eltwise_full_and_ones.py using this command:
pytest tests/ttnn/python_api_testing/non_working_unit_tests/wormhole/test_eltwise_full_and_ones.py

Expected behavior There are few test cases presented in the unit test, which are failing with:

TT_FATAL @ ../tt_metal/impl/buffers/buffer.cpp:31: valid_page_size
info:
For valid non-interleaved buffers page size 2048 must equal buffer size 156408. For interleaved-buffers page size should be divisible by buffer size
backtrace:
 --- tt::tt_metal::Buffer::Buffer(tt::tt_metal::Device*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::optional<tt::tt_metal::ShardSpecBuffer>, bool)
 --- tt::tt_metal::tensor_impl::detail::allocate_interleaved_buffer_on_device(unsigned int, tt::tt_metal::Device*, tt::tt_metal::Shape const&, tt::tt_metal::DataType, tt::tt_metal::Layout, tt::tt_metal::MemoryConfig const&)
 --- tt::tt_metal::tensor_impl::allocate_buffer_on_device(unsigned int, tt::tt_metal::Device*, tt::tt_metal::Shape const&, tt::tt_metal::DataType, tt::tt_metal::Layout, tt::tt_metal::MemoryConfig const&, std::optional<tt::tt_metal::ShardSpecBuffer>)
 --- /home/ubuntu/tt-metal/build/lib/libtt_eager.so(+0x6dab76) [0x7f6b51f6ab76]
 --- /home/ubuntu/tt-metal/build/lib/libtt_eager.so(_ZNSt8__detail9__variant17__gen_vtable_implILb1ENS0_12_Multi_arrayIPFSt10shared_ptrIN2tt8tt_metal6BufferEEOZNS5_11tensor_impl16to_device_bufferI8bfloat16EES7_RKSt7variantIJNS5_12OwnedStorageENS5_13DeviceStorageENS5_15BorrowedStorageENS5_22MultiDeviceHostStorageENS5_18MultiDeviceStorageEEEPNS5_6DeviceERKNS5_5ShapeENS5_8DataTypeENS5_6LayoutERKNS5_12MemoryConfigESt8optionalINS5_15ShardSpecBufferEESU_ISt17reference_wrapperINS5_12CommandQueueEEEEUlOT_E_SJ_EJEEESt5tupleIJSJ_EESt16integer_sequenceImJLm0EEEE14__visit_invokeES14_SJ_+0x22) [0x7f6b51f6b0c2]
 --- /home/ubuntu/tt-metal/build/lib/libtt_eager.so(+0x6a8140) [0x7f6b51f38140]
 --- std::_Function_handler<tt::tt_metal::Tensor (tt::tt_metal::Tensor const&, tt::tt_metal::Device*, tt::tt_metal::MemoryConfig const&, std::optional<std::reference_wrapper<tt::tt_metal::CommandQueue> >), tt::tt_metal::Tensor (*)(tt::tt_metal::Tensor const&, tt::tt_metal::Device*, tt::tt_metal::MemoryConfig const&, std::optional<std::reference_wrapper<tt::tt_metal::CommandQueue> >)>::_M_invoke(std::_Any_data const&, tt::tt_metal::Tensor const&, tt::tt_metal::Device*&&, tt::tt_metal::MemoryConfig const&, std::optional<std::reference_wrapper<tt::tt_metal::CommandQueue> >&&)
 --- tt::tt_metal::tensor_impl::to_device_wrapper(tt::tt_metal::Tensor const&, tt::tt_metal::Device*, tt::tt_metal::MemoryConfig const&, std::optional<std::reference_wrapper<tt::tt_metal::CommandQueue> >)
 --- /home/ubuntu/tt-metal/build/lib/libtt_eager.so(+0x764d86) [0x7f6b51ff4d86]
 --- tt::tt_metal::Device::push_work(std::function<void ()>&&, bool)
 --- tt::tt_metal::Tensor::to(tt::tt_metal::Device*, tt::tt_metal::MemoryConfig const&) const
 --- /home/ubuntu/tt-metal/build/lib/libtt_eager.so(+0x562a17) [0x7f6b51df2a17]
 --- tt::tt_metal::full(tt::tt_metal::Shape, float, tt::tt_metal::DataType, tt::tt_metal::Layout, tt::tt_metal::Device*, tt::tt_metal::MemoryConfig const&)
 --- /home/ubuntu/tt-metal/tt_eager/tt_lib/_C.so(+0x1d13f9) [0x7f6b523683f9]
 --- /home/ubuntu/tt-metal/tt_eager/tt_lib/_C.so(+0x1038e1) [0x7f6b5229a8e1]
 --- python(PyCFunction_Call+0x59) [0x5d5499]
 --- python(_PyEval_EvalFrameDefault+0x6eb0) [0x54d9f0]
 --- python(_PyEval_EvalCodeWithName+0x26a) [0x54552a]
 --- python(_PyFunction_Vectorcall+0x393) [0x5d5a23]
 --- python(PyObject_Call+0x62) [0x5d4c12]
 --- python(_PyEval_EvalFrameDefault+0x1f26) [0x548a66]
 --- python(_PyEval_EvalCodeWithName+0x26a) [0x54552a]
 --- python(_PyFunction_Vectorcall+0x393) [0x5d5a23]
 --- python(PyObject_Call+0x62) [0x5d4c12]
 --- python(_PyEval_EvalFrameDefault+0x1f26) [0x548a66]
 --- python(_PyEval_EvalCodeWithName+0x26a) [0x54552a]
 --- python(_PyFunction_Vectorcall+0x393) [0x5d5a23]
 --- python() [0x57a4af]
 --- python(_PyObject_MakeTpCall+0x296) [0x5d6066]
 --- python(_PyEval_EvalFrameDefault+0x690a) [0x54d44a]
 --- python(_PyEval_EvalCodeWithName+0x26a) [0x54552a]
 --- python(_PyFunction_Vectorcall+0x393) [0x5d5a23]
 --- python(PyObject_Call+0x62) [0x5d4c12]
 --- python(_PyEval_EvalFrameDefault+0x1f26) [0x548a66]
 --- python(_PyEval_EvalCodeWithName+0x26a) [0x54552a]
 --- python(_PyFunction_Vectorcall+0x393) [0x5d5a23]
 --- python(PyObject_Call+0x62) [0x5d4c12]
 --- python(_PyEval_EvalFrameDefault+0x1f26) [0x548a66]
 --- python(_PyEval_EvalCodeWithName+0x26a) [0x54552a]
 --- python(_PyFunction_Vectorcall+0x393) [0x5d5a23]
 --- python(PyObject_Call+0x62) [0x5d4c12]
 --- python(_PyEval_EvalFrameDefault+0x1f26) [0x548a66]
 --- python(_PyEval_EvalCodeWithName+0x26a) [0x54552a]
 --- python(_PyFunction_Vectorcall+0x393) [0x5d5a23]
 --- python() [0x57a4af]
 --- python(_PyObject_MakeTpCall+0x296) [0x5d6066]
 --- python(_PyEval_EvalFrameDefault+0x690a) [0x54d44a]
 --- python(_PyEval_EvalCodeWithName+0x26a) [0x54552a]
 --- python(_PyFunction_Vectorcall+0x393) [0x5d5a23]
 --- python(PyObject_Call+0x62) [0x5d4c12]
 --- python(_PyEval_EvalFrameDefault+0x1f26) [0x548a66]
 --- python(_PyEval_EvalCodeWithName+0x26a) [0x54552a]
 --- python(_PyFunction_Vectorcall+0x393) [0x5d5a23]
 --- python(PyObject_Call+0x62) [0x5d4c12]
 --- python(_PyEval_EvalFrameDefault+0x1f26) [0x548a66]
 --- python(_PyEval_EvalCodeWithName+0x26a) [0x54552a]
 --- python(_PyFunction_Vectorcall+0x393) [0x5d5a23]
 --- python(PyObject_Call+0x62) [0x5d4c12]
 --- python(_PyEval_EvalFrameDefault+0x1f26) [0x548a66]
 --- python(_PyEval_EvalCodeWithName+0x26a) [0x54552a]
 --- python(_PyFunction_Vectorcall+0x393) [0x5d5a23]
 --- python(_PyEval_EvalFrameDefault+0x725) [0x547265]
 --- python(_PyEval_EvalCodeWithName+0x26a) [0x54552a]
 --- python(_PyFunction_Vectorcall+0x393) [0x5d5a23]
 --- python(_PyEval_EvalFrameDefault+0x725) [0x547265]
 --- python(_PyFunction_Vectorcall+0x1b6) [0x5d5846]
 --- python(_PyEval_EvalFrameDefault+0x725) [0x547265]
 --- python(_PyEval_EvalCodeWithName+0x26a) [0x54552a]
 --- python(PyEval_EvalCode+0x27) [0x684327]
 --- python() [0x673a41]
 --- python() [0x673abb]
 --- python() [0x673b61]
 --- python(PyRun_SimpleFileExFlags+0x197) [0x6747e7]
 --- python(Py_RunMain+0x212) [0x6b4072]
 --- python(Py_BytesMain+0x2d) [0x6b43fd]
 --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f6b915b9083]
 --- python(_start+0x2e) [0x5da67e]

which is followed by crash:

corrupted size vs. prev_size
Fatal Python error: Aborted

Current thread 0x00007f1d5c860740 (most recent call first):
  File "/home/ubuntu/tt-metal/tests/tt_eager/python_api_testing/sweep_tests/common.py", line 50 in run_tt_lib_test
  File "/home/ubuntu/tt-metal/tests/tt_eager/python_api_testing/sweep_tests/common.py", line 74 in _try_except_wrapper
  File "/home/ubuntu/tt-metal/tests/tt_eager/python_api_testing/sweep_tests/common.py", line 93 in run_test_and_save_results
  File "/home/ubuntu/tt-metal/tests/tt_eager/python_api_testing/sweep_tests/run_pytorch_test.py", line 241 in run_sweep_test
  File "/home/ubuntu/tt-metal/tests/tt_eager/python_api_testing/sweep_tests/run_pytorch_test.py", line 301 in run_sweep_tests
  File "/home/ubuntu/tt-metal/tests/ttnn/python_api_testing/sweep_tests/run_sweep_test.py", line 27 in test_run_sweep
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/python.py", line 195 in pytest_pyfunc_call
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/python.py", line 1789 in runtest
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 167 in pytest_runtest_call
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 260 in <lambda>
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 339 in from_call
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 259 in call_runtest_hook
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 220 in call_and_report
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 131 in runtestprotocol
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 112 in pytest_runtest_protocol
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 349 in pytest_runtestloop
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 324 in _main
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 270 in wrap_session
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/config/__init__.py", line 167 in main
  File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/config/__init__.py", line 190 in console_main
  File "/home/ubuntu/tt-metal/python_env/bin/pytest", line 8 in <module>
Aborted (core dumped)

Note to observe this crash (and not only assertion fail), you must run sweep:

Running sweeps To get additional information and results for different combinations of input shapes, types, layouts and memory configs for which this operation was tested you can also run locally sweeps and check the results. To do this you should:

  1. Run non working sweep by using pytest tests/ttnn/python_api_testing/sweep_tests/run_sweep_test.py --input-path tests/ttnn/python_api_testing/sweep_tests/test_configs/ci_sweep_tests_broken/wormhole/ttnn_eltwise_full_test.yaml --input-method cli --cli-input results_ttnn_full
  2. After the run is completed all test sweeps results should be available inside specified output directory.

There are more sweeps which you can try by changing the above command to target files:

tests/ttnn/python_api_testing/sweep_tests/test_configs/ci_sweep_tests_broken/wormhole/ttnn_eltwise_full_test.yaml
tests/ttnn/python_api_testing/sweep_tests/test_configs/ci_sweep_tests_broken/wormhole/ttnn_eltwise_ones_test.yaml
tests/ttnn/python_api_testing/sweep_tests/test_configs/ci_sweep_tests_broken/grayskull/ttnn_eltwise_full_test.yaml
tests/ttnn/python_api_testing/sweep_tests/test_configs/ci_sweep_tests_broken/grayskull/ttnn_eltwise_ones_test.yaml

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15 (15 by maintainers)

Commits related to this issue

Most upvoted comments

Merged PR #12921 to main. can we close this issue @nemanjagrujic

@VirdhatchaniKN Have in mind that the limitation is not added to docs:

https://docs.tenstorrent.com/tt-metal/latest/ttnn/ttnn/api/ttnn.full.html

and, example given in docs does not work since we have this limitation.