tt-metal: [Bug Report] Reciprocal unary op causes crash after all_reduce
Describe the bug
Crash while running this test. Substituting ttnn::reciprocal with any other unary op doesn’t result in crash.
TEST_F(N300UtilsTest, TestXTensorReplicateAllReduce_96_768) {
auto* device = &ttml::autograd::ctx().get_device();
auto mesh_shape = device->shape();
std::cout << "I AM HERE IN TEST 0\n";
xt::xarray<float> xtensor = xt::random::rand({96 * 768}, -0.05, 0.05).reshape({1, 1, 96, 768});
ttml::core::XTensorToMeshVariant<float> replicate_composer = ttml::core::ReplicateXTensorToMesh<float>(mesh_shape);
auto tensor = ttml::core::from_xtensor(xtensor, device, replicate_composer);
std::cout << "I AM HERE IN TEST 1\n";
tensor = ttnn::add(tensor, 100.0F);
std::cout << "I AM HERE IN TEST 2\n";
tensor = ttnn::reciprocal(tensor);
std::cout << "I AM HERE IN TEST 3\n";
}
Output:
I AM HERE IN TEST 0
I AM HERE IN TEST 1
I AM HERE IN TEST 2
Segmentation fault (core dumped)
.bashrc
export ARCH_NAME=wormhole_b0
export TT_METAL_HOME=/home/ubuntu/tt-metal
export PYTHONPATH=/home/ubuntu/tt-metal
export TT_METAL_ENV=dev
To Reproduce
./build_metal.sh -b Release --build-tt-traincd build/tt-train/ctest .(I didn’t use./build/tt-train/tests/ttml_tests)- you can run only one test with this command
ctest -R N300UtilsTest.TestXTensorReplicateAllReduce_96_768 - See crash and prints before last print
Expected behavior Should work without crash.
Please complete the following environment information:
- OS: Ubuntu 20.04
- Version of software: 9b84fb8295e922cdf784e06aa21b2bfce3e2f255
- Device: N300
Additional context
There might be a deeper problem, so we would like to understand what’s going on with this random operation. Thanks in advance.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 16 (15 by maintainers)
Commits related to this issue
- #16646: Fix dangling reference in sharded tensor args — committed to tenstorrent/tt-metal by patrickroberts 2 years ago
- #16646: Revert reciprocal to auto launch — committed to tenstorrent/tt-metal by deleted user 2 years ago
- #16646: Revert reciprocal to auto launch — committed to tenstorrent/tt-metal by deleted user 2 years ago
- #16646: Revert reciprocal to auto launch — committed to tenstorrent/tt-metal by deleted user 2 years ago
I’ve got a core dump, will rebuild in RelWithDebInfo so I can investigate more thoroughly, but for now here’s a backtrace at least:
Hey @patrickroberts I’ve noticed a few things:
In overload it overloads vs complex parameters composite op. As result reciprocal is registered without auto launch. It might be a reason why yo get some race condition like access to the tensor which is not yet set.
Overall it means we should make sure that no simple ops are registered without auto launch. There might be more ops where people forget to do it. I am not sure 100% that it is the reason of the crash. cc @ayerofieiev-tt and @sminakov-tt because it might be related to the compute output specs.
💥
I might have identified the problem, I’m compiling an attempted fix to test, will let you know if it works.