tch-rs: I use tch-rs rewritten Bark, am I running out of memory? Torch("CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
Hello, I try to using tch-rs rewrite bark:
https://github.com/ThisSeanZhang/bark-rs
Rust: rustc 1.71.0-nightly (f9a6b7158 2023-05-05) Torch version: 2.0.0 System: Window 10 CUDA: 11.8 GPU: 2070 super (8GB)
But I got this error:
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Torch("CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Exception raised from c10_cuda_check_implementation at C:\\actions-runner\\_work\\pytorch\\pytorch\\builder\\windows\\pytorch\\c10\\cuda\\CUDAException.cpp:44 (most recent call first):
00007FF8BBE3D24200007FF8BBE3D1E0 c10.dll!c10::Error::Error [<unknown file> @ <unknown line number>]
00007FF8BBE3CE8A00007FF8BBE3CE30 c10.dll!c10::detail::torchCheckFail [<unknown file> @ <unknown line number>]
00007FF8B8DF834F00007FF8B8DF8040 c10_cuda.dll!c10::cuda::c10_cuda_check_implementation [<unknown file> @ <unknown line number>]
00007FF82719BDCA00007FF827103020 torch_cuda.dll!at::cuda::zero_ [<unknown file> @ <unknown line number>]
00007FF82718F0A900007FF827103020 torch_cuda.dll!at::cuda::zero_ [<unknown file> @ <unknown line number>]
00007FF827176C2100007FF827103020 torch_cuda.dll!at::cuda::zero_ [<unknown file> @ <unknown line number>]
00007FF8271A8FEA00007FF827103020 torch_cuda.dll!at::cuda::zero_ [<unknown file> @ <unknown line number>]
00007FF8271AD4F800007FF8271AD4E0 torch_cuda.dll!at::native::structured_ufunc_add_CUDA::impl [<unknown file> @ <unknown line number>]
00007FF8270B234A00007FF827080670 torch_cuda.dll!at::cuda::where_outf [<unknown file> @ <unknown line number>]
00007FF826FC7ABB00007FF826F63DA0 torch_cuda.dll!at::cuda::bucketize_outf [<unknown file> @ <unknown line number>]
00007FF86BAC4DA800007FF86BAC4D10 torch_cpu.dll!at::_ops::add_Tensor::redispatch [<unknown file> @ <unknown line number>]
00007FF86CEE565C00007FF86CD3E910 torch_cpu.dll!torch::autograd::GraphRoot::apply [<unknown file> @ <unknown line number>]
00007FF86CF102D900007FF86CD3E910 torch_cpu.dll!torch::autograd::GraphRoot::apply [<unknown file> @ <unknown line number>]
00007FF86BA37A9600007FF86BA37980 torch_cpu.dll!at::_ops::add_Tensor::call [<unknown file> @ <unknown line number>]
00007FF60CFD87E200007FF60CFD87B0 bark-rs.exe!at::add [D:\\libtorch\\200_cu118\\include\\ATen\\ops\\add.h @ 27]
00007FF60CEFA9DF00007FF60CEFA9A0 bark-rs.exe!atg_add [C:\\Users\\Sean\\.cargo\\registry\\src\\index.crates.io-6f17d22bba15001f\\torch-sys-0.12.0\\libtch\\torch_api_generated.cpp.h @ 3353]
00007FF60CD928A300007FF60CD927C0 bark-rs.exe!tch::wrappers::tensor::Tensor::f_add [C:\\Users\\Sean\\.cargo\\registry\\src\\index.crates.io-6f17d22bba15001f\\tch-0.12.0\\src\\wrappers\\tensor_fallible_generated.rs @ 8164]
00007FF60CD97E7E00007FF60CD97E60 bark-rs.exe!tch::wrappers::tensor::Tensor::g_add [C:\\Users\\Sean\\.cargo\\registry\\src\\index.crates.io-6f17d22bba15001f\\tch-0.12.0\\src\\wrappers\\tensor_generated.rs @ 4780]
00007FF60CD9AD3300007FF60CD9AD10 bark-rs.exe!tch::tensor::ops::impl$14::add [C:\\Users\\Sean\\.cargo\\registry\\src\\index.crates.io-6f17d22bba15001f\\tch-0.12.0\\src\\tensor\\ops.rs @ 48]
00007FF60C3CE83F00007FF60C3CE6F0 bark-rs.exe!bark_rs::bark_gpt::Block::forward_t [D:\\projects-rust\\bark-rs\\src\\bark_gpt.rs @ 317]
00007FF60C3D051200007FF60C3CF260 bark-rs.exe!bark_rs::bark_gpt::BarkGPT::forward_t [D:\\projects-rust\\bark-rs\\src\\bark_gpt.rs @ 452]
00007FF60C3D7E8700007FF60C3D7BB0 bark-rs.exe!bark_rs::generation::generate_text_semantic::closure$0 [D:\\projects-rust\\bark-rs\\src\\generation.rs @ 350]
00007FF60C3C3CCB00007FF60C3C3C60 bark-rs.exe!tch::wrappers::tensor::no_grad<tch::wrappers::tensor::Tensor,bark_rs::generation::generate_text_semantic::closure_env$0> [C:\\Users\\Sean\\.cargo\\registry\\src\\index.crates.io-6f17d22bba15001f\\tch-0.12.0\\src\\wrappers\\tensor.rs @ 814]
00007FF60C3CC24F00007FF60C3CB9A0 bark-rs.exe!bark_rs::generation::generate_text_semantic [D:\\projects-rust\\bark-rs\\src\\generation.rs @ 334]
00007FF60C3D3C8000007FF60C3D3BB0 bark-rs.exe!bark_rs::api::text_to_semantic [D:\\projects-rust\\bark-rs\\src\\api.rs @ 78]
00007FF60C3D3B0C00007FF60C3D3A90 bark-rs.exe!bark_rs::api::generate_audio [D:\\projects-rust\\bark-rs\\src\\api.rs @ 37]
00007FF60C3C669E00007FF60C3C65B0 bark-rs.exe!bark_rs::main [D:\\projects-rust\\bark-rs\\src\\main.rs @ 22]
00007FF60C3D4D0B00007FF60C3D4D00 bark-rs.exe!core::ops::function::FnOnce::call_once<void (*)(),tuple$<> > [/rustc/f9a6b71580cd53dd4491d9bb6400f7ee841d9c22\\library\\core\\src\\ops\\function.rs @ 250]
00007FF60C3D7AFE00007FF60C3D7AF0 bark-rs.exe!std::sys_common::backtrace::__rust_begin_short_backtrace<void (*)(),tuple$<> > [/rustc/f9a6b71580cd53dd4491d9bb6400f7ee841d9c22\\library\\std\\src\\sys_common\\backtrace.rs @ 137]
00007FF60C3DA57100007FF60C3DA560 bark-rs.exe!std::rt::lang_start::closure$0<tuple$<> > [/rustc/f9a6b71580cd53dd4491d9bb6400f7ee841d9c22\\library\\std\\src\\rt.rs @ 166]
00007FF60CE9E2B800007FF60CE9E200 bark-rs.exe!std::rt::lang_start_internal [/rustc/f9a6b71580cd53dd4491d9bb6400f7ee841d9c22/library\\std\\src\\rt.rs @ 148]
00007FF60C3DA54A00007FF60C3DA510 bark-rs.exe!std::rt::lang_start<tuple$<> > [/rustc/f9a6b71580cd53dd4491d9bb6400f7ee841d9c22\\library\\std\\src\\rt.rs @ 165]
00007FF60C3C66D900007FF60C3C66C0 bark-rs.exe!main [<unknown file> @ <unknown line number>]
00007FF60D0D2CA000007FF60D0D2B94 bark-rs.exe!__scrt_common_main_seh [D:\\a\\_work\\1\\s\\src\\vctools\\crt\\vcstartup\\src\\startup\\exe_common.inl @ 288]
00007FF907CB761400007FF907CB7600 KERNEL32.DLL!BaseThreadInitThunk
[<unknown file> @ <unknown line number>]
00007FF9094626A100007FF909462680 ntdll.dll!RtlUserThreadStart [<unknown file> @ <unknown line number>]
")', C:\Users\Sean\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tch-0.12.0\src\wrappers\tensor_generated.rs:4780:27
stack backtrace:
0: std::panicking::begin_panic_handler
at /rustc/f9a6b71580cd53dd4491d9bb6400f7ee841d9c22/library\std\src\panicking.rs:578
1: core::panicking::panic_fmt
at /rustc/f9a6b71580cd53dd4491d9bb6400f7ee841d9c22/library\core\src\panicking.rs:67
2: core::result::unwrap_failed
at /rustc/f9a6b71580cd53dd4491d9bb6400f7ee841d9c22/library\core\src\result.rs:1651
3: enum2$<core::result::Result<tch::wrappers::tensor::Tensor,enum2$<tch::error::TchError> > >::unwrap<tch::wrappers::tensor::Tensor,enum2$<tch::error::TchError> >
at /rustc/f9a6b71580cd53dd4491d9bb6400f7ee841d9c22\library\core\src\result.rs:1076
4: tch::wrappers::tensor::Tensor::g_add
at C:\Users\Sean\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tch-0.12.0\src\wrappers\tensor_generated.rs:4780
5: tch::tensor::ops::impl$14::add
at C:\Users\Sean\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tch-0.12.0\src\tensor\ops.rs:48
6: bark_rs::bark_gpt::Block::forward_t
at .\src\bark_gpt.rs:317
7: bark_rs::bark_gpt::BarkGPT::forward_t
at .\src\bark_gpt.rs:452
8: bark_rs::generation::generate_text_semantic::closure$0
at .\src\generation.rs:350
9: tch::wrappers::tensor::no_grad<tch::wrappers::tensor::Tensor,bark_rs::generation::generate_text_semantic::closure_env$0>
at C:\Users\Sean\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tch-0.12.0\src\wrappers\tensor.rs:814
10: bark_rs::generation::generate_text_semantic
at .\src\generation.rs:334
11: bark_rs::api::text_to_semantic
at .\src\api.rs:78
12: bark_rs::api::generate_audio
at .\src\api.rs:37
13: bark_rs::main
at .\src\main.rs:21
14: core::ops::function::FnOnce::call_once<void (*)(),tuple$<> >
at /rustc/f9a6b71580cd53dd4491d9bb6400f7ee841d9c22\library\core\src\ops\function.rs:250
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [316,0,0], thread: [96,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [316,0,0], thread: [97,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [316,0,0], thread: [98,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [316,0,0], thread: [99,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [316,0,0], thread: [100,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [316,0,0], thread: [101,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [316,0,0], thread: [102,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [316,0,0], thread: [103,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [316,0,0], thread: [104,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [316
Am I exceeding the memory limit?
sometimes with different error:
“thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Torch("CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`
Exception raised from gemm at C:\\actions-runner\\_work\\pytorch\\pytorch\\builder\\windows\\pytorch\\aten\\src\\ATen\\cuda\\CUDABlas.cpp:340 (most recent call first):
00007FF8E81CD24200007FF8E81CD1E0 c10.dll!c10::Error::Error [<unknown file> @ <unknown line number>]
00007FF8E81CCE8A00007FF8E81CCE30 c10.dll!c10::detail::torchCheckFail [<unknown file> @ <unknown line number>]
00007FF83B63C6B300007FF83B63C380 torch_cuda.dll!at::cuda::blas::gelsBatched<c10::complex<double> > [<unknown file> @ <unknown line number>]
00007FF83B64F8D400007FF83B64DA20 torch_cuda.dll!at::native::log_sigmoid_forward_out_cuda [<unknown file> @ <unknown line number>]
00007FF83B651F3100007FF83B64DA20 torch_cuda.dll!at::native::log_sigmoid_forward_out_cuda [<unknown file> @ <unknown line number>]
00007FF83B6533B700007FF83B653350 torch_cuda.dll!at::native::structured_mm_out_cuda::impl [<unknown file> @ <unknown line number>]
00007FF83B52B46A00007FF83B4D0670 torch_cuda.dll!at::cuda::where_outf [<unknown file> @ <unknown line number>]
00007FF83B41725600007FF83B3B3DA0 torch_cuda.dll!at::cuda::bucketize_outf [<unknown file> @ <unknown line number>]
00007FF88004993500007FF8800498B0 torch_cpu.dll!at::_ops::mm::redispatch [<unknown file> @ <unknown line number>]
00007FF88146C4ED00007FF88118E910 torch_cpu.dll!torch::autograd::GraphRoot::apply [<unknown file> @ <unknown line number>]
00007FF88143AC7C00007FF88118E910 torch_cpu.dll!torch::autograd::GraphRoot::apply [<unknown file> @ <unknown line number>]
00007FF87FFE161200007FF87FFE1510 torch_cpu.dll!at::_ops::mm::call [<unknown file> @ <unknown line number>]
00007FF87F6A4EDE00007FF87F693BE0 torch_cpu.dll!at::native::tensordot_out [<unknown file> @ <unknown line number>]
00007FF87F6B4CC700007FF87F6B4C60 torch_cpu.dll!at::native::matmul [<unknown file> @ <unknown line number>]
00007FF88056E42E00007FF88056A890 torch_cpu.dll!at::compositeimplicitautograd::where [<unknown file> @ <unknown line number>]
00007FF88055545600007FF880510BF0 torch_cpu.dll!at::compositeimplicitautograd::broadcast_to_symint [<unknown file> @ <unknown line number>]
00007FF8800D556200007FF8800D5460 torch_cpu.dll!at::_ops::matmul::call [<unknown file> @ <unknown line number>]
00007FF87F691CB800007FF87F691870 torch_cpu.dll!at::native::linear [<unknown file> @ <unknown line number>]
00007FF88056E2CE00007FF88056A890 torch_cpu.dll!at::compositeimplicitautograd::where [<unknown file> @ <unknown line number>]
00007FF88055569B00007FF880510BF0 torch_cpu.dll!at::compositeimplicitautograd::broadcast_to_symint [<unknown file> @ <unknown line number>]
00007FF87FC03D6600007FF87FC03C50 torch_cpu.dll!at::_ops::linear::call [<unknown file> @ <unknown line number>]
00007FF75E7C36C200007FF75E7C3690 bark-rs.exe!at::linear [D:\\libtorch\\200_cu118\\include\\ATen\\ops\\linear.h @ 27]
00007FF75E70733C00007FF75E707290 bark-rs.exe!atg_linear [C:\\Users\\Sean\\.cargo\\registry\\src\\index.crates.io-6f17d22bba15001f\\torch-sys-0.12.0\\libtch\\torch_api_generated.cpp.h @ 10281]
00007FF75E56321700007FF75E5630D0 bark-rs.exe!tch::wrappers::tensor::Tensor::f_linear<ref$<tch::wrappers::tensor::Tensor> > [C:\\Users\\Sean\\.cargo\\registry\\src\\index.crates.io-6f17d22bba15001f\\tch-0.12.0\\src\\wrappers\\tensor_fallible_generated.rs @ 21116]
00007FF75E5666A600007FF75E566680 bark-rs.exe!tch::wrappers::tensor::Tensor::linear<ref$<tch::wrappers::tensor::Tensor> > [C:\\Users\\Sean\\.cargo\\registry\\src\\index.crates.io-6f17d22bba15001f\\tch-0.12.0\\src\\wrappers\\tensor_generated.rs @ 10865]
00007FF75E5717E600007FF75E5717B0 bark-rs.exe!tch::nn::linear::impl$1::forward [C:\\Users\\Sean\\.cargo\\registry\\src\\index.crates.io-6f17d22bba15001f\\tch-0.12.0\\src\
n\\linear.rs @ 50]
00007FF75E568FC000007FF75E568FA0 bark-rs.exe!tch::wrappers::tensor::Tensor::apply<tch::nn::linear::Linear> [C:\\Users\\Sean\\.cargo\\registry\\src\\index.crates.io-6f17d22bba15001f\\tch-0.12.0\\src\
n\\module.rs @ 48]
00007FF75DBC264F00007FF75DBC2460 bark-rs.exe!bark_rs::bark_gpt::CausalSelfAttention::forward_t [D:\\projects-rust\\hello-ai\\bark-rs\\src\\bark_gpt.rs @ 189]
00007FF75DBC3D7800007FF75DBC3C70 bark-rs.exe!bark_rs::bark_gpt::Block::forward_t [D:\\projects-rust\\hello-ai\\bark-rs\\src\\bark_gpt.rs @ 316]
00007FF75DBC5A9200007FF75DBC47E0 bark-rs.exe!bark_rs::bark_gpt::BarkGPT::forward_t [D:\\projects-rust\\hello-ai\\bark-rs\\src\\bark_gpt.rs @ 452]
00007FF75DBCAB9300007FF75DBCA780 bark-rs.exe!bark_rs::generation::generate_text_semantic::closure$0 [D:\\projects-rust\\hello-ai\\bark-rs\\src\\generation.rs @ 348]
00007FF75DBD3F0B00007FF75DBD3EA0 bark-rs.exe!tch::wrappers::tensor::no_grad<tch::wrappers::tensor::Tensor,bark_rs::generation::generate_text_semantic::closure_env$0> [C:\\Users\\Sean\\.cargo\\registry\\src\\index.crates.io-6f17d22bba15001f\\tch-0.12.0\\src\\wrappers\\tensor.rs @ 814]
00007FF75DBDACDF00007FF75DBDA430 bark-rs.exe!bark_rs::generation::generate_text_semantic [D:\\projects-rust\\hello-ai\\bark-rs\\src\\generation.rs @ 333]
00007FF75DBCA5E000007FF75DBCA510 bark-rs.exe!bark_rs::api::text_to_semantic [D:\\projects-rust\\hello-ai\\bark-rs\\src\\api.rs @ 78]
00007FF75DBCA46C00007FF75DBCA3F0 bark-rs.exe!bark_rs::api::generate_audio [D:\\projects-rust\\hello-ai\\bark-rs\\src\\api.rs @ 37]
00007FF75DBD62CE00007FF75DBD61E0 bark-rs.exe!bark_rs::main [D:\\projects-rust\\hello-ai\\bark-rs\\src\\main.rs @ 22]
00007FF75DBC110B00007FF75DBC1100 bark-rs.exe!core::ops::function::FnOnce::call_once<void (*)(),tuple$<> > [/rustc/f9a6b71580cd53dd4491d9bb6400f7ee841d9c22\\library\\core\\src\\ops\\function.rs @ 250]
00007FF75DBC1E7E00007FF75DBC1E70 bark-rs.exe!std::sys_common::backtrace::__rust_begin_short_backtrace<void (*)(),tuple$<> > [/rustc/f9a6b71580cd53dd4491d9bb6400f7ee841d9c22\\library\\std\\src\\sys_common\\backtrace.rs @ 137]
00007FF75DBD432100007FF75DBD4310 bark-rs.exe!std::rt::lang_start::closure$0<tuple$<> > [/rustc/f9a6b71580cd53dd4491d9bb6400f7ee841d9c22\\library\\std\\src\\rt.rs @ 166]
00007FF75E66F4A800007FF75E66F3F0 bark-rs.exe!std::rt::lang_start_internal [/rustc/f9a6b71580cd53dd4491d9bb6400f7ee841d9c22/library\\std\\src\\rt.rs @ 148]
00007FF75DBD42FA00007FF75DBD42C0 bark-rs.exe!std::rt::lang_start<tuple$<> > [/rustc/f9a6b71580cd53dd4491d9bb6400f7ee841d9c22\\library\\std\\src\\rt.rs @ 165]
00007FF75DBD630900007FF75DBD62F0 bark-rs.exe!main [<unknown file> @ <unknown line number>]
00007FF75E8A39D000007FF75E8A38C4 bark-rs.exe!__scrt_common_main_seh [D:\\a\\_work\\1\\s\\src\\vctools\\crt\\vcstartup\\src\\startup\\exe_common.inl @ 288]
00007FF907CB761400007FF907CB7600 KERNEL32.DLL!BaseThreadInitThunk [<unknown file> @ <unknown line number>]
00007FF9094626A100007FF909462680 ntdll.dll!RtlUserThreadStart [<unknown file> @ <unknown line number>]
")', C:\Users\Sean\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tch-0.12.0\src\wrappers\tensor_generated.rs:10865:37
stack backtrace:
0: std::panicking::begin_panic_handler
at /rustc/f9a6b71580cd53dd4491d9bb6400f7ee841d9c22/library\std\src\panicking.rs:578
1: core::panicking::panic_fmt
at /rustc/f9a6b71580cd53dd4491d9bb6400f7ee841d9c22/library\core\src\panicking.rs:67
2: core::result::unwrap_failed
at /rustc/f9a6b71580cd53dd4491d9bb6400f7ee841d9c22/library\core\src\result.rs:1651
3: enum2$<core::result::Result<tch::wrappers::tensor::Tensor,enum2$<tch::error::TchError> > >::unwrap<tch::wrappers::tensor::Tensor,enum2$<tch::error::TchError> >
at /rustc/f9a6b71580cd53dd4491d9bb6400f7ee841d9c22\library\core\src\result.rs:1076
4: tch::wrappers::tensor::Tensor::linear<ref$<tch::wrappers::tensor::Tensor> >
at C:\Users\Sean\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tch-0.12.0\src\wrappers\tensor_generated.rs:10865
5: tch::nn::linear::impl$1::forward
at C:\Users\Sean\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tch-0.12.0\src\nn\linear.rs:49
6: tch::wrappers::tensor::Tensor::apply<tch::nn::linear::Linear>
at C:\Users\Sean\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tch-0.12.0\src\nn\module.rs:47
7: bark_rs::bark_gpt::CausalSelfAttention::forward_t
at .\src\bark_gpt.rs:189
8: bark_rs::bark_gpt::Block::forward_t
at .\src\bark_gpt.rs:316
9: bark_rs::bark_gpt::BarkGPT::forward_t
at .\src\bark_gpt.rs:452
10: bark_rs::generation::generate_text_semantic::closure$0
at .\src\generation.rs:348
11: tch::wrappers::tensor::no_grad<tch::wrappers::tensor::Tensor,bark_rs::generation::generate_text_semantic::closure_env$0>
at C:\Users\Sean\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tch-0.12.0\src\wrappers\tensor.rs:814
12: bark_rs::generation::generate_text_semantic
at .\src\generation.rs:333
13: bark_rs::api::text_to_semantic
at .\src\api.rs:78
14: bark_rs::api::generate_audio
at .\src\api.rs:37
15: bark_rs::main
at .\src\main.rs:21
16: core::ops::function::FnOnce::call_once<void (*)(),tuple$<> >
at /rustc/f9a6b71580cd53dd4491d9bb6400f7ee841d9c22\library\core\src\ops\function.rs:250
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [316,0,0], thread: [64,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [316,0,0], thread: [65,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [316,0,0], thread: [66,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [316,0,0], thread: [67,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1146: block: [316,0,0], thread: [68,0,0] Assertion `srcIndex <
Thanks in advance for any pointers!❤
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 16 (8 by maintainers)
It looks like the relevant bits are the assertion failures at the bottom of your error messages.
The assert can be seen in this file and look indexing related, the stacktraces indeed seem a bit off mentioning some linear layer, did you try running with
CUDA_LAUNCH_BLOCKING=1as suggested in the error message to see if you get something more accurate? Other suggestions would be to run on the cpu rather than on a gpu to see if the errors are better there. Another thing to try would be to trim down the code as much as possible to get a simpler repro, hopefully with just a couple operations and then it would be easier for others to potentially investigate.