midi-model: RuntimeError: probability tensor contains either `inf`, `nan` or element < 0
我更新了代码下来,生成的时候大概率会碰到这个错误。 Scores 会传入nan的情况
CUDA Version: 11.8 torch==2.0.1cu118 transformers==4.33.1 pytorch-lightning==2.0.7
具体:
打印sample_top_p_k参数,model.sample_top_p_k(scores, top_p, top_k) def sample_top_p_k(self, probs, p, k): # Debugging code print(“probs:”, probs) print(“Contains inf:”, torch.isinf(probs).any()) print(“Contains nan:”, torch.isnan(probs).any()) print(“Contains negative:”, (probs < 0).any())
probs: tensor([[[nan, nan, nan, …, nan, nan, nan]]], device=‘cuda:0’)
Contains inf: tensor(False, device=‘cuda:0’)
Contains nan: tensor(True, device=‘cuda:0’)
Contains negative: tensor(False, device=‘cuda:0’)
generating: 77%|███████▋ | 1149/1493 [01:55<00:34, 9.94it/s]
Traceback (most recent call last):
File “C:\Users\HOLA\AppData\Local\Programs\Python\Python310\lib\site-packages\gradio\routes.py”, line 427, in run_predict
output = await app.get_blocks().process_api(
File “C:\Users\HOLA\AppData\Local\Programs\Python\Python310\lib\site-packages\gradio\blocks.py”, line 1323, in process_api
result = await self.call_function(
File “C:\Users\HOLA\AppData\Local\Programs\Python\Python310\lib\site-packages\gradio\blocks.py”, line 1067, in call_function
prediction = await utils.async_iteration(iterator)
File “C:\Users\HOLA\AppData\Local\Programs\Python\Python310\lib\site-packages\gradio\utils.py”, line 336, in async_iteration
return await iterator.anext()
File “C:\Users\HOLA\AppData\Local\Programs\Python\Python310\lib\site-packages\gradio\utils.py”, line 329, in anext
return await anyio.to_thread.run_sync(
File “C:\Users\HOLA\AppData\Local\Programs\Python\Python310\lib\site-packages\anyio\to_thread.py”, line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File “C:\Users\HOLA\AppData\Local\Programs\Python\Python310\lib\site-packages\anyio_backends_asyncio.py”, line 937, in run_sync_in_worker_thread
return await future
File “C:\Users\HOLA\AppData\Local\Programs\Python\Python310\lib\site-packages\anyio_backends_asyncio.py”, line 867, in run
result = context.run(func, *args)
File “C:\Users\HOLA\AppData\Local\Programs\Python\Python310\lib\site-packages\gradio\utils.py”, line 312, in run_sync_iterator_async
return next(iterator)
File “D:\Data\Git Repository\midi-model\app.py”, line 128, in run
for i, token_seq in enumerate(generator):
File “C:\Users\HOLA\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\utils_contextlib.py”, line 56, in generator_context
response = gen.send(request)
File “D:\Data\Git Repository\midi-model\app.py”, line 62, in generate
sample = model.sample_top_p_k(scores, top_p, top_k)
File “D:\Data\Git Repository\midi-model\midi_model.py”, line 72, in sample_top_p_k
next_token = torch.multinomial(probs_sort.reshape(-1, shape[-1]), num_samples=1).reshape(*shape[:-1], 1)
RuntimeError: probability tensor contains either inf, nan or element < 0
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 18 (7 by maintainers)
这就奇怪了,我之前用的版本代码好好的,现在推理也不行