sentencepiece: `sentencepiece==0.1.92` seems breaking something

with newly released sentencepiece==0.1.92

Python 3.6.9 (default, Nov  7 2019, 10:44:02)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import transformers, torch
>>> transformers.__version__
'2.9.1'
>>> torch.__version__
'1.4.0'
>>> torch.rand(3)
Segmentation fault (core dumped)

However, downgrade to sentencepiece==0.1.91 solves this issue

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 11
Comments: 16 (1 by maintainers)

Most upvoted comments

I can confirm, I have been having the exact same issue with version 0.1.92 which I don’t with 0.1.91. In my case, I’m having a segmentation fault on pytorch (1.3.1) and transformers (2.11.0) torch.nn.Module init. Using faulthandler I managed to get the details that it creates a seg fault at the following:

Current thread 0x00007fa347061280 (most recent call first):
  File ".../lib/python3.7/site-packages/torch/nn/modules/module.py", line 74 in __init__
  File ".../lib/python3.7/site-packages/torch/nn/modules/loss.py", line 10 in __init__
  File ".../lib/python3.7/site-packages/torch/nn/modules/loss.py", line 593 in __init__

That line happens to be:

torch._C._log_api_usage_once("python.nn_module")

I suspect something in the sentencepiece new version prevent torch binaries from normally register its resources.

Downgrading to version 0.1.91 fixes the issue for me as well.

hugoperrin on Jun 15, 2020

Not sure how related this is but it might help. installing & importing transformers would cause a segmentation fault. I fixed it by pip install sentencepiece==0.1.91.

Here’s my stack trace

wassimseif on Nov 8, 2020

Is it the same as https://github.com/pytorch/pytorch/issues/8358? This looks like another case of incompatibility with PyTorch which releases non standard manylinux1 packages, not SentencePiece fault.

Sometimes the import order helps working around the issue:

$ python
>>> import torch
>>> import sentencepiece
>>> torch.nn.Module()
Segmentation fault
$ python
>>> import sentencepiece
>>> import torch
>>> torch.nn.Module()
Module()

guillaumekln on Jun 18, 2020

Downgraded.

taku910 on Jun 9, 2020