neuralcoref: binary incompatibility

I’m using the current spaCy from the master branch, and getting this error:

RuntimeWarning: spacy.tokens.span.Span size changed, may indicate binary incompatibility. Expected 72 from C header, got 80 from PyObject

I’m assuming this happens because span.pxd has changed after the 2.1 release: https://github.com/explosion/spaCy/commits/master/spacy/tokens/span.pxd

I tried reinstalling with

pip install neuralcoref --no-binary neuralcoref

But the warning remains and the program crashes when I run nlp(doc):

Process finished with exit code -1073741819 (0xC0000005)

Any idea on how to fix this? I’m compiling spaCy from sources too, so I was hoping not to have to do the same for neuralcoref …

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 7
Comments: 29 (2 by maintainers)

Most upvoted comments

Yes, downgrading to 2.1.0 does seems to work

Gunjitbedi on May 28, 2019

According to the docs, this should be fixed by installing with pip install neuralcoref --no-binary neuralcoref instead, but that doesn’t seem to fix it for me.

The --no-binary flag will tell pip to install the sdist, rather than the binary wheel. The problem is that the sdist of a Cython package like neuralcoref isn’t really the source: it already has some compilation — the Python-to-Cython processing stage.

The problem is that the sdist includes .cpp files that were generated using the .pxd files of a particular spaCy version. These are basically headers that get copied into the .cpp source when you run Cython. Those headers specify a different size for some objects, e.g. Span.

So what’s needed is a new Neuralcoref release that pins against the new spaCy. As soon as the new sdist is generated, everything should work correctly — because Cython just needs to be rerun with the new spaCy. That’s why building from master works, as @svlandeg reports above.

honnibal on Aug 15, 2019

@svlandeg can you confirm that if you build neuralcoref’s current master branche from source while having a spacy > 2.1.0 it works?

Yep, when I build neuralcoref’s master branch (4.0) from source, in combination with the current spaCy master branch (2.1.6) from source, it does work.

svlandeg on Jul 25, 2019

@thomwolf: So just to summarize, I think there are two main issues discussed in this thread:

Using neuralcoref out-of-the-box will install the latest spaCy version because of spacy>=2.1.0 in the requirements.txt, but the latest version of spaCy has changes in some of its core Cython files (such as span.pxd) so this raises incompatibility issues. For most users, downgrading to spaCy 2.1.0 will be fine for now, so the PR by @BramVanroy makes sense to address this issue as a sort of temporary fix: https://github.com/huggingface/neuralcoref/pull/186
When you do want to use a newer spaCy version > 2.1.0, like I do, a binary incompatibility RuntimeWarning is thrown by neuralcoref after installing it with pip install neuralcoref. According to the docs, this should be fixed by installing with pip install neuralcoref --no-binary neuralcoref instead, but that doesn’t seem to fix it for me. The docs do mention this as a fix for a warning around spacy.strings.StringStore size changed while I might add that the current incompatibility is caused by spacy.tokens.span.Span. Then when I instead build neuralcoref’s current master branch from source, it does work. So I feel like the issue is with the pip install neuralcoref --no-binary neuralcoref command.

svlandeg on Aug 15, 2019

Does not fix the issue for me. Going to try to build from sources…

Edit: compiling from source cleared up the error.

joshuakfarrar on Aug 30, 2019

That fixes it for me.

MRocholl on Aug 15, 2019

How is Java involved in this?

BramVanroy on Jul 24, 2019

Yes, this is not fixed because I currently don’t have a clear understanding of the root of the problem and how way to fix it. I will try to discuss with @honnibal and @ines and see if we can work out the source of the problem.

thomwolf on Jul 23, 2019