rdf4j: LuceneSailConnection is not threadsafe

The LuceneSail sends the same LuceneIndex to each LuceneSailConnection created from it. Each of the connections are implemented to call LuceneIndex.endReading when they are closed, which fails sporadically with the following (or similar) stacktrace:

Exception in thread "Thread-1" java.lang.AssertionError: org.eclipse.rdf4j.repository.RepositoryException: org.eclipse.rdf4j.sail.SailException: org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed
    at org.eclipse.rdf4j.sail.lucene.AbstractLuceneSailTest$1.run(AbstractLuceneSailTest.java:1069)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.eclipse.rdf4j.repository.RepositoryException: org.eclipse.rdf4j.sail.SailException: org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed
    at org.eclipse.rdf4j.repository.sail.SailRepositoryConnection.commit(SailRepositoryConnection.java:172)
    at org.eclipse.rdf4j.repository.base.AbstractRepositoryConnection.conditionalCommit(AbstractRepositoryConnection.java:366)
    at org.eclipse.rdf4j.repository.base.AbstractRepositoryConnection.add(AbstractRepositoryConnection.java:491)
    at org.eclipse.rdf4j.sail.lucene.AbstractLuceneSailTest$1.run(AbstractLuceneSailTest.java:1063)
    ... 1 more
Caused by: org.eclipse.rdf4j.sail.SailException: org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed
    at org.eclipse.rdf4j.sail.lucene.LuceneSailConnection.commit(LuceneSailConnection.java:237)
    at org.eclipse.rdf4j.repository.sail.SailRepositoryConnection.commit(SailRepositoryConnection.java:169)
    ... 4 more
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed
    at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
    at org.apache.lucene.index.SegmentReader.getPostingsReader(SegmentReader.java:227)
    at org.apache.lucene.index.CodecReader.fields(CodecReader.java:102)
    at org.apache.lucene.index.LeafReader.terms(LeafReader.java:207)
    at org.apache.lucene.index.LeafReader.termDocsEnum(LeafReader.java:318)
    at org.eclipse.rdf4j.sail.lucene.LuceneIndex.addDocuments(LuceneIndex.java:461)
    at org.eclipse.rdf4j.sail.lucene.LuceneIndex.getDocuments(LuceneIndex.java:452)
    at org.eclipse.rdf4j.sail.lucene.LuceneIndex.getDocuments(LuceneIndex.java:338)
    at org.eclipse.rdf4j.sail.lucene.AbstractSearchIndex.addRemoveStatements(AbstractSearchIndex.java:237)
    at org.eclipse.rdf4j.sail.lucene.LuceneSailConnection.addRemoveStatements(LuceneSailConnection.java:251)
    at org.eclipse.rdf4j.sail.lucene.LuceneSailConnection.commit(LuceneSailConnection.java:217)
    ... 5 more
org.eclipse.rdf4j.repository.RepositoryException: org.eclipse.rdf4j.sail.SailException: org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed
    at org.eclipse.rdf4j.repository.sail.SailRepositoryConnection.commit(SailRepositoryConnection.java:172)
    at org.eclipse.rdf4j.repository.base.AbstractRepositoryConnection.conditionalCommit(AbstractRepositoryConnection.java:366)
    at org.eclipse.rdf4j.repository.base.AbstractRepositoryConnection.add(AbstractRepositoryConnection.java:491)
    at org.eclipse.rdf4j.sail.lucene.AbstractLuceneSailTest$1.run(AbstractLuceneSailTest.java:1063)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.eclipse.rdf4j.sail.SailException: org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed
    at org.eclipse.rdf4j.sail.lucene.LuceneSailConnection.commit(LuceneSailConnection.java:237)
    at org.eclipse.rdf4j.repository.sail.SailRepositoryConnection.commit(SailRepositoryConnection.java:169)
    ... 4 more
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed
    at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
    at org.apache.lucene.index.SegmentReader.getPostingsReader(SegmentReader.java:227)
    at org.apache.lucene.index.CodecReader.fields(CodecReader.java:102)
    at org.apache.lucene.index.LeafReader.terms(LeafReader.java:207)
    at org.apache.lucene.index.LeafReader.termDocsEnum(LeafReader.java:318)
    at org.eclipse.rdf4j.sail.lucene.LuceneIndex.addDocuments(LuceneIndex.java:461)
    at org.eclipse.rdf4j.sail.lucene.LuceneIndex.getDocuments(LuceneIndex.java:452)
    at org.eclipse.rdf4j.sail.lucene.LuceneIndex.getDocuments(LuceneIndex.java:338)
    at org.eclipse.rdf4j.sail.lucene.AbstractSearchIndex.addRemoveStatements(AbstractSearchIndex.java:237)
    at org.eclipse.rdf4j.sail.lucene.LuceneSailConnection.addRemoveStatements(LuceneSailConnection.java:251)
    at org.eclipse.rdf4j.sail.lucene.LuceneSailConnection.commit(LuceneSailConnection.java:217)
    ... 5 more

The failure is not deterministic for whatever reason.

In general RepositoryConnection is not specified to be threadsafe, but closing a repository connection should not affect other connections before the entire Repository/Sail is shutdown. In this test the shutdown should not happen before the test completes. The bug is reproducible fairly consistently for me, happening in at least 1 in 20 builds with a change to randomly vary the number of insertions for each of the threads so they don’t all complete at a similar time consistently.

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 21 (19 by maintainers)

Commits related to this issue

Merge pull request #309 from eclipse/issues/#301-fix-testsuite-lock Issues/#301 fix testsuite lock — committed to eclipse-rdf4j/rdf4j by abrokenjester 8 years ago
Merge pull request #322 from ansell/issues/#301-lucene-threading issue#301 : Work on tightening the contract for lucene sail components — committed to eclipse-rdf4j/rdf4j by abrokenjester 8 years ago

Most upvoted comments

Sure, we can reopen or open a follow up if/when it pops up again. To clarify that was 30 runs just of LuceneSailTest, where the issue was occurring, inside of Eclipse JUnit runner. I didn’t run the maven build 30 times and it was only on my 2 CPU VirtualBox virtual machine. But it was occurring consistently for me in the 2 CPU virtual machine previously, so in that way it was a valid test host.

ansell on Sep 5, 2016

Just merged it because why the hell not 😃

abrokenjester on Sep 4, 2016