orientdb: Stored row in cluster shows different record count after ODB hangs

OrientDB Version, operating system, or hardware.

v2.2.7

Operating System

Linux

Expected behavior and actual behavior

Orient DB running normally but after the some hours , hangs with cpu usage 100% . the following trace ,

com.orientechnologies.orient.core.exception.OPaginatedClusterException: Error during record creation DB name="BSNTS_DB" Component Name="fiscaltransaction" at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.createRecord(OPaginatedCluster.java:494) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doCreateRecord(OAbstractPaginatedStorage.java:3091) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.createRecord(OAbstractPaginatedStorage.java:960) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeSaveRecord(ODatabaseDocumentTx.java:2075) at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveNew(OTransactionNoTx.java:246) at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:179) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:2598) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:103) at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1800) at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1791) at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:63) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLInsert.saveRecord(OCommandExecutorSQLInsert.java:330) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLInsert.execute(OCommandExecutorSQLInsert.java:231) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:72) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:2680) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:2626) at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.command(ONetworkProtocolBinary.java:1515) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:2680) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:2626) at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.command(ONetworkProtocolBinary.java:1515) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:577) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:318) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:196) at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77) Caused by: java.lang.NullPointerException at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.findFreePage(OPaginatedCluster.java:1970) at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.addEntry(OPaginatedCluster.java:1906) at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.createRecord(OPaginatedCluster.java:475) ... 21 more $ANSI{green {db=BSNTS_DB}} Error on creating record in cluster: plocal cluster: fiscaltransaction com.orientechnologies.orient.core.exception.OPaginatedClusterException: Error during record creation DB name="BSNTS_DB" Component Name="fiscaltransaction" at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.createRecord(OPaginatedCluster.java:494) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.createRecord(OAbstractPaginatedStorage.java:960) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeSaveRecord(ODatabaseDocumentTx.java:2075) at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveNew(OTransactionNoTx.java:246) at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:179) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:2598) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:103) at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1800) at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1791) at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:63) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLInsert.saveRecord(OCommandExecutorSQLInsert.java:330) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLInsert.execute(OCommandExecutorSQLInsert.java:231) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:72) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:2680) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:2626) at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.command(ONetworkProtocolBinary.java:1515) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:577) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:318) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:196) at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77) Caused by: java.lang.NullPointerException at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.findFreePage(OPaginatedCluster.java:1970) at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.addEntry(OPaginatedCluster.java:1906) at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.createRecord(OPaginatedCluster.java:475)

and i’m wondered when run the select count command result that show as below :

So when i insert new Record in FiscalTransaction class , randomly raised error and not saving the record .

Please help me .

Steps to reproduce the problem .

I don’t know , this behavior is unexpected .

It seems we have a null pointer for a record in storage after the database hanging . i guess there is a problem with cluster 29 because when i insert some record after 3 successful insert the following error appears :

Error during saving of record with rid #29:-1

Thanks Saeed Tabrizi

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 24 (24 by maintainers)

Commits related to this issue

Issue #6583 , was partially fixed — committed to orientechnologies/orientdb by deleted user 8 years ago
Issue #6583 , was partially fixed — committed to orientechnologies/orientdb by deleted user 8 years ago

Most upvoted comments

@saeedtabrizi seems I have found problem of your issue and working on it.

laa on Aug 29, 2016

@saeedtabrizi I close this issue because I suppose that it can be confirmed as fixed. If you reproduce the problem on a stress test, feel free to reopen issue.

laa on Sep 28, 2016

@saeedtabrizi that is cool, thank you very much for your time.

laa on Sep 20, 2016

@saeedtabrizi could you update till 2.2.10 version, it should solve the problem which you found, I keep the issue open in order to do not forget to fix minor issue inside of this big one it is related to free space usage statistic, and it should not affect your operations on a database.

laa on Sep 13, 2016

@saeedtabrizi thank you. Your issue should be already fixed in upcoming release. Still will be a minor issue about free space usage statistic. I will try to use a database which you sent to fix it too. But your major issue about DB freeze should gone.

laa on Sep 5, 2016

@saeedtabrizi OK, then could you do followings steps.

If CPU usage is 100%, please get information which we discussed.
Kill the process as usual and because we did work with you in online mode, please keep the exact/untouched copy of the database (that is critical so I will be able to look at it and experiment with it).

Sure I do not want to harm your users one more time, but we need to reproduce it at least once and get all information to fix the issue. Once I post it, on next day at most I will look at it to provide a fix ASAP. Sorry for inconveniences.

laa on Aug 18, 2016

@saeedtabrizi is it possible to send us database of files of “fiscaltransaction” cluster ?

laa on Aug 18, 2016