quartz: Triggers are getting blocked permanently

Dear Quartz Team,

We are using Quartz 2.2.1 in clustered-mode with JDBC job store to schedule jobs marked as @DisallowConcurrentExecution.

We have observed that occasionally triggers are getting stuck in trigger state BLOCKED without ever recovering automatically. Looking into the job store DB tables, the pattern is always the same:

• The TRIGGER_STATE on <PREFIX>_TRIGGERS is in state BLOCKED

• There is no corresponding record in <PREFIX>_FIRED_TRIGGERS

Obviously org.quartz.impl.jdbcjobstore.JobStoreSupport.clusterRecover(Connection, List<SchedulerStateRecord>) will not recover such triggers, so the only way to get out of this inconsistent state is to manually set the TRIGGER_STATE back to WAITING.

It is not yet clear under which circumstances this error occurs. However, our log files indicate that jobs getting stuck coincides with temporary database problems.

Below you can find an example of a NullPointerException in org.quartz.impl.jdbcjobstore.JobStoreSupport.triggersFired(List<OperableTrigger>). The exception itself was caused somewhere in the JDBC driver (Sybase jConnect) when trying to invoke rollback() on a JDBC connection. The log entry’s timestamp correlates exactly with the time the trigger got stuck.

2017 05 01 20:20:02#+00#ERROR#org.quartz.core.QuartzSchedulerThread##anonymous#ItOpScheduler_Clustered_QuartzSchedulerThread#Runtime error occurred in main trigger firing loop.java.lang.NullPointerException: while trying to invoke the method com.sybase.jdbc4.tds.TdsCursor.setRowNum(int) of a null object loaded from field com.sybase.jdbc4.tds.CurInfo3Token._cursor of an object loaded from local variable 'this'
	at com.sybase.jdbc4.tds.CurInfo3Token.getMetaInformation(CurInfo3Token.java:85)
	at com.sybase.jdbc4.tds.CurInfoToken.<init>(CurInfoToken.java:130)
	at com.sybase.jdbc4.tds.CurInfo3Token.<init>(CurInfo3Token.java:45)
	at com.sybase.jdbc4.tds.Tds.nextResult(Tds.java:3239)
	at com.sybase.jdbc4.tds.Tds.readCommandResults(Tds.java:4459)
	at com.sybase.jdbc4.tds.Tds.doCommand(Tds.java:4444)
	at com.sybase.jdbc4.tds.Tds.endTransaction(Tds.java:2602)
	at com.sybase.jdbc4.jdbc.SybConnection.rollback(SybConnection.java:1953)
	at sun.reflect.GeneratedMethodAccessor492.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.sap.core.persistence.jdbc.trace.TraceableBase$1.invoke(TraceableBase.java:44)
	at com.sun.proxy.$Proxy17.rollback(Unknown Source)
	at com.sap.core.persistence.jdbc.trace.TraceableConnection.rollback(TraceableConnection.java:239)
	at org.apache.commons.dbcp.DelegatingConnection.rollback(DelegatingConnection.java:368)
	at org.apache.commons.dbcp.DelegatingConnection.rollback(DelegatingConnection.java:368)
	at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.rollback(PoolingDataSource.java:323)
	at sun.reflect.GeneratedMethodAccessor492.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.quartz.impl.jdbcjobstore.AttributeRestoringConnectionInvocationHandler.invoke(AttributeRestoringConnectionInvocationHandler.java:73)
	at com.sun.proxy.$Proxy143.rollback(Unknown Source)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.rollbackConnection(JobStoreSupport.java:3658)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3817)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.triggersFired(JobStoreSupport.java:2908)
	at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:336)
|

Please let me know if you need additional details.

Thanks for your support, Sebastian