AxonFramework: TrackingEventProcessor cannot be reset immediately after shutdown in very rare cases

Hi,

We noticed that resetting the TrackingEventProcessor after shutdown leads to an IllegalStateException in very rare cases. This might depend on the actual timeout setting, but I can reproduce the problem with almost the default settings.

Basic information

• Axon Framework version: 4.5.14
• JDK version: 11

Steps to reproduce

Please take a look at the following code which repeatedly resets a tracking event processor.

@SpringBootApplication
public class DemoApplication {

	public static void main( final String[] args ) {
		final var applicationContext = new SpringApplicationBuilder( DemoApplication.class )
				.properties( "axon.axonserver.enabled=false" )
				.run( args );

		final EventProcessingConfiguration eventProcessingConfiguration = applicationContext.getBean( EventProcessingConfiguration.class );
		final Optional<TrackingEventProcessor> optionalEventProcessor = eventProcessingConfiguration.eventProcessor( MyEventHandler.PROCESSING_GROUP, TrackingEventProcessor.class );
		final TrackingEventProcessor eventProcessor = optionalEventProcessor.get( );
		while ( true ) {
			eventProcessor.shutDown( );
			eventProcessor.resetTokens( );
			eventProcessor.start( );
		}
	}

	@Autowired
	private void configure( final EventProcessingConfigurer configurer ) {
		configurer.registerTrackingEventProcessorConfiguration( c -> TrackingEventProcessorConfiguration
				.forParallelProcessing( 4 ) );
	}

	@Component
	@ProcessingGroup( MyEventHandler.PROCESSING_GROUP )
	static class MyEventHandler {

		public static final String PROCESSING_GROUP = "MyEventHandler";

		@EventHandler
		public void on( final Integer i ) {
		}

	}

}

On my machine it takes only a few seconds until the application crashes with an exception:

Exception in thread “main” java.lang.IllegalStateException: TrackingProcessor must be shut down before triggering a reset at org.axonframework.common.Assert.state(Assert.java:44) at org.axonframework.eventhandling.TrackingEventProcessor.resetTokens(TrackingEventProcessor.java:616) at org.axonframework.eventhandling.TrackingEventProcessor.resetTokens(TrackingEventProcessor.java:610) at org.axonframework.eventhandling.TrackingEventProcessor.resetTokens(TrackingEventProcessor.java:597) at org.axonframework.eventhandling.TrackingEventProcessor.resetTokens(TrackingEventProcessor.java:586) at com.example.demo.DemoApplication.main(DemoApplication.java:32)

Here is what I know: By changing the assertions in the reset method slightly I am now pretty sure that the failing assertion is activeProcessorThreads( ) == 0. The segments seem to be modified by another thread (the TrackingSegmentWorker). The shutdownAsync method does not seem to wait for either the thread or the active segments being zero. Maybe this would be an approach to solve the issue at hand.

Expected behaviour

A reset is possible immediately after shutdown.

Actual behaviour

A reset leads somtimes to an exception after shutdown.