tracing: IllegalStateException: Span wasn't started - an observation must be started (not only created)

Hi,

I have a spring boot webflux application which uses this functionality in an autoconfiguration to enable tracing information in access logs:

@Configuration
class TracingAutoConfiguration {

    init {
        Hooks.enableAutomaticContextPropagation()
    }
}

Unfortunately we are running into problems and our service every few hours logs this error message and we are clueless why?

Error:

java.lang.IllegalStateException: Span wasn't started - an observation must be started (not only created)
	at io.micrometer.tracing.handler.TracingObservationHandler.getRequiredSpan(TracingObservationHandler.java:195)
	at io.micrometer.tracing.handler.DefaultTracingObservationHandler.onStop(DefaultTracingObservationHandler.java:50)
	at io.micrometer.observation.ObservationHandler$FirstMatchingCompositeObservationHandler.onStop(ObservationHandler.java:197)
	at io.micrometer.observation.SimpleObservation.lambda$notifyOnObservationStopped$0(SimpleObservation.java:276)
	at java.base/java.util.ArrayDeque$DescendingIterator.forEachRemaining(Unknown Source)
	at io.micrometer.observation.SimpleObservation.notifyOnObservationStopped(SimpleObservation.java:276)
	at io.micrometer.observation.SimpleObservation.stop(SimpleObservation.java:196)
	at org.springframework.security.web.server.ObservationWebFilterChainDecorator$AroundWebFilterObservation$SimpleAroundWebFilterObservation$ObservationReference.stop(ObservationWebFilterChainDecorator.java:426)
	at org.springframework.security.web.server.ObservationWebFilterChainDecorator$AroundWebFilterObservation$SimpleAroundWebFilterObservation.stop(ObservationWebFilterChainDecorator.java:293)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableConditionalSubscriber.cancel(FluxPeekFuseable.java:445)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.MonoContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.cancel(MonoContextWriteRestoringThreadLocals.java:164)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:207)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.FluxDoOnEach$DoOnEachSubscriber.cancel(FluxDoOnEach.java:113)
	at reactor.core.publisher.FluxPeek$PeekSubscriber.cancel(FluxPeek.java:153)
	at reactor.core.publisher.MonoContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.cancel(MonoContextWriteRestoringThreadLocals.java:164)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.cancel(MonoIgnoreThen.java:143)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.Operators.terminate(Operators.java:1277)
	at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:481)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Unknown Source)

Anyone an idea what we can do? We are using Spring boot 3.1.1 with micrometer-tracing-1.1.4

About this issue

  • Original URL
  • State: closed
  • Created 10 months ago
  • Reactions: 2
  • Comments: 16 (3 by maintainers)

Most upvoted comments

@marcingrzejszczak the problem is still there. We updated to implementation("io.micrometer:micrometer-observation:1.11.5"), but still get errors like this one.

java.lang.IllegalStateException: Span wasn't started - an observation must be started (not only created)
	at io.micrometer.tracing.handler.TracingObservationHandler.getRequiredSpan(TracingObservationHandler.java:202)
	at io.micrometer.tracing.handler.DefaultTracingObservationHandler.onStop(DefaultTracingObservationHandler.java:50)
	at io.micrometer.observation.ObservationHandler$FirstMatchingCompositeObservationHandler.onStop(ObservationHandler.java:197)
	at io.micrometer.observation.SimpleObservation.lambda$notifyOnObservationStopped$0(SimpleObservation.java:276)
	at java.base/java.util.ArrayDeque$DescendingIterator.forEachRemaining(Unknown Source)
	at io.micrometer.observation.SimpleObservation.notifyOnObservationStopped(SimpleObservation.java:276)
	at io.micrometer.observation.SimpleObservation.stop(SimpleObservation.java:196)
	at org.springframework.security.web.server.ObservationWebFilterChainDecorator$AroundWebFilterObservation$SimpleAroundWebFilterObservation$ObservationReference.stop(ObservationWebFilterChainDecorator.java:426)
	at org.springframework.security.web.server.ObservationWebFilterChainDecorator$AroundWebFilterObservation$SimpleAroundWebFilterObservation.stop(ObservationWebFilterChainDecorator.java:293)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableConditionalSubscriber.cancel(FluxPeekFuseable.java:445)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.MonoContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.cancel(MonoContextWriteRestoringThreadLocals.java:164)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:207)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.FluxDoOnEach$DoOnEachSubscriber.cancel(FluxDoOnEach.java:113)
	at reactor.core.publisher.FluxPeek$PeekSubscriber.cancel(FluxPeek.java:153)
	at reactor.core.publisher.MonoContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.cancel(MonoContextWriteRestoringThreadLocals.java:164)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:654)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2399)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2179)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.cancel(MonoIgnoreThen.java:143)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
	at reactor.core.publisher.Operators.terminate(Operators.java:1277)
	at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:481)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Unknown Source)

Can we reopen this one?

@oemergenc can you file an issue in Spring Security and reference this one there?