cmssw: seemingly-random crash of one HLT test in IB

IB CMSSW_12_5_X_2022-05-29-0000 had a seemingly-random crash in one HLT job: https://cmssdt.cern.ch/SDT/jenkins-artifacts/HLT-Validation/CMSSW_12_5_X_2022-05-29-0000/el8_amd64_gcc10/HLT_Integration_GRun_MC_hlt.log

Similar tests in the same IB did not show this issue, and all HLT tests in the ensuing IB passed without problems.

This might be a transient problem, but I document it here in case it reappears, and in case others have any feedback.

Copy of the full log file:

%MSG-i ThreadStreamSetup:  (NoModuleName) 29-May-2022 18:47:30 UTC pre-events
setting # threads 4
setting # streams 4
%MSG
%MSG-i CUDAService:  (NoModuleName) 29-May-2022 18:47:32 UTC pre-events
CUDAService disabled by configuration
%MSG
29-May-2022 18:47:39 UTC  Initiating request to open file file:../RelVal_Raw_GRun_MC.root
29-May-2022 18:47:40 UTC  Successfully opened file file:../RelVal_Raw_GRun_MC.root
terminate called after throwing an instance of 'edm::Exception'
  what():  An exception of category 'NotFound' occurred.
Exception Message:
Service no ServiceRegistry has been set for this thread 



A fatal system signal has occurred: abort signal
The following is the call stack containing the origin of the signal.

Sun May 29 18:48:02 UTC 2022
Thread 8 (Thread 0x2b1fff000700 (LWP 48626) "cmsRun"):
#0  0x00002b1fa412f44c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002b1fa3c7f73c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/bootstrap/BUILD/el8_amd64_gcc10/external/gcc/10.3.0-84898dea653199466402e67d73657f10/gcc-10.3.0/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00002b1fe18fd00b in Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/external/el8_amd64_gcc10/lib/libtensorflow_cc.so.2
#4  0x00002b1fe18fd838 in Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/external/el8_amd64_gcc10/lib/libtensorflow_cc.so.2
#5  0x00002b1fe18fa0c8 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/external/el8_amd64_gcc10/lib/libtensorflow_cc.so.2
#6  0x00002b1fedfa97e0 in tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/external/el8_amd64_gcc10/lib/libtensorflow_framework.so.2
#7  0x00002b1fa41291cf in start_thread () from /lib64/libpthread.so.0
#8  0x00002b1fa437ad83 in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x2b1ffcc09700 (LWP 48625) "cmsRun"):
#0  0x00002b1fa412f44c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002b1fa3c7f73c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/bootstrap/BUILD/el8_amd64_gcc10/external/gcc/10.3.0-84898dea653199466402e67d73657f10/gcc-10.3.0/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00002b1fe18fd00b in Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/external/el8_amd64_gcc10/lib/libtensorflow_cc.so.2
#4  0x00002b1fe18fd838 in Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/external/el8_amd64_gcc10/lib/libtensorflow_cc.so.2
#5  0x00002b1fe18fa0c8 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/external/el8_amd64_gcc10/lib/libtensorflow_cc.so.2
#6  0x00002b1fedfa97e0 in tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/external/el8_amd64_gcc10/lib/libtensorflow_framework.so.2
#7  0x00002b1fa41291cf in start_thread () from /lib64/libpthread.so.0
#8  0x00002b1fa437ad83 in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x2b1ffca08700 (LWP 48624) "cmsRun"):
#0  0x00002b1fa412f44c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002b1fa3c7f73c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/bootstrap/BUILD/el8_amd64_gcc10/external/gcc/10.3.0-84898dea653199466402e67d73657f10/gcc-10.3.0/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00002b1fe18fd00b in Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/external/el8_amd64_gcc10/lib/libtensorflow_cc.so.2
#4  0x00002b1fe18fd838 in Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/external/el8_amd64_gcc10/lib/libtensorflow_cc.so.2
#5  0x00002b1fe18fa0c8 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/external/el8_amd64_gcc10/lib/libtensorflow_cc.so.2
#6  0x00002b1fedfa97e0 in tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/external/el8_amd64_gcc10/lib/libtensorflow_framework.so.2
#7  0x00002b1fa41291cf in start_thread () from /lib64/libpthread.so.0
#8  0x00002b1fa437ad83 in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x2b1ff3000700 (LWP 48601) "cmsRun"):
#0  0x00002b1fa437a8cd in syscall () from /lib64/libc.so.6
#1  0x00002b1fa34ac2c7 in tbb::detail::r1::futex_wait (comparand=2, futex=0x2b1fa8dcb02c) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc10/external/tbb/v2021.5.0-e966a5acb1e4d5fd7605074bafbb079c/tbb-v2021.5.0/src/tbb/semaphore.h:103
#2  tbb::detail::r1::binary_semaphore::P (this=0x2b1fa8dcb02c) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc10/external/tbb/v2021.5.0-e966a5acb1e4d5fd7605074bafbb079c/tbb-v2021.5.0/src/tbb/semaphore.h:290
#3  0x00002b1fa34be8f1 in tbb::detail::r1::rml::internal::thread_monitor::commit_wait (c=..., this=0x2b1fa8dcb020) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc10/external/tbb/v2021.5.0-e966a5acb1e4d5fd7605074bafbb079c/tbb-v2021.5.0/src/tbb/rml_thread_monitor.h:243
#4  tbb::detail::r1::rml::private_worker::run (this=0x2b1fa8dcb000) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc10/external/tbb/v2021.5.0-e966a5acb1e4d5fd7605074bafbb079c/tbb-v2021.5.0/src/tbb/private_server.cpp:274
#5  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x2b1fa8dcb000) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc10/external/tbb/v2021.5.0-e966a5acb1e4d5fd7605074bafbb079c/tbb-v2021.5.0/src/tbb/private_server.cpp:221
#6  0x00002b1fa41291cf in start_thread () from /lib64/libpthread.so.0
#7  0x00002b1fa437ad83 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x2b1ff1d16700 (LWP 48600) "cmsRun"):
#0  0x00002b1fa437a8cd in syscall () from /lib64/libc.so.6
#1  0x00002b1fa34ac2c7 in tbb::detail::r1::futex_wait (comparand=2, futex=0x2b1fa8dcb12c) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc10/external/tbb/v2021.5.0-e966a5acb1e4d5fd7605074bafbb079c/tbb-v2021.5.0/src/tbb/semaphore.h:103
#2  tbb::detail::r1::binary_semaphore::P (this=0x2b1fa8dcb12c) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc10/external/tbb/v2021.5.0-e966a5acb1e4d5fd7605074bafbb079c/tbb-v2021.5.0/src/tbb/semaphore.h:290
#3  0x00002b1fa34be8f1 in tbb::detail::r1::rml::internal::thread_monitor::commit_wait (c=..., this=0x2b1fa8dcb120) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc10/external/tbb/v2021.5.0-e966a5acb1e4d5fd7605074bafbb079c/tbb-v2021.5.0/src/tbb/rml_thread_monitor.h:243
#4  tbb::detail::r1::rml::private_worker::run (this=0x2b1fa8dcb100) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc10/external/tbb/v2021.5.0-e966a5acb1e4d5fd7605074bafbb079c/tbb-v2021.5.0/src/tbb/private_server.cpp:274
#5  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x2b1fa8dcb100) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc10/external/tbb/v2021.5.0-e966a5acb1e4d5fd7605074bafbb079c/tbb-v2021.5.0/src/tbb/private_server.cpp:221
#6  0x00002b1fa41291cf in start_thread () from /lib64/libpthread.so.0
#7  0x00002b1fa437ad83 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x2b1ff1315700 (LWP 48599) "cmsRun"):
#0  0x00002b1fa437a8cd in syscall () from /lib64/libc.so.6
#1  0x00002b1fa34ac2c7 in tbb::detail::r1::futex_wait (comparand=2, futex=0x2b1fa8dcb0ac) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc10/external/tbb/v2021.5.0-e966a5acb1e4d5fd7605074bafbb079c/tbb-v2021.5.0/src/tbb/semaphore.h:103
#2  tbb::detail::r1::binary_semaphore::P (this=0x2b1fa8dcb0ac) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc10/external/tbb/v2021.5.0-e966a5acb1e4d5fd7605074bafbb079c/tbb-v2021.5.0/src/tbb/semaphore.h:290
#3  0x00002b1fa34be8f1 in tbb::detail::r1::rml::internal::thread_monitor::commit_wait (c=..., this=0x2b1fa8dcb0a0) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc10/external/tbb/v2021.5.0-e966a5acb1e4d5fd7605074bafbb079c/tbb-v2021.5.0/src/tbb/rml_thread_monitor.h:243
#4  tbb::detail::r1::rml::private_worker::run (this=0x2b1fa8dcb080) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc10/external/tbb/v2021.5.0-e966a5acb1e4d5fd7605074bafbb079c/tbb-v2021.5.0/src/tbb/private_server.cpp:274
#5  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x2b1fa8dcb080) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc10/external/tbb/v2021.5.0-e966a5acb1e4d5fd7605074bafbb079c/tbb-v2021.5.0/src/tbb/private_server.cpp:221
#6  0x00002b1fa41291cf in start_thread () from /lib64/libpthread.so.0
#7  0x00002b1fa437ad83 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x2b1fd3100700 (LWP 48464) "cmsRun"):
#0  0x00002b1fa4133662 in waitpid () from /lib64/libpthread.so.0
#1  0x00002b1faefd5477 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/lib/el8_amd64_gcc10/pluginFWCoreServicesPlugins.so
#2  0x00002b1faefd5fea in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/lib/el8_amd64_gcc10/pluginFWCoreServicesPlugins.so
#3  0x00002b1fa3c84c50 in std::execute_native_thread_routine (__p=0x2b1faa742990) at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#4  0x00002b1fa41291cf in start_thread () from /lib64/libpthread.so.0
#5  0x00002b1fa437ad83 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2b1fa519a040 (LWP 48358) "cmsRun"):
#0  0x00002b1fa4465ae1 in poll () from /lib64/libc.so.6
#1  0x00002b1faefd572f in full_read.constprop () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/lib/el8_amd64_gcc10/pluginFWCoreServicesPlugins.so
#2  0x00002b1faefd60bc in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/lib/el8_amd64_gcc10/pluginFWCoreServicesPlugins.so
#3  0x00002b1faefd8a0b in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/lib/el8_amd64_gcc10/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00002b1fa438fa4f in raise () from /lib64/libc.so.6
#6  0x00002b1fa4362db5 in abort () from /lib64/libc.so.6
#7  0x00002b1fa3c507dc in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:95
#8  0x00002b1fa3c5b7b6 in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
#9  0x00002b1fa3c5a899 in __cxa_call_terminate (ue_header=ue_header@entry=0x2b200b3ded60) at ../../../../libstdc++-v3/libsupc++/eh_call.cc:54
#10 0x00002b1fa3c5b1d1 in __cxxabiv1::__gxx_personality_v0 (version=<optimized out>, actions=6, exception_class=5138137972254386944, ue_header=0x2b200b3ded60, context=<optimized out>) at ../../../../libstdc++-v3/libsupc++/eh_personality.cc:685
#11 0x00002b1fa411770f in _Unwind_RaiseException_Phase2 (exc=0x2b200b3ded60, context=0x7ffce9d4bf40, frames_p=0x7ffce9d4be48) at ../../../libgcc/unwind.inc:64
#12 0x00002b1fa41180b6 in _Unwind_Resume (exc=0x2b200b3ded60) at ../../../libgcc/unwind.inc:241
#13 0x00002b1fa1fd8a3e in edm::Exception::throwThis(edm::errors::ErrorCodes, char const*, char const*, char const*, char const*, char const*) [clone .cold] () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/lib/el8_amd64_gcc10/libFWCoreUtilities.so
#14 0x00002b2006b5d1b1 in CAHitNtupletGeneratorOnGPU::~CAHitNtupletGeneratorOnGPU() () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/lib/el8_amd64_gcc10/pluginRecoPixelVertexingPixelTripletsPlugins.so
#15 0x00002b2006b4a834 in virtual thunk to CAHitNtupletCUDA::~CAHitNtupletCUDA() () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/lib/el8_amd64_gcc10/pluginRecoPixelVertexingPixelTripletsPlugins.so
#16 0x00002b1fdb0c27bd in std::_Sp_counted_deleter<edm::global::EDProducerBase*, std::default_delete<edm::global::EDProducerBase>, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/lib/el8_amd64_gcc10/pluginRecoPixelVertexingPixelLowPtUtilitiesPlugins.so
#17 0x00002b1fdb0c2a72 in std::_Sp_counted_ptr_inplace<edm::maker::ModuleHolderT<edm::global::EDProducerBase>, std::allocator<edm::maker::ModuleHolderT<edm::global::EDProducerBase> >, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/lib/el8_amd64_gcc10/pluginRecoPixelVertexingPixelLowPtUtilitiesPlugins.so
#18 0x00002b1fa1cd5de2 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) [clone .isra.0] () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/lib/el8_amd64_gcc10/libFWCoreFramework.so
#19 0x00002b1fa1cd5d72 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) [clone .isra.0] () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/lib/el8_amd64_gcc10/libFWCoreFramework.so
#20 0x00002b1fa1cd5d72 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) [clone .isra.0] () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/lib/el8_amd64_gcc10/libFWCoreFramework.so
#21 0x00002b1fa1ceae19 in std::_Sp_counted_ptr<edm::ModuleRegistry*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/lib/el8_amd64_gcc10/libFWCoreFramework.so
#22 0x00002b1fa1bddd17 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/lib/el8_amd64_gcc10/libFWCoreFramework.so
#23 0x00002b1fa1c1ea99 in edm::Schedule::~Schedule() () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/lib/el8_amd64_gcc10/libFWCoreFramework.so
#24 0x00002b1fa1bbdfc7 in std::default_delete<edm::Schedule>::operator()(edm::Schedule*) const [clone .part.0] () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/lib/el8_amd64_gcc10/libFWCoreFramework.so
#25 0x00002b1fa1bbebbe in edm::EventProcessor::EventProcessor(std::shared_ptr<edm::ProcessDesc>, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) [clone .cold] () from /cvmfs/cms-ib.cern.ch/week1/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_X_2022-05-29-0000/lib/el8_amd64_gcc10/libFWCoreFramework.so
#26 0x000000000040a190 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#27 0x00002b1fa34b30eb in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc10/external/tbb/v2021.5.0-e966a5acb1e4d5fd7605074bafbb079c/tbb-v2021.5.0/src/tbb/arena.cpp:698
#28 0x000000000040b094 in main::{lambda()#1}::operator()() const ()
#29 0x000000000040970c in main ()

Current Modules:

Module: none (crashed)

A fatal system signal has occurred: abort signal