cmssw: Segfaults in workflow 11834.21 step 2
Workflow 11834.21 step 2 crashes in CMSSW_12_2_X_2021-12-01-2300
Thu Dec 2 05:25:32 CET 2021
Thread 10 (Thread 0x2af25c000700 (LWP 2095) "cmsRun"):
#2 0x00002af217aa43f0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3 <signal handler called>
#4 0x00002af20fbada62 in TBufferFile::ReadClassBuffer(TClass const*, void*, TClass const*) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/external/slc7_amd64_gcc900/lib/libRIO.so
#5 0x00002af20fdf90ba in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/external/slc7_amd64_gcc900/lib/libRIO.so
#6 0x00002af20fcab224 in TStreamerInfoActions::VectorLooper::GenericRead(TBuffer&, void*, void const*, TStreamerInfoActions::TLoopConfiguration const*, TStreamerInfoActions::TConfiguration const*) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/external/slc7_amd64_gcc900/lib/libRIO.so
#7 0x00002af20fba5c44 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*, void*) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/external/slc7_amd64_gcc900/lib/libRIO.so
#8 0x00002af20fcf5842 in int TStreamerInfoActions::ReadSTL<&TStreamerInfoActions::ReadSTLMemberWiseSameClass, &TStreamerInfoActions::ReadSTLObjectWiseFastArray>(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/external/slc7_amd64_gcc900/lib/libRIO.so
#9 0x00002af20fba61c5 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/external/slc7_amd64_gcc900/lib/libRIO.so
#10 0x00002af20f6ab067 in TBranchElement::ReadLeavesMember(TBuffer&) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/external/slc7_amd64_gcc900/lib/libTree.so
#11 0x00002af20f6a32f9 in TBranch::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/external/slc7_amd64_gcc900/lib/libTree.so
#12 0x00002af20f6b4d08 in TBranchElement::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/external/slc7_amd64_gcc900/lib/libTree.so
#13 0x00002af20f6b4cad in TBranchElement::GetEntry(long long, int) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/external/slc7_amd64_gcc900/lib/libTree.so
#14 0x00002af25393214b in edm::RootTree::getEntry(TBranch*, long long) const () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/pluginIOPoolInput.so
#15 0x00002af25390366a in edm::RootDelayedReader::getProduct_(edm::BranchID const&, edm::EDProductGetter const*) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/pluginIOPoolInput.so
#16 0x00002af20f1867c4 in edm::DelayedReader::getProduct(edm::BranchID const&, edm::EDProductGetter const*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#17 0x00002af20f25e874 in edm::DelayedReaderInputProductResolver::resolveProduct_(edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#18 0x00002af20f24dd49 in edm::Principal::findProductByLabel(edm::KindOfType, edm::TypeID const&, edm::InputTag const&, edm::EDConsumerBase const*, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#19 0x00002af20f24df15 in edm::Principal::findProductByTag(edm::TypeID const&, edm::InputTag const&, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#20 0x00002af27dcd2dca in edm::Adjuster<std::vector<PCaloHit, std::allocator<PCaloHit> > >::doOffset(int, int, edm::EventPrincipal const&, edm::ModuleCallingContext const*, unsigned int, int) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/pluginSimGeneralMixingModulePlugins.so
#21 0x00002af27dcca35b in edm::MixingModule::pileAllWorkers(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/pluginSimGeneralMixingModulePlugins.so
#22 0x00002af27dcd56af in void edm::PileUp::readPileUp<std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)> >(edm::EventID const&, std::vector<edm::SecondaryEventIDAndFileInfo, std::allocator<edm::SecondaryEventIDAndFileInfo> >&, std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)>, int, edm::StreamID const&) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/pluginSimGeneralMixingModulePlugins.so
#23 0x00002af27dccc00a in edm::MixingModule::doPileUp(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/pluginSimGeneralMixingModulePlugins.so
#24 0x00002af27dd641fa in edm::BMixingModule::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libMixingBase.so
#25 0x00002af20f304b6c in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#26 0x00002af20f2e58bd in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#27 0x00002af20f23fc35 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#28 0x00002af20f23fded in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#29 0x00002af20f2400f6 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#30 0x00002af20f242486 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#31 0x00002af20efb8e15 in tbb::detail::d1::function_task<edm::WaitingTaskWithArenaHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}::operator()() const::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libFWCoreConcurrency.so
Thread 9 (Thread 0x2af25b001700 (LWP 2094) "cmsRun"):
#2 0x00002af217aa43f0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3 <signal handler called>
#4 0x00002af211972d19 in syscall () from /lib64/libc.so.6
#5 0x00002af210a63b47 in tbb::detail::r1::futex_wait (comparand=2, futex=0x2af21652c12c) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_12_1_0_pre5-slc7_amd64_gcc900/build/CMSSW_12_1_0_pre5-build/BUILD/slc7_amd64_gcc900/external/tbb/v2021.4.0-651a6efca0c94b3c25e36a8faa72480b/tbb-v2021.4.0/src/tbb/semaphore.h:104
#6 tbb::detail::r1::binary_semaphore::P (this=0x2af21652c12c) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_12_1_0_pre5-slc7_amd64_gcc900/build/CMSSW_12_1_0_pre5-build/BUILD/slc7_amd64_gcc900/external/tbb/v2021.4.0-651a6efca0c94b3c25e36a8faa72480b/tbb-v2021.4.0/src/tbb/semaphore.h:290
#7 0x00002af210a5d8f8 in tbb::detail::r1::rml::internal::thread_monitor::commit_wait (c=..., this=0x2af21652c120) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_12_1_0_pre5-slc7_amd64_gcc900/build/CMSSW_12_1_0_pre5-build/BUILD/slc7_amd64_gcc900/external/tbb/v2021.4.0-651a6efca0c94b3c25e36a8faa72480b/tbb-v2021.4.0/src/tbb/rml_thread_monitor.h:242
#8 tbb::detail::r1::rml::private_worker::run (this=0x2af21652c100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_12_1_0_pre5-slc7_amd64_gcc900/build/CMSSW_12_1_0_pre5-build/BUILD/slc7_amd64_gcc900/external/tbb/v2021.4.0-651a6efca0c94b3c25e36a8faa72480b/tbb-v2021.4.0/src/tbb/private_server.cpp:274
#9 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x2af21652c100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_12_1_0_pre5-slc7_amd64_gcc900/build/CMSSW_12_1_0_pre5-build/BUILD/slc7_amd64_gcc900/external/tbb/v2021.4.0-651a6efca0c94b3c25e36a8faa72480b/tbb-v2021.4.0/src/tbb/private_server.cpp:221
#10 0x00002af211665ea5 in start_thread () from /lib64/libpthread.so.0
#11 0x00002af2119789fd in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x2af25a600700 (LWP 2093) "cmsRun"):
#0 0x00002af21196dccd in poll () from /lib64/libc.so.6
#1 0x00002af217aa49d7 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#2 0x00002af217aa530c in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3 0x00002af217aa87ab in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#4 <signal handler called>
#5 0x00002af25dc26696 in EcalHitResponse::analogSignalAmplitude(DetId const&, double, CLHEP::HepRandomEngine*) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libSimCalorimetryEcalSimAlgos.so
#6 0x00002af27fe74fa3 in EBHitResponseImpl<ecalPh1>::putAnalogSignal(PCaloHit const&, CLHEP::HepRandomEngine*) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libSimCalorimetryEcalSimProducers.so
#7 0x00002af25dc24024 in EcalTDigitizer<EBDigitizerTraits>::add(std::vector<PCaloHit, std::allocator<PCaloHit> > const&, int, CLHEP::HepRandomEngine*) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libSimCalorimetryEcalSimAlgos.so
#8 0x00002af27fe6bcc1 in EcalDigiProducer::accumulateCaloHits(edm::Handle<std::vector<PCaloHit, std::allocator<PCaloHit> > > const&, edm::Handle<std::vector<PCaloHit, std::allocator<PCaloHit> > > const&, edm::Handle<std::vector<PCaloHit, std::allocator<PCaloHit> > > const&, int) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libSimCalorimetryEcalSimProducers.so
#9 0x00002af27fe6c76d in EcalDigiProducer::accumulate(PileUpEventPrincipal const&, edm::EventSetup const&, edm::StreamID const&) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libSimCalorimetryEcalSimProducers.so
#10 0x00002af27dcca22e in edm::MixingModule::accumulateEvent(PileUpEventPrincipal const&, edm::EventSetup const&, edm::StreamID const&) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/pluginSimGeneralMixingModulePlugins.so
#11 0x00002af27dcca396 in edm::MixingModule::pileAllWorkers(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/pluginSimGeneralMixingModulePlugins.so
#12 0x00002af27dcd56af in void edm::PileUp::readPileUp<std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)> >(edm::EventID const&, std::vector<edm::SecondaryEventIDAndFileInfo, std::allocator<edm::SecondaryEventIDAndFileInfo> >&, std::_Bind<bool (edm::MixingModule::*(std::reference_wrapper<edm::MixingModule>, std::_Placeholder<1>, edm::ModuleCallingContext const*, int, std::_Placeholder<2>, int, std::reference_wrapper<edm::EventSetup const>, edm::StreamID))(edm::EventPrincipal const&, edm::ModuleCallingContext const*, int, int, int&, edm::EventSetup const&, edm::StreamID const&)>, int, edm::StreamID const&) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/pluginSimGeneralMixingModulePlugins.so
#13 0x00002af27dccc00a in edm::MixingModule::doPileUp(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/pluginSimGeneralMixingModulePlugins.so
#14 0x00002af27dd641fa in edm::BMixingModule::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libMixingBase.so
#15 0x00002af20f304b6c in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#16 0x00002af20f2e58bd in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#17 0x00002af20f23fc35 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#18 0x00002af20f23fded in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#19 0x00002af20f2400f6 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#20 0x00002af20f242486 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#21 0x00002af20efb8e15 in tbb::detail::d1::function_task<edm::WaitingTaskWithArenaHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}::operator()() const::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libFWCoreConcurrency.so
Thread 1 (Thread 0x2af2139f40c0 (LWP 1781) "cmsRun"):
#2 0x00002af217aa43f0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3 <signal handler called>
#4 0x00002af23ec8704a in RecHitsSortedInPhi::RecHitsSortedInPhi(std::vector<BaseTrackerRecHit const*, std::allocator<BaseTrackerRecHit const*> > const&, Point3DBase<float, GlobalTag> const&, DetLayer const*) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libRecoTrackerTkHitPairs.so
#5 0x00002af23ec83c3c in LayerHitMapCache::operator()(SeedingLayerSetsHits::SeedingLayer const&, TrackingRegion const&) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libRecoTrackerTkHitPairs.so
#6 0x00002af23ec81d0a in HitPairGeneratorFromLayerPair::doublets(TrackingRegion const&, edm::Event const&, edm::EventSetup const&, SeedingLayerSetsHits::SeedingLayer const&, SeedingLayerSetsHits::SeedingLayer const&, LayerHitMapCache&) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libRecoTrackerTkHitPairs.so
#7 0x00002af268919cc5 in (anonymous namespace)::Impl<(anonymous namespace)::DoNothing, (anonymous namespace)::ImplIntermediateHitDoublets, (anonymous namespace)::RegionsLayersSeparate>::produce(bool, edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/pluginRecoTrackerTkHitPairsPlugins.so
#8 0x00002af268915e3a in HitPairEDProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/pluginRecoTrackerTkHitPairsPlugins.so
#9 0x00002af20f304b6c in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#10 0x00002af20f2e58bd in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#11 0x00002af20f23fc35 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#12 0x00002af20f23fded in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02709/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_2_X_2021-12-01-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
Current Modules:
Module: MixingModule:mix (crashed)
Module: none
Module: MixingModule:mix
Module: HitPairEDProducer:hltDisplacedhltIter4PFlowPixelLessHitDoublets
A fatal system signal has occurred: segmentation violation
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 32 (32 by maintainers)
Good point, indeed 😃 Refreshing the PR tests, merge will follow
@swagata87 Good catch! I checked one random succeeding 11834.21 step 2 in the IBs and that did not show
TrajectoryNaNwarnings. I have been suspectingRecHitsSortedInPhiconstructor to be the culprit as it, viaHitPairEDProducer:hltDisplacedhltIter4PFlowPixelLessHitDoubletsmodule, is the only piece of code present in all of the collected stack traces. But I didn’t manage to make the connection to #35489. To me your hypothesis sounds very plausible.11834 is a pileup workflow, and the pileup mixing is not fully reproducible when run with multiple threads (because the MinBias files are processed per edm stream, and assignment of input GEN events to streams is not deterministic; MixingModule has a specific playback mode that allows to rerun the pileup mixing in exactly the same way, but it needs an already mixed file, i.e. that from step2, as an input).