cmssw: Crash in GEMDAQStatusSource::analyze() in Tier0
A crash in Tier0 was reported in https://cms-talk.web.cern.ch/t/paused-jobs-streamexpresscosmics/36382 with the stack trace (from step2 of the workflow, i.e. cmsRun2)
Thread 9 (Thread 0x14f2c8bff700 (LWP 2105) "cmsRun"):
#2 0x000014f313d63410 in sig_pause_for_stacktrace () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3 <signal handler called>
#4 0x000014f317a1182d in __lll_lock_wait () from /lib64/libpthread.so.0
#5 0x000014f317a0aba4 in pthread_mutex_lock () from /lib64/libpthread.so.0
#6 0x000014f317cbad9c in dqm::implementation::DQMStore::enterLumi(unsigned int, unsigned int, unsigned long) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libDQMServicesCore.so
#7 0x000014f2c1841730 in non-virtual thunk to DQMOneEDAnalyzer<>::accumulate(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/pluginDQMServicesComponentsPlugins.so
#8 0x000014f31ac3afce in edm::one::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libFWCoreFramework.so
#9 0x000014f31ac259de in edm::WorkerT<edm::one::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libFWCoreFramework.so
Thread 8 (Thread 0x14f2ca5fe700 (LWP 2104) "cmsRun"):
#2 0x000014f313d63410 in sig_pause_for_stacktrace () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3 <signal handler called>
#4 0x000014f3176cee8c in __memmove_avx_unaligned_erms () from /lib64/libc.so.6
#5 0x000014f31834334b in std::char_traits<char>::copy (__n=1, __s2=<optimized out>, __s1=<optimized out>) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_2_0_pre2-el8_amd64_gcc12/build/CMSSW_13_2_0_pre2-build/BUILD/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/gcc-12.3.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/char_traits.h:435
#6 std::basic_streambuf<char, std::char_traits<char> >::xsgetn (this=0x14f2ca5f7af0, __s=<optimized out>, __n=1) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_2_0_pre2-el8_amd64_gcc12/build/CMSSW_13_2_0_pre2-build/BUILD/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/gcc-12.3.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/streambuf.tcc:56
#7 0x000014f2f07c6a25 in boost::enable_if<boost::is_integral<unsigned int>, void>::type eos::portable_iarchive::load<unsigned int>(unsigned int&, eos::portable_iarchive::dummy<2>) [clone .constprop.0] () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libCondFormatsGeometryObjects.so
#8 0x000014f2f07bff73 in boost::archive::detail::iserializer<eos::portable_iarchive, std::vector<float, std::allocator<float> > >::load_object_data(boost::archive::detail::basic_iarchive&, void*, unsigned int) const () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libCondFormatsGeometryObjects.so
#9 0x000014f2f1bd4c07 in boost::archive::detail::basic_iarchive::load_object(void*, boost::archive::detail::basic_iserializer const&) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/external/el8_amd64_gcc12/lib/libboost_serialization.so.1.80.0
#10 0x000014f2ed120f39 in void SiPixel2DTemplateDBObject::serialize<eos::portable_iarchive>(eos::portable_iarchive&, unsigned int) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libCondFormatsSiPixelObjects.so
#11 0x000014f2f1bd4c07 in boost::archive::detail::basic_iarchive::load_object(void*, boost::archive::detail::basic_iserializer const&) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/external/el8_amd64_gcc12/lib/libboost_serialization.so.1.80.0
#12 0x000014f2cd9b8049 in cond::persistency::PayloadProxy<SiPixel2DTemplateDBObject>::loadPayload() () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/pluginCondCoreSiPixelPluginsPlugin.so
#13 0x000014f2cd9b4a17 in cond::ProductResolver<SiPixel2DTemplateDBObjectRcd, SiPixel2DTemplateDBObject, cond::DefaultInitializer<SiPixel2DTemplateDBObject> >::prefetch(edm::eventsetup::DataKey const&, edm::EventSetupRecordDetails) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/pluginCondCoreSiPixelPluginsPlugin.so
#14 0x000014f31ab37fee in edm::eventsetup::ESSourceProductResolverBase::doPrefetchAndSignals(edm::eventsetup::EventSetupRecordImpl const&, edm::eventsetup::DataKey const&, edm::ESParentContext const&) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libFWCoreFramework.so
#15 0x000014f31ab380b5 in edm::SerialTaskQueue::QueuedTask<edm::eventsetup::ESSourceProductResolverBase::prefetchAsyncImplTemplate<edm::eventsetup::ESSourceProductResolverNonConcurrentBase::prefetchAsyncImpl(edm::WaitingTaskHolder, edm::eventsetup::EventSetupRecordImpl const&, edm::eventsetup::DataKey const&, edm::EventSetupImpl const*, edm::ServiceToken const&, edm::ESParentContext const&)::{lambda(auto:1&, auto:2)#1}, edm::eventsetup::ESSourceProductResolverNonConcurrentBase::prefetchAsyncImpl(edm::WaitingTaskHolder, edm::eventsetup::EventSetupRecordImpl const&, edm::eventsetup::DataKey const&, edm::EventSetupImpl const*, edm::ServiceToken const&, edm::ESParentContext const&)::{lambda()#2}>(edm::eventsetup::ESSourceProductResolverNonConcurrentBase::prefetchAsyncImpl(edm::WaitingTaskHolder, edm::eventsetup::EventSetupRecordImpl const&, edm::eventsetup::DataKey const&, edm::EventSetupImpl const*, edm::ServiceToken const&, edm::ESParentContext const&)::{lambda(auto:1&, auto:2)#1}, edm::eventsetup::ESSourceProductResolverNonConcurrentBase::prefetchAsyncImpl(edm::WaitingTaskHolder, edm::eventsetup::EventSetupRecordImpl const&, edm::eventsetup::DataKey const&, edm::EventSetupImpl const*, edm::ServiceToken const&, edm::ESParentContext const&)::{lambda()#2}, edm::WaitingTaskHolder, edm::eventsetup::EventSetupRecordImpl const&, edm::eventsetup::DataKey const&, edm::ESParentContext const&)::{lambda()#1}>::execute() () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libFWCoreFramework.so
Thread 7 (Thread 0x14f2c9bfd700 (LWP 2103) "cmsRun"):
#2 0x000014f313d63410 in sig_pause_for_stacktrace () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3 <signal handler called>
#4 0x000014f317a1182d in __lll_lock_wait () from /lib64/libpthread.so.0
#5 0x000014f317a0aba4 in pthread_mutex_lock () from /lib64/libpthread.so.0
#6 0x000014f317cbad9c in dqm::implementation::DQMStore::enterLumi(unsigned int, unsigned int, unsigned long) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libDQMServicesCore.so
#7 0x000014f2b73ace7b in DQMEDAnalyzer::beginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libDQMDTMonitorModule.so
#8 0x000014f31ac487dd in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamBeginRun(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libFWCoreFramework.so
#9 0x000014f31ac2dfc0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamBegin(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libFWCoreFramework.so
Thread 6 (Thread 0x14f2cafff700 (LWP 2102) "cmsRun"):
#3 0x000014f313d66e70 in sig_dostack_then_abort () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#4 <signal handler called>
#5 0x000014f2b5469714 in GEMDAQStatusSource::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/pluginDQMGEMPlugins.so
#6 0x000014f31ac48281 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libFWCoreFramework.so
#7 0x000014f31ac2ccee in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libFWCoreFramework.so
Thread 5 (Thread 0x14f2cbfff700 (LWP 2101) "cmsRun"):
#2 0x000014f313d63410 in sig_pause_for_stacktrace () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3 <signal handler called>
#4 0x000014f3196937f3 in sizedDeleteImpl (size=56, ptr=0x14f2406c03c0) at src/jemalloc_cpp.cpp:192
#5 operator delete (ptr=0x14f2406c03c0, size=56) at src/jemalloc_cpp.cpp:200
#6 0x000014f319c4f22b in THashTableIter::Next() () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/external/el8_amd64_gcc12/lib/libCore.so
#7 0x000014f319c4e335 in THashTable::Rehash(int, bool) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/external/el8_amd64_gcc12/lib/libCore.so
#8 0x000014f319c4c800 in THashList::Rehash(int) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/external/el8_amd64_gcc12/lib/libCore.so
#9 0x000014f3131dc1fb in TAxis::SetBinLabel(int, char const*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/external/el8_amd64_gcc12/lib/libHist.so
#10 0x000014f317cda082 in dqm::impl::MonitorElement::setBinLabel(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libDQMServicesCore.so
#11 0x000014f2c2f5d58e in tadqm::TrackAnalyzer::bookHistosForHitProperties(dqm::implementation::IBooker&) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/pluginDQMTrackingMonitor.so
#12 0x000014f2c2f5f9ee in tadqm::TrackAnalyzer::initHisto(dqm::implementation::IBooker&, edm::EventSetup const&, edm::ParameterSet const&) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/pluginDQMTrackingMonitor.so
#13 0x000014f2c2fa3dbe in TrackingMonitor::bookHistograms(dqm::implementation::IBooker&, edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/pluginDQMTrackingMonitor.so
#14 0x000014f2c2f3e894 in DQMEDAnalyzer::beginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/pluginDQMTrackingMonitor.so
#15 0x000014f31ac487dd in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamBeginRun(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libFWCoreFramework.so
#16 0x000014f31ac2dfc0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamBegin(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libFWCoreFramework.so
Thread 4 (Thread 0x14f2cce97700 (LWP 2100) "cmsRun"):
#2 0x000014f313d63410 in sig_pause_for_stacktrace () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3 <signal handler called>
#4 0x000014f317a1182d in __lll_lock_wait () from /lib64/libpthread.so.0
#5 0x000014f317a0aba4 in pthread_mutex_lock () from /lib64/libpthread.so.0
#6 0x000014f317cbd151 in dqm::implementation::DQMStore::initLumi(unsigned int, unsigned int, unsigned long) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libDQMServicesCore.so
#7 0x000014f2990d1a6b in DQMEDAnalyzer::beginLuminosityBlock(edm::LuminosityBlock const&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/pluginDQMOfflineTriggerPlugins.so
#8 0x000014f31ac48dbd in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamBeginLuminosityBlock(edm::StreamID, edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libFWCoreFramework.so
#9 0x000014f31ac2e000 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamBegin(edm::StreamID, edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libFWCoreFramework.so
Thread 3 (Thread 0x14f2cd898700 (LWP 2099) "cmsRun"):
#2 0x000014f313d63410 in sig_pause_for_stacktrace () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3 <signal handler called>
#4 0x000014f317a1182d in __lll_lock_wait () from /lib64/libpthread.so.0
#5 0x000014f317a0aba4 in pthread_mutex_lock () from /lib64/libpthread.so.0
#6 0x000014f317cbd151 in dqm::implementation::DQMStore::initLumi(unsigned int, unsigned int, unsigned long) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libDQMServicesCore.so
#7 0x000014f2990d1a6b in DQMEDAnalyzer::beginLuminosityBlock(edm::LuminosityBlock const&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/pluginDQMOfflineTriggerPlugins.so
#8 0x000014f31ac48dbd in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamBeginLuminosityBlock(edm::StreamID, edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libFWCoreFramework.so
#9 0x000014f31ac2e000 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamBegin(edm::StreamID, edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libFWCoreFramework.so
Thread 1 (Thread 0x14f3198e6640 (LWP 1981) "cmsRun"):
#2 0x000014f313d63410 in sig_pause_for_stacktrace () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3 <signal handler called>
#4 0x000014f2b4fe462f in std::__detail::_NFA<std::__cxx11::regex_traits<char> >::_M_insert_repeat(long, long, bool) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libDQML1TMonitor.so
#5 0x000014f2b4fd83c9 in std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_quantifier() () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libDQML1TMonitor.so
#6 0x000014f2b4fdd5a8 in std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative() () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libDQML1TMonitor.so
#7 0x000014f2b4fdfe69 in std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction() () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libDQML1TMonitor.so
#8 0x000014f2b4fe16a2 in L1TdeStage2uGT::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libDQML1TMonitor.so
#9 0x000014f31ac48281 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libFWCoreFramework.so
#10 0x000014f31ac2ccee in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_13_3_3/lib/el8_amd64_gcc12/libFWCoreFramework.so
Current Modules:
Module: GEMDAQStatusSource:GEMDAQStatusSource (crashed)
Module: HLTTauDQMOfflineSource:hltTauOfflineMonitor_TagAndProbe
Module: none
Module: DTSegmentAnalysisTask:dtSegmentAnalysisMonitor
Module: DQMEventInfo:dqmInfoTracking
Module: TrackingMonitor:MonitorTrackSTACosmicMuons
Module: L1TdeStage2uGT:l1tdeStage2uGT
Module: JetMETHLTOfflineSource:jetMETHLTOfflineSourceAK8Fwd
About this issue
- Original URL
- State: closed
- Created 4 months ago
- Comments: 15 (15 by maintainers)
+alca
assign alca
I added the details in https://cms-talk.web.cern.ch/t/paused-jobs-streamexpresscosmics/36382/3
As Yechan already mentioned the issue was caused due to an inadvertant update of the GEM Channel mapping tag GEMChMap_online_v2 in the HLT/Express/Prompt GT from Run 377418. Since GEM was included in global only starting Run 377453 the crash is observed from then (Snapshot below)
The conditions are now reversed but only starting Run 377464. So it should not be observed after that. For the recovery of the failed jobs, we are coordinating with the T0 experts.
The issue is caused by the appending new GEMChMap which includes the GE21 production chambers, which doesn’t exist in the current geometry. The problem will be gone after adding the chambers in the geometry. But I don’t know when it would be happened. So I talked with @saumyaphor4252 to revert the GEMChMap to old version. From the Run 377464, we are working with the GEMChMap that we were using last years.
assign dqm