cmssw: Segmentation violation in RelVal wf `25234.911` step 2

Hello,

RelVal wf 25234.911 is failing due to segmentation violation in GEANT4 IBs (CMSSW_13_3_GEANT4_X_2023-08-02-2300) with the following stacktrace:

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thu Aug  3 07:51:47 CEST 2023
Thread 8 (Thread 0x14be326f0700 (LWP 2179363) "cmsRun"):
#0  0x000014be96cff45c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000014be97380d20 in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865
#2  std::__condvar::wait (__m=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155
#3  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41
#4  0x000014be5cd9b76b in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2
#5  0x000014be5cd9bd8a in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2
#6  0x000014be5cd99468 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2
#7  0x000014be4ad33430 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/external/el8_amd64_gcc11/lib/libtensorflow_framework.so.2
#8  0x000014be96cf91ca in start_thread () from /lib64/libpthread.so.0
#9  0x000014be96965e73 in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x14be328f1700 (LWP 2179356) "cmsRun"):
#0  0x000014be96cff45c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000014be97380d20 in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865
#2  std::__condvar::wait (__m=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155
#3  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41
#4  0x000014be5cd9b76b in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2
#5  0x000014be5cd9bd8a in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2
#6  0x000014be5cd99468 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2
#7  0x000014be4ad33430 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/external/el8_amd64_gcc11/lib/libtensorflow_framework.so.2
#8  0x000014be96cf91ca in start_thread () from /lib64/libpthread.so.0
#9  0x000014be96965e73 in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x14be32af2700 (LWP 2179351) "cmsRun"):
#0  0x000014be96cff45c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000014be97380d20 in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865
#2  std::__condvar::wait (__m=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/gcc-11.4.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155
#3  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41
#4  0x000014be5cd9b76b in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2
#5  0x000014be5cd9bd8a in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2
#6  0x000014be5cd99468 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2
#7  0x000014be4ad33430 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/external/el8_amd64_gcc11/lib/libtensorflow_framework.so.2
#8  0x000014be96cf91ca in start_thread () from /lib64/libpthread.so.0
#9  0x000014be96965e73 in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x14be4190a700 (LWP 2170218) "cmsRun"):
#0  0x000014be96a25868 in nanosleep () from /lib64/libc.so.6
#1  0x000014be96a2576e in sleep () from /lib64/libc.so.6
#2  0x000014be94caab10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  base::subtle::NoBarrier_Load (ptr=0x14be97688cc0 <tcmalloc::Static::central_cache_+15808>) at ./src/base/atomicops-internals-x86.h:151
#5  SpinLock::SpinLoop (this=this@entry=0x14be97688cc0 <tcmalloc::Static::central_cache_+15808>) at src/base/spinlock.cc:80
#6  0x000014be974f010c in SpinLock::SlowLock (this=0x14be97688cc0 <tcmalloc::Static::central_cache_+15808>) at src/base/spinlock.cc:88
#7  0x000014be974e8e35 in SpinLock::Lock (this=0x14be97688cc0 <tcmalloc::Static::central_cache_+15808>) at src/base/spinlock.h:69
#8  SpinLockHolder::SpinLockHolder (l=0x14be97688cc0 <tcmalloc::Static::central_cache_+15808>, this=0x14be41903e48) at src/base/spinlock.h:124
#9  tcmalloc::CentralFreeList::InsertRange (this=this@entry=0x14be97688cc0 <tcmalloc::Static::central_cache_+15808>, start=0x11da51c80, end=0x11da8b440, N=N@entry=32) at src/central_freelist.cc:230
#10 0x000014be974ecbea in tcmalloc::ThreadCache::ReleaseToCentralCache (this=this@entry=0x490100, src=src@entry=0x4902a0, cl=<optimized out>, N=N@entry=32) at src/thread_cache.cc:207
#11 0x000014be974ecf2c in tcmalloc::ThreadCache::ListTooLong (this=0x490100, list=0x4902a0, cl=<optimized out>) at src/thread_cache.cc:164
#12 0x000014be32309064 in std::_Hashtable<int, std::pair<int const, l1t::HGCalTowerMap>, std::allocator<std::pair<int const, l1t::HGCalTowerMap> >, std::__detail::_Select1st, std::equal_to<int>, std::hash<int>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::clear() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/pluginL1TriggerL1THGCalPlugins_fe_be.so
#13 0x000014be3230cdce in void HGCalTowerMap2DImpl::buildTowerMap2D<l1t::HGCalTriggerSums>(std::vector<edm::Ptr<l1t::HGCalTriggerSums>, std::allocator<edm::Ptr<l1t::HGCalTriggerSums> > > const&, BXVector<l1t::HGCalTowerMap>&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/pluginL1TriggerL1THGCalPlugins_fe_be.so
#14 0x000014be3230d308 in HGCalTowerMapProcessor::run(edm::Handle<BXVector<l1t::HGCalTriggerSums> > const&, BXVector<l1t::HGCalTowerMap>&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/pluginL1TriggerL1THGCalPlugins_fe_be.so
#15 0x000014be75224da7 in HGCalTowerMapProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/pluginL1TriggerL1THGCalPlugins.so
#16 0x000014be993582bd in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so
#17 0x000014be9933ea22 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so
#18 0x000014be992c949a in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so
#19 0x000014be992c9948 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so
#20 0x000014be99483f79 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/libFWCoreConcurrency.so
#21 0x000014be97d8096f in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14be332efe00, waiter=..., this=0x14be95b95500) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-9a8ff57e6fe1e16621ddeb827f96f436/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#22 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14be95b95500) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-9a8ff57e6fe1e16621ddeb827f96f436/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#23 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-9a8ff57e6fe1e16621ddeb827f96f436/tbb-v2021.9.0/src/tbb/arena.cpp:137
#24 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-9a8ff57e6fe1e16621ddeb827f96f436/tbb-v2021.9.0/src/tbb/market.cpp:599
#25 0x000014be97d82bb5 in tbb::detail::r1::rml::private_worker::run (this=0x14be952f8000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-9a8ff57e6fe1e16621ddeb827f96f436/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#26 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14be952f8000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-9a8ff57e6fe1e16621ddeb827f96f436/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#27 0x000014be96cf91ca in start_thread () from /lib64/libpthread.so.0
#28 0x000014be96965e73 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x14be4239c700 (LWP 2170195) "cmsRun"):
#0  0x000014be96a25868 in nanosleep () from /lib64/libc.so.6
#1  0x000014be96a2576e in sleep () from /lib64/libc.so.6
#2  0x000014be94caab10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014be974e8992 in tcmalloc::CentralFreeList::ReleaseToSpans (this=0x14be97688cc0 <tcmalloc::Static::central_cache_+15808>, object=0x11ce538c0) at src/central_freelist.cc:110
#5  0x000014be974e8ae6 in tcmalloc::CentralFreeList::ReleaseListToSpans (this=this@entry=0x14be97688cc0 <tcmalloc::Static::central_cache_+15808>, start=<optimized out>, start@entry=0x11ce6b680) at src/central_freelist.cc:85
#6  0x000014be974e8dab in tcmalloc::CentralFreeList::InsertRange (this=this@entry=0x14be97688cc0 <tcmalloc::Static::central_cache_+15808>, start=0x11ce6b680, end=0x11ce7c9c0, N=N@entry=32) at src/central_freelist.cc:241
#7  0x000014be974ecbea in tcmalloc::ThreadCache::ReleaseToCentralCache (this=this@entry=0x48e080, src=src@entry=0x48e220, cl=<optimized out>, N=N@entry=32) at src/thread_cache.cc:207
#8  0x000014be974ecf2c in tcmalloc::ThreadCache::ListTooLong (this=0x48e080, list=0x48e220, cl=<optimized out>) at src/thread_cache.cc:164
#9  0x000014be32309064 in std::_Hashtable<int, std::pair<int const, l1t::HGCalTowerMap>, std::allocator<std::pair<int const, l1t::HGCalTowerMap> >, std::__detail::_Select1st, std::equal_to<int>, std::hash<int>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::clear() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/pluginL1TriggerL1THGCalPlugins_fe_be.so
#10 0x000014be3230cdce in void HGCalTowerMap2DImpl::buildTowerMap2D<l1t::HGCalTriggerSums>(std::vector<edm::Ptr<l1t::HGCalTriggerSums>, std::allocator<edm::Ptr<l1t::HGCalTriggerSums> > > const&, BXVector<l1t::HGCalTowerMap>&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/pluginL1TriggerL1THGCalPlugins_fe_be.so
#11 0x000014be3230d308 in HGCalTowerMapProcessor::run(edm::Handle<BXVector<l1t::HGCalTriggerSums> > const&, BXVector<l1t::HGCalTowerMap>&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/pluginL1TriggerL1THGCalPlugins_fe_be.so
#12 0x000014be75224da7 in HGCalTowerMapProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/pluginL1TriggerL1THGCalPlugins.so
#13 0x000014be993582bd in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so
#14 0x000014be9933ea22 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so
#15 0x000014be992c949a in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so
#16 0x000014be992c9948 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so
#17 0x000014be99483f79 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/libFWCoreConcurrency.so
#18 0x000014be97d8096f in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14be3347a200, waiter=..., this=0x14be95b95480) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-9a8ff57e6fe1e16621ddeb827f96f436/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#19 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14be95b95480) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-9a8ff57e6fe1e16621ddeb827f96f436/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#20 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-9a8ff57e6fe1e16621ddeb827f96f436/tbb-v2021.9.0/src/tbb/arena.cpp:137
#21 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-9a8ff57e6fe1e16621ddeb827f96f436/tbb-v2021.9.0/src/tbb/market.cpp:599
#22 0x000014be97d82bb5 in tbb::detail::r1::rml::private_worker::run (this=0x14be952f8100) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-9a8ff57e6fe1e16621ddeb827f96f436/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#23 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14be952f8100) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-9a8ff57e6fe1e16621ddeb827f96f436/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#24 0x000014be96cf91ca in start_thread () from /lib64/libpthread.so.0
#25 0x000014be96965e73 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x14be42d9d700 (LWP 2170194) "cmsRun"):
#0  0x000014be96a4ff41 in poll () from /lib64/libc.so.6
#1  0x000014be94ce5d0f in full_read.constprop () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#2  0x000014be94cad8ec in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  0x000014be94cae34b in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x000014be3761fd02 in TICLLayerTileProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/pluginRecoHGCalTICLPlugins.so
#6  0x000014be993582bd in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so
#7  0x000014be9933ea22 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so
#8  0x000014be992c949a in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so
#9  0x000014be992c9948 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so
#10 0x000014be99483f79 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/libFWCoreConcurrency.so
#11 0x000014be97d8096f in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14be332d5000, waiter=..., this=0x14be95b95400) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-9a8ff57e6fe1e16621ddeb827f96f436/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#12 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14be95b95400) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-9a8ff57e6fe1e16621ddeb827f96f436/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#13 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-9a8ff57e6fe1e16621ddeb827f96f436/tbb-v2021.9.0/src/tbb/arena.cpp:137
#14 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-9a8ff57e6fe1e16621ddeb827f96f436/tbb-v2021.9.0/src/tbb/market.cpp:599
#15 0x000014be97d82bb5 in tbb::detail::r1::rml::private_worker::run (this=0x14be952f8080) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-9a8ff57e6fe1e16621ddeb827f96f436/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#16 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14be952f8080) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-9a8ff57e6fe1e16621ddeb827f96f436/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#17 0x000014be96cf91ca in start_thread () from /lib64/libpthread.so.0
#18 0x000014be96965e73 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x14be7c354700 (LWP 2163837) "cmsRun"):
#0  0x000014be96d03672 in waitpid () from /lib64/libpthread.so.0
#1  0x000014be94caae87 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#2  0x000014be94cad81a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  0x000014be97386894 in std::execute_native_thread_routine (__p=0x525dfb0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x000014be96cf91ca in start_thread () from /lib64/libpthread.so.0
#5  0x000014be96965e73 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x14be95e88c80 (LWP 2156792) "cmsRun"):
#0  0x000014be96a25868 in nanosleep () from /lib64/libc.so.6
#1  0x000014be96a2576e in sleep () from /lib64/libc.so.6
#2  0x000014be94caab10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014be3019c2b3 in std::_Hashtable<unsigned int, unsigned int, std::allocator<unsigned int>, std::__detail::_Identity, std::equal_to<unsigned int>, std::hash<unsigned int>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node<unsigned int, false>*, unsigned long) [clone .constprop.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/pluginL1TriggerL1THGCalPlugins_geometries.so
#5  0x000014be3019dfa9 in std::pair<std::__detail::_Node_iterator<unsigned int, true, false>, bool> std::_Hashtable<unsigned int, unsigned int, std::allocator<unsigned int>, std::__detail::_Identity, std::equal_to<unsigned int>, std::hash<unsigned int>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_insert<unsigned int const&, std::__detail::_AllocNode<std::allocator<std::__detail::_Hash_node<unsigned int, false> > > >(unsigned int const&, std::__detail::_AllocNode<std::allocator<std::__detail::_Hash_node<unsigned int, false> > > const&, std::integral_constant<bool, true>) [clone .constprop.0] [clone .isra.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/pluginL1TriggerL1THGCalPlugins_geometries.so
#6  0x000014be3019193c in HGCalTriggerGeometryV9Imp3::getCellsFromModule(unsigned int) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/pluginL1TriggerL1THGCalPlugins_geometries.so
#7  0x000014be30191a4c in HGCalTriggerGeometryV9Imp3::getModulePosition(unsigned int) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/pluginL1TriggerL1THGCalPlugins_geometries.so
#8  0x000014be751d46e2 in HGCalConcentratorTrigSumImpl::doSum(unsigned int, std::vector<l1t::HGCalTriggerCell, std::allocator<l1t::HGCalTriggerCell> > const&, std::vector<l1t::HGCalTriggerSums, std::allocator<l1t::HGCalTriggerSums> >&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/libL1TriggerL1THGCal.so
#9  0x000014be3231632b in HGCalConcentratorProcessorSelection::run(edm::Handle<BXVector<l1t::HGCalTriggerCell> > const&, std::tuple<BXVector<l1t::HGCalTriggerCell>, BXVector<l1t::HGCalTriggerSums>, BXVector<l1t::HGCalConcentratorData> >&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/pluginL1TriggerL1THGCalPlugins_fe_be.so
#10 0x000014be752215e2 in HGCalConcentratorProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/pluginL1TriggerL1THGCalPlugins.so
#11 0x000014be993582bd in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so
#12 0x000014be9933ea22 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so
#13 0x000014be992c949a in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so
#14 0x000014be992c9948 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so
#15 0x000014be99483f79 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/libFWCoreConcurrency.so
#16 0x000014be97d88a6d in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x14be95b90e00, this=0x14be95b95380) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-9a8ff57e6fe1e16621ddeb827f96f436/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#17 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x14be95b95380) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-9a8ff57e6fe1e16621ddeb827f96f436/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#18 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-9a8ff57e6fe1e16621ddeb827f96f436/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#19 0x000014be9924f39d in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so
#20 0x000014be9925ce81 in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so
#21 0x000014be9925d411 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_GEANT4_X_2023-08-02-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so
#22 0x000000000040c17d in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#23 0x000014be97d75877 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-9a8ff57e6fe1e16621ddeb827f96f436/tbb-v2021.9.0/src/tbb/arena.cpp:688
#24 0x00000000004101e0 in main::{lambda()#1}::operator()() const ()
#25 0x0000000000407d6c in main ()

Current Modules:

Module: TICLLayerTileProducer:ticlLayerTileProducer (crashed)
Module: HGCalTowerMapProducer:l1tHGCalTowerMapProducer
Module: HGCalTowerMapProducer:l1tHGCalTowerMapProducer
Module: HGCalConcentratorProducer:l1tHGCalConcentratorProducer

A fatal system signal has occurred: segmentation violation

I cannot reproduce the failure locally, but I am reporting to keep track of the failures specially since this failure has appeared as of the move to tcmalloc.

Thanks, Andrea.