cmssw: Crash in DD4hep geometry, step2

As reported by @bsunanda at the latest SIM meeting, DD4hep geometries succeed in step1, but fail in phase2 step2, see geometry D77 wf 35034.911

PersistencyIO    INFO  +++ Set Streamer to dd4hep::OpaqueDataBlock
DD4hep           WARN  ++ Using globally Geant4 unit system (mm,ns,MeV)
CompactLoader    INFO  +++ Processing compact file: /gpfs/cms/users/cossutti/Timing/geometry/CMSSW_12_3_X_2022-01-27-2300/src/Geometry/CMSCommonData/data/dd4hep/cmsExtendedGeometry2026D77.xml with flag (null)
DD4CMS           INFO  +++ Processing the CMS detector description file:///gpfs/cms/users/cossutti/Timing/geometry/CMSSW_12_3_X_2022-01-27-2300/src/Geometry/CMSCommonData/data/dd4hep/cmsExtendedGeometry2026D77.xml
Detector         INFO  *********** Created World volume with size: 101000 101000 450000
PlacedVolume     INFO  REFLECTION: (x.Cross(y)).Dot(z):       -1 Parent: eregalgo:EFAW [TGeoVolume] Daughter: eregalgo:EHAWR [TGeoVolumeAssembly]
Detector         INFO  +++ Patching names of anonymous shapes....
DDDefinition     INFO  +++ Finished processing file:///gpfs/cms/users/cossutti/Timing/geometry/CMSSW_12_3_X_2022-01-27-2300/src/Geometry/CMSCommonData/data/dd4hep/cmsExtendedGeometry2026D77.xml
DD4hep           WARN  ++ Using globally Geant4 unit system (mm,ns,MeV)
DD4CMS           INFO  +++ Processing the CMS detector description xml-memory-buffer
Detector         INFO  *********** Created World volume with size: 101000 101000 450000
Detector         INFO  +++ Patching names of anonymous shapes....
DDDefinition     INFO  +++ Finished processing xml-memory-buffer
----- Begin Fatal Exception 28-Jan-2022 19:05:20 CET-----------------------
An exception of category 'InvalidReference' occurred while
   [0] Processing global begin Run run: 1
   [1] Prefetching for module L1FPGATrackProducer/'TTTracksFromExtendedTrackletEmulation'
   [2] Calling method for EventSetup module trackerDTC::ProducerES/'TrackTriggerSetup'
Exception Message:
NullPointer 
----- End Fatal Exception -------------------------------------------------

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 38 (38 by maintainers)

Most upvoted comments

@ianna @cvuosalo @civanch #36970 proposes a possible solution to the problem found by @srimanob, which should allow us to move forward in catching other geometry problems in the DD4hep version of D88 step2.

fabiocos on Feb 15, 2022

Following the discussion at last simulation meeting, I checked scenario D88 for MTD only (see https://github.com/fabiocos/cmssw/tree/fc-testmtdD88 ) and I confirm there is a crash in the reconstruction geometry of MTD:

Begin processing the 1st record. Run 1, Event 1, LumiSection 1 on stream 0 at 14-Feb-2022 10:15:53.539 CET
PersistencyIO    INFO  +++ Set Streamer to dd4hep::OpaqueDataBlock
DD4hep           WARN  ++ Using globally Geant4 unit system (mm,ns,MeV)
CompactLoader    INFO  +++ Processing compact file: /cvmfs/cms-ib.cern.ch/nweek-02719/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-02-11-1100/src/Geometry/CMSCommonData/data/dd4hep/cmsExtendedGeometry2026D88.xml with flag (null)
DD4CMS           INFO  +++ Processing the CMS detector description file:///cvmfs/cms-ib.cern.ch/nweek-02719/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-02-11-1100/src/Geometry/CMSCommonData/data/dd4hep/cmsExtendedGeometry2026D88.xml
Detector         INFO  *********** Created World volume with size: 101000 101000 450000
PlacedVolume     INFO  REFLECTION: (x.Cross(y)).Dot(z):       -1 Parent: eregalgo:EFAW [TGeoVolume] Daughter: eregalgo:EHAWR [TGeoVolumeAssembly]
Detector         INFO  +++ Patching names of anonymous shapes....


A fatal system signal has occurred: bus error
The following is the call stack containing the origin of the signal.

Mon Feb 14 10:16:01 CET 2022
Thread 2 (Thread 0x7fa2ddab6700 (LWP 22715) "cmsRun"):
#0  0x00007fa3061cb1d9 in waitpid () from /lib64/libpthread.so.0
#1  0x00007fa2f8e5ce07 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/nweek-02719/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-02-10-1100/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#2  0x00007fa2f8e5d97a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02719/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-02-10-1100/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#3  0x00007fa3067c3f90 in std::execute_native_thread_routine (__p=0x7fa2e8972540) at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#4  0x00007fa3061c3ea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa305eecb0d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fa30403f540 (LWP 22518) "cmsRun"):
#0  0x00007fa305ee1ddd in poll () from /lib64/libc.so.6
#1  0x00007fa2f8e5d0bf in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02719/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-02-10-1100/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#2  0x00007fa2f8e5da4c in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02719/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-02-10-1100/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#3  0x00007fa2f8e6039b in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02719/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-02-10-1100/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007fa2dcb99900 in cms::DDFilteredView::parameters() const () from /cvmfs/cms-ib.cern.ch/nweek-02719/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-02-10-1100/lib/slc7_amd64_gcc10/libDetectorDescriptionDDCMS.so
#6  0x00007fa2dac8427b in GeometricTimingDet::GeometricTimingDet(cms::DDFilteredView*, GeometricTimingDet::GTDEnumType) () from /gpfs/cms/users/cossutti/Timing/geometry/CMSSW_12_3_X_2022-02-11-1100/lib/slc7_amd64_gcc10/libGeometryMTDNumberingBuilder.so
#7  0x00007fa2da51eae0 in CmsMTDConstruction<cms::DDFilteredView>::buildSubdet(cms::DDFilteredView&) () from /gpfs/cms/users/cossutti/Timing/geometry/CMSSW_12_3_X_2022-02-11-1100/lib/slc7_amd64_gcc10/pluginGeometryMTDNumberingBuilderPlugins.so
#8  0x00007fa2da52026d in DDCmsMTDConstruction::construct(cms::DDCompactView const&) () from /gpfs/cms/users/cossutti/Timing/geometry/CMSSW_12_3_X_2022-02-11-1100/lib/slc7_amd64_gcc10/pluginGeometryMTDNumberingBuilderPlugins.so
#9  0x00007fa2da5245a3 in MTDGeometricTimingDetESModule::produce(IdealGeometryRecord const&) () from /gpfs/cms/users/cossutti/Timing/geometry/CMSSW_12_3_X_2022-02-11-1100/lib/slc7_amd64_gcc10/pluginGeometryMTDNumberingBuilderPlugins.so
#10 0x00007fa2da52ac79 in edm::eventsetup::Callback<MTDGeometricTimingDetESModule, std::unique_ptr<GeometricTimingDet, std::default_delete<GeometricTimingDet> >, IdealGeometryRecord, edm::eventsetup::CallbackSimpleDecorator<IdealGeometryRecord> >::runProducerAsync(tbb::detail::d1::task_group*, std::__exception_ptr::exception_ptr const*, edm::eventsetup::EventSetupRecordImpl const*, edm::EventSetupImpl const*, edm::ServiceToken const&)::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const () from /gpfs/cms/users/cossutti/Timing/geometry/CMSSW_12_3_X_2022-02-11-1100/lib/slc7_amd64_gcc10/pluginGeometryMTDNumberingBuilderPlugins.so
#11 0x00007fa2da52b336 in decltype ({parm#1}()) edm::convertException::wrap<edm::eventsetup::Callback<MTDGeometricTimingDetESModule, std::unique_ptr<GeometricTimingDet, std::default_delete<GeometricTimingDet> >, IdealGeometryRecord, edm::eventsetup::CallbackSimpleDecorator<IdealGeometryRecord> >::runProducerAsync(tbb::detail::d1::task_group*, std::__exception_ptr::exception_ptr const*, edm::eventsetup::EventSetupRecordImpl const*, edm::EventSetupImpl const*, edm::ServiceToken const&)::{lambda()#1}::operator()() const::{lambda()#1}>(edm::eventsetup::Callback<MTDGeometricTimingDetESModule, std::unique_ptr<GeometricTimingDet, std::default_delete<GeometricTimingDet> >, IdealGeometryRecord, edm::eventsetup::CallbackSimpleDecorator<IdealGeometryRecord> >::runProducerAsync(tbb::detail::d1::task_group*, std::__exception_ptr::exception_ptr const*, edm::eventsetup::EventSetupRecordImpl const*, edm::EventSetupImpl const*, edm::ServiceToken const&)::{lambda()#1}::operator()() const::{lambda()#1}) () from /gpfs/cms/users/cossutti/Timing/geometry/CMSSW_12_3_X_2022-02-11-1100/lib/slc7_amd64_gcc10/pluginGeometryMTDNumberingBuilderPlugins.so
#12 0x00007fa2da52b454 in edm::eventsetup::Callback<MTDGeometricTimingDetESModule, std::unique_ptr<GeometricTimingDet, std::default_delete<GeometricTimingDet> >, IdealGeometryRecord, edm::eventsetup::CallbackSimpleDecorator<IdealGeometryRecord> >::runProducerAsync(tbb::detail::d1::task_group*, std::__exception_ptr::exception_ptr const*, edm::eventsetup::EventSetupRecordImpl const*, edm::EventSetupImpl const*, edm::ServiceToken const&)::{lambda()#1}::operator()() const () from /gpfs/cms/users/cossutti/Timing/geometry/CMSSW_12_3_X_2022-02-11-1100/lib/slc7_amd64_gcc10/pluginGeometryMTDNumberingBuilderPlugins.so
#13 0x00007fa2da52c44f in void edm::SerialTaskQueueChain::actionToRun<edm::eventsetup::Callback<MTDGeometricTimingDetESModule, std::unique_ptr<GeometricTimingDet, std::default_delete<GeometricTimingDet> >, IdealGeometryRecord, edm::eventsetup::CallbackSimpleDecorator<IdealGeometryRecord> >::runProducerAsync(tbb::detail::d1::task_group*, std::__exception_ptr::exception_ptr const*, edm::eventsetup::EventSetupRecordImpl const*, edm::EventSetupImpl const*, edm::ServiceToken const&)::{lambda()#1}&>(edm::eventsetup::Callback<MTDGeometricTimingDetESModule, std::unique_ptr<GeometricTimingDet, std::default_delete<GeometricTimingDet> >, IdealGeometryRecord, edm::eventsetup::CallbackSimpleDecorator<IdealGeometryRecord> >::runProducerAsync(tbb::detail::d1::task_group*, std::__exception_ptr::exception_ptr const*, edm::eventsetup::EventSetupRecordImpl const*, edm::EventSetupImpl const*, edm::ServiceToken const&)::{lambda()#1}&) () from /gpfs/cms/users/cossutti/Timing/geometry/CMSSW_12_3_X_2022-02-11-1100/lib/slc7_amd64_gcc10/pluginGeometryMTDNumberingBuilderPlugins.so
#14 0x00007fa2da52c4c1 in edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::eventsetup::Callback<MTDGeometricTimingDetESModule, std::unique_ptr<GeometricTimingDet, std::default_delete<GeometricTimingDet> >, IdealGeometryRecord, edm::eventsetup::CallbackSimpleDecorator<IdealGeometryRecord> >::runProducerAsync(tbb::detail::d1::task_group*, std::__exception_ptr::exception_ptr const*, edm::eventsetup::EventSetupRecordImpl const*, edm::EventSetupImpl const*, edm::ServiceToken const&)::{lambda()#1}>(tbb::detail::d1::task_group&, edm::eventsetup::Callback<MTDGeometricTimingDetESModule, std::unique_ptr<GeometricTimingDet, std::default_delete<GeometricTimingDet> >, IdealGeometryRecord, edm::eventsetup::CallbackSimpleDecorator<IdealGeometryRecord> >::runProducerAsync(tbb::detail::d1::task_group*, std::__exception_ptr::exception_ptr const*, edm::eventsetup::EventSetupRecordImpl const*, edm::EventSetupImpl const*, edm::ServiceToken const&)::{lambda()#1}&&)::{lambda()#1}>::execute() () from /gpfs/cms/users/cossutti/Timing/geometry/CMSSW_12_3_X_2022-02-11-1100/lib/slc7_amd64_gcc10/pluginGeometryMTDNumberingBuilderPlugins.so
#15 0x00007fa308a8f075 in tbb::detail::d1::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02719/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-02-10-1100/lib/slc7_amd64_gcc10/libFWCoreConcurrency.so
#16 0x00007fa306feeb8c in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x7fa302a9f300, this=0x7fa302acb700) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/task_dispatcher.h:322
#17 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x7fa302acb700) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/task_dispatcher.h:463
#18 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/task_dispatcher.cpp:168
#19 0x00007fa30880a988 in edm::EventProcessor::processLumis(std::shared_ptr<void> const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-02-11-1100/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#20 0x00007fa3088157db in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_3_X_2022-02-11-1100/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#21 0x000000000040a266 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#22 0x00007fa306fdd15b in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-75e6d730601d8461f20893321f4f7660/tbb-v2021.4.0/src/tbb/arena.cpp:698
#23 0x000000000040b094 in main::{lambda()#1}::operator()() const ()
#24 0x000000000040971c in main ()

To be understood and fixed

fabiocos on Feb 14, 2022