cmssw: buffer overflow in RecHitsSortedInPhi on NaN

ASAN found a heap buffer overflow in RecHitsSortedInPhi::RecHitsSortedInPhi(), very likely because phi for the track was NaN, which doesn’t sort well.

%MSG-e TrajectoryNaN:  TrackProducer:hltIterL3OIMuCtfWithMaterialTracksNoVtx  30-Sep-2021 06:32:04 CEST Run: 1 Event: 109
Trajectory has NaN
%MSG
%MSG-e TrajectoryNaN:  TrackProducer:hltIterL3OIMuCtfWithMaterialTracksNoVtx  30-Sep-2021 06:32:04 CEST Run: 1 Event: 109
Trajectory has NaN
%MSG
%MSG-e TrajectoryNaN:  TrackProducer:hltIterL3OIMuCtfWithMaterialTracks  30-Sep-2021 06:32:05 CEST Run: 1 Event: 109
Trajectory has NaN
%MSG
%MSG-e TrajectoryNaN:   TrackProducer:hltIterL3OIMuCtfWithMaterialTracksScoutingNoVtx  30-Sep-2021 06:32:11 CEST Run: 1 Event: 109
Trajectory has NaN
%MSG
%MSG-e TrajectoryNaN:   TrackProducer:hltIterL3OIMuCtfWithMaterialTracksScoutingNoVtx  30-Sep-2021 06:32:11 CEST Run: 1 Event: 109
Trajectory has NaN
%MSG
=================================================================
==3648==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x61b0013c9d78 at pc 0x2b614f17ac19 bp 0x2b616f95a810 sp 0x2b616f95a808
READ of size 4 at 0x61b0013c9d78 thread T7
    #0 0x2b614f17ac18 in RecHitsSortedInPhi::RecHitsSortedInPhi(std::vector<BaseTrackerRecHit const*, std::allocator<BaseTrackerRecHit const*> > const&, Point3DBase<float, GlobalTag> const&, DetLayer const*) (/cvmfs/cms-ib.cern.ch/nweek-02700/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_ASAN_X_2021-09-29-2300/lib/slc7_amd64_gcc10/libRecoTrackerTkHitPairs.so+0x82c18)
    #1 0x2b614f16984a in LayerHitMapCache::operator()(SeedingLayerSetsHits::SeedingLayer const&, TrackingRegion const&, edm::EventSetup const&) (/cvmfs/cms-ib.cern.ch/nweek-02700/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_ASAN_X_2021-09-29-2300/lib/slc7_amd64_gcc10/libRecoTrackerTkHitPairs.so+0x7184a)
    #2 0x2b614f163383 in HitPairGeneratorFromLayerPair::doublets(TrackingRegion const&, edm::Event const&, edm::EventSetup const&, SeedingLayerSetsHits::SeedingLayer const&, SeedingLayerSetsHits::SeedingLayer const&, LayerHitMapCache&) (/cvmfs/cms-ib.cern.ch/nweek-02700/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_ASAN_X_2021-09-29-2300/lib/slc7_amd64_gcc10/libRecoTrackerTkHitPairs.so+0x6b383)
    #3 0x2b6180807612 in (anonymous namespace)::Impl<(anonymous namespace)::DoNothing, (anonymous namespace)::ImplIntermediateHitDoublets, (anonymous namespace)::RegionsLayersSeparate>::produce(bool, edm::Event&, edm::EventSetup const&) (/cvmfs/cms-ib.cern.ch/nweek-02700/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_ASAN_X_2021-09-29-2300/lib/slc7_amd64_gcc10/pluginRecoTrackerTkHitPairsPlugins.so+0x31612)
    #4 0x2b61807ed63a in HitPairEDProducer::produce(edm::Event&, edm::EventSetup const&) (/cvmfs/cms-ib.cern.ch/nweek-02700/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_ASAN_X_2021-09-29-2300/lib/slc7_amd64_gcc10/pluginRecoTrackerTkHitPairsPlugins.so+0x1763a)
    #5 0x2b611cdb3f1e in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/nweek-02700/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_ASAN_X_2021-09-29-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so+0x904f1e)
    #6 0x2b611cd1ee72 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/nweek-02700/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_ASAN_X_2021-09-29-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so+0x86fe72)
    #7 0x2b611ca13004 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) (/cvmfs/cms-ib.cern.ch/nweek-02700/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_ASAN_X_2021-09-29-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so+0x564004)
    #8 0x2b611ca1384a in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) (/cvmfs/cms-ib.cern.ch/nweek-02700/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_ASAN_X_2021-09-29-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so+0x56484a)
    #9 0x2b611ca1f2a6 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() (/cvmfs/cms-ib.cern.ch/nweek-02700/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_ASAN_X_2021-09-29-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so+0x5702a6)
    #10 0x2b611d719231 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (/cvmfs/cms-ib.cern.ch/nweek-02700/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_ASAN_X_2021-09-29-2300/lib/slc7_amd64_gcc10/libFWCoreConcurrency.so+0x11231)
    #11 0x2b611ed433de in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_12_1_0_pre3-slc7_amd64_gcc10/build/CMSSW_12_1_0_pre3-build/BUILD/slc7_amd64_gcc10/external/tbb/v2021.3.0-d08d4722ccbb5e1470ca0b76226c3cf7/tbb-v2021.3.0/src/tbb/task_dispatcher.h:321
    #12 0x2b611ed433de in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_12_1_0_pre3-slc7_amd64_gcc10/build/CMSSW_12_1_0_pre3-build/BUILD/slc7_amd64_gcc10/external/tbb/v2021.3.0-d08d4722ccbb5e1470ca0b76226c3cf7/tbb-v2021.3.0/src/tbb/task_dispatcher.h:462
    #13 0x2b611ed433de in tbb::detail::r1::arena::process(tbb::detail::r1::thread_data&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_12_1_0_pre3-slc7_amd64_gcc10/build/CMSSW_12_1_0_pre3-build/BUILD/slc7_amd64_gcc10/external/tbb/v2021.3.0-d08d4722ccbb5e1470ca0b76226c3cf7/tbb-v2021.3.0/src/tbb/arena.cpp:138
    #14 0x2b611ed528ea in tbb::detail::r1::market::process(rml::job&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_12_1_0_pre3-slc7_amd64_gcc10/build/CMSSW_12_1_0_pre3-build/BUILD/slc7_amd64_gcc10/external/tbb/v2021.3.0-d08d4722ccbb5e1470ca0b76226c3cf7/tbb-v2021.3.0/src/tbb/market.cpp:596
    #15 0x2b611ed562d3 in tbb::detail::r1::rml::private_worker::run() /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_12_1_0_pre3-slc7_amd64_gcc10/build/CMSSW_12_1_0_pre3-build/BUILD/slc7_amd64_gcc10/external/tbb/v2021.3.0-d08d4722ccbb5e1470ca0b76226c3cf7/tbb-v2021.3.0/src/tbb/private_server.cpp:267
    #16 0x2b611ed564f8 in tbb::detail::r1::rml::private_worker::thread_routine(void*) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_12_1_0_pre3-slc7_amd64_gcc10/build/CMSSW_12_1_0_pre3-build/BUILD/slc7_amd64_gcc10/external/tbb/v2021.3.0-d08d4722ccbb5e1470ca0b76226c3cf7/tbb-v2021.3.0/src/tbb/private_server.cpp:221
    #17 0x2b611f955ea4 in start_thread (/lib64/libpthread.so.0+0x7ea4)
    #18 0x2b611fc689fc in __clone (/lib64/libc.so.6+0xfe9fc)

0x61b0013c9d78 is located 8 bytes to the left of 1552-byte region [0x61b0013c9d80,0x61b0013ca390)
allocated by thread T7 here:
    #0 0x2b611bb923d7 in operator new(unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cpp:99
    #1 0x2b614f17a99a in RecHitsSortedInPhi::RecHitsSortedInPhi(std::vector<BaseTrackerRecHit const*, std::allocator<BaseTrackerRecHit const*> > const&, Point3DBase<float, GlobalTag> const&, DetLayer const*) (/cvmfs/cms-ib.cern.ch/nweek-02700/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_ASAN_X_2021-09-29-2300/lib/slc7_amd64_gcc10/libRecoTrackerTkHitPairs.so+0x8299a)
    #2 0x2b614f16984a in LayerHitMapCache::operator()(SeedingLayerSetsHits::SeedingLayer const&, TrackingRegion const&, edm::EventSetup const&) (/cvmfs/cms-ib.cern.ch/nweek-02700/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_ASAN_X_2021-09-29-2300/lib/slc7_amd64_gcc10/libRecoTrackerTkHitPairs.so+0x7184a)
    #3 0x2b614f163383 in HitPairGeneratorFromLayerPair::doublets(TrackingRegion const&, edm::Event const&, edm::EventSetup const&, SeedingLayerSetsHits::SeedingLayer const&, SeedingLayerSetsHits::SeedingLayer const&, LayerHitMapCache&) (/cvmfs/cms-ib.cern.ch/nweek-02700/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_ASAN_X_2021-09-29-2300/lib/slc7_amd64_gcc10/libRecoTrackerTkHitPairs.so+0x6b383)
    #4 0x2b6180807612 in (anonymous namespace)::Impl<(anonymous namespace)::DoNothing, (anonymous namespace)::ImplIntermediateHitDoublets, (anonymous namespace)::RegionsLayersSeparate>::produce(bool, edm::Event&, edm::EventSetup const&) (/cvmfs/cms-ib.cern.ch/nweek-02700/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_ASAN_X_2021-09-29-2300/lib/slc7_amd64_gcc10/pluginRecoTrackerTkHitPairsPlugins.so+0x31612)
    #5 0x2b61807ed63a in HitPairEDProducer::produce(edm::Event&, edm::EventSetup const&) (/cvmfs/cms-ib.cern.ch/nweek-02700/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_ASAN_X_2021-09-29-2300/lib/slc7_amd64_gcc10/pluginRecoTrackerTkHitPairsPlugins.so+0x1763a)
    #6 0x2b611cdb3f1e in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/nweek-02700/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_ASAN_X_2021-09-29-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so+0x904f1e)
    #7 0x2b611cd1ee72 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/nweek-02700/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_ASAN_X_2021-09-29-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so+0x86fe72)
    #8 0x2b611ca13004 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) (/cvmfs/cms-ib.cern.ch/nweek-02700/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_ASAN_X_2021-09-29-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so+0x564004)
    #9 0x2b611ca1384a in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) (/cvmfs/cms-ib.cern.ch/nweek-02700/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_ASAN_X_2021-09-29-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so+0x56484a)
    #10 0x2b611ca1f2a6 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() (/cvmfs/cms-ib.cern.ch/nweek-02700/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_ASAN_X_2021-09-29-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so+0x5702a6)
    #11 0x2b611d719231 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (/cvmfs/cms-ib.cern.ch/nweek-02700/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_ASAN_X_2021-09-29-2300/lib/slc7_amd64_gcc10/libFWCoreConcurrency.so+0x11231)
    #12 0x2b611ed433de in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_12_1_0_pre3-slc7_amd64_gcc10/build/CMSSW_12_1_0_pre3-build/BUILD/slc7_amd64_gcc10/external/tbb/v2021.3.0-d08d4722ccbb5e1470ca0b76226c3cf7/tbb-v2021.3.0/src/tbb/task_dispatcher.h:321
    #13 0x2b611ed433de in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_12_1_0_pre3-slc7_amd64_gcc10/build/CMSSW_12_1_0_pre3-build/BUILD/slc7_amd64_gcc10/external/tbb/v2021.3.0-d08d4722ccbb5e1470ca0b76226c3cf7/tbb-v2021.3.0/src/tbb/task_dispatcher.h:462
    #14 0x2b611ed433de in tbb::detail::r1::arena::process(tbb::detail::r1::thread_data&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_12_1_0_pre3-slc7_amd64_gcc10/build/CMSSW_12_1_0_pre3-build/BUILD/slc7_amd64_gcc10/external/tbb/v2021.3.0-d08d4722ccbb5e1470ca0b76226c3cf7/tbb-v2021.3.0/src/tbb/arena.cpp:138
    #15 0x2b611ed528ea in tbb::detail::r1::market::process(rml::job&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_12_1_0_pre3-slc7_amd64_gcc10/build/CMSSW_12_1_0_pre3-build/BUILD/slc7_amd64_gcc10/external/tbb/v2021.3.0-d08d4722ccbb5e1470ca0b76226c3cf7/tbb-v2021.3.0/src/tbb/market.cpp:596

Thread T7 created by T0 here:
    #0 0x2b611bb3c0a2 in __interceptor_pthread_create ../../../../libsanitizer/asan/asan_interceptors.cpp:214
    #1 0x2b611ed5608e in tbb::detail::r1::rml::internal::thread_monitor::launch(void* (*)(void*), void*, unsigned long) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_12_1_0_pre3-slc7_amd64_gcc10/build/CMSSW_12_1_0_pre3-build/BUILD/slc7_amd64_gcc10/external/tbb/v2021.3.0-d08d4722ccbb5e1470ca0b76226c3cf7/tbb-v2021.3.0/src/tbb/rml_thread_monitor.h:208
    #2 0x2b611ed5608e in tbb::detail::r1::rml::private_worker::wake_or_launch() /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_12_1_0_pre3-slc7_amd64_gcc10/build/CMSSW_12_1_0_pre3-build/BUILD/slc7_amd64_gcc10/external/tbb/v2021.3.0-d08d4722ccbb5e1470ca0b76226c3cf7/tbb-v2021.3.0/src/tbb/private_server.cpp:299
    #3 0x2b611ed5608e in tbb::detail::r1::rml::private_server::wake_some(int) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_12_1_0_pre3-slc7_amd64_gcc10/build/CMSSW_12_1_0_pre3-build/BUILD/slc7_amd64_gcc10/external/tbb/v2021.3.0-d08d4722ccbb5e1470ca0b76226c3cf7/tbb-v2021.3.0/src/tbb/private_server.cpp:397

SUMMARY: AddressSanitizer: heap-buffer-overflow (/cvmfs/cms-ib.cern.ch/nweek-02700/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_ASAN_X_2021-09-29-2300/lib/slc7_amd64_gcc10/libRecoTrackerTkHitPairs.so+0x82c18) in RecHitsSortedInPhi::RecHitsSortedInPhi(std::vector<BaseTrackerRecHit const*, std::allocator<BaseTrackerRecHit const*> > const&, Point3DBase<float, GlobalTag> const&, DetLayer const*)
Shadow bytes around the buggy address:
  0x0c3680271350: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c3680271360: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c3680271370: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c3680271380: 00 00 00 00 fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c3680271390: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c36802713a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa[fa]
  0x0c36802713b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c36802713c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c36802713d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c36802713e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c36802713f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==3648==ABORTING

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 21 (19 by maintainers)

Most upvoted comments

back to 1)

added

iff --git a/RecoTracker/TkHitPairs/src/RecHitsSortedInPhi.cc b/RecoTracker/TkHitPairs/src/RecHitsSortedInPhi.cc
index 3d806501f56..0969c0daa2a 100644
--- a/RecoTracker/TkHitPairs/src/RecHitsSortedInPhi.cc
+++ b/RecoTracker/TkHitPairs/src/RecHitsSortedInPhi.cc
@@ -1,5 +1,6 @@
 #include "RecoTracker/TkHitPairs/interface/RecHitsSortedInPhi.h"
 #include "DataFormats/TrackerRecHit2D/interface/BaseTrackerRecHit.h"
+#include "FWCore/Utilities/interface/isFinite.h"

 #include <algorithm>
 #include <cassert>
@@ -21,9 +22,13 @@ RecHitsSortedInPhi::RecHitsSortedInPhi(const std::vector<Hit>& hits, GlobalPoint
   // assert(origin.x()==0 && origin.y()==0);

   theHits.reserve(hits.size());
-  for (auto const& hp : hits)
+  for (auto const& hp : hits) {
+    auto gp = hp->globalPosition();
+    auto phi = gp.barePhi();
+    if (!edm::isFinite(phi))
+      std::cout << "NAN in layer " << layer << ' ' << hp->rawId() << ' ' << hp-> type() << ' ' << phi << ' ' << gp << ' ' << hp->localPosition() << std::endl;
     theHits.emplace_back(hp);
-
+  }
   std::sort(theHits.begin(), theHits.end(), HitLessPhi());

   for (unsigned int i = 0; i != theHits.size(); ++i) {

generated this

[innocent@patatrack02 innocent]$ dasgoclient --limit 0 --query 'file dataset=/RelValTTbar_14TeV/CMSSW_12_0_0_pre4-120X_mcRun3_2021_realistic_v2-v1/GEN-SIM' | sort -u > step1_dasquery.log
[innocent@patatrack02 innocent]$ cmsDriver.py step2 -s DIGI:pdigi_valid,L1,DIGI2RAW,HLT:@relval2021 --conditions auto:phase1_2021_realistic --datatier GEN-SIM-DIGI-RAW -n 10 --eventcontent FEVTDEBUGHLT --geometry DB:Extended --era Run3 --pileup Run3_Flat55To75_PoissonOOTPU --pileup_input das:/RelValMinBias_14TeV/CMSSW_12_0_0_pre4-120X_mcRun3_2021_realistic_v2-v1/GEN-SIM --filein filelist:step1_dasquery.log --fileout file:step2.root --no_exec
DIGI:pdigi_valid,L1,DIGI2RAW,HLT:@relval2021,ENDJOB
We have determined that this is simulation (if not, rerun cmsDriver.py with --data)
with DB:
entry filelist:step1_dasquery.log
found files:  ['/store/relval/CMSSW_12_0_0_pre4/RelValTTbar_14TeV/GEN-SIM/120X_mcRun3_2021_realistic_v2-v1/00000/361e4fff-3935-4325-9891-62841915ccd1.root', '/store/relval/CMSSW_12_0_0_pre4/RelValTTbar_14TeV/GEN-SIM/120X_mcRun3_2021_realistic_v2-v1/00000/5a5f9bff-e48b-47c3-9081-5b77513d1bff.root', '/store/relval/CMSSW_12_0_0_pre4/RelValTTbar_14TeV/GEN-SIM/120X_mcRun3_2021_realistic_v2-v1/00000/826f9b32-706e-467e-a51b-f3636d7d4147.root', '/store/relval/CMSSW_12_0_0_pre4/RelValTTbar_14TeV/GEN-SIM/120X_mcRun3_2021_realistic_v2-v1/00000/99843aa4-3246-443b-88be-da487ad3edc8.root']
the query is file dataset = /RelValMinBias_14TeV/CMSSW_12_0_0_pre4-120X_mcRun3_2021_realistic_v2-v1/GEN-SIM
DAS succeeded after 1 attempts 0
found files:  ['/store/relval/CMSSW_12_0_0_pre4/RelValMinBias_14TeV/GEN-SIM/120X_mcRun3_2021_realistic_v2-v1/00000/1f2432c4-b04b-4be9-a5f4-d6121657ad61.root', '/store/relval/CMSSW_12_0_0_pre4/RelValMinBias_14TeV/GEN-SIM/120X_mcRun3_2021_realistic_v2-v1/00000/5188ff57-8de1-4731-958d-0206eefb725e.root', '/store/relval/CMSSW_12_0_0_pre4/RelValMinBias_14TeV/GEN-SIM/120X_mcRun3_2021_realistic_v2-v1/00000/8ffd7938-eb07-44b5-9694-ea018ac892db.root', '/store/relval/CMSSW_12_0_0_pre4/RelValMinBias_14TeV/GEN-SIM/120X_mcRun3_2021_realistic_v2-v1/00000/ace00a2c-259f-4ac1-b449-9fed3dc70e17.root', '/store/relval/CMSSW_12_0_0_pre4/RelValMinBias_14TeV/GEN-SIM/120X_mcRun3_2021_realistic_v2-v1/00000/b1db77ed-9af1-41e1-8acf-6aeb3a4ffefd.root', '/store/relval/CMSSW_12_0_0_pre4/RelValMinBias_14TeV/GEN-SIM/120X_mcRun3_2021_realistic_v2-v1/00000/e9794f10-c40e-45ad-8677-0f3f7410d966.root']

and see no error report

Begin processing the 2nd record. Run 1, Event 101, LumiSection 2 on stream 0 at 01-Oct-2021 14:59:26.932 CEST
[2021-10-01 15:00:42.020547 +0200][Error  ][PostMaster        ] [cmsxrootd-site3.fnal.gov:1093 #0] Forcing error on disconnect: [ERROR] Operation interrupted.
[2021-10-01 15:00:51.071862 +0200][Error  ][PostMaster        ] [cmsxrootd-site1.fnal.gov:1093 #0] Forcing error on disconnect: [ERROR] Operation interrupted.
Begin processing the 3rd record. Run 1, Event 103, LumiSection 2 on stream 0 at 01-Oct-2021 15:02:11.713 CEST
Begin processing the 4th record. Run 1, Event 104, LumiSection 2 on stream 0 at 01-Oct-2021 15:05:33.020 CEST
Begin processing the 5th record. Run 1, Event 105, LumiSection 2 on stream 0 at 01-Oct-2021 15:08:32.481 CEST
[2021-10-01 15:09:39.030113 +0200][Error  ][PostMaster        ] [cmsxrootd-site2.fnal.gov:1093 #0] Forcing error on disconnect: [ERROR] Operation interrupted.
%MSG-e TrajectoryNotPosDef:   TrackProducer:hltDoubletRecoveryPFlowCtfWithMaterialTracksForBTag  01-Oct-2021 15:11:16 CEST Run: 1 Event: 105
Trajectory covariance is not positive-definite
%MSG
Begin processing the 6th record. Run 1, Event 106, LumiSection 2 on stream 0 at 01-Oct-2021 15:11:28.326 CEST
[2021-10-01 15:12:56.048811 +0200][Error  ][PostMaster        ] [eoscms.cern.ch:1094 #0] Forcing error on disconnect: [ERROR] Operation interrupted.
Begin processing the 7th record. Run 1, Event 107, LumiSection 2 on stream 0 at 01-Oct-2021 15:14:13.954 CEST
[2021-10-01 15:15:42.037775 +0200][Error  ][PostMaster        ] [cmsxrootd-site3.fnal.gov:1094 #0] Forcing error on disconnect: [ERROR] Operation interrupted.
Begin processing the 8th record. Run 1, Event 109, LumiSection 2 on stream 0 at 01-Oct-2021 15:16:57.392 CEST
Begin processing the 9th record. Run 1, Event 108, LumiSection 2 on stream 0 at 01-Oct-2021 15:19:48.384 CEST
[2021-10-01 15:21:56.335535 +0200][Error  ][PostMaster        ] [cmsxrootd-site2.fnal.gov:1093 #0] Forcing error on disconnect: [ERROR] Operation interrupted.
Begin processing the 10th record. Run 1, Event 110, LumiSection 2 on stream 0 at 01-Oct-2021 15:22:42.009 CEST
01-Oct-2021 15:25:42 CEST  Closed file root://xrootd-cms.infn.it//store/relval/CMSSW_12_0_0_pre4/RelValTTbar_14TeV/GEN-SIM/120X_mcRun3_2021_realistic_v2-v1/00000/361e4fff-3935-4325-9891-62841915ccd1.root
01-Oct-2021 15:25:47 CEST  Closed file root://xrootd-cms.infn.it//store/relval/CMSSW_12_0_0_pre4/RelValMinBias_14TeV/GEN-SIM/120X_mcRun3_2021_realistic_v2-v1/00000/ace00a2c-259f-4ac1-b449-9fed3dc70e17.root

We can put a warning (or even an error) and see if ever triggers…

VinInn on Oct 1, 2021

It was seen in step2 of the ASAN builds of WF 11834.0, logs here:

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/raw/slc7_amd64_gcc10/CMSSW_12_1_ASAN_X_2021-09-29-2300/pyRelValMatrixLogs/run/11834.0_TTbar_14TeV+2021PU+TTbar_14TeV_TuneCP5_GenSimINPUT+DigiPU+RecoPU+HARVESTPU+Nano/step2_TTbar_14TeV+2021PU+TTbar_14TeV_TuneCP5_GenSimINPUT+DigiPU+RecoPU+HARVESTPU+Nano.log

To reproduce, you’d need an ASAN build, e.g. CMSSW_12_1_ASAN_X_2021-09-29-2300, and probably the exact step1 input for that relval, as this appears to be a rare condition. Since this is a rare condition, I’d recommend just putting in some protection against NaNs.

dan131riley on Sep 30, 2021