cmssw: Ecal and Hcal Local GPU Reco crashes on missing detector
A bug was exposed last MWGR in that both HCAL and ECAL local reconstruction on a GPU do not have protections when the respective detector is out.
This is explicitly in the HBHERecHitProducerGPU
cmsRun: /build/cmsbld/jenkins/workspace/auto-builds/CMSSW_11_3_1-slc7_amd64_gcc900/build/CMSSW_11_3_1-build/tmp/BUILDROOT/ebcff69f73237d0d563ea1e7c33f2d83/opt/cmssw/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_1/src/RecoLocalCalo/HcalRecProducers/src/MahiGPU.cu:1064: void hcal::reconstruction::entryPoint(const hcal::reconstruction::InputDataGPU&, hcal::reconstruction::OutputDataGPU&, const hcal::reconstruction::ConditionsProducts&, hcal::reconstruction::ScratchDataGPU&, const hcal::reconstruction::ConfigParameters&, cudaStream_t): Assertion `startingSample == 0 || startingSample == 2’ failed.
and a similar crash was observed in ECAL
To reproduce simply run over any run with the appropriate detector missing.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 53 (53 by maintainers)
While keeping all paths enabled, the job always crashes. So there seems to be some destructive interference between the ECAL and Pixel reconstruction 😕
Dear all,
As initially reported by us during yesterday’s Run Organization meeting and during today’s Daily Run Meeting, we are seeing HCAL and Pixel crashes when running over data that does not include HCAL, ECAL and Pixel in the run. It appears that unfortunately the additional protections implemented in CMSSW_11_3_3 were insufficient. The details of the crashes are included in the following e-log, and it appears that the nature of these are different than the ones that were initially reported in this ticket above. We also saw that Pixel has recreated the issues and are investigating in https://github.com/cms-sw/cmssw/issues/34659.
We are continuing our tests to isolate and cross-check the ECAL protections, however, we wanted to make you aware of this issue.
Best regards, Mateusz on behalf of TSG FOG