cmssw: failure of HLT-Validation tests in reading `edm::HepMCProduct`

The HLT-Validation tests failed in CMSSW_12_4_X_2022-03-28-1100, with the error in [1], e.g. https://cmssdt.cern.ch/SDT/jenkins-artifacts/HLT-Validation/CMSSW_12_4_X_2022-03-28-1100/slc7_amd64_gcc10/RelVal_DigiL1Raw_GRun_MC.log

The same tests passed in the previous 12_4_X IB, and they are passing in earlier releases (e.g. 12_3_X).
Locally, the error can be reproduced in CMSSW_12_4_X_2022-03-28-1100, and the same job completes correctly for CMSSW_12_4_X_2022-03-27-2300 (as per IB tests).
I don’t see any change in CMSSW itself that can explain it.
The only update that looks related to the error message is one in cmsdist having to do with pythia8 and HepMC (but I don’t really know that this is related to this issue): https://github.com/cms-sw/cmsdist/pull/7724 .

It’s worth noting that the input EDM file used in the failing HLT test is relatively old (GEN-SIM from 11_2_0_pre8).

A quick test suggests that using a more recent GEN-SIM file (from 12_3_0_pre6) fixes the problem.

I’m opening this issue for documentation, and to get insight from others on what the issue actually is here.

[1]

Begin processing the 1st record. Run 1, Event 55, LumiSection 2 on stream 0 at 28-Mar-2022 23:32:50.956 CEST                                                                                               
----- Begin Fatal Exception 28-Mar-2022 23:32:57 CEST-----------------------                                                                                                                               
An exception of category 'FileReadError' occurred while                                                                                                                                                    
   [0] Processing  Event run: 1 lumi: 2 event: 55 stream: 0                                                                                                                                                
   [1] Running path 'RAWSIMoutput_step'                                                                                                                                                                    
   [2] Prefetching for module PoolOutputModule/'RAWSIMoutput'                                                                                                                                              
   [3] While reading from source edm::HepMCProduct generatorSmeared '' SIM                                                                                                                                 
   [4] Reading branch edmHepMCProduct_generatorSmeared__SIM.                                                                                                                                               
Exception Message:                                                                                                                                                                                         
basic_string::_M_construct null not valid                                                                                                                                                                  
----- End Fatal Exception -------------------------------------------------

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 32 (32 by maintainers)

Most upvoted comments

@pcanal , https://github.com/root-project/root/pull/10280 looks good, PR tests passed https://github.com/cms-sw/root/pull/167#issuecomment-1084043391 and also the additonal HLT tests mentioned https://github.com/cms-sw/cmssw/issues/37392#issuecomment-1081454800 worked too

smuzaffar on Mar 31, 2022

Could you give a recipe to reproduce?

Here it goes:

cmsrel CMSSW_12_4_X_2022-03-28-1100
cd CMSSW_12_4_X_2022-03-28-1100/src
cmsenv

cmsDriver.py RelValNEW --step=DIGI:pdigi,L1,DIGI2RAW --conditions=auto:run3_mc_GRun \
--custom_conditions= --fileout=RelVal_DigiL1Raw_GRun_MC.root --number=100 --mc --datatier GEN-SIM-DIGI-RAW --eventcontent=RAWSIM \
--customise=HLTrigger/Configuration/CustomConfigs.L1T --era=Run3 --customise= --scenario=pp --python_filename=RelValNEW_DigiL1Raw_GRun_MC.py \
--filein=root://eoscms.cern.ch//eos/cms/store/group/dpg_trigger/comm_trigger/TriggerStudiesGroup/STORM/GEN-SIM/CMSSW_12/29297788-21c3-4bf2-afe0-5382f08fbcd5.root \
&> RelValNEW.log

cmsDriver.py RelValOLD --step=DIGI:pdigi,L1,DIGI2RAW --conditions=auto:run3_mc_GRun \
--custom_conditions= --fileout=RelVal_DigiL1Raw_GRun_MC.root --number=100 --mc --datatier GEN-SIM-DIGI-RAW --eventcontent=RAWSIM \
--customise=HLTrigger/Configuration/CustomConfigs.L1T --era=Run3 --customise= --scenario=pp --python_filename=RelValOLD_DigiL1Raw_GRun_MC.py \
--filein=root://eoscms.cern.ch//eos/cms/store/group/dpg_trigger/comm_trigger/TriggerStudiesGroup/STORM/GEN-SIM/CMSSW_11/3ee9ba1e-0ef8-4242-8343-cff886c9f7b3.root \
&> RelValOLD.log

RelValOLD will fail, while RelValNEW will not. The only difference between the two is the input file (and HLT does not actually run here, see the steps).

(this test is not part of addOnTests, right?)

Indeed, it is not. It is part of additional tests designed by HLT which run as “HLT-Validation” tests in IB [*]. Afaiu, the HLT-Validation tests run in “DEFAULT” IBs, but not in other IB flavours (e.g. DEVEL IBs). https://github.com/cms-sw/cmsdist/pull/7724 looks like a suspect because the issue showed up as soon as that cmsdist update made it into default IBs; it was already in DEVEL IBs, but it seems we don’t run HLT-Validation tests for those (I’m not saying we should).

[*] For reference, those tests can be run locally with

cd HLTrigger/Configuration/test
./runAll.csh IB

missirol on Mar 29, 2022