cmssw: Covariance Version Conflict between HLT Module and miniAOD Module

By trying to use new HLT paths made in my user area to generate small MiniAOD file for DQM testing, I found what I believed was an issue between an HLT module and later MiniAOD modules. I sent a plea and description of my problem to Michal Bluj (observing this issue) and he describes the situation as the following:

• DeepBoostedJetTagInfoProducer [1] builds internally pat::PackedCandidates from reco::PFCandidates when run with “use_hlt_features” on (see [1] lines 797-876). The PackedCandidates are built with a hardcoded version of track covariance packing (see l. 829 and calls of setTrackProperties(track, qual, cov_ver), e.g. in ll.845,847). This version is hardcoded to be equal to 0;
• At offline, within the miniAOD sequence, there are three producers of pat::PackedCandidates (pluginName/producerName): PATPackedCandidateProducer/packedPFCandidates, PATLostTracks/lostTracks and PATTracksToPackedCandidates/hiPixelTracks. All of them have configurable version of track covariance, which is set to 1 for Run-3;
• When the two settings are adjusted, i.e. either value hardcoded in DeepBoostedJetTagInfoProducer is set to 1 or value used by miniAOD modules set to 0 [2], the configuration works fine (in the sense that it does not throw an exception - tested with full file of ~1.5k of events).
• What I do not understand well is the fact how and why building (transient) pat::PackedCandidates internally in DeepBoostedJetTagInfoProducer interferes with production of (persistent) pat::PackedCandidates by miniAOD modules. I checked that it also happens with a workflow w/o DeepBoostedJetTagInfoProducer with enabled “use_hlt_features” when covariance versions are different in different miniAOD modules (1 for packedPFCandidates and 0 for lostTracks). This can suggest that covariance packing schema is a kind of global, which in my opinion is a bug.
• [1] RecoBTag/FeatureTools/plugins/DeepBoostedJetTagInfoProducer.cc

Here is a set of commands to reproduce the issue.

Clean cmssw release.

cmsrel CMSSW_14_0_0_pre2
cd CMSSW_14_0_0_pre2/src
cmsenv
git cms-addpkg HLTrigger/Configurations

Download user cff and place in configurations

hltGetConfiguration /users/ballmond/PNet/BaseV4GRun_w_PNet/V13 \
--globaltag auto:phase1_2024_realistic --mc --unprescale --max-events 10 \
--input /store/mc/Run3Winter24Digi/VBFHToTauTau_M125_TuneCP5_13p6TeV_powheg-pythia8/GEN-SIM-RAW/133X_mcRun3_2024_realistic_v8-v2/2560000/000af33b-2a68-4c78-b544-e1c0e3f4f576.root\
--cff > HLT_PNet_cff.py

cp HLT_PNet_cff.py HLTrigger/Configuration/python

Run the cmsDriver command to take a RAW file to MiniAOD level, using the user cff (It’s possible you will need to download the file locally first to ensure that it can be opened promptly. It is from a recent VBFHTauTau dataset to ensure the PNet paths fire)

cmsDriver.py step2 -s RAW2DIGI,L1Reco,HLT:PNet,RECO,PAT \
--eventcontent MINIAOD \
--conditions auto:phase1_2024_realistic \
--era Run3 --geometry DB:Extended \
--process reMINI --filein /store/mc/Run3Winter24Digi/VBFHToTauTau_M125_TuneCP5_13p6TeV_powheg-pythia8/GEN-SIM-RAW/133X_mcRun3_2024_realistic_v8-v2/2560000/000af33b-2a68-4c78-b544-e1c0e3f4f576.root \
-n 5 --mc

The error only appears if a Pnet trigger fires, so be wary if you decide to use a different file for some reason. Here is the error:

Begin processing the 2nd record. Run 1, Event 844217, LumiSection 1284 on stream 0 at 03-Feb-2024 18:59:15.805 CET
----- Begin Fatal Exception 03-Feb-2024 18:59:24 CET-----------------------
An exception of category 'UnimplementedFeature' occurred while
   [0] Processing  Event run: 1 lumi: 1284 event: 844217 stream: 0
   [1] Running path 'HLT_DoublePNetTauhPFJet30_Medium_L2NN_eta2p3_v1'
   [2] Calling method for module DeepBoostedJetTagInfoProducer/'hltParticleNetJetTagInfos'
Exception Message:
Attempting to load multiple covariance version in same process. This is not supported.
----- End Fatal Exception -------------------------------------------------

As I understood, this is some bug with an apparently global variable that shouldn’t be global. For now, a way to circumvent the error is to add the following lines to the cmsDriver config file (step2_RAW2DIGI_L1Reco_HLT_RECO_PAT.py)

cov_ver = 0
process.packedPFCandidates.covarianceVersion = cov_ver
process.lostTracks.covarianceVersion = cov_ver
process.hiPixelTracks.covarianceVersion = cov_ver

But ideally we would like to resolve this issue now so it doesn’t appear later. Thanks for looking into this.