cmssw: large number of tracking-related warning/error messages in HLT jobs

CMSSW jobs with recent HLT menus [0] show a very large number of LogWarnings and LogErrors coming mainly from the HLT e/gamma [1] and muon [2] reconstruction. The warnings/errors seem to be mainly related to tracking. A recipe to reproduce some of these warnings/errors is in [3].

In CMSHLT-2449, it was noted that this creates significant size increase of HLT log files, and the short-term ‘solution’ was to silence these warnings (not the errors) via the MessageLogger module of the HLT menu.

This issue is about understanding from RECO and POG experts what the best solution is.

• Are these messages a sign of issues that should be addressed with some urgency?

• If not, should they be downgraded to “info” or “debug” messages, rather than being silenced by HLT?

FYI: @fwyzard @Martin-Grunewald @silviodonato @swagata87 @JanFSchulte

---

[0] I currently don’t know when this started, and if it is different with earlier releases or menus.

[1]

%MSG-w BasicTrajectoryState:  GsfTrackProducer:hltEgammaGsfTracksForBParking  29-Aug-2022 14:51:04 CEST Run: 355558 Event: 215872193
curv error not pos-def
[      -663558      29.038      483458   1.241e+07     -314550
        29.038   -0.245501    -20.6987      -535.1     6.49207
        483458    -20.6987     -352241-9.04176e+06      229186
     1.241e+07      -535.1-9.04176e+06-2.32095e+08   5.883e+06
       -314550     6.49207      229186   5.883e+06     -149378 ]
pos/mom/mf  (30.1516,49.9366,-35.9222)   (2.33339,3.34254,-2.93736)   (-0.00243308,-0.00402963,3.81145)
%MSG
%MSG-e PosteriorWeightsCalculator:   GsfTrackProducer:hltEgammaGsfTracksForBParking 29-Aug-2022 14:51:04 CEST  Run: 355558 Event: 215872193
PosteriorWeightsCalculator: sumWeight < DBL_MIN
%MSG
%MSG-e GsfMultiStateUpdator:  GsfTrackProducer:hltEgammaGsfTracksForBParking  29-Aug-2022 14:51:04 CEST Run: 355558 Event: 215872193
 no weights could be retreived. invalid updated state !.
%MSG
%MSG-e InvalidState:  GsfTrackProducer:hltEgammaGsfTracksForBParking  29-Aug-2022 14:51:04 CEST Run: 355558 Event: 215872193
inside hit

[2]

%MSG-w RecoMuon:  L3MuonProducer:hltL3NoFiltersNoVtxMuonsOIHit  29-Aug-2022 14:50:53 CEST Run: 355558 Event: 216016013                                                                             
     Failed to load tracker track trajectory                                                                                                                                                       

[3]

#!/bin/bash

cmsrel CMSSW_12_4_8
cd CMSSW_12_4_8/src
cmsenv

hltGetConfiguration /online/collisions/2022/2e34/v1.3/HLT/V1 \
  --globaltag 124X_dataRun3_HLT_v4 \
  --data \
  --unprescale \
  --output none \
  --max-events 200 \
  --eras Run3 --l1-emulator uGT --l1 L1Menu_Collisions2022_v1_3_0-d1_xml \
  --input root://eoscms.cern.ch//eos/cms/store/group/dpg_trigger/comm_trigger/TriggerStudiesGroup/STORM/RAW/Run2022B_HLTPhysics0_run355558/cd851cf4-0fca-4d76-b80e-1d33e1371929.root \
  > hlt.py

cmsRun hlt.py &> hlt.log