cmssw: opportunity to cleanup LogErrors and LogWarnings in 2023 data promptReco processing

This may be just for taxonomy, but could be used as for investigation of corresponding modules

I was looking at message logger details in 2023 data for run 366498 in Muon0 and ZeroBias LogErrorMonitor skim files /store/data/Run2023B/Muon0/USER/LogErrorMonitor-PromptReco-v1/000/366/498/00000/0e204272-49fb-42de-a373-65747ff240b4.root and /store/data/Run2023B/ZeroBias/USER/LogErrorMonitor-PromptReco-v1/000/366/498/00000/3f9f69f5-7933-47bb-a5a4-d1f9ffbf7331.root have about 25K events (the run is fairly short)

Warnings

235 EcalRecHitWorkerRecover
358 TrackNaN
616 BeamSpotFromDB
1126 Configuration
1136 L1TStage2uGTTiming
1224 BadMuon
3681 BasicTrajectoryState
24298 TopMonitor
26727 L1TTauOffline
407208 L1T

Errors

1 InvalidGhostTrackState
2 BadRange
4 InvalidState
5 GsfMultiStateUpdator
7 PFTrackTransformer
16 TrackerValidationVariables
225 FailedPropagation
542 TrajectoryNotPosDef
3251 CTPPSPixelDataFormatter
10392 HOCalib

plain text printout in fwlite with (later post-processed to get totals):

f = open("zb.ews.pyscan.txt", "a")
en = ROOT.TChain("Events")
n = en.Add("root://cms-xrd-global.cern.ch/LFN")
for ev in range(en.GetEntries()):
  n = en.GetEntry(ev)
  mv = en.edmErrorSummaryEntrys_logErrorHarvester__RECO.product()
  e = en.EventAuxiliary
  for mm in mv: print(e.run(), e.luminosityBlock(), e.event(), mm.severity.getName().data(), mm.count, mm.module, mm.category, file=f)
f.close()

Full outputs with run lumi event sevLev count module category are available for Muon0 and ZeroBias These exclude MemoryCheck

an example log from one job in ZeroBias can be found here

an example of Muon0 job log is here

About this issue

Original URL
State: open
Created a year ago
Comments: 50 (50 by maintainers)

Most upvoted comments

@francescobrivio I can comment on this maybe.

The modules like simEmtfDigis are run in data for data/emulator comparisons in DQM. These are the ones labeled val...Digis as in valEmtfStage2Digis.
The problem we see here is due to EMTF unpacker not cleanining quality 0 LCTs. So these make their way into the emulator which prints out the warning as it removes them from the LCT collection. Technically they don’t go into the core EMTF algorithm, so it’s not a major problem.

I’ll submit a PR soon that adds cleaning in the unpacker, so the warnings will go away. This happens when EMTF recieves an HMT bit from CSCs, but no LCT in the same payload. So, it’s not corrupt data, but something we’re not handling properly in the unpacker.

eyigitba on May 16, 2023

I’m no expert on that, but I could do something similar in the past with some plugins…

I am not sure you have control on that at the level of Tier-0 configuration.

RecoTLR.customisePrompt and customiseExpress can be used https://github.com/cms-sw/cmssw/blob/437b0e222bd2b10003d0ab4bd51604eb310b103e/Configuration/DataProcessing/python/RecoTLR.py#L155

But wouldn’t these be needed in reprocessing as well? In that case I’d configure limited issue of errors by default and enable elsewhere for the tests.

Similar to the tracker, it may be more practical to produce error objects and rely on analysis of theese data products instead of text log files

slava77 on Jul 12, 2023

I now used python to analyze this in fwlite the updated toplist is

Warnings

235 EcalRecHitWorkerRecover
358 TrackNaN
616 BeamSpotFromDB
1126 Configuration
1136 L1TStage2uGTTiming
1224 BadMuon
3681 BasicTrajectoryState
24298 TopMonitor
26727 L1TTauOffline
407208 L1T

Errors

1 InvalidGhostTrackState
2 BadRange
4 InvalidState
5 GsfMultiStateUpdator
7 PFTrackTransformer
16 TrackerValidationVariables
225 FailedPropagation
542 TrajectoryNotPosDef
3251 CTPPSPixelDataFormatter
10392 HOCalib

f = open("zb.ews.pyscan.txt", "a")
en = ROOT.TChain("Events")
n = en.Add("root://cms-xrd-global.cern.ch/LFN")
for ev in range(en.GetEntries()):
  n = en.GetEntry(ev)
  mv = en.edmErrorSummaryEntrys_logErrorHarvester__RECO.product()
  e = en.EventAuxiliary
  for mm in mv: print(e.run(), e.luminosityBlock(), e.event(), mm.severity.getName().data(), mm.count, mm.module, mm.category, file=f)
f.close

slava77 on May 2, 2023