cmssw: Multiple RelVal failures due to file being registered in DAS but not present
RelVals 20834.x, 21034.x are broken since 2022-02-26-0000 due to inaccessible file:
Failed to open file at URL root://eoscms.cern.ch:1094//eos/cms/store/user/cmsbuild/store/relval/CMSSW_12_3_0_pre5/RelValTTbar_14TeV/GEN-SIM/123X_mcRun4_realistic_v4_2026D88noPU-v1/10000/49e54274-4298-4576-b47b-866e2247eab5.root
This file is a part of /RelValTTbar_14TeV/CMSSW_12_3_0_pre5-123X_mcRun4_realistic_v4_2026D88noPU-v1/GEN-SIM dataset, and registered in DAS as accessible on T2_CH_CERN:
$ dasgoclient --limit 0 --query 'file dataset=/RelValTTbar_14TeV/CMSSW_12_3_0_pre5-123X_mcRun4_realistic_v4_2026D88noPU-v1/GEN-SIM site=T2_CH_CERN'
/store/relval/CMSSW_12_3_0_pre5/RelValTTbar_14TeV/GEN-SIM/123X_mcRun4_realistic_v4_2026D88noPU-v1/10000/49e54274-4298-4576-b47b-866e2247eab5.root
but it is not actually present on EOS:
$ ls /eos/cms/store/user/cmsbuild/store/relval/CMSSW_12_3_0_pre5/RelValTTbar_14TeV/GEN-SIM/123X_mcRun4_realistic_v4_2026D88noPU-v1/10000/49e54274-4298-4576-b47b-866e2247eab5.root
ls: cannot access /eos/cms/store/user/cmsbuild/store/relval/CMSSW_12_3_0_pre5/RelValTTbar_14TeV/GEN-SIM/123X_mcRun4_realistic_v4_2026D88noPU-v1/10000/49e54274-4298-4576-b47b-866e2247eab5.root: No such file or directory
Previously, no files from that dataset were registered as present on T2_CH_CERN, and DAS was returning a full list of files, so RelVal was using a different file (2c4c1ca9-73fe-4648-982f-e773c9ec91e9.root), which is cached on EOS.
About this issue
- Original URL
- State: open
- Created a year ago
- Comments: 18 (18 by maintainers)
@makortel , bot keeps the old results if das returns empty list or error for a query.
I see now in the DAS web GUI that the
site dataset=/RelValTTbar_14TeV/CMSSW_12_3_0_pre5-123X_mcRun4_realistic_v4_2026D88noPU-v1/GEN-SIMshows T2_IN_TIFRfile dataset=/RelValTTbar_14TeV/CMSSW_12_3_0_pre5-123X_mcRun4_realistic_v4_2026D88noPU-v1/GEN-SIM site=T2_CH_CERNindeed shows/store/relval/CMSSW_12_3_0_pre5/RelValTTbar_14TeV/GEN-SIM/123X_mcRun4_realistic_v4_2026D88noPU-v1/10000/49e54274-4298-4576-b47b-866e2247eab5.rootsite file=/store/relval/CMSSW_12_3_0_pre5/RelValTTbar_14TeV/GEN-SIM/123X_mcRun4_realistic_v4_2026D88noPU-v1/10000/49e54274-4298-4576-b47b-866e2247eab5.rootshows T2_IN_TIFRClearly there is some inconsistency between
file dataset=... site=T2_CH_CERNandsite file=...on the same file. As far as I can tell, DAS is picking all this site information from Rucio. Let me add @ericvaandering in case he’d have an idea where to look further (or who could help further)