rucio: Incorrect RSE shown in FTS Opensearch

Description

Katy from RAL asked the FTS team to investigate erroneous RSE values within the central WLCG monitoring system. After investigating the problem, the FTS team discovered that RUCIO is the source of the incorrect information. I suspect that at an early point in time the information was correct and was probably made invalid when the concept of multi-source file transfers was added.

RUCIO passes incorrect RSE information into FTS which gets passed through to the following WLCG monitoring field:

  • Index pattern monit_prod_fts_enr_complete* Field: data.file_metadata

The following elastic search link gives an example of the problem:

The above monitoring page shows different source storage endpoints (davs://t2-xrdcms.lnl.infn.it and davs://cms-t2-se01.sdfarm.kr) but the RSE (RUCIO storage endpoint) is always the same (T2_KR_KISTI). The RSE is therefore incorrect.

FTS allows RUCIO to pass a single JSON “metadata” document for each file transfer it submits. An FTS file transfer can have many source files. Here is the problem. Each source file is located at a different RSE, however RUCIO has only written a single RSE value into the single file metadata entry.

FTS treats the JSON “metadata” document of each file transfer as opaque. The FTS software has no concept of RSE (RUCIO Storage Endpoint). The FTS team and service do not and should not police the contents of the JSON “metadata” document of each file transfer.

I can guess show this problem came about. Most probably we all started with single source file transfers and then one day the multi-source transfers were added and boom.

I do not know where the problem lies. Is it in the RUCIO code base? Is it in the experiment specific deployment of RUCIO? Is it in the enrichment system of the CERN WLCG monitoring system? Is it in the experiment’s CRIC system which contains the mapping from storage endpoint URL to RSE?

The bottom-line is that RUCIO should stop injecting invalid RSE information into FTS and an alternative mapping solution from storage endpoint to RSE should implemented, most probably within the enrichment system of CERN WLCG monitoring system.

Please feel free to move/copy this ticket to another system if this is not the appropriate place.

Steps to reproduce

  1. Look at the following elastic search link:

  2. Not the above monitoring page shows different source storage endpoints (davs://t2-xrdcms.lnl.infn.it and davs://cms-t2-se01.sdfarm.kr) but the RSE (RUCIO storage endpoint) is always the same (T2_KR_KISTI). The RSE is therefore incorrect.

Rucio Version

No response

Additional Information

No response

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 16 (9 by maintainers)

Most upvoted comments

Hello,

We also had the idea of a mapping in place. FTS doesn’t put any limits on the file_metadata field. All of this will be sent untouched to MONIT, where it’s the MONIT system who will have to look into the mapping and extract the correct value.

Other systems who rely on the FTS monitoring messages directly would also need to be made aware of the change. But I guess not many non-Rucio systems really parsed the file_metadata.src_rse / file_metadata.dst_rse fields.

At this moment, it’s a question for MONIT on what they prefer:

  1. Obtain the src_rse / dst_rse via CRIC OR
  2. Obtain it from the Rucio file_metadata multi-replica mapping

Cheers, Mihai

Hi guys, work in progress. The change in Rucio is trivial, I’m more worried about the dashboard implications, specially for CMS, I’m following up with them. I’ll then come back with a date where sending this feature could be stopped.

Cheers, Hugo

Hello,

I don’t think this can be fixed in Rucio.

The FTS submission for multi-replica would only allow you to submit file_metadata once, despite creating multiple transfers from your submission. Consider the following multi-replica example:

{
  "files": [
    {
      "sources": [
        "mock://mock_src_rse_one/file_0001",
        "mock://mock_src_rse_two/file_0002"
      ],
      "destinations": [
        "mock://mock_dst_rse/file_0003"
      ],
      "selection_strategy": "orderly",
      "metadata": "Multi-replica per-file metadata set only once!"
    }
  ]
}

Submitting and then querying FTS:

$ fts-rest-transfer-submit -s https://fts3-pilot.cern.ch:8446/ --file fts-submission-multireplica.json
$ davix-get --cert /tmp/x509up_u($id -u) "https://fts3-pilot.cern.ch:8446/jobs/<job-id>/files" | jq '.[] | .source_surl + " --> " + .dest_surl + "  --  " + .file_metadata.label'

"mock://mock_src_rse_one/file_0001 --> mock://mock_dst_rse/file_0003  --  Multi-replica per-file metadata set only once!"
"mock://mock_src_rse_two/file_0002 --> mock://mock_dst_rse/file_0003  --  Multi-replica per-file metadata set only once!"

Simply put, relying on Rucio-provided file_metadata for multi-replica transfers is not valid. At this point, maybe it’s better to avoid file_metadata in multi-replica submissions.

My thoughts on the matter.

Cheers, Mihai