azure-pipelines-tasks: Publish Test Result: JUnit result with multiple test suites not displayed properly

@dnlopes like you quoted:

We’ll take it up in the coming months.

That was 3 years ago. So, in this case, people are just starting to question Microsoft’s backlog grooming, I guess. 🤷‍♂️

Real-world public example: https://www.npmjs.com/package/prettier-plugin-elm

Contents of junit.xml:

<testsuites name="jest tests" tests="18" failures="0" time="18.103">
  <testsuite name="/dist/cache.test.js" errors="0" failures="0" skipped="0" timestamp="2018-12-23T11:16:25" time="14.348" tests="1">
    <testcase classname=" correctly deals with cache" name=" correctly deals with cache" time="13.494">
    </testcase>
  </testsuite>
  <testsuite name="/dist/index.test.js" errors="0" failures="0" skipped="0" timestamp="2018-12-23T11:16:25" time="16.507" tests="17">
    <testcase classname=" formats fixture broken-code.elm" name=" formats fixture broken-code.elm" time="0.902">
    </testcase>
    <testcase classname=" formats fixture broken-code.md" name=" formats fixture broken-code.md" time="0.547">
    </testcase>
    ...
    <testcase classname=" formats fixture only-import.md" name=" formats fixture only-import.md" time="0.417">
    </testcase>
  </testsuite>
</testsuites>

Task:

  - task: PublishTestResults@2
    inputs:
      testResultsFormat: JUnit
      testResultsFiles: $(System.DefaultWorkingDirectory)/junit.xml
    displayName: "Publish test results"
    condition: and(variables.REPORT_TEST_RESULTS_AND_COVERAGE, succeededOrFailed())

Result (test suites are joined, total duration is broken):

HALP 🙏

Hi , We have considered this feedback and added this to our backlog. We’ll take it up in the coming months. I’ll post the updates once its done. Thanks.

@divais I still have the same issue. Test suite names are just ignored, only test cases are shown when publishing the test result. Is there any ETA when this will be available?

Any update on this?

We have just moved from an on-premise Jenkins to Azure DevOps just to discover that in this shiny new cloud world we cannot even review test results anymore due to said flattened list of undistinguishable test cases.

This is a crucial feature!

Any news, when this is planned?

Is this going to be supported in the future, or is it on the roadmap?

Some promised to share a workaround, but I couldn’t find anyone, so I wrote one myself. I’m unsure if I will have to wait until the 6th anniversary of this bug to publish it… (just kidding). Here you are!

It depends on xmlstarlet - a command line XML toolkit

Use your own job.xml - fetching one from the internet as I do here is just an example.

pipeline.yaml:

trigger:
  - '*'

jobs:
- job: fetch_result
  displayName: 'Fetch and Publish Job Result'
  pool: 'Build-Linux'
  workspace:
    clean: outputs

  steps:
  - checkout: self

  - bash: |
      wget -O job.xml https://validation.linaro.org/api/v0.2/jobs/4059960/junit
    workingDirectory: $(Build.BinariesDirectory)
    displayName: Get Job Result from LAVA Instance

  - task: Bash@3
    displayName: 'Split jUnit XML to separate files because of nasty ADO bug'
    inputs:
      filePath: $(Build.SourcesDirectory)/split_junit_testsuites.sh
      arguments: $(Build.BinariesDirectory)/job.xml
      workingDirectory: $(Build.BinariesDirectory)
      noProfile: false

  - task: PublishTestResults@2
    displayName: 'Publish Test Results'
    inputs:
      testResultsFormat: JUnit
      testResultsFiles: '**/test_*.xml'
      searchFolder: $(Build.BinariesDirectory)

split_junit_testsuites.sh:

#!/usr/bin/env bash
#
# This script will split a jUnit XML into files containing only one testsuite.
# This is necessary to work around a many years old bug in Azure DevOps
# See https://github.com/microsoft/azure-pipelines-tasks/issues/7659 for details
#
# License: GPLv2
# Author: stefan-muc (GitHub) in 2024-04

command -v xmlstarlet --version >/dev/null 2>&1 || { echo >&2 "This script requires xmlstarlet in PATH, but couldn't find it. Aborting."; exit 1; }

if [[ $# -ne 1 ]]; then
    echo "Please provide path to jUnit XML as first parameter."
    exit 2
fi

if [[ ! -f "$1" ]]; then
    echo "The file you provided doesn't exist. Aborting."
    exit 3
fi

testsuites=$(xmlstarlet sel -t -v "count(/testsuites/testsuite)" "$1")

if [[ $testsuites -gt 0 ]]; then
    echo "Found $testsuites testsuites"
else
    echo "Couldn't find any testsuite. Exiting."
    exit 3
fi

trap cleanup EXIT
tempdir=$(mktemp -d --tmpdir "ADO-junit-split.XXXXXXXXXX")

if [[ ! -d "$tempdir" ]]; then
    echo "Creating '$tempdir' failed."
    exit 4
fi

tempfile=$tempdir/tempfile.xml
for testsuite in $(seq 1 $testsuites); do
    name=$(xmlstarlet sel -t -m "/testsuites/testsuite[$testsuite]" -v @name "$1")
    echo "Exporting testsuite $testsuite '$name'"

    cp "$1" "$tempfile"
    for i in $(seq $testsuites -1 1); do
        if [[ $i -ne $testsuite ]]; then
            xmlstarlet ed --delete "/testsuites/testsuite[$i]" "$tempfile" > "$tempfile.new"
            mv "$tempfile.new" "$tempfile"
        fi
    done

    mv "$tempfile" "./test_$name.xml"
done

rm "$1" # Delete original file to not export this in a later ADO step

function cleanup()
{
    echo "Cleaning up $tempdir"
    rm -r $tempdir
}

Use at your own risk.

Any updates? Test results UI looks really inconvenient

Hi , We have considered this feedback and added this to our backlog. We’ll take it up in the coming months. I’ll post the updates once its done. Thanks.

Yet another person asking for an update on this issue. As said above, this was added to the backlog.

Any update on this? We are using Azure DevOps Services and all test cases gather in a single test run is not usable when we are reviewing the report.

This is a critical one for us as well! So please give us some updates!

kicking this issue down the road like a can

Any updates? How long is your roadmap backlog, since we entered the 4th year of this issue.

Microsoft isn’t interested in building Azure features that are complete. They are only interested in building features enough of the way so that they can claim that Azure supports said feature. So this, like so many other things, will never get fixed.

But it clearly says in your official documentation that you support test suites (e.g., under Test Results > Outcome):

Failed: if exists /Testsuites/testsuite/testcase/failure

Is this repo even being maintained?

it’s marketing for peopleless computing 😄

it’s on the backlog since 2019. please be patient 😁

Our team would love if this was worked on or we could find a workaround as well. An update would be great

four years… duh.

Hi,

I’ve found a workaround for this “problem”. If I put a single inside a -tag, it’ll work (when grouped by test run). But I have to produce a single file for each testsuite I want to create.

It would help avoiding this workaround (and multiple result files), if I were able to put multiple inside a -tag.

Could you explain where did you put the -tag, please?

@shailesh-sk could you please take a look?