tmt: Record each failure reported within one testcase

One testcase can report multiple failures by {tmt,rstrnt,rhts}-report-result helper.

Now the junit reporter only reports one failure message for a failed testcase, no matter how many sub failures reported.

By adding this we can make the result more accurate and convenient for analysis.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 22

Most upvoted comments

Hi @jencce, I’ve made a few more code pushes to 2689 and I’d like to know does the following meet your requirements.

main.fmf content used for testing

root@fedora:~/2086# cat main.fmf 
/plan:
    provision:
        how: container
    discover:
        how: fmf
    execute:
        how: tmt
    report:
        how: junit
/check:
    test: |
        touch /tmp/pass_log ; echo "Content of pass_log" >> /tmp/pass_log; echo "Content of command exe within /XFSTESTS_START:RedHatEnterprise9:ext4:"; tmt-report-result "/XFSTESTS_START:RedHatEnterprise9:ext4:" PASS -o /tmp/pass_log
        touch /tmp/bad_log ; echo "Content of bad_log" >> /tmp/bad_log; echo "Content of command exe within /test/bad"; tmt-report-result "/test/bad" FAIL -o /tmp/bad_log
        touch /tmp/weird_log ; echo "Content of weird_log" >> /tmp/weird_log; echo "Content of command exe within /test/weird"; tmt-report-result "/test/weird" WARN -o /tmp/weird_log
        touch /tmp/main_additional_log; echo "Content of main_additional_log" >> /tmp/main_additional_log; echo "Content of command exe within /check"; tmt-report-result "" PASS -o /tmp/main_additional_log
    result: custom
root@fedora:~/2086#

Output of the tmt test execution.

root@fedora:~/2086# tmt run -vvv --debug
Using tree '/root/2086'.
/var/tmp/tmt/run-450
tmt version: 1.33.dev9+g6f828393
tmt command line: /usr/bin/tmt run -vvv --debug
Run data not found.
Enabled steps: prepare, discover, execute, finish, report and provision
Workdir '/var/tmp/tmt/run-450/plan' created.
Found 1 plan.

/plan
    Workdir '/var/tmp/tmt/run-450/plan/discover' created.
    Workdir '/var/tmp/tmt/run-450/plan/provision' created.
    Workdir '/var/tmp/tmt/run-450/plan/prepare' created.
    Workdir '/var/tmp/tmt/run-450/plan/execute' created.
    Workdir '/var/tmp/tmt/run-450/plan/report' created.
    Workdir '/var/tmp/tmt/run-450/plan/finish' created.
    discover
    workdir: /var/tmp/tmt/run-450/plan/discover
        how: fmf
        order: 50
        Workdir '/var/tmp/tmt/run-450/plan/discover/default-0' created.
        Git root not found, using '/root/2086.'
        directory: /root/2086
        Copy '/root/2086' to '/var/tmp/tmt/run-450/plan/discover/default-0/tests'.
        Check metadata tree in '/var/tmp/tmt/run-450/plan/discover/default-0/tests'.
        summary: 1 test selected
            /check
    provision
    workdir: /var/tmp/tmt/run-450/plan/provision
        queued provision.provision task #1: default-0
        
        provision.provision task #1: default-0
        how: container
        order: 50
        primary address: tmt-450-tRfEZThr
        topology address: tmt-450-tRfEZThr
        Workdir '/var/tmp/tmt/run-450/plan/provision/default-0' created.
        Check for container image 'fedora'.
        name: tmt-450-tRfEZThr
        Create network 'tmt-450-network'.
        Start container 'fedora'.
        multihost name: default-0
        arch: x86_64
        distro: Fedora Linux 38 (Container Image)
        kernel: 6.7.4-200.fc39.x86_64
        package manager: dnf
        selinux: yes
        is superuser: yes
    
        summary: 1 guest provisioned
    prepare
    workdir: /var/tmp/tmt/run-450/plan/prepare
        queued push task #1: push to default-0
        
        push task #1: push to default-0
    
        queued prepare task #1: requires on default-0
        
        prepare task #1: requires on default-0
        how: install
        summary: Install required packages
        name: requires
        order: 70
        where: default-0
        package: 1 package requested
            /usr/bin/flock
            cmd: rpm -q --whatprovides /usr/bin/flock || dnf install -y  /usr/bin/flock
            out: util-linux-core-2.38.1-4.fc38.x86_64
    
        queued pull task #1: pull from default-0
        
        pull task #1: pull from default-0
    
        summary: 1 preparation applied
    execute
    workdir: /var/tmp/tmt/run-450/plan/execute
        queued execute task #1: default-0 on default-0
        
        execute task #1: default-0 on default-0
        how: tmt
        order: 50
        exit-first: false
        Workdir '/var/tmp/tmt/run-450/plan/execute/default-0' created.
        /check [1/1]
            test: /check
        Execute '/check' as a 'shell' test.
        test wrapper: /var/tmp/tmt/run-450/plan/discover/default-0/tests/tmt-test-wrapper.sh-default-0-default-0
        test topology: /var/tmp/tmt/run-450/plan/execute/data/guest/default-0/check-1/tmt-test-topology.yaml
        test topology: /var/tmp/tmt/run-450/plan/execute/data/guest/default-0/check-1/tmt-test-topology.sh
                cmd:
                    touch /tmp/pass_log ; echo "Content of pass_log" >> /tmp/pass_log; echo "Content of command exe within /XFSTESTS_START:RedHatEnterprise9:ext4:"; tmt-report-result "/XFSTESTS_START:RedHatEnterprise9:ext4:" PASS -o /tmp/pass_log
                    touch /tmp/bad_log ; echo "Content of bad_log" >> /tmp/bad_log; echo "Content of command exe within /test/bad"; tmt-report-result "/test/bad" FAIL -o /tmp/bad_log
                    touch /tmp/weird_log ; echo "Content of weird_log" >> /tmp/weird_log; echo "Content of command exe within /test/weird"; tmt-report-result "/test/weird" WARN -o /tmp/weird_log
                    touch /tmp/main_additional_log; echo "Content of main_additional_log" >> /tmp/main_additional_log; echo "Content of command exe within /check"; tmt-report-result "" PASS -o /tmp/main_additional_log
                out: Content of command exe within /XFSTESTS_START:RedHatEnterprise9:ext4:
                out: Content of command exe within /test/bad
                out: Content of command exe within /test/weird
                out: Content of command exe within /check
        Extract results of '/check'.
                         pass /check/XFSTESTS_START:RedHatEnterprise9:ext4 (on default-0) [1/1]
                         fail /check/test/bad (on default-0) [1/1]
                         warn /check/test/weird (on default-0) [1/1]
                         pass /check/check (on default-0) [1/1]

    
        summary: 4 tests executed
    report
    workdir: /var/tmp/tmt/run-450/plan/report
        how: junit
        order: 50
        Workdir '/var/tmp/tmt/run-450/plan/report/default-0' created.
        output: /var/tmp/tmt/run-450/plan/report/default-0/junit.xml
        summary: 2 tests passed, 1 test failed and 1 warn
    finish
    workdir: /var/tmp/tmt/run-450/plan/finish
    
        container: stopped
        container: removed
        Remove network 'tmt-450-network'.
        container: network removed
    Prune '/plan' plan workdir '/var/tmp/tmt/run-450/plan'.
        Workdir '/var/tmp/tmt/run-450/plan/provision/default-0' already exists.
        Workdir '/var/tmp/tmt/run-450/plan/prepare/requires' created.
        Workdir '/var/tmp/tmt/run-450/plan/execute/default-0' already exists.
        summary: 0 tasks completed

total: 2 tests passed, 1 test failed and 1 warn
root@fedora:~/2086#

Generated junit.xml file

root@fedora:~/2086# cat /var/tmp/tmt/run-450/plan/report/default-0/junit.xml
<?xml version="1.0" ?>
<testsuites disabled="0" errors="1" failures="1" tests="4" time="0.0">
	<testsuite disabled="0" errors="1" failures="1" name="/plan" skipped="0" tests="4" time="0">
		<testcase name="/check/XFSTESTS_START:RedHatEnterprise9:ext4">
			<system-out>Content of pass_log
</system-out>
		</testcase>
		<testcase name="/check/test/bad">
			<failure type="failure" message="fail">Content of bad_log
</failure>
			<system-out>Content of bad_log
</system-out>
		</testcase>
		<testcase name="/check/test/weird">
			<error type="error" message="warn">Content of weird_log
</error>
			<system-out>Content of weird_log
</system-out>
		</testcase>
		<testcase name="/check/check">
			<system-out>Content of main_additional_log
</system-out>
		</testcase>
	</testsuite>
</testsuites>
root@fedora:~/2086#

Able to reproduce with a simple config:

/plan:
    provision:
        how: container
    discover:
        how: fmf
    execute:
        how: tmt
/test:
    test: |
        tmt-report-result /test/good PASS
        tmt-report-result /test/bad FAIL
        tmt-report-result /test/weird WARN

This results into two tests reported by the display plugin:

report
    how: display
        fail /test
        pass /test
    summary: 1 test passed and 1 test failed

And just one reported by the junit plugin:

<?xml version="1.0" ?>
<testsuites disabled="0" errors="0" failures="1" tests="2" time="0.0">
    <testsuite disabled="0" errors="0" failures="1" name="/plan" skipped="0" tests="2" time="0">
        <testcase name="/test">
            <failure type="failure" message="fail"/>
        </testcase>
        <testcase name="/test"/>
    </testsuite>
</testsuites>

@happz, what’s interesting that we get the same serialnumber for both:

  - name: /test
    result: fail
    note:
    log:
      - data/guest/default-0/test-1/output.txt
    serialnumber: 1
    fmf_id:
        name: /test
    ids:
        id:
        extra-nitrate:
        extra-task:
    guest:
        name: default-0
        role:
        __class__:
            module: tmt.result
            name: ResultGuestData
    check: []
    starttime: '2023-09-14T13:58:29.629448+00:00'
    endtime: '2023-09-14T13:58:29.845018+00:00'
    duration: 00:00:00
    __class__:
        module: tmt.result
        name: Result
  - name: /test
    result: pass
    note:
    log:
      - data/guest/default-0/test-1/output.txt
    serialnumber: 1
    fmf_id:
        name: /test
    ids:
        id:
        extra-nitrate:
        extra-task:
    guest:
        name: default-0
        role:
        __class__:
            module: tmt.result
            name: ResultGuestData
    check: []
    starttime: '2023-09-14T13:58:29.629448+00:00'
    endtime: '2023-09-14T13:58:29.845018+00:00'
    duration: 00:00:00
    __class__:
        module: tmt.result
        name: Result

@pfdaly, fyi.