testng: Incorrect listener's onStart(ISuite suite) execution order in case of inner suites

TestNG Version

Note: only the latest version is supported

6.11

Expected behavior

The listener is called for each suite, if the parent suite contains child suites then the parent suites should run first before running the child suite.

Actual behavior

The child suites are first run before running the parent suite.

Is the issue reproducible on runner?

  • Shell
  • Maven
  • Gradle
  • Ant
  • Eclipse
  • IntelliJ
  • NetBeans

Test case sample

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="ParentSuite" parallel="classes" thread-count="10">

    <suite-files>
        <suite-file path="child_suite.xml"/>
    </suite-files>

    <test verbose="1" name="SomeTest">
        <classes>
            <class name="com.example.SomeTest"/>
        </classes>
    </test>

</suite>

About this issue

  • Original URL
  • State: open
  • Created 7 years ago
  • Reactions: 3
  • Comments: 25 (16 by maintainers)

Most upvoted comments

@krmahadevan I am not the bug reporter but I would expect the following order of execution:

  1. ISuiteListener#onStart - all_suites
  2. ISuiteListener#onStart - suite1
  3. ISuiteListener#onFinish - suite1
  4. ISuiteListener#onStart - suite2
  5. ISuiteListener#onFinish - suite2
  6. ISuiteListener#onFinish - all_suites

So yes, all_suites should finish last but it should start first, because it contains all other suites! However, currently all_suites behaves not as a root container but as a container of any <test> items not included in the nested suites.

The same problem happened on version 7.0.0-SNAPSHOT attaching example maven project testng-reportportal-example.zip