testng: group-by-instances with test dependencies causes instantiation of tests to exponentially slow

I recently ran into an issue where TestNG is taking a really long time graphing the order to run tests in. If I don’t use dependsOnMethods then by 1600 test instances (3200 test methods plus some before/after methods) start to run after a second or so but once I add a single dependsOnMethods declaration my tests take over 5 minutes to get ordered and that’s even before anything is run.

# instances	total duration (s)
100	0.4504
200	1.449
300	2.740
400	5.498
500	10.28
600	17.82
700	28.59
800	45.10
900	59.39
1000	81.54
1100	109.56
…	…
1600	364.92

As you can tell, the growth rate is exponential. Is there any way to speed up this process?

Here is some test code which demonstrates the issue:

import com.google.common.base.Stopwatch
import org.testng.annotations.*

data class TestInstance(val id: Int) {
    companion object {
        init {
            val stopwatch = Stopwatch.createStarted()
            Runtime.getRuntime().addShutdownHook(Thread(Runnable { println(stopwatch) }))
        }

        @Factory fun instances(): Array<Any> = Array(1600) { id -> TestInstance(id) }
    }

    @BeforeMethod fun beforeMethod() = Unit
    @Test fun something() = Unit
    @Test(dependsOnMethods = arrayOf("something")) fun somethingElse() = Unit
    @AfterMethod fun afterMethod() = Unit
    @AfterSuite fun afterSuite() = Unit
}

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 15 (11 by maintainers)

Commits related to this issue

Fix topologicalSort for more efficiency Closes #1077 and #1081 This fix reduces the scope of MethodHelper toplologicalSort to be more efficient. The search scope for "methoddependedupon" was cha... — committed to krmahadevan/testng by Aheiss 6 years ago

Most upvoted comments

Hey Guys,

I have been looking into this issue since it is causing problems with being able to run our full test suite. I found a fix that improves performance significantly when running a suite that contains a large number of classes. When trying to run our full suite which consists of about 700 classes containing a total of 8500 test methods on version 6.14.2 the suite never loads similar to what is described in #1077. As pointed out above the issue appears to be be in MethodHelper#topologicalSort looking even further down it looks like MethodHelper#matchmethod gets bogged down when trying to match a regex when a large array is passed, this is true especially when the matching method is far down in the array. I was able to get around this issue by modifying MethodHelper#topologicalSort to first splitting all the methods into their associated class then adding an additional for loop to iterate through all the classes then using the existing for loop to iterates through all the methods in the class to find the dependeduponmethod.

         //Create Map of classes that contain test methods
	LinkedHashMap<ITestClass, ArrayList<ITestNGMethod>> classMethods =  sortIntoClasses(methods);	
	//
	// Create the graph
	//
	for (Map.Entry<ITestClass, ArrayList<ITestNGMethod>> testClass : classMethods.entrySet()) {
		
		//Changed to sort methods in class instead of all methods
		// for (ITestNGMethod m : methods) {
		
		for (ITestNGMethod m : testClass.getValue()) {
			ITestNGMethod[] testList = testClass.getValue().toArray(new ITestNGMethod[testClass.getValue().size()]);
			
			result.addNode(m);

			List<ITestNGMethod> predecessors = Lists.newArrayList();

			String[] methodsDependedUpon = m.getMethodsDependedUpon();
			String[] groupsDependedUpon = m.getGroupsDependedUpon();

			if (methodsDependedUpon.length > 0) {
				ITestNGMethod[] methodsNamed = MethodHelper.findDependedUponMethods(m, testList);
				for (ITestNGMethod pred : methodsNamed) {
					predecessors.add(pred);
				}
			}

I would like to note that this does NOT fix the performance when creating tests using a @factory. This only improves performance when test are grouped by classes. I am just starting to look into the factory issue and will add any new information if I find it.

Here is the sortIntoClasses method is being used above :

private static LinkedHashMap<ITestClass, ArrayList<ITestNGMethod>> sortIntoClasses(ITestNGMethod[] methods) {
	LinkedHashMap<ITestClass, ArrayList<ITestNGMethod>> result = new LinkedHashMap<>();
	for (ITestNGMethod method : methods) {
		// Extract key
		ITestClass methodClass = method.getTestClass();
		// Class already in map
		if (result.containsKey(methodClass)) {
			ArrayList<ITestNGMethod> methodList = result.get(methodClass);
			methodList.add(method);
			result.put(methodClass, methodList);
		} else {
			ArrayList<ITestNGMethod> methodList = new ArrayList<>();
			methodList.add(method);
			result.put(methodClass, methodList);
		}
	}
	return result;

}

Aheiss on Feb 19, 2018