testng: BUG: Parameter alwaysRun=true for before-methods forces execution of those methods

Guys, sorry but I just want to prevent breaking changes in testng, so will try to explain my opinion: Before testng v6.9.5 parameter alwaysRun had very different meaning in @Before- and @After- methods. Below I tried to describe behavior for that version (based on official testng documentation, javadocs and implementation):

• In context of @BeforeSuite, @BeforeTest, @BeforeClass, @BeforeMethod : schedule this method for execution regardless to which group it belongs / does not belong (of cause if corresponding @Tests were scheduled) Please note: (by implementation up to 6.9.5): – method will be skipped if any parent or preceding configurations failed. – this behavior does not depend on config-failure-policy

• In context of @AfterSuite, @AfterTest, @AfterClass, @AfterMethod : (from javadoc and official documentation): configuration method will be run even if one or more methods invoked previously failed or was skipped

• In context of @Test: (from javadoc): run test-method even if it depends on a method that failed. This attribute ignored if test doesn’t depend on any method or group. Please note: – test-method will be skipped (will not run) if it depends on skipped methods. – test-method will be skipped in every case when any of parent configurations failed.

Last note is very impotent. It means: even when if you force to execute @BeforeMethod after @BeforeClass was failed , all the methods which belongs to class will be skipped. So why we need to force execution of @BeforeMethod if test ignored in any way???

I think this behavior ( v6.9.5) should be saved because it was well-designed by @cbeust and it have meaning. There are no reasonable situations in testing when you should continue to execute ‘child’ prerequisites after ‘parent’ failed.

@krmahadevan, If I understand properly from PR 9d5e6d7, you want to force execution for before-methods only when configfailurepolicy = continue. Please use SRP principle for parameters in testng. Currently (v13.1) configfailurepolicy used to avoid multiple execution of the same configuration method. (Same means: method with concrete signature which belongs to concrete instance) Please don’t use this parameter for any other purpose. This will break compatibility and will make process of testng configuration ugly.

Finally: IMHO to fix current issue you should revert changes which were made in scope of #420 (328620f) - it was wrong fix. Maybe it fixed behavior in particular case but it does not fix root cause. I’m almost sure that issue #420 can be reproduced with or without alwaysRun on @BeforeSuite

@kool79 It looks you have a good knowledge of the issue. Maybe you can propose a pull request with the fix?

I’ve found this bug was introduced in testng-6.9.5 when fixing #420 Below the log for v6.9.4 (note that no other before methods were invoked except before-suite):

  D:\dev\AC\git\test-testng\testng.xml
I'm beforeSuite. I should be alone in console log. Going to fail...

java.lang.RuntimeException
	at qa.ok.testng.AlwaysRunTests.failedBeforeSuite(AlwaysRunTests.java:14)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:85)
	at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:517)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:213)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:140)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:296)
	at org.testng.SuiteRunner.run(SuiteRunner.java:259)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1199)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1124)
	at org.testng.TestNG.run(TestNG.java:1032)
	at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:72)
	at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)
Test ignored.
===============================================
my-suite
Total tests run: 1, Failures: 0, Skips: 1
Configuration Failures: 1, Skips: 3
===============================================