robotframework: `--runemptysuite` should work with `--rerunfailed` when there are no failed tests
Hi,
The Collecting failed tests from 'output.xml' failed: All tests passed. is pretty misleading: In reality nothing fails, it finds as just many failed tests as it is supposed to find. Maybe Could not find failed tests, exiting or similar would be better. Now it indicates that it would give some non-zero status code and so on.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 15 (9 by maintainers)
Commits related to this issue
- Document that uing --RunEmptySuite w/ --ReRunFailed requires RF 5.0.1+ Part of #3862. — committed to robotframework/robotframework by pekkaklarck 2 years ago
Yeah, you should get 0 return code when using
--runemptysuite --rerunfailed out.xmland there are no failed tests. That’s how--rerunfailedworks in other contexts as well. Notice that you also get log and report after such execution.I can try to test this, is there a pull request etc?
Fixing this was a bit more challenging than I thought due to how
--rerunfailedhas been implemented. Now I got a pretty nice fix done but still need to create acceptance tests for it.The fix will contain a subtle change to
TestSuite.filterbehavior. This method basically implements--include/--excludeand--suite/--testbehavior and it accepts lists of include/exclude tags and suite/test names as arguments. Earlier passing, for example, test names asNoneor an empty list caused no tests to be filtered. After the forthcoming changeNonestill works the same way, but passing an empty list means no test will be selected. That’s obviously a backwards incompatible change, but because the default behavior doesn’t change and because usingNoneis anyway better if you want to explicitly disable filtering using a certain criteria, I don’t think this change needs to be mentioned in the release notes.