tox: option to fail fast

I would find this option useful as well. I’d like my linter, unit tests, and end-to-end tests in different environments, and there should be an option to stop early if the linter or unit tests fail to avoid running the long e2e tests. E.g. something like this: tox -e lint,py36,py36-e2e --stop-on-fail

This would be really useful for running tox within git-bisect

right now i’m dong a big refactor on a big python project i move one function into place, run tox, wait 3 minutes, and then scroll about 18000 lines to see what next to do…

on the one hand, this leaves a lot of time for contemplation: why am i doing this? why am i not doing something i get paid for? why didn’t i listen to my parents and learn something sensible like goat herding? on the other hand it would be quite useful to get this done quickly and have less existential angst.

For development perspective I would find this very useful because I don’t want to scroll lots of screens in order to see what failed. I just want to get to the failure as quick as possible, fix it and run again.

I can understand that for CI the desire may be different but if this feature would be controlable using both tox.ini entry and a TOX_ variable it would be ideal, as I could just define the variable for my development machine and not have to touch the codebase.

If it needs to only stop some envs if one envs fails
and not to stop other envs

Sample Situation :

If we need to only stop all other PyPys if any one of PyPy fails, still continue other envs and only stop other CPythons if any one of CPython fails.

Maybe this approach helps 😃

[tox]
envlist =  {py27,py35}-batch1,{pypy2,pypy3}-batch2

[testenv:batch1]
stoponfail = True

[testenv:batch2]
stoponfail = True

Next thought is obviously to introduce a flag that tox should stop running if one testenv fails, but I see trouble that way (e.g. what is detox supposed to do, which will end up in core at some point).

Until this will be implemented, I’m using for loop for it like:

for env in `tox -l`; do echo $env; tox -e $env || break; done

Last paragraph states:

Tox >= 2.0 will abort on an error by default, which is safer and more typical of CI and command execution tools, as it doesn’t make sense to run tests if installing some prerequisite failed and it doesn’t >make sense to try to deploy if tests failed.

So that’s where I got impression that tox stops on error.

Not sure about wording but that part should somehow emphasize the fact that only current environment execution is stopped and if there are more defined, either in envlist configuration or via -e switch from the cli rest will be executed.

@jdahlin those thousand test running are handled by your test runner, not by tox; so the test runner provides the availability to stop, not tox. All we can do is that if py37 failed we don’t start py36 too… but that’s tox environment count, not test count.

I would love to see this feature as well, we have thousands of tests and it’s quite hard to scroll up in the list to find the previous failures.

indeed, running only one of the envs greatly cut down on waiting and scrolling time, and i was able to make some actual progress!

thank you all for the advice!

Hi @jtiai as already said: this is how it’s supposed to be 😃

You could get to the wanted behaviour by doing something like this on the command line (same semantics on POSIX and Windows):

tox -e env1 && tox -e env2 && tox -e env3

If any of those runs fail the rest of the chain will not be executed anymore.

On CI systems you usually have the concept of stages that are executed serially and if one stage fails the whole plan fails and nothing else is executed. So there you would create a stage for each of the tox environments that depend on each other.