zephyr: twister fails on several stm32 boards with tests/arch/arm testcases

I disagree with you. In my opinion it is definitely test problem. Every test scenario in yaml file should be standalone and shouldn’t impact on another tests. Without such assumption we will never be able to make stable CI process.

Sure, if you disagree then please provide a solution within the test itself to address this problem.

Depending on the hardware feature you are testing, further steps are needed to cleanup the environment to allow additional tests to run. We have seen this happening with features like power management, watchdig, interrupts and so on.

How do you imagine handling such situation by Twister? How Twister could recognize if such output: PROJECT EXECUTION SUCCESSFUL

This output is not Ok and is not enough to evaluate the issue, we agree on this.

A test runner, read twister should do the setup/teardown for each test to guarantee that the environment is sane and not corrupted by previous tests. Twister need to know how to deal with such cases. The extreme case could be a hardware/power reset after each test, and complete erase of the flash after each test and a power reset, or something along those lines. The above output can come from any test and until now this was acceptable. If we enforce verification of the test being run we need to do more than btw… Your solution assume that if the testsuite name at the beginning and at the end match, then it is ok, but you could be parsing some console output of some other test that is stuck in the hardware, so we are not fully green here, but I guess such situations do not happen very often, however, it is something we need to fix.

This goes back to what I suggested some time ago where each test is assigned an ID during build which is tracked during execution, meaning that twister knows what to expect, because it is the source of the ID, so to illustrate this:

1. twister builds the test with a unique ID
2. test executes and prints the ID on the screen
3. twister matches the id of that test with the one printed on the screen
4. twister only passes the test if the id matches, an old output will have a non-matching ID which would fail the test.

It seems the failure is always on twister-out/nucleo_wb55rg/tests/arch/arm/arm_interrupt/arch.interrupt.extra_exception_info :

handler.log

@FRASTM okey, so for me now it is clear, that this is not a problem with Twister and “testsuite name verification” it is rather problem with nmi test. In my opinion we shouldn’t revert “testsuite name verification”, because it help us to discover problem with this test (it probably exists before introduction this additional verification). Now this test should be fixed. I don’t know how NMI works and how this test should be changed to not impact on other one. Eventually it could be a problem with nucleo_wb55rg board, but probably not because this issue was observed on other platforms yet.

@FRASTM could you create new issue relevant to this nmi test, and describe exactly what to do to reproduce this error by run scripts/twister --device-testing --device-serial /dev/ttyACM0 -p nucleo_wb55rg -T tests/arch/arm -v -v -j 1