foundry: `fail_on_revert` does not fail test suite when revert happens inside handler function

Hey folks - ACKing the issue and thanks for the repros. We’ll fix. We are in the middle of the alloy transition so some things have been slower, appreciate the patience. We’ll prio this in next week’s sprint, so should be solved soon.

@Evalir @gakonst can this bugfix be prioritized pls? The reproducible example is the simplest possible and it is confirmed by several independent developers. This is definitely not an expected behavior of foundry

Hey all! fix is on #6199 and will go on main soon—would be great to get some testers!

I also just ran into this. The workaround for now is just to consider any reverts as a failed test, even though it says pass. For CI/CD you’d have to read the text of the number of reverts, and if it’s not 0 consider it failed.

This regression may have been introduced in #5676

hey @lucas-manuel @0xfarhaan could y’all confirm this is still happening? trying to repro on latest foundry and the repros do fail with fail_on_revert = true.

I can confirm that the issue is still happening on version: forge 0.2.0 (893fc9f 2023-09-12T00:22:45.184609469Z)

I just had to run the following test contract (as described in this issue’s description):

// SPDX-License-Identifier: UNLICENSED
pragma solidity ^0.8.0;

import "forge-std/Test.sol";

contract BrokenInvariantTest is Test {
    bytes4[] internal selectors;

    function setUp() public {
        selectors.push(this.doSomething.selector);

        targetSelector(FuzzSelector({addr: address(this), selectors: selectors}));
    }

    function doSomething() public {
        require(false);
    }

    function invariantBroken() public {}
}

Output is:

Same would have expected this to fail with fail_on_revert set true. Updated the example @Rubilmax posted to use the assertion helper which does post in the logs an assertion failure but the test suite passes.

Code:

// SPDX-License-Identifier: UNLICENSED
pragma solidity ^0.8.0;

import "forge-std/Test.sol";

contract Handler is Test {

    function doSomething() public {
        assertTrue(false);
    }

}

contract BrokenInvariantTest is Test {
    
    bytes4[] internal selectors;

    Handler handler;

    function setUp() public {
        handler = new Handler();

        selectors.push(Handler.doSomething.selector);

        targetSelector(FuzzSelector({addr: address(handler), selectors: selectors}));
    }

    function statefulFuzz_BrokenInvariant() public {}

}

~/rep/personal/forge-playground main ?1 ❯ forge t --mt statefulFuzz_Broken                           
[⠒] Compiling...
No files changed, compilation skipped

Running 1 test for tests/unit/5.t.sol:BrokenInvariantTest
[PASS] statefulFuzz_BrokenInvariant() (runs: 1, calls: 1, reverts: 1)
Logs:
  Error: Assertion Failed
  
Test result: ok. 1 passed; 0 failed; 0 skipped; finished in 2.20ms
Ran 1 test suites: 1 tests passed, 0 failed, 0 skipped (1 total tests

We did ask for this behaviour in https://github.com/foundry-rs/foundry/issues/4718 with the solution implemented in https://github.com/foundry-rs/foundry/pull/5445.