casper-node: Non-det failure of `run_equivocator_network` test.

https://drone-auto-casper-network.casperlabs.io/casper-network/casper-node/1453/3/3 failed with

failures:

---- reactor::participating::tests::run_equivocator_network stdout ----
thread 'reactor::participating::tests::run_equivocator_network' panicked at 'assertion failed: `(left == right)`
  left: `[]`,
 right: `[PublicKey::Ed25519(5bc5542a73a3fd4e53c2f3c84838156cf8ce5de0db828a5644b1c6d883056aa6)]`', node/src/reactor/participating/tests.rs:319:13

source: https://github.com/casper-network/casper-node/blob/dev/node/src/reactor/participating/tests.rs#L319

4.11.23 can we create equivocation through the diagnostic port? Goal: make this deterministic and then address. Use the equivocation mechanism available in Zug

About this issue

Original URL
State: open
Created 3 years ago
Comments: 22 (17 by maintainers)

Commits related to this issue

Merge #1862 1862: Add more asserts to run_equivocator_network. r=afck a=afck Assert that the equivocator is not a validator anymore after up to 20 eras. The probability of no equivocation happening ... — committed to casper-network/casper-node by casperlabs-bors-ng[bot] 3 years ago
Merge #1921 1921: Make run_equivocator_network test simpler and less flaky. r=afck a=afck This wraps the `Participating` reactor in a `FilterReactor` in the test, and delays the first few messages t... — committed to casper-network/casper-node by casperlabs-bors-ng[bot] 3 years ago
Merge #1985 1985: Make run_equivocator_network test more reliable. r=goral09 a=afck This is still flaky, possibly because sometimes the nodes only start a bit after the genesis timestamp. With this ... — committed to casper-network/casper-node by casperlabs-bors-ng[bot] 3 years ago
Merge #2133 2133: Improve and add logging to run_equivocator_network test. r=afck a=afck The test is flaky on CI but always passes locally. This adds some logging so we can see whether it's a timing... — committed to casper-network/casper-node by casperlabs-bors-ng[bot] 3 years ago
Merge #2269 2269: Fix run_equivocator_network test. r=afck a=afck Delay only _consensus_ messages in the test, not other network requests. The test was flaky before, and I suspect it had to do wi... — committed to casper-network/casper-node by casperlabs-bors-ng[bot] 3 years ago
Merge #2436 2436: Temporarily disable the `run_equivocator_network` test r=rafal-ch a=rafal-ch This PR temporarily disables the `run_equivocator_network` test as it is giving the non-deterministic r... — committed to casper-network/casper-node by casperlabs-bors-ng[bot] 3 years ago
Merge #2465 2465: Re-enable run_equivocator_network test. r=afck a=afck This re-enables the test, but doesn't fail if we didn't manage to equivocate. The check for equivcation should be re-enable... — committed to casper-network/casper-node by casperlabs-bors-ng[bot] 3 years ago
Merge #3638 3638: Reinstate early exit from run_equivocator_network test r=Fraser999 a=Fraser999 This reinstates the workaround recently removed from the `run_equivocator_network` pending a proper f... — committed to casper-network/casper-node by casperlabs-bors-ng[bot] a year ago

Most upvoted comments

This test is going to be temporarily disabled. It fails so often, that we tend to just retry the build and don’t investigate if it uncovered a real problem.

@Fraser999 @piotr-dziubecki @EdHastingsCasperLabs We think this test is very important, though, so please consider increasing priority of the ticket.

rafal-ch on Dec 3, 2021

Experienced this as well, just fyi: https://drone-auto-casper-network.casperlabs.io/casper-network/casper-node/1988

TomVasile on Sep 1, 2021

I think the least invasive way to make Alice equivocate is to delay all messages to and from her clone by one round length: She’d then send a witness unit citing the first proposed block, and the clone should send one that doesn’t. In the second round, the clone’s messages are delivered and all nodes detect the equivocation. The tests could be simplified to always expect an equivocation in the first era.

One way to implement this is to wrap the reactor in a new FilteringReactor that in dispatch_event applies a configurable filter

Box<dyn Fn(Event) -> Either<Effects<Event>, Event>

and either returns the effects directly or forwards to the inner reactor’s dispatch_event.

In run_equivocator_network the filter would wrap all events containing IncomingMessage or NetworkRequest in

self.effect_builder.set_timeout(round_len).event(move |_| event)

to delay them.

afck on Aug 12, 2021