cometbft: tests: `make test` fails locally while Github CI checks pass

Bug Report

Setup

Branch main, clean repository:

$ git rev-parse --verify HEAD
2d097e134ff29ee2ad6ed9653b7dc788fe4a0555
$ git status --ignored
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean

Have you tried the latest version: it is the main branch. Did not test on release branches.

ABCI app (name for built-in, URL for self-written if it’s publicly available): not relevant

Environment:

  • OS (e.g. from /etc/os-release): Darwin 22.6.0 Darwin Kernel Version 22.6.0: Fri Sep 15 13:41:28 PDT 2023; root:xnu-8796.141.3.700.8~1/RELEASE_ARM64_T6000 arm64
  • Install tools: go version go1.21.3 darwin/arm64

Config

Standard.

How to reproduce it

  1. Download the repository
  2. Run make test
  3. Go grab (or make) a coffee

What happened?

Some tests fail, after very very long 18:47.13 minutes.

What did you expect to happen?

The tests succeed. Maybe the execution to be more expedite.

Logs

make test &> output.txt  1083.17s user 326.12s system 125% cpu 18:47.13 total

Output: https://gist.github.com/cason/036e2fe1738157f2a452d9bb588ff051

dump_consensus_state output

Irrelevant.

Anything else we need to know

Checks performed by Github CI pass, including unit tests. Refer, for instance, to a recent pull request, like https://github.com/cometbft/cometbft/pull/2250.

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 1
  • Comments: 16 (16 by maintainers)

Commits related to this issue

Most upvoted comments

It does not fail in CI because CI sets the CI environment variable, which causes the test to be skipped. To succeed locally, to CI=1 make test or similar.

Developers reading the tests and seeing checkTxsInOrder being used all around might think FIFO is guaranteed. I would keep the test here, as a reminder that this is not true.

My previous suggestion doesn’t improve a lot:

$ time make test &> output3.txt
make test &> output3.txt  277.29s user 37.73s system 56% cpu 9:21.39 total

Moreover, tests still fail.

Apparently, the CI job uses the Makefile target test-group. Which uses go test -mod=readonly -timeout=15m -race -coverprofile=$(BUILDDIR)/$*.profile.out as testing command. Maybe make test should use the same command.

I created the somehow related PR #2263 just before seeing this issue 🙂