prometheus: Promtool promql unittests get slower the more there are of them

Proposal: Parallel test execution

Total tests in suite increase testing time almost exponentially

We are sitting at around 130 tests right now, and tests take 21 mins on our CI/CD pipeline (10 mins on my 8 core local machine)…

By adding a --workers or similar setting to promtool test rules file.json this could be reduced by a big margin.

Would this be suitable/possible to incorporate into promtool test execution?

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 17 (9 by maintainers)

Commits related to this issue

promtool: Calculate mint and maxt per test Previously a single test that used a later eval time would make all other tests in the file share the [mint, maxt] and potentially evaluate far more samples... — committed to dgl/prometheus by dgl 4 years ago
promtool: Calculate mint and maxt per test (#8096) * promtool: Calculate mint and maxt per test Previously a single test that used a later eval time would make all other tests in the file share t... — committed to prometheus/prometheus by dgl 4 years ago

Most upvoted comments

Yeah, splitting up tests across different promtool invocations should only ever be (slightly) slower due to the overheads of starting up new processes so something is wrong here. This will need further investigation.

brian-brazil on Oct 7, 2020

Hi again @dgl , just had a chance to run the whole test suite with the build from your PR and it does seem to improve it:

We have the following:

112 alerts distributed in 20 groups
460 test cases in 112 alert rule tests

With “current” promtool running a promtool test rules tests.json takes roughly 20 minutes.

with your fix:

$ time ../../GoLang/prometheus/promtool test rules tests.json
Unit Testing:  tests.json
  SUCCESS

../../GoLang/prometheus/promtool test rules tests.json  79.98s user 5.70s system 147% cpu 58.135 total

ivannpaz on Oct 22, 2020

Unittest execution time should be measured in seconds, not minutes. For example the fairly exhaustive tests for PromQL itself take less than 10s.

It sounds like you’ve created some very expensive tests with far more data than is needed to test that your rules are working correctly, I’d suggest trimming them down.

brian-brazil on Oct 7, 2020