sktime: [BUG] logic problem with new CI - coverage of `TestAllEstimators` and `TestAllObjects` incorrect
I think there is a bug in the new CI. The coverage is not affected since “old CI” is still running.
The issue occurs when, say, a forecaster changes.
The new CI will then trigger tests in sktime.forecasting, but not in sktime.tests which contains TestAllEstimator.
So, we run the forecaster specific tests for the changed forecaster, but not the object and estimator specific tests.
FYI @yarnabrina, is there a way to fix this?
The “one liner” to “run all tests for estimator X” is check_estimator - however, this is intentionally avoided in the original setup as an entry point since this distributes less well across workers, via pytest-xdist.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 16 (6 by maintainers)
Commits related to this issue
- [MNT] include unit tests in `sktime/tests` in per module tests (#6353) Fixes #6352 This PR adds coverage of `sktime/tests` for per module CI workflows. — committed to sktime/sktime by yarnabrina 3 years ago
should we move the test times discussion to here? https://github.com/sktime/sktime/issues/6344
The solution could be, to run
testsin all “module” tests, that will at least ensure that the estimator/object tests are run for changed forecasters.We cannot turn them off in “other”, as estimators may live in “other” modules. But that will trigger only if an “other” and a “module” estimator is changed in the same PR, so perhaps low incidence.