test-infra: testgrid shows PASSING for tests that don't run
The testgrid summary seems to mark a test as PASSING if it can’t find any results at all. This makes it hard to notice when a job stops working entirely, because it looks like everything is fine at a glance.
For example:
https://k8s-testgrid.appspot.com/release-1.6-all#Summary
kubeadm-gce-1.6: PASSING
No results in the past week
No passing build found
Tests have not run
Last update: 09-11 10:41
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 16 (13 by maintainers)
Update for the STALE status: TestGrid should support STALE as a new status, if 1) there are no results at all (i.e. no column headers), or 2) there’s a ‘stale’ warning on the tab.
It looks like a lot of dashboards are still going to dodge STALE status because they have column headers but no actual results; I’ll see about changing that, but what would probably be more useful for everyone is to set ‘alert_stale_results_hours’ on your test group to have the tab be considered stale if there are no results in X hours.