test-infra: some testgrid dashboards missing newer test results
eg: https://storage.googleapis.com/k8s-gubernator/triage/index.html#6028a5ad633695a738fc
- shows bump in test results starting around 2018-09-27
- Pick the latest job failure: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-cri-containerd-e2e-gci-gce/7727#sig-api-machinery-admissionwebhook-should-be-able-to-deny-attaching-pod
- Click the recent runs link to get to: https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/logs/ci-cri-containerd-e2e-gci-gce/
- Click the job name to get to testgrid: https://k8s-testgrid.appspot.com/sig-node-containerd#e2e-gci
- Expected: latest run in testgrid matches latest run in gubernator link
- Actual: latest run in testgrid is from 2018-09-26, gubernator shows multiple failing runs from 2018-09-27
Since the results are getting picked up by triage and gubernator, we can conclude they’re landing in GCS.
The summary page https://k8s-testgrid.appspot.com/sig-node-containerd#Summary shows the following for “e2e-gci”. The “Tests last ran on” is indicative of our problem
Last green run: 7706
Tests last ran on: 09-26 10:26
Last update: 09-27 09:21
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 17 (17 by maintainers)
I’m going to be removing recently-generated builds that have snowflake IDs (i.e. within the last 3 days) from kubernetes-jenkins/logs; ETA for identifying and removing all of them about an hour. Once that’s done, TestGrid should go update builds it missed within the last day. If you’re missing results and really need to check up on them, feel free to regenerate your dashboard by bumping up ‘days_of_results’ in your test group config by 1 (which will force TestGrid to re-collect those results).
And having double-checked some dashboards, looks like they’re properly showing recent results from today. Think we can consider this closed, and probably work on a more long-term item to prevent snowflake vs. tot IDs from doing this again.
@krzyzacy:
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
Update:
During Tuesday’s outage tot was obviously down for a while, so we saw snowflake IDs, and when tot was restored, the build IDs picks up but we sort everything with descending so…
@michelle192837 is doing some clean up and, we should kill tot 🙃