kubernetes: API call latency regression
There is yet another regression for API-call latencies for 5k-node test: https://testgrid.k8s.io/sig-scalability-gce#gce-scale-performance
The last 3 runs of load test failed there.
I didn’t yet have time to look into it deeper, but (especially the last run) seems suspicious, due to this one:
Resource:nodes Subresource: Verb:POST Scope:cluster Latency:{Perc50:4.749ms Perc90:1.807403s Perc99:1.807403s} Count:16}
It seems some nodes were added during the test, which isn’t expected.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 51 (51 by maintainers)
Commits related to this issue
- gce-5000-performance - disable load run density more often Ref. https://github.com/kubernetes/kubernetes/issues/76579 As the regression is cleary visible in the density tests, we'll disable the long... — committed to mm4tt/test-infra by mm4tt 5 years ago
- gce-5000-performance - disable load run density more often Ref. https://github.com/kubernetes/kubernetes/issues/76579 As the regression is cleary visible in the density tests, we'll disable the long... — committed to mm4tt/test-infra by mm4tt 5 years ago
- gce-5000-performance - disable load & run density more often Ref. https://github.com/kubernetes/kubernetes/issues/76579 As the regression is cleary visible in the density tests, we'll disable the lo... — committed to mm4tt/test-infra by mm4tt 5 years ago
- Revert fluentd-gcp version to 0.6-1.6.0-1 in gce-5000. This is essentially reverting https://github.com/kubernetes/kubernetes/pull/77224/ in our gce 5000 performance tests. We believe it may be rela... — committed to mm4tt/test-infra by mm4tt 5 years ago
- Re-enable prometheus in GCE 5K Performance tests. We've ruled out that the prometheus was a cause of the regressions a long time ago. Actually it helps a lot when debugging. The positive side effect... — committed to mm4tt/test-infra by mm4tt 5 years ago
- Enable prometheus snapshot for gce-5K-performance Ref. https://github.com/kubernetes/kubernetes/issues/76579 — committed to mm4tt/test-infra by mm4tt 5 years ago
- gce-5000-performance - disable load & run density more often Ref. https://github.com/kubernetes/kubernetes/issues/76579 As the regression is cleary visible in the density tests, we'll disable the lo... — committed to chases2/test-infra by mm4tt 5 years ago
- Revert fluentd-gcp version to 0.6-1.6.0-1 in gce-5000. This is essentially reverting https://github.com/kubernetes/kubernetes/pull/77224/ in our gce 5000 performance tests. We believe it may be rela... — committed to chases2/test-infra by mm4tt 5 years ago
- Re-enable prometheus in GCE 5K Performance tests. We've ruled out that the prometheus was a cause of the regressions a long time ago. Actually it helps a lot when debugging. The positive side effect... — committed to chases2/test-infra by mm4tt 5 years ago
- Enable prometheus snapshot for gce-5K-performance Ref. https://github.com/kubernetes/kubernetes/issues/76579 — committed to chases2/test-infra by mm4tt 5 years ago
@liggitt I wasn’t precise with my previous comment: I am running version Kuberntes as of 99a52350a43b8824db655779a7d34d8c8f761dfe with 1cca3f9 reverted. It reverted cleanly for me.