kubernetes: Serial / Flaky / Slow tests should not be marked [Conformance]

We have a new ci/pull job that runs conformance tests with a single node started using hack/local-up-cluster.sh. Comparing runs of pull-kubernetes-e2e-gce and ci-kubernetes-local-e2e there are a few tests that we do not run on the GCE job because they are marked as [Serial] / [Flaky] / [Slow].

Questions:

Are these tests actually needed?
Do we need a blocking pre-submit job that runs the full suite? (given that it takes time > 1 hour to run)

[dims@dims-mac 20:19] ~/Downloads ⟩ diff -Bbwu e2e-gce-2.txt e2e-local-2.txt
--- e2e-gce-2.txt	2018-04-04 20:19:18.000000000 -0400
+++ e2e-local-2.txt	2018-04-04 20:19:31.000000000 -0400
@@ -12,12 +11,14 @@
 [k8s.io] Probing container should *not* be restarted with a exec "cat /tmp/health" liveness probe [Conformance]
 [k8s.io] Probing container should be restarted with a /healthz http liveness probe [Conformance]
 [k8s.io] Probing container should be restarted with a exec "cat /tmp/health" liveness probe [Conformance]
+[k8s.io] Probing container should have monotonically increasing restart count [Slow] [Conformance]
 [k8s.io] Probing container with readiness probe should not be ready before initial delay and never restart [Conformance]
 [k8s.io] Probing container with readiness probe that fails should never be ready and never restart [Conformance]
 [k8s.io] Variable Expansion should allow composing env vars into new env vars [Conformance]
 [k8s.io] Variable Expansion should allow substituting values in a container's args [Conformance]
 [k8s.io] Variable Expansion should allow substituting values in a container's command [Conformance]
 [k8s.io] [sig-node] Events should be sent by kubelets and the scheduler about pods scheduling and running [Conformance]
+[k8s.io] [sig-node] Pods Extended [k8s.io] Delete Grace Period should be submitted and removed [Flaky] [Conformance]
 [k8s.io] [sig-node] Pods Extended [k8s.io] Pods Set QOS Class should be submitted and removed [Conformance]
 [k8s.io] [sig-node] PreStop should call prestop when killing a pod [Conformance]
 [sig-api-machinery] ConfigMap should be consumable via environment variable [Conformance]
@@ -37,6 +38,10 @@
 [sig-api-machinery] Garbage collector should orphan pods created by rc if delete options say so [Conformance]
 [sig-api-machinery] Secrets should be consumable from pods in env vars [Conformance]
 [sig-api-machinery] Secrets should be consumable via the environment [Conformance]
+[sig-apps] Daemon set [Serial] should retry creating failed daemon pods [Conformance]
+[sig-apps] Daemon set [Serial] should run and stop complex daemon [Conformance]
+[sig-apps] Daemon set [Serial] should run and stop simple daemon [Conformance]
+[sig-apps] Daemon set [Serial] should update pod when spec was updated and update strategy is RollingUpdate [Conformance]
 [sig-apps] ReplicaSet should serve a basic image on each replica with a public image [Conformance]
 [sig-apps] ReplicationController should serve a basic image on each replica with a public image [Conformance]
 [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] Burst scaling should run to completion even with unhealthy pods [Conformance]
@@ -81,6 +86,9 @@
 [sig-network] Services should provide secure master service [Conformance]
 [sig-network] Services should serve a basic endpoint from pods [Conformance]
 [sig-network] Services should serve multiport endpoints from pods [Conformance]
+[sig-scheduling] SchedulerPredicates [Serial] validates resource limits of pods that are allowed to run [Conformance]
+[sig-scheduling] SchedulerPredicates [Serial] validates that NodeSelector is respected if matching [Conformance]
+[sig-scheduling] SchedulerPredicates [Serial] validates that NodeSelector is respected if not matching [Conformance]
 [sig-storage] ConfigMap optional updates should be reflected in volume [Conformance]
 [sig-storage] ConfigMap should be consumable from pods in volume [Conformance]
 [sig-storage] ConfigMap should be consumable from pods in volume as non-root [Conformance]

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 29 (28 by maintainers)

Commits related to this issue

Remove recommended skips in conformance README. Work has been done to clean up the conformance suite such that no tests are marked as flaky or disruptive, so updating the documentation so that downst... — committed to randomvariable/test-infra by randomvariable 4 years ago

Most upvoted comments

By that I mean I’d like us to update https://github.com/kubernetes/kubernetes/blob/master/test/conformance/walk.go

I think it should reject tests that have [Flaky] or [Slow] or [Feature:.*] tags

I would code this in such a way as to anticipate more tags down the line, and warn/inform on tags that we may consider blocking on in the future but not today (eg: [Privileged], [Serial], etc)

spiffxp on Jan 24, 2019

ok, nevermind, it was quicker than I thought to list these out… instead @mgdevstack I’m curious what the impact of these tests are… how slow are they really, and do they really need to be serial?

Using junit.xml from a recent run of ci-kubernetes-gce-conformance. It will list fully expanded test names for all e2e tests, skipped or not. Then, I swear I tried to be nice and use xpath string(//testcase/@name) but my copy of xpath wasn’t having it. So grep/sed it is.

$ wget https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-gce-conformance/822/artifacts/junit_01.xml
$ for tag in Flaky Slow Serial; do \
  echo --$tag--; cat junit_01.xml | grep name= | sed -e 's/.* name="\([^"]*\)".*/\1/' | \
  grep '\[Conformance\]' | sort | grep "\[$tag\]"; done
--Flaky--
--Slow--
[k8s.io] Probing container should have monotonically increasing restart count [Slow][NodeConformance] [Conformance]
[sig-storage] EmptyDir wrapper volumes should not cause race condition when used for configmaps [Serial] [Slow] [Conformance]
--Serial--
[sig-api-machinery] Namespaces [Serial] should ensure that all pods are removed when a namespace is deleted [Conformance]
[sig-api-machinery] Namespaces [Serial] should ensure that all services are removed when a namespace is deleted [Conformance]
[sig-apps] Daemon set [Serial] should retry creating failed daemon pods [Conformance]
[sig-apps] Daemon set [Serial] should rollback without unnecessary restarts [Conformance]
[sig-apps] Daemon set [Serial] should run and stop complex daemon [Conformance]
[sig-apps] Daemon set [Serial] should run and stop simple daemon [Conformance]
[sig-apps] Daemon set [Serial] should update pod when spec was updated and update strategy is RollingUpdate [Conformance]
[sig-scheduling] SchedulerPredicates [Serial] validates resource limits of pods that are allowed to run  [Conformance]
[sig-scheduling] SchedulerPredicates [Serial] validates that NodeSelector is respected if matching  [Conformance]
[sig-scheduling] SchedulerPredicates [Serial] validates that NodeSelector is respected if not matching  [Conformance]
[sig-storage] EmptyDir wrapper volumes should not cause race condition when used for configmaps [Serial] [Slow] [Conformance]

spiffxp on Nov 1, 2018