pipeline: Nightly unit test failure due to race condition when initializing config stores

Expected Behavior

Nightly build unit test passes

Actual Behavior

Nightly build unit test failed:

TaskRun is: pipeline-release-nightly-gkd24-unit-tests-l2pdb

Failure:

  --- FAIL: TestReconcile_ExplicitDefaultSA/success (0.08s)

Additional Info

Tekton Pipeline version: v0.13.2
I tried to reproduce this locally and failed:

 go test -race -cover -count=10 -run="TestReconcile_ExplicitDefaultSA" ./pkg/reconciler/taskrun

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 17 (5 by maintainers)

Commits related to this issue

Repro race causing #2815 This commit hacks in a test that tries to reproduce the issue in #2815. Even then, it only reproduced the issue every once in a while - running doit.sh will usually cause the... — committed to bobcatfish/pipeline by bobcatfish 4 years ago
Fix race: Make informed watcher start wait for Add event 🏎️ When using the informed watcher to watch a config map, previously add events were being processed in a goroutine with no syncrhonization m... — committed to bobcatfish/pkg by bobcatfish 3 years ago
Fix race: Make informed watcher start wait for Add event 🏎️ When using the informed watcher to watch a config map, previously add events were being processed in a goroutine with no syncrhonization m... — committed to bobcatfish/pkg by bobcatfish 3 years ago
Fix race: Make informed watcher start wait for Add event 🏎️ When using the informed watcher to watch a config map, previously add events were being processed in a goroutine with no syncrhonization m... — committed to bobcatfish/pkg by bobcatfish 3 years ago
Fix race: Make informed watcher start wait for Add event 🏎️ When using the informed watcher to watch a config map, previously add events were being processed in a goroutine with no syncrhonization m... — committed to bobcatfish/pkg by bobcatfish 3 years ago
Fix race: Make informed watcher start wait for Add event 🏎️ When using the informed watcher to watch a config map, previously add events were being processed in a goroutine with no syncrhonization m... — committed to bobcatfish/pkg by bobcatfish 3 years ago
Fix race: Make informed watcher start wait for Add event 🏎️ When using the informed watcher to watch a config map, previously add events were being processed in a goroutine with no syncrhonization m... — committed to bobcatfish/pkg by bobcatfish 3 years ago
Fix race: Make informed watcher start wait for Add event 🏎️ (#2036) * Fix race: Make informed watcher start wait for Add event 🏎️ When using the informed watcher to watch a config map, previously ... — committed to knative/pkg by bobcatfish 3 years ago

Most upvoted comments

This test has failed during nightlies on two of my build cop rotations in a row. I’ll assign this to myself and see if I can figure out what’s triggering the random behaviour.

ghost on Jul 24, 2020

I think the point was that this is non-deterministic - let’s leave it open until we feel we understand why the race occurred.

bobcatfish on Jul 20, 2020

Nothing unfortunately:

[unit-test] --- FAIL: TestReconcile_ExplicitDefaultSA (0.14s)
[unit-test]     --- FAIL: TestReconcile_ExplicitDefaultSA/success (0.08s)
[unit-test] FAIL
[unit-test] FAIL        github.com/tektoncd/pipeline/pkg/reconciler/taskrun     22.590s
[unit-test] FAIL

The command I used is (you have to be connected to the dogfooding cluster) :

tkn tr logs pipeline-release-nightly-gkd24-unit-tests-l2pdb

dibyom on Jun 15, 2020