pipeline: e2e TestGitPipelineRun test fails on newly installed pipeline

Expected Behavior

TestGitPipelineRun test should pass

Actual Behavior

TestGitPipelineRun test fails because of timeout. Corresponding taskrun is created but no pod is started. Usually second attempt is successful.

# oc get tr -A
NAMESPACE         NAME                                     SUCCEEDED   REASON   STARTTIME   COMPLETIONTIME
arendelle-8nxtw   git-check-pipeline-run-git-check-72mth

# oc get pod -n arendelle-8nxtw
No resources found in arendelle-8nxtw namespace.

Steps to Reproduce the Problem

Install k8s cluster using kind
Install nightly pipeline build

kubectl apply --filename https://storage.googleapis.com/tekton-releases-nightly/pipeline/latest/release.yaml

Run the test just after pipeline is installed

# go test -v -tags=e2e -count=1 ./test -run ^TestGitPipelineRun/tekton_@_master_with_PR_refspec

Wait 10mins and get the error

# go test -v -tags=e2e -count=1 ./test -run ^TestGitPipelineRun/tekton_@_master_with_PR_refspec
Using kubeconfig at `/root/.kube/config` with cluster ``
Cluster information
- Version: v1.19.1
- BuildDate: 2020-09-14T07:30:52Z
- GoVersion: go1.15
- Platform: linux/amd64
=== RUN   TestGitPipelineRun
=== RUN   TestGitPipelineRun/tekton_@_master_with_PR_refspec
=== PAUSE TestGitPipelineRun/tekton_@_master_with_PR_refspec
=== CONT  TestGitPipelineRun/tekton_@_master_with_PR_refspec
    init_test.go:147: Create namespace arendelle-8nxtw to deploy to
    init_test.go:163: Verify SA "default" is created in namespace "arendelle-8nxtw"
    git_checkout_test.go:103: Creating Git PipelineResource git-source-resource
    git_checkout_test.go:119: Creating PipelineRun git-check-pipeline-run
panic: test timed out after 10m0s
...
FAIL	github.com/tektoncd/pipeline/test	600.088s
FAIL

Additional Info

Kubernetes version:

# kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38", GitTreeState:"clean", BuildDate:"2020-12-08T17:59:43Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.1", GitCommit:"206bcadf021e76c27513500ca24182692aabd17e", GitTreeState:"clean", BuildDate:"2020-09-14T07:30:52Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}

Tekton Pipeline version:

# tkn version
Client version: 0.11.0
Pipeline version: v20201210-d58a1ae68f
Triggers version: unknown

/kind bug

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 22 (12 by maintainers)

Commits related to this issue

Set fake AWS credentials on controller to workaround aws-sdk bug Several issues have now reared their head which are directly caused by an update to the aws-sdk. The update results in extremely long ... — committed to tektoncd/pipeline by deleted user 3 years ago

Most upvoted comments

Using the runtime.trace package I’m able to isolate 8x 75s-long calls to http.dialConnFor, that look like they’re all spawned from the aws-sdk-go package. So at least in my case I think this accounts for the ten minute delay:

0.000342165	 .         	198	task k8schain (id 1, parent 0) created
0.000345207	 .     3042	198	namespace=default
0.000346755	 .     1548	198	saName=default
0.000348249	 .     1494	198	region k8schain.New started (duration: 10m2.20341841s)
0.005701440	 .  5353191	198	new goroutine 260: github.com/aws/aws-sdk-go/internal/sync/singleflight.(*Group).doCall
0.005805444	 .   104004	260	new goroutine 261: net/http.(*Transport).dialConnFor
75.329407376	75.323601932	260	new goroutine 464: net/http.(*Transport).dialConnFor
150.583420726	75.254013350	260	new goroutine 785: net/http.(*Transport).dialConnFor
225.953546323	75.370125597	260	new goroutine 991: net/http.(*Transport).dialConnFor
301.129792264	75.176245941	260	new goroutine 1258: net/http.(*Transport).dialConnFor
376.398534187	75.268741923	260	new goroutine 1497: net/http.(*Transport).dialConnFor
451.715118090	75.316583903	260	new goroutine 1809: net/http.(*Transport).dialConnFor
527.040350774	75.325232684	260	new goroutine 2069: net/http.(*Transport).dialConnFor
602.203773864	75.163423090	198	total-k8schain-time=10m2.203418063s
602.203775323	 .     1459	198	task end
GC:22.952305ms

ghost on Jul 2, 2021