pipeline: Overriding Pipeline timeout does not work
Expected Behavior
A PipelineRun, created by a Pipeline should not timeout after 1 hour, but rather after 2h30m.
I used the following configuration:
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
name: pipeline-name
spec:
tasks:
- name: task-name
taskRef:
kind: Task
name: task-name
timeout: "2h30m0s"
Actual Behavior
A PipelineRun, created by a Pipeline runs out after 1 hour:
PipelineRun "pipeline-name-id" failed to finish within "1h0m0s"
The PipelineRun yaml configuration contains the following:
spec:
pipelineRef:
name: pipeline-name
taskRunTemplate:
serviceAccountName: pipeline
timeouts:
pipeline: 1h0m0s
The pipeline timeout is not overwritten and the pipeline fails.
Steps to Reproduce the Problem
- Create a Pipeline with a task that takes over an hour to complete.
- Set the timeout of the task to more than 1 hour (as shown in the “Expected Behavior” section).
- Run the Pipeline. Check the created PipelineRun config for the timeout limits
Additional Info
I am running a pipeline with 2 tasks, one of which takes longer than an hour. I have specified the timeout limit for this task. However, when a PipelineRun is created by the Pipeline, the default value of 1 hour is not overwritten. I have followed the documentation on how to set a timeout for a pipeline, which seems to have to be done on the task-level.
In the PipelineRun yaml, created by the Pipeline, both:
tasks:
- name: task-name
taskRef:
kind: Task
name: task-name
timeout: 2h30m0s
and
spec:
pipelineRef:
name: pipeline-name
taskRunTemplate:
serviceAccountName: pipeline
timeouts:
pipeline: 1h0m0s
are present. Because the pipeline timeout is shorter than the task timeout, the pipeline will fail after an hour.
I have also tried to set the default timeout value to 2 hours via a ConfigMap. This did not work either:
apiVersion: v1
kind: ConfigMap
metadata:
name: config-defaults
namespace: tekton-pipelines
data:
default-timeout-minutes: "150"
This still resulted in a 1 hour timeout limit.
PipelineRun
Using the following configuration for a PipelineRun does work. In this case the default value of 1 hour is overwritten by 2h40m. However, I would like to not have to create my PipelineRuns manually.
apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerTemplate
metadata:
name: trigger
spec:
resourcetemplates:
- apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
generateName: triggered
spec:
pipelineRef:
name: pipeline-name
timeout: "2h40m0s"
-
Kubernetes version: v1.27.10+28ed2d7 (OpenShift 4.4)
Output of
kubectl version:
Client Version: v1.27.4
Kustomize Version: v5.0.1
Server Version: v1.27.10+28ed2d7
- Tekton Pipeline version: tekton.dev/v1beta1
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 16 (7 by maintainers)
Commits related to this issue
- Add PipelineRun pipeline-timeout note to pipeline docs The Pipelines docs' timeout section did not mention that the Pipeline itself had its timeout configured by the PipelineRun, causing some confusi... — committed to aThorp96/tekton-pipeline by aThorp96 2 years ago
- Add PipelineRun pipeline-timeout note to pipeline docs The Pipelines docs' timeout section did not mention that the Pipeline itself had its timeout configured by the PipelineRun, causing some confusi... — committed to tektoncd/pipeline by aThorp96 2 years ago
@vdemeester do you know someone who’d be interested to looking into this issue?
@aThorp96 I have not been able to reproduce this bug as well using your code example and a deadline in both pipeline and pipeline run.
As I see you have already updated docs for this issue. Guess the issue may be closed ✌️
@tampler I was unable to reproduce this issue, where the configured timeout-minutes is not applied and not respected, with the following setup:
Pipeline:
Defaults-config:
After applying the above, when creating a pipeline run via the Tekton Dashboard and also
tkn pipeline start, the Pipeline Run was created correctly with the specified default timeout of150minutes: Pipeline Run:Similarly if I switched the Task timeout (defined in the Pipeline) to be greater than the Pipeline timeout (defined in the Pipeline Run) then the timelines behaved as expected: the Pipeline timed out before the Task’s configured timeout
Two things to note though:
@MarijnJV you mention
how to set a timeout for a pipeline, which seems to have to be done on the task-level. I think this is the source of some confusion here. The Pipeline specifies the timeouts for each Task, but the Pipeline Run specifies the timeout for the Pipeline, and the two timeouts are orthogonal. If the Pipeline timeout lapses, the Task timeout is not relevant as all of the Pipeline Run’s Task Runs are immediately stopped as “timed-out”. This could be clearer in the Pipeline docs you linked, and is more clearly explained in the Pipeline Run docs. I can maybe improve the Pipeline’s timeout docs so that this distinction is clearer.@tampler Given the above, I was looking into how a PipelineRun may be created and the
default-timeout-minutesnot be applied, and I was unable to do so with valid configuration. What I did notice however was that theconfig-defaultsconfigmap’s pre-populated values are not applied. The values in the config-map by default are nested under a_examplekey. So if you just changed the value in the config map from “60” to “150” then it will have no effect. You need to movedefault-timeout-minutesout of_exampleso that it is at.data.default-timeout-minutes. After doing that, all pipeline-runs were created with the 150-minute pipeline-timeout. If you still experience the issue after ensuring the configmap is correct, do you mind manually creating the pipeline usingtkn pipeline start <pipeline name> --pipeline-timeout "2h30m0s"and confirming that the pipeline run still times-out after1h0m0s?@aThorp96 Thanks for looking into this 👀
tekton-pipelines/config-defaultsconfig map. Bumped default timeout to 600 minstask timeoutin the pipeline like I showed upper (see quotation)The thing is that the
timeoutdirective WORKS, however there’s a hardcoded upper limit somewhere in your code base which you should move into the global config map.As I told, decreasing timeout to a lower value (say,
3 mins) works fine. Bumping timeout to1h5m0swon’t work due to your ceiling hardcoded valueI can look into this @afrittoli @vdemeester