spinnaker: Bake timeout settings
Issue Summary:
Bake stage fails with timeout after 20 seconds
Cloud Provider(s):
GCP
Environment:
Spinnaker 1.26.3 deployed fully distributed on kubernetes (GKE 1.17) via Halyard 1.42.1
Feature Area:
Pipelines
Description:
Bake stage fails with timeout after 20 seconds when trying to clone a large GitHub repository containing the kubernetes manifests on which to run a Kustomize step
Steps to Reproduce:
Large GitHub repo used as manifest artifact source for kubernetes manifests used by pipeline’s Bake stage doing Kustomize
Additional Details:
I’ve been scouring the documentation but can’t find timeout settings, this page is closest but doesn’t appear to have a complete list of settings?
https://spinnaker.io/reference/halyard/custom/#tweakable-service-settings
I also looked through past tickets such as #5518 and #4498, but the timeout settings shown there have not worked for me when putting them under ~/.hal/default/service-settings/ and then running hal deploy apply. I’ve tried both orca.yml / rosco.yml as well as orca-local.yml / rosco-local.yml and sections ok-http-client and okHttpClient as mentioned in those tickets.
Could we please get clearly documented settings to tune timeouts?
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 4
- Comments: 15 (6 by maintainers)
This fix worked for me as well, though I did 60 seconds because I didn’t need 120. I do feel like this should either be the default, be more easily configurable, and/or handled through some means other than a long-running HTTP request.
None of the above did work for me because the issue is not the client but the server. After some debugging I found out that the server’s async request handling has timeouts as well and the default is 30 seconds. This is a problem for slow git clone jobs that are done in that async requests. To fix this I had to increase the timeout with the following config in
.hal/default/service-settings/clouddriver.ymlThe
-XX:MaxRAMPercentage=50.0is the default that is set and we have to extend it with-Dspring.mvc.async.request-timeout=180000to increase the async request timeout (value is in ms).EDIT: The error you will see in clouddriver is
Cannot dispatch without an AsyncContextif you have this problem.