spinnaker: Bake timeout settings

Issue Summary:

Bake stage fails with timeout after 20 seconds

Cloud Provider(s):

GCP

Environment:

Spinnaker 1.26.3 deployed fully distributed on kubernetes (GKE 1.17) via Halyard 1.42.1

Feature Area:

Pipelines

Description:

Bake stage fails with timeout after 20 seconds when trying to clone a large GitHub repository containing the kubernetes manifests on which to run a Kustomize step

Steps to Reproduce:

Large GitHub repo used as manifest artifact source for kubernetes manifests used by pipeline’s Bake stage doing Kustomize

Additional Details:

I’ve been scouring the documentation but can’t find timeout settings, this page is closest but doesn’t appear to have a complete list of settings?

https://spinnaker.io/reference/halyard/custom/#tweakable-service-settings

I also looked through past tickets such as #5518 and #4498, but the timeout settings shown there have not worked for me when putting them under ~/.hal/default/service-settings/ and then running hal deploy apply. I’ve tried both orca.yml / rosco.yml as well as orca-local.yml / rosco-local.yml and sections ok-http-client and okHttpClient as mentioned in those tickets.

Could we please get clearly documented settings to tune timeouts?

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 4
  • Comments: 15 (6 by maintainers)

Most upvoted comments

I added this additional configuration to local/orca-local.yml and local/rosco-local.yml when using github.com/spinnaker/kustomization-base as my configuration for the project and it’s been working great for a couple of months. This is for the next person who runs into this, best of luck with ArgoCD.

# Add a timeout to okHttp so that it has 120s to clone repos which tend to
# time out with their default of 20s
# https://github.com/spinnaker/rosco/pull/594#pullrequestreview-395576692
okHttpClient:
    readTimeoutMs: 120000

This fix worked for me as well, though I did 60 seconds because I didn’t need 120. I do feel like this should either be the default, be more easily configurable, and/or handled through some means other than a long-running HTTP request.

None of the above did work for me because the issue is not the client but the server. After some debugging I found out that the server’s async request handling has timeouts as well and the default is 30 seconds. This is a problem for slow git clone jobs that are done in that async requests. To fix this I had to increase the timeout with the following config in .hal/default/service-settings/clouddriver.yml

env:
  JAVA_OPTS: "-XX:MaxRAMPercentage=50.0 -Dspring.mvc.async.request-timeout=180000"

The -XX:MaxRAMPercentage=50.0 is the default that is set and we have to extend it with -Dspring.mvc.async.request-timeout=180000 to increase the async request timeout (value is in ms).

EDIT: The error you will see in clouddriver is Cannot dispatch without an AsyncContext if you have this problem.