spinnaker: `hal --ready` CLI command can not deserialize Halyard health check response
Issue Summary:
The hal --ready command can not deserialize health check response returned by Halyard when a groups field is included in said health check.
Cloud Provider(s):
Kubernetes.
Environment:
We use the Helm chart provided by OpsMx/spinnaker-helm version 2.2.6 to deploy Spinnaker version 1.26.7 and Halyard version 1.45.0 (which is using this Docker image).
Feature Area:
Halyard, hal CLI.
Description:
During efforts yesterday to upgrade Halyard from 1.44.1 to 1.45.0 we noticed a change in the /health endpoint response body from our Halyard pod. Previously, running the following command inside said pod returned:
$ curl http://spinnaker-spinnaker-halyard:8064/health
{
"status" : "UP"
}
Now with version 1.45.0 it returns:
$ curl http://spinnaker-spinnaker-halyard:8064/health
{
"status" : "UP",
"groups" : [ "liveness", "readiness" ]
}
This causes the hal --ready command to error as it expects a Map<String, String>:
hal --ready
! ERROR
com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot
deserialize value of type `java.lang.String` from Array value (token
`JsonToken.START_ARRAY`)
at [Source: (retrofit.ExceptionCatchingTypedInput$ExceptionCatchingInputStream);
line: 3, column: 14] (through reference chain:
java.util.LinkedHashMap["groups"])
? Try the command again with the --debug flag.
This stalls deployment with the OpsMx/spinnaker-helm chart linked before and prevents us from upgrading further.
I would expect hal CLI to be able to parse this response as it only uses the status field’s value to determine healthiness.
Steps to Reproduce:
Unfortunately I could not narrow down the (Spring) configuration to have Halyard return "groups" : [ "liveness", "readiness" ] in its /health response body by running the Docker image linked before locally.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 5
- Comments: 18 (1 by maintainers)
Commits related to this issue
- Disable health probe Disable probe (which is enabled in 1.45+) because it breaks the client See the following links for more info: https://github.com/spinnaker/spinnaker/issues/6655 https://github.co... — committed to jfrabaute/spinnaker-helm by jfrabaute 2 years ago
- Disable health probe Disable probe (which is enabled in 1.45+) because it breaks the client See the following links for more info: https://github.com/spinnaker/spinnaker/issues/6655 https://github.co... — committed to jfrabaute/spinnaker-helm by jfrabaute 2 years ago
In case, using Opsmx helm chart, as a workaround you can fork or copy the chart and remove this line make it work with hal 1.45 and spinnaker 1.27: https://github.com/OpsMx/spinnaker-helm/blob/21c1b4fa02c5f378fdce529afa1772d6e5b95997/charts/spinnaker/templates/configmap/halyard-config.yaml#L14
@Badbond - if this Helm Chart deploy is similar to the original Helm Chart, as a workaround you can exec onto the
spinnaker-install-using-hal-<id>pod and manually run the script to configure and install Spinnaker. The steps below may be slightly different on your pod, even though the Helm Chart looks identical.Copying the install script to home on the pod (allowing modifications),deleting the failing line, then running it will allow Spinnaker to be installed / configured e.g