spinnaker: `hal --ready` CLI command can not deserialize Halyard health check response

Issue Summary:

The hal --ready command can not deserialize health check response returned by Halyard when a groups field is included in said health check.

Cloud Provider(s):

Kubernetes.

Environment:

We use the Helm chart provided by OpsMx/spinnaker-helm version 2.2.6 to deploy Spinnaker version 1.26.7 and Halyard version 1.45.0 (which is using this Docker image).

Feature Area:

Halyard, hal CLI.

Description:

During efforts yesterday to upgrade Halyard from 1.44.1 to 1.45.0 we noticed a change in the /health endpoint response body from our Halyard pod. Previously, running the following command inside said pod returned:

$ curl http://spinnaker-spinnaker-halyard:8064/health 
{
  "status" : "UP"
}

Now with version 1.45.0 it returns:

$ curl http://spinnaker-spinnaker-halyard:8064/health 
{
  "status" : "UP",
  "groups" : [ "liveness", "readiness" ]
}

This causes the hal --ready command to error as it expects a Map<String, String>:

hal --ready
! ERROR
  com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot
  deserialize value of type `java.lang.String` from Array value (token
  `JsonToken.START_ARRAY`)
at [Source: (retrofit.ExceptionCatchingTypedInput$ExceptionCatchingInputStream);
  line: 3, column: 14] (through reference chain:
  java.util.LinkedHashMap["groups"])
? Try the command again with the --debug flag.

This stalls deployment with the OpsMx/spinnaker-helm chart linked before and prevents us from upgrading further.

I would expect hal CLI to be able to parse this response as it only uses the status field’s value to determine healthiness.

Steps to Reproduce:

Unfortunately I could not narrow down the (Spring) configuration to have Halyard return "groups" : [ "liveness", "readiness" ] in its /health response body by running the Docker image linked before locally.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 5
  • Comments: 18 (1 by maintainers)

Commits related to this issue

Most upvoted comments

In case, using Opsmx helm chart, as a workaround you can fork or copy the chart and remove this line make it work with hal 1.45 and spinnaker 1.27: https://github.com/OpsMx/spinnaker-helm/blob/21c1b4fa02c5f378fdce529afa1772d6e5b95997/charts/spinnaker/templates/configmap/halyard-config.yaml#L14

@Badbond - if this Helm Chart deploy is similar to the original Helm Chart, as a workaround you can exec onto the spinnaker-install-using-hal-<id> pod and manually run the script to configure and install Spinnaker. The steps below may be slightly different on your pod, even though the Helm Chart looks identical.

Copying the install script to home on the pod (allowing modifications),deleting the failing line, then running it will allow Spinnaker to be installed / configured e.g

cp /opt/halyard/scripts/install.sh ~/copy_install.sh
vi ~/copy_install.sh <delete line 5>
chmod +x ~/copy_install.sh
~/copy_install.sh