spinnaker: Deck will not start with ssl enabled

Issue Summary:

I’ve seen a handful of issues on here in the past, but they’ve all been resolved so I’m opening a new one.

I have a Kubernetes-installed environment with Halyard and am currently using SAML+Okta with no problems. Otherwise a pretty basic configuration. That part works fine, but we need programatic access so I started looking to set up SSL + X.509 as a way to access the environment with Spin (are there any easier paths?)

After this, gate starts up just fine but deck does not.

Cloud Provider(s):

AWS

Environment:

EKS (Kubernetes 1.15) Halyard (Spinnaker 1.18.9)

Feature Area:

Halyard
SSL
X.509

Steps to Reproduce:

Presumably just what is described above:

Install Spinnaker on Kubernetes with Halyard and version 1.18.9
Set up SSL using the documentation
Observe that spin-deck gets stuck in CrashLoopBackOff
Disable ssl for the ui and observe that spin-deck starts up correctly

Additional Details:

I do have an SSL-terminating ELB in front of things (Ingress) but since this is just a startup problem, that shouldn’t be relevant. I just wanted to mention it - and I did make the Ingress annotation changes to proxy SSL traffic upstream to deck/gate.

The entire log from a failed deck pod follows – note there is no information about problems binding to a port, which I noticed a lot of the old issues had.

Considering dependency setenvif for ssl:
Module setenvif already enabled
Considering dependency mime for ssl:
Module mime already enabled
Considering dependency socache_shmcb for ssl:
Enabling module socache_shmcb.
Enabling module ssl.
See /usr/share/doc/apache2/README.Debian.gz on how to configure SSL and create self-signed certificates.
To activate the new configuration, you need to run:
  service apache2 restart
Enabling site spinnaker.
To activate the new configuration, you need to run:
  service apache2 reload
Site 000-default disabled.
To activate the new configuration, you need to run:
  service apache2 reload
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 10.40.163.36. Set the 'ServerName' directive globally to suppress this message
Action '-D FOREGROUND' failed.
The Apache error log may have more information.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 17

Commits related to this issue

Change encryption to sha256 from des3 As per: https://github.com/spinnaker/spinnaker/issues/5730 Deck will not start if we use `des3` as the cipher. — committed to snalvi/docs by snalvi 4 years ago
Update ssl.md This issue was supposed to be resolved in the PR but was not: https://github.com/spinnaker/spinnaker/issues/5730 — committed to snalvi/spinnaker.github.io by snalvi 4 years ago
Update index.md This was supposed to be updated in https://github.com/spinnaker/spinnaker/issues/5730 but was not. — committed to snalvi/spinnaker.github.io by snalvi 4 years ago

Most upvoted comments

Doc updates needed: https://www.spinnaker.io/setup/security/ssl/ - two instances of “openssl x509” https://www.spinnaker.io/setup/security/authentication/x509/ - similarly for client certs for good measure

And for the Armory folks, the same problem is on this page: https://docs.armory.io/spinnaker-install-admin-guides/api-endpoint/#obtain-or-generate-a-server-certificate-for-the-deck-ui-service

jaygorrell on May 4, 2020

If @aleon1220 doesn’t get around to the change, I can make it next week. Thank you so much for raising the issue and providing the solution @jaygorrell 😃 I’ll ping you for when we can close this @bcornils

317brian on May 14, 2020

Thank you so much, that was exactly the issue.

Adding -sha256 to openssl x509 -req calls (specifically to sign Deck’s cert) is all it took.

To verify if this is your issue or not, you can use this:

openssl x509 -in deck.crt -text -noout | grep "Signature Algorithm"
    Signature Algorithm: sha1WithRSAEncryption

You want it to come back with sha256WithRSAEncryption to be valid.

I’m going to leave this open because there’s a clear (and small) documentation miss that needs captured.

jaygorrell on May 3, 2020

I encountered this problem as well, the issue was the certs which were generated for SSL had a weak cipher. Don’t use des3 as per the documentation to sign the certs and ensure your openssl client is updated to the latest (so it’s not using SHA1 to sign).

To see the errors for this you need to exec into the pod and look at the apache error logs - it would say something along the lines of weak cipher.

snalvi on May 3, 2020