emissary: istio 1.5 breaks secret handling for mTLS

Describe the bug

istio 1.5 introduces a breaking change to how secrets are handled. Secrets are no longer written into the namespace, which is how ambassador currently consumes the secrets for use in mTLS.

This only applies to STRICT mode in mTLS and only applies if you directly install istio 1.5 as a fresh install via istioctl. If you are upgrading from 1.4 to 1.5 then the old secret infrastructure remains. See https://discuss.istio.io/t/istios-helm-support-in-2020/5535/12 for confirmation of this behavior.

See the upgrade notes for more in-depth explanations of what is changing in regards to secrets.

To Reproduce Steps to reproduce the behavior:

Install istio 1.5 using istioctl as a fresh install. Ensure you are using strict mTLS.
Install Ambassador (any version post 1.0)
Configure TLS by following the istio documentation
Configure a mapping to use the new TLSContext
Attempt to curl that mapping. You will see a ‘connection reset’ error due to the lack of secret availability as the istio sidecar will reject the request.

Expected behavior

Ambassador needs to either change to read via the new grpc secret method or use istio sidecars so that the current deployment solutions continue to work without interruption when using strict mTLS.

Additional context

We will also need to upgrade documentation to provide paths for upgrading istio installations. The decision on the part of the istio team to have different behavior depending on whether you perform a fresh install or are upgrading means that we will need to provide instructions on both paths.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 16 (6 by maintainers)

Commits related to this issue

(from AES) Merge pull request #2587 from datawire/rhs/changelog-fixup Changelog fixup. — committed to emissary-ingress/emissary by deleted user 3 years ago
(from AES) Merge pull request #2587 from datawire/rhs/changelog-fixup Changelog fixup. — committed to emissary-ingress/emissary by deleted user 3 years ago

Most upvoted comments

To add to the comment above I have a feeling that I am also going to be in a similar boat sometime in the near future. We’re heavily invested in both Ambassador and istio but to be honest I’m going to have a tough time convincing people that we shouldn’t ditch Ambassador since istio functionality usage is more widespread.

Unfortunately I don’t have the bandwidth right now to really dive deep into Ambassador source to look for a solution. If that changes in the future I’ll post my findings here.

ppeble on Oct 30, 2020

This should be fixed in Ambassador 1.10. Check out https://www.getambassador.io/docs/latest/howtos/istio/ for the docs. 🙂

kflynn on Jan 4, 2021

I have been investigating on the path of new ambassador and istio installations too in the last two weeks and here’s what I can say. I have been able to run ambassador 1.6.2 with istio 1.6.8 with automatic sidecar injection (label istio.io/rev: canary on ambassador namespace) and these values added to my values.yaml:

env:
  AMBASSADOR_ENVOY_BASE_ID: "1"

podAnnotations:
  proxy.istio.io/config: |
    proxyMetadata:
      OUTPUT_CERTS: /etc/istio-certs
  sidecar.istio.io/userVolume: '[{"name": "istio-certs", "emptyDir": {"medium": "Memory"}}]'
  sidecar.istio.io/userVolumeMount: '[{"name": "istio-certs", "mountPath": "/etc/istio-certs"}]'
  traffic.sidecar.istio.io/includeInboundPorts: ''
  traffic.sidecar.istio.io/includeOutboundIPRanges: ''
  
volumes:
  - name: istio-certs
    emptyDir:
      medium: Memory

volumeMounts:
  - mountPath: /etc/istio-certs/
    name: istio-certs

Everything is running fine for 24 hours, and then comes the trouble:

ambassador/ambassador-57c99fd4b8-hnf2s[istio-proxy]: 2020-08-19T06:24:53.635384Z	info	sds	resource:default pushed key/cert pair to proxy

The cert rotation happens as it should, but the refreshed certificate is not being picked up by ambassador, which results in errors

"upstream_transport_failure_reason":"TLS error: 268436501:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_EXPIRED"

Ambassador still runs with the expired mTLS certificate. It’s pretty clear that the mounted certificate is not getting refreshed as it is rotated by the istio-proxy sidecar… Restarting the pods will indeed fix the issue until the next rotation…

To me, it’s the only piece of the puzzle left to solve to give this migration a real go.

nicolasbelanger on Aug 19, 2020

As shown above the base-id change has been merged. With it you can run both Ambassador and an istio-proxy sidecar, which gets us a step closer. I’m going to continue to test this and then hopefully provide some updated doc once it is fully released.

Another update: It was pointed out by @kflynn that the grpc ‘secret discovery service’ is in fact not an istio-specific thing but rather a piece of envoy functionality: https://www.envoyproxy.io/docs/envoy/latest/configuration/security/secret

I did not realize this! I was assuming we would have to almost reverse-engineer the grpc setup in istio but instead we might be able to leverage existing clients and/or configuration documentation to use built-in envoy functionality. I will have to spend some time researching this option in the near future.

ppeble on May 11, 2020