istio: Certificate Challenges failing with Istio + Istio Gateway when installed via Helm

Bug Description

I have installed Istio on Azure AKS via Helm and Flux. (base, istiod, gateway)

I am currently trying to migrate to Istio Gateway for exposes our applications to the outside world. Currently we do this via Nginx Ingress Controller (and works fine), but when using Istio Gateway we are getting problems when trying to create Certificates for our sub domains.

My scenario is that I’m currently trying to create a Certificate for the a Concourse url e.g. concourse-istio1.tooling.domain.com. (-istio1 is just to test a cert gets created and I can access the URL, I will be switching to a more better named domain when I know this all works)

Without getting into the details of creating Istio Gateways I am following this guide and by the time I even get to the Gateway part, I can see that the Challenge created doesn’t work due to error.

Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://concourse-istio1.tooling.domain.com/.well-known/acme-challenge/gW6Ky1oqRmLvAzUw....

Now, I’ve read a tonne on people having these problems, and they recommended putting general gateways in front of it, which I’ve done via this

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: general-gateway
  namespace: istio-system
  labels:
    app: gateway
spec:
  selector:
    istio: gateway
  servers:
    - port:
        number: 80
        protocol: HTTP
        name: http
      hosts:
        - "*"

And got to a point where the error I got was waiting for http-01 challenge propagation: wrong status code '404', expected '200'

I then read more and people described that I had to add a VirtualService that took that challenge verification and directed it to the acme-solver-xxxx like so:

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: http-redirect
  namespace: istio-system
spec:
  gateways:
  - general-gateway
  hosts:
  - '*'
  http:
    - match:
        - uri:
            prefix: /.well-known/acme-challenge/ 
      route:
        - destination:
            port:
              number: 8089
            host: cm-acme-http-solver-nvcvc.istio-system.svc.cluster.local

Which is a terrible solution for automation as we’re expected to know the pod name ahead of time meaning manual intervention (and we want all of this to be done automatically). But the error I get from this is waiting for http-01 challenge propagation: wrong status code 503 expected '200 which I’m assuming is down to some mTLS setting. But again, I don’t understand how and why I’m expected to do all of this routing? Especially when the Istio Guide on Cert-manager doesn’t include any of the above. I’ve also read many many examples of multiple people doing multiple things and I’ve tried many combinations and just cannot get it to work. Many of which are examples on old versions of Istio where stuff like global.k8sIngress.enabled aren’t even options anymore as I’m using the newer istio/gateway Helm chart.

Issuer yaml for brevity alongside the certificate.

Certificate

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: concourse-ingress-cert
  namespace: istio-system
spec:
  secretName: concourse-ingress-cert
  dnsNames:
  - concourse-istio1.domain.com
  issuerRef:
    name: letsencrypt-prod
    kind: Issuer
    group: cert-manager.io

Issuer

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-prod
  namespace: istio-system
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: blah.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      # An empty 'selector' means that this solver matches all domains
      - selector: {}
        http01:
          ingress:
            class: istio

And IngressClass because apparently I read somewhere I needed it as the old ingress.class annotation is being deprecated.

IngressClass

---
apiVersion: networking.k8s.io/v1beta1
kind: IngressClass
metadata:
  name: istio
  namespace: istio-system
spec:
  controller: istio.io/ingress-controller

Cert-manager HelmRelease

---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  name: jetstack
  namespace: cert-manager
spec:
  interval: 1m
  url: https://charts.jetstack.io
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: cert-manager
  namespace: cert-manager
spec:
  chart:
    spec:
      chart: cert-manager
      sourceRef:
        kind: HelmRepository
        name: jetstack
  interval: 1m
  values:
    version: v1.7.1
    installCRDs: true

Not sure why all of this is getting superfluous, I’m hoping it’s a simple thing to do, but at the moment, it seems that there’s a lack of documentation on this, (something I’m happy to add - but I need to get it working first) especially in an area that seems to attract so many issues.

EDIT:

I will add the Flux config for how I’m installing Istio, in case there is a config variable I am missing.

HelmRelease for IstioBase and IstioD

---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  name: istio-system
  namespace: istio-system
spec:
  interval: 1m
  timeout: 2m0s
  url: https://istio-release.storage.googleapis.com/charts
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: istio-base
  namespace: istio-system
spec:
  chart:
    spec:
      chart: base
      version: 1.12.2
      sourceRef:
        kind: HelmRepository
        name: istio-system
  interval: 1m
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: istiod
  namespace: istio-system
spec:
  chart:
    spec:
      chart: istiod
      version: 1.12.2
      sourceRef:
        kind: HelmRepository
        name: istio-system
  interval: 1m

HelmRelease for Istio Gateway (Ingress)

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: istio-gateway
  namespace: istio-system
spec:
  chart:
    spec:
      chart: gateway
      version: 1.12.2
      sourceRef:
        kind: HelmRepository
        name: istio-system
  interval: 1m
  values:
    service: 
      loadBalancerIP: [our Azure AKS LB Public IP]

Version

> istioctl version                              
client version: 1.12.2
control plane version: 1.12.2
data plane version: 1.12.2 (32 proxies)

> kubectl version --short                       
Client Version: v1.21.4
Server Version: v1.21.9

Additional Information

No response