cilium: Ingress: incompatible with cert-manager ACME HTTP-01

Is there an existing issue for this?

• I have searched the existing issues

What happened?

With cilium 1.14.3, set the default ingress class to shared. Create a basic ClusterIssuer for the Let’s Encrypt ACME HTTP01 challenge. Create an Ingress with the appropriate annotation and spec.tls setup properly. Cilium Install

export CILIUM_VERSION="v1.14.3"
helm upgrade --install \
    cilium \
    cilium/cilium \
    --version ${CILIUM_VERSION} \
    --namespace kube-system \
    --set ipam.mode=kubernetes \
    --set kubeProxyReplacement=true \
    --set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
    --set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
    --set bpf.autoMount.enabled=true \
    --set cgroup.autoMount.enabled=false \
    --set cgroup.hostRoot=/sys/fs/cgroup \
    --set k8sServiceHost=localhost \
    --set k8sServicePort=7445 \
    --set bpf.masquerade=true \
    --set bandwidthManager.enabled=true \
    --set bandwidthManager.bbr=true \
    --set prometheus.enabled=true \
    --set operator.prometheus.enabled=true \
    --set hubble.relay.enabled=true \
    --set hubble.ui.enabled=true \
    --set hubble.enabled=true \
    --set hubble.metrics.enableOpenMetrics=true \
    --set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,httpV2:exemplars=true;labelsContext=source_ip\,source_namespace\,source_workload\,destination_ip\,destination_namespace\,destination_workload\,traffic_direction}" \
    --set ingressController.enabled=true \
    --set ingressController.default=true \
    --set ingressController.loadbalancerMode=shared \
    --set ingressController.service.allocateLoadBalancerNodePorts="true" \
    --set ingressController.service.loadBalancerIP="10.0.0.153" \
    --set l2announcements.enabled=true \
    --set l2podAnnouncements.enabled=true \
    --set l2podAnnouncements.interface=eth0 \
    --set k8sClientRateLimit.qps=${QPS} \
    --set k8sClientRateLimit.burst=${BURST} \
    --set envoy.enabled=true \
    --set envoy.prometheus.enabled=true \
    --set routingMode=native \
    --set ipv4NativeRoutingCIDR=10.0.0.0/8 \
    --set policyEnforcementMode=default \
    --set debug.enabled=false \
    --set autoDirectNodeRoutes=true \
    --set hostFirewall.enabled=true \
    --set loadBalancer.algorithm=maglev \
    --set maglev.tableSize=65521 \
    --set maglev.hashSeed=${SEED} \
    --set loadBalancer.mode=dsr \
    --set loadBalancer.dsrDispatch=opt \
    --values <(cat <<EOF
ingressController:
  service:
    labels:
      cilium.loadbalancer.ips.service/name: ingress-gateway-pool
EOF
)

Cert-manager install

export CERT_MANAGER_VERSION="v1.13.1"
helm upgrade --install cert-manager jetstack/cert-manager \
    --namespace cert-manager \
    --create-namespace \
    --set installCRDs=true \
    --version $CERT_MANAGER_VERSION  \
    --set "extraArgs={--feature-gates=ExperimentalGatewayAPISupport=true,--feature-gates=AdditionalCertificateOutputFormats=true}" \
    --set webhook.extraArgs={--feature-gates="AdditionalCertificateOutputFormats=true"}

ZeroSSL Cluster Issuer

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: cilium-zerossl
spec:
  acme:
    # ZeroSSL ACME server
    server: https://acme.zerossl.com/v2/DV90
    email: ${CLUSTER_ISSUER_EMAIL}

    privateKeySecretRef:
      name: zerossl-private-key

    externalAccountBinding:
      keyID: ${ZERO_SSL_EAB_KEY_ID}
      keySecretRef:
        name: zerossl-eab-secret
        key: secret

    # ACME HTTP01 Ingress solver
    solvers:
    - http01:
        ingress:
          ingressClassName: cilium

LB Pool and Announcement Policy (No BGP)

apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
  name: ingress-gateway-pool
spec:
  disabled: false
  cidrs:
    - cidr: 10.0.0.152/30
  serviceSelector:
    matchLabels:
      cilium.loadbalancer.ips.service/name: ingress-gateway-pool
---
apiVersion: cilium.io/v2alpha1
kind: CiliumL2AnnouncementPolicy
metadata:
  name: ingress-gateway-pool-announcement-policy
spec:
  loadBalancerIPs: true
  interfaces:
  - eth0
  serviceSelector:
    matchLabels:
      cilium.loadbalancer.ips.service/name: ingress-gateway-pool

Ingress Service

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ${INGRESS_NAME}
  namespace: ${NAMESPACE}
  annotations:
    cert-manager.io/cluster-issuer: cilium-zerossl
spec:
  ingressClassName: cilium
  rules:
  - host: ${SERVICE_HOST}
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: ${SERVICE_NAME}
              port:
                number: 3001
  tls:
  - hosts:
    - ${SERVICE_HOST}
    secretName: cilium-zerossl-${INGRESS_NAME}-tls

Observe as cert manager spins in a loop and creates hundreds of ingresses.

Cilium Version

cilium-cli: v0.15.11 compiled with go1.21.3 on linux/amd64 cilium image (default): v1.14.2 cilium image (stable): v1.14.3 cilium image (running): 1.14.3

Kernel Version

Linux: 6.1.58

Kubernetes Version

Client Version: v1.28.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.3

Sysdump

No response

Relevant log output

No response

Anything else?

No response

Code of Conduct

• I agree to follow this project’s Code of Conduct