istio: istiod crashes when using wildcard hostname in Gateway listener (regression from v1.26.0)

Is this the right place to submit this?

  • This is not a security vulnerability or a crashing bug
  • This is not a question about how to use Istio

Bug Description

Hi,

I’m encountering a crash in istiod when using the Gateway API with a wildcard hostname in the listener.

Here is the Gateway manifest triggering the issue:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: internal
spec:
  gatewayClassName: istio
  listeners:
  - allowedRoutes:
      kinds:
      - group: gateway.networking.k8s.io
        kind: HTTPRoute
      namespaces:
        from: All
    hostname: '*.<dns>'
    name: http
    port: 80
    protocol: HTTP

This configuration causes istiod to crash from Istio v1.26.0 and onwards (tested with 1.1.0 to 1.3.0 of the Gateway API), regardless of the actual HTTPRoute resources attached. The logs show the following panic:

panic: runtime error: index out of range [1] with length 1

goroutine 118 [running]:
istio.io/istio/pilot/pkg/config/kube/gateway.referenceAllowed({{0x381e200, 0x4001f6c300}, {{{0x385ab80, 0x4000a4c460}, {0x384d450, 0x4000a63950}}, {{0x385ad30, 0x4000a4c7e0}, {0x384d488, 0x4000d604b0}}, ...}}, ...)
        istio.io/istio/pilot/pkg/config/kube/gateway/conversion.go:617 +0xac8
istio.io/istio/pilot/pkg/config/kube/gateway.extractParentReferenceInfo.func1(...)
        istio.io/istio/pilot/pkg/config/kube/gateway/conversion.go:700
istio.io/istio/pilot/pkg/config/kube/gateway.extractParentReferenceInfo({{0x381e200, 0x4001f6c300}, {{{0x385ab80, 0x4000a4c460}, {0x384d450, 0x4000a63950}}, {{0x385ad30, 0x4000a4c7e0}, {0x384d488, 0x4000d604b0}}, ...}}, ...)
        istio.io/istio/pilot/pkg/config/kube/gateway/conversion.go:716 +0x500
istio.io/istio/pilot/pkg/config/kube/gateway.computeRoute[...]({{0x381e200, 0x4001f6c300}, {{{0x385ab80, 0x4000a4c460}, {0x384d450, 0x4000a63950}}, {{0x385ad30, 0x4000a4c7e0}, {0x384d488, 0x4000d604b0}}, ...}}, ...)
        istio.io/istio/pilot/pkg/config/kube/gateway/route_collections.go:438 +0x6c
istio.io/istio/pilot/pkg/config/kube/gateway.HTTPRouteCollection.func1({0x381e200, 0x4001f6c300}, 0x4001877980)
        istio.io/istio/pilot/pkg/config/kube/gateway/route_collections.go:53 +0x1d4
istio.io/istio/pkg/kube/krt.NewStatusManyCollection[...].func2(0x4001877980?)
        istio.io/istio/pkg/kube/krt/status.go:54 +0x44
istio.io/istio/pkg/kube/krt.(*manyCollection[...]).onPrimaryInputEventLocked(0x38d9220, {0x400157c198, 0x1, 0x6b4fbbf77})
        istio.io/istio/pkg/kube/krt/collection.go:418 +0x18c
istio.io/istio/pkg/kube/krt.(*manyCollection[...]).onPrimaryInputEvent(0x38d9220, {0x400157c198, 0x1, 0x1})
        istio.io/istio/pkg/kube/krt/collection.go:399 +0x140
istio.io/istio/pkg/kube/krt.(*manyCollection[...]).runQueue.func1.1()
        istio.io/istio/pkg/kube/krt/collection.go:623 +0x34
istio.io/istio/pkg/queue.(*queueImpl).processNextItem(0x4000252770)
        istio.io/istio/pkg/queue/instance.go:141 +0x34
istio.io/istio/pkg/queue.(*queueImpl).Run(0x4000252770, 0x4000743260)
        istio.io/istio/pkg/queue/instance.go:180 +0x134
istio.io/istio/pkg/kube/krt.(*manyCollection[...]).runQueue(0x38d9220)
        istio.io/istio/pkg/kube/krt/collection.go:631 +0xdc
created by istio.io/istio/pkg/kube/krt.newManyCollection[...] in goroutine 1
        istio.io/istio/pkg/kube/krt/collection.go:606 +0x578

It appears the crash happens in referenceAllowed() in conversion.go. Interestingly, the issue does not occur with Istio v1.25.2, and the configuration works as expected.

Workaround: Removing the hostname field from the listener entirely avoids the crash. However, this is not ideal for production use cases that rely on hostname-based routing.

This looks like a regression and istiod should not panic due to a malformed or unexpected hostname value. Let me know if I can provide any additional information to help reproduce or debug this.

Thanks!

Version

$ istioctl version                                             
client version: 1.24.2
control plane version: 1.26.0
data plane version: 1.26.0 (1 proxies)
$ kubectl version                                                                              
Client Version: v1.32.1
Kustomize Version: v5.5.0
Server Version: v1.32.2

Additional Information

Target cluster context: minikube

Running with the following config:

istio-namespace: istio-system full-secrets: false timeout (mins): 30 include: { } exclude: { Namespaces: kube-node-lease,kube-public,kube-system,local-path-storage } end-time: 2025-05-14 14:51:08.348283 +0200 CEST

Cluster endpoint: https://localhost:53889 CLI version: version.BuildInfo{Version:“1.24.2”, GitRevision:“Homebrew”, GolangVersion:“go1.23.4”, BuildStatus:“Homebrew”, GitTag:“1.24.2”}

The following Istio control plane revisions/versions were found in the cluster: Revision default: &version.MeshInfo{version.ServerInfo{Component:“pilot”, Revision:“default”, Info:version.BuildInfo{Version:“1.26.0”, GitRevision:“c2e9871f340c0e0b114bcd1b73208284f1d17c9e”, GolangVersion:“”, BuildStatus:“Clean”, GitTag:“1.26.0”}}}

The following proxy revisions/versions were found in the cluster: Revision default: Versions {1.26.0}

Fetching logs for the following containers:

gateway/internal-istio/internal-istio-74c9c6bf88-9klrw/istio-proxy istio-system/istiod/istiod-74cfdfbc96-jrb4h/discovery istio-system/istiod/istiod-74cfdfbc96-pwhx9/discovery istio-system/istiod/istiod-c7584855f-pj469/discovery

Fetching Istio control plane information from cluster.

Fetching CNI logs from cluster.

Running Istio analyze on all namespaces and report as below: Analysis Report: Warning [IST0107] (Deployment gateway/internal-istio) Misplaced annotation: networking.istio.io/service-type can only be applied to Gateway Warning [IST0107] (Deployment gateway/internal-istio) Misplaced annotation: proxy.istio.io/config can only be applied to Pod Warning [IST0107] (Pod gateway/internal-istio-74c9c6bf88-9klrw) Misplaced annotation: networking.istio.io/service-type can only be applied to Gateway Warning [IST0107] (Service gateway/internal-istio) Misplaced annotation: networking.istio.io/service-type can only be applied to Gateway Warning [IST0107] (Service gateway/internal-istio) Misplaced annotation: proxy.istio.io/config can only be applied to Pod Info [IST0102] (Namespace default) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace default istio-injection=enabled’ to enable it, or ‘kubectl label namespace default istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0102] (Namespace gateway) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace gateway istio-injection=enabled’ to enable it, or ‘kubectl label namespace gateway istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0102] (Namespace kube-node-lease) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace kube-node-lease istio-injection=enabled’ to enable it, or ‘kubectl label namespace kube-node-lease istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0102] (Namespace kube-public) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace kube-public istio-injection=enabled’ to enable it, or ‘kubectl label namespace kube-public istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0102] (Namespace kube-system) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace kube-system istio-injection=enabled’ to enable it, or ‘kubectl label namespace kube-system istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0102] (Namespace user) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace user istio-injection=enabled’ to enable it, or ‘kubectl label namespace user istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0118] (Service kube-system/kube-dns) Port name metrics (port: 9153, targetPort: 9153) doesn’t follow the naming convention of Istio port. Creating an archive at /Users/NICOLAS/bug-report.tar.gz. Time used for creating the tar file is 101.599166ms. Cleaning up temporary files in /var/folders/by/_27mm6514p11b90nfjd8_v640000gp/T/bug-report. Done.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 3
  • Comments: 18 (14 by maintainers)

Most upvoted comments

Ok I was live tweeting earlier so most of it is wrong or irrelevant. But I think @deveshdama and know what the issue is at a high level.

repro

Install istio, install gatewayapi crds, deploy bookinfo, and the following

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: bookinfo-gateway
spec:
  gatewayClassName: istio
  listeners:
  - name: http
    hostname: "*.bookinfo.local"
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: All
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: bookinfo-gateway2
spec:
  gatewayClassName: istio
  listeners:
  - name: http
    hostname: "*.bookinfo.local"
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: All
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: bookinfo
spec:
  hostnames:
  - "book1.bookinfo.local"
  parentRefs:
  - name: bookinfo-gateway
  - name: bookinfo-gateway2
  rules:
  - backendRefs:
    - name: productpage
      port: 9080

Then change the the hostname of one of the Gateways from *.bookinfo.local to book1.bookinfo.local and apply the config. If istiod hasn’t crashed yet, revert the change you just did and so on and so forth until it crashes.

the issue

At a high level, there are two pieces of code

https://github.com/istio/istio/blob/328a06172db5800b997fd96ef9b1550f67720cba/pilot/pkg/model/gateway.go#L564-L582

https://github.com/istio/istio/blob/328a06172db5800b997fd96ef9b1550f67720cba/pilot/pkg/config/kube/gateway/conversion.go#L616-L617

The first line runs when we push config to gateways, the second one when we are building the config.

Fundamentally, the issue is that there are two ways to represent namespaced hostnames: ns/hostname and hostname which is equivalent to */hostname. The latter piece of code only handles the ns/hostname version.

So here is why the repro works and why its so convoluted. The first time the conversion.go code runs, all the hostnames are in the correct format. However, sanitizeServerHostNamespace changes the values of the hostnames in place. When you edit the Gateway object, now modified hostnames get reused (sometimes) causing the crash.

Its still not clear to me why this is so racy, but I think a fix that would solve the issue is to change the latter piece of code to handle both formats of hostnames.

Fixed by https://github.com/istio/istio/pull/56373, will be included in 1.26.1

Thanks @Stevenjin8 for the detailed writeup.

The race occurs because sanitizeServerHostNamespace modifies the Server.Hosts slice in-place, and this modified data is reused in subsequent reconciliations.

What could be happening is multiple events (e.g., updates to Gateway and HTTPRoute) are queued and processed concurrently, and the order of operations isn’t guaranteed.

For example:

  1. The Gateway update triggers a reconciliation that runs sanitizeServerHostNamespace, modifying hostnames (like changing */bookinfo.local to just bookinfo.local).
  2. Before the updated configuration is fully propagated, the HTTPRoute reconciliation runs referenceAllowed.
  3. The conversion.go code assumes all hostnames are in format namespace/host and accesses index 1 after splitting on “/” causing panic.

Agreed, the fix seems straightforward - we need to make referenceAllowed resilient to hostnames with or without namespace prefixes.

Sorry for not being able to answer you about this issue

Many thanks @Stevenjin8 and @deveshdama !

ok was able to repro

Yes sure, I got the problem with the following configuration :

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  annotations:
    networking.istio.io/service-type: ClusterIP
    proxy.istio.io/config: '{"gatewayTopology":{"numTrustedProxies":1}}'
  name: internal
  namespace: gateway
spec:
  gatewayClassName: istio
  listeners:
  - allowedRoutes:
      kinds:
      - group: gateway.networking.k8s.io
        kind: HTTPRoute
      namespaces:
        from: All
    hostname: '*.bookinfo.local'
    name: http
    port: 80
    protocol: HTTP
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: bookinfo-1
  namespace: gateway
spec:
  hostnames:
    - book1.bookinfo.local
  parentRefs:
    - name: internal
      namespace: gateway
  rules:
    - backendRefs:
        - name: book1
          port: 8080

It appears it definitely happens when hostnames is defined on HTTPRoute as well as a wildcard hostname is defined on gateway listener

Can you add the YAML for any attached HTTPRoute too?

It looks like the panic is during validation of the HTTPRoute spec.hostnames field and comparing it to valid hostnames on the Gateway parentRef, not just the presence of the hostname field on a listener. There may be some mixup between singular/plural here, HTTPRoutes can specify an array of hostnames, but the hostname field on Gateway listeners is singular.