istio: istiod crashes when using wildcard hostname in Gateway listener (regression from v1.26.0)
Is this the right place to submit this?
- This is not a security vulnerability or a crashing bug
- This is not a question about how to use Istio
Bug Description
Hi,
I’m encountering a crash in istiod when using the Gateway API with a wildcard hostname in the listener.
Here is the Gateway manifest triggering the issue:
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: internal
spec:
gatewayClassName: istio
listeners:
- allowedRoutes:
kinds:
- group: gateway.networking.k8s.io
kind: HTTPRoute
namespaces:
from: All
hostname: '*.<dns>'
name: http
port: 80
protocol: HTTP
This configuration causes istiod to crash from Istio v1.26.0 and onwards (tested with 1.1.0 to 1.3.0 of the Gateway API), regardless of the actual HTTPRoute resources attached. The logs show the following panic:
panic: runtime error: index out of range [1] with length 1
goroutine 118 [running]:
istio.io/istio/pilot/pkg/config/kube/gateway.referenceAllowed({{0x381e200, 0x4001f6c300}, {{{0x385ab80, 0x4000a4c460}, {0x384d450, 0x4000a63950}}, {{0x385ad30, 0x4000a4c7e0}, {0x384d488, 0x4000d604b0}}, ...}}, ...)
istio.io/istio/pilot/pkg/config/kube/gateway/conversion.go:617 +0xac8
istio.io/istio/pilot/pkg/config/kube/gateway.extractParentReferenceInfo.func1(...)
istio.io/istio/pilot/pkg/config/kube/gateway/conversion.go:700
istio.io/istio/pilot/pkg/config/kube/gateway.extractParentReferenceInfo({{0x381e200, 0x4001f6c300}, {{{0x385ab80, 0x4000a4c460}, {0x384d450, 0x4000a63950}}, {{0x385ad30, 0x4000a4c7e0}, {0x384d488, 0x4000d604b0}}, ...}}, ...)
istio.io/istio/pilot/pkg/config/kube/gateway/conversion.go:716 +0x500
istio.io/istio/pilot/pkg/config/kube/gateway.computeRoute[...]({{0x381e200, 0x4001f6c300}, {{{0x385ab80, 0x4000a4c460}, {0x384d450, 0x4000a63950}}, {{0x385ad30, 0x4000a4c7e0}, {0x384d488, 0x4000d604b0}}, ...}}, ...)
istio.io/istio/pilot/pkg/config/kube/gateway/route_collections.go:438 +0x6c
istio.io/istio/pilot/pkg/config/kube/gateway.HTTPRouteCollection.func1({0x381e200, 0x4001f6c300}, 0x4001877980)
istio.io/istio/pilot/pkg/config/kube/gateway/route_collections.go:53 +0x1d4
istio.io/istio/pkg/kube/krt.NewStatusManyCollection[...].func2(0x4001877980?)
istio.io/istio/pkg/kube/krt/status.go:54 +0x44
istio.io/istio/pkg/kube/krt.(*manyCollection[...]).onPrimaryInputEventLocked(0x38d9220, {0x400157c198, 0x1, 0x6b4fbbf77})
istio.io/istio/pkg/kube/krt/collection.go:418 +0x18c
istio.io/istio/pkg/kube/krt.(*manyCollection[...]).onPrimaryInputEvent(0x38d9220, {0x400157c198, 0x1, 0x1})
istio.io/istio/pkg/kube/krt/collection.go:399 +0x140
istio.io/istio/pkg/kube/krt.(*manyCollection[...]).runQueue.func1.1()
istio.io/istio/pkg/kube/krt/collection.go:623 +0x34
istio.io/istio/pkg/queue.(*queueImpl).processNextItem(0x4000252770)
istio.io/istio/pkg/queue/instance.go:141 +0x34
istio.io/istio/pkg/queue.(*queueImpl).Run(0x4000252770, 0x4000743260)
istio.io/istio/pkg/queue/instance.go:180 +0x134
istio.io/istio/pkg/kube/krt.(*manyCollection[...]).runQueue(0x38d9220)
istio.io/istio/pkg/kube/krt/collection.go:631 +0xdc
created by istio.io/istio/pkg/kube/krt.newManyCollection[...] in goroutine 1
istio.io/istio/pkg/kube/krt/collection.go:606 +0x578
It appears the crash happens in referenceAllowed() in conversion.go. Interestingly, the issue does not occur with Istio v1.25.2, and the configuration works as expected.
Workaround: Removing the hostname field from the listener entirely avoids the crash. However, this is not ideal for production use cases that rely on hostname-based routing.
This looks like a regression and istiod should not panic due to a malformed or unexpected hostname value. Let me know if I can provide any additional information to help reproduce or debug this.
Thanks!
Version
$ istioctl version
client version: 1.24.2
control plane version: 1.26.0
data plane version: 1.26.0 (1 proxies)
$ kubectl version
Client Version: v1.32.1
Kustomize Version: v5.5.0
Server Version: v1.32.2
Additional Information
Target cluster context: minikube
Running with the following config:
istio-namespace: istio-system full-secrets: false timeout (mins): 30 include: { } exclude: { Namespaces: kube-node-lease,kube-public,kube-system,local-path-storage } end-time: 2025-05-14 14:51:08.348283 +0200 CEST
Cluster endpoint: https://localhost:53889 CLI version: version.BuildInfo{Version:“1.24.2”, GitRevision:“Homebrew”, GolangVersion:“go1.23.4”, BuildStatus:“Homebrew”, GitTag:“1.24.2”}
The following Istio control plane revisions/versions were found in the cluster: Revision default: &version.MeshInfo{version.ServerInfo{Component:“pilot”, Revision:“default”, Info:version.BuildInfo{Version:“1.26.0”, GitRevision:“c2e9871f340c0e0b114bcd1b73208284f1d17c9e”, GolangVersion:“”, BuildStatus:“Clean”, GitTag:“1.26.0”}}}
The following proxy revisions/versions were found in the cluster: Revision default: Versions {1.26.0}
Fetching logs for the following containers:
gateway/internal-istio/internal-istio-74c9c6bf88-9klrw/istio-proxy istio-system/istiod/istiod-74cfdfbc96-jrb4h/discovery istio-system/istiod/istiod-74cfdfbc96-pwhx9/discovery istio-system/istiod/istiod-c7584855f-pj469/discovery
Fetching Istio control plane information from cluster.
Fetching CNI logs from cluster.
Running Istio analyze on all namespaces and report as below: Analysis Report: Warning [IST0107] (Deployment gateway/internal-istio) Misplaced annotation: networking.istio.io/service-type can only be applied to Gateway Warning [IST0107] (Deployment gateway/internal-istio) Misplaced annotation: proxy.istio.io/config can only be applied to Pod Warning [IST0107] (Pod gateway/internal-istio-74c9c6bf88-9klrw) Misplaced annotation: networking.istio.io/service-type can only be applied to Gateway Warning [IST0107] (Service gateway/internal-istio) Misplaced annotation: networking.istio.io/service-type can only be applied to Gateway Warning [IST0107] (Service gateway/internal-istio) Misplaced annotation: proxy.istio.io/config can only be applied to Pod Info [IST0102] (Namespace default) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace default istio-injection=enabled’ to enable it, or ‘kubectl label namespace default istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0102] (Namespace gateway) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace gateway istio-injection=enabled’ to enable it, or ‘kubectl label namespace gateway istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0102] (Namespace kube-node-lease) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace kube-node-lease istio-injection=enabled’ to enable it, or ‘kubectl label namespace kube-node-lease istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0102] (Namespace kube-public) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace kube-public istio-injection=enabled’ to enable it, or ‘kubectl label namespace kube-public istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0102] (Namespace kube-system) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace kube-system istio-injection=enabled’ to enable it, or ‘kubectl label namespace kube-system istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0102] (Namespace user) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace user istio-injection=enabled’ to enable it, or ‘kubectl label namespace user istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0118] (Service kube-system/kube-dns) Port name metrics (port: 9153, targetPort: 9153) doesn’t follow the naming convention of Istio port. Creating an archive at /Users/NICOLAS/bug-report.tar.gz. Time used for creating the tar file is 101.599166ms. Cleaning up temporary files in /var/folders/by/_27mm6514p11b90nfjd8_v640000gp/T/bug-report. Done.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 3
- Comments: 18 (14 by maintainers)
Ok I was live tweeting earlier so most of it is wrong or irrelevant. But I think @deveshdama and know what the issue is at a high level.
repro
Install istio, install gatewayapi crds, deploy bookinfo, and the following
Then change the the hostname of one of the Gateways from
*.bookinfo.localtobook1.bookinfo.localand apply the config. If istiod hasn’t crashed yet, revert the change you just did and so on and so forth until it crashes.the issue
At a high level, there are two pieces of code
https://github.com/istio/istio/blob/328a06172db5800b997fd96ef9b1550f67720cba/pilot/pkg/model/gateway.go#L564-L582
https://github.com/istio/istio/blob/328a06172db5800b997fd96ef9b1550f67720cba/pilot/pkg/config/kube/gateway/conversion.go#L616-L617
The first line runs when we push config to gateways, the second one when we are building the config.
Fundamentally, the issue is that there are two ways to represent namespaced hostnames:
ns/hostnameandhostnamewhich is equivalent to*/hostname. The latter piece of code only handles thens/hostnameversion.So here is why the repro works and why its so convoluted. The first time the
conversion.gocode runs, all the hostnames are in the correct format. However,sanitizeServerHostNamespacechanges the values of the hostnames in place. When you edit theGatewayobject, now modified hostnames get reused (sometimes) causing the crash.Its still not clear to me why this is so racy, but I think a fix that would solve the issue is to change the latter piece of code to handle both formats of hostnames.
Fixed by https://github.com/istio/istio/pull/56373, will be included in 1.26.1
Thanks @Stevenjin8 for the detailed writeup.
The race occurs because
sanitizeServerHostNamespacemodifies theServer.Hostsslice in-place, and this modified data is reused in subsequent reconciliations.What could be happening is multiple events (e.g., updates to Gateway and HTTPRoute) are queued and processed concurrently, and the order of operations isn’t guaranteed.
For example:
sanitizeServerHostNamespace, modifying hostnames (like changing*/bookinfo.localto justbookinfo.local).referenceAllowed.conversion.gocode assumes all hostnames are in formatnamespace/hostand accesses index 1 after splitting on “/” causing panic.Agreed, the fix seems straightforward - we need to make
referenceAllowedresilient to hostnames with or without namespace prefixes.Sorry for not being able to answer you about this issue
Many thanks @Stevenjin8 and @deveshdama !
ok was able to repro
Yes sure, I got the problem with the following configuration :
It appears it definitely happens when hostnames is defined on HTTPRoute as well as a wildcard hostname is defined on gateway listener
Can you add the YAML for any attached HTTPRoute too?
It looks like the panic is during validation of the HTTPRoute
spec.hostnamesfield and comparing it to valid hostnames on the Gateway parentRef, not just the presence of the hostname field on a listener. There may be some mixup between singular/plural here, HTTPRoutes can specify an array of hostnames, but thehostnamefield on Gateway listeners is singular.