istio: istioctl bug-report include filtering logic not working with namespace and pod filtering

Is this the right place to submit this?

• This is not a security vulnerability or a crashing bug
• This is not a question about how to use Istio

Bug Description

When trying to gather information about specific pods within a namespace from multiple namespaces, istioctl bug-report does not properly get all proxy information.

This is important because some environments we work with have over 600 services in a single cluster. We would like to grab the istio proxy information of just the control plane, gateways and workload pod having issues. Using the filtering logic in istioctl bug-report does not allow us to do that in a single command or we have to grab proxy information and logs from ALL pods running in the namespaces.

Use cases

• istioctl bug-report - grabs all information as expected

> istioctl bug-report
Running with the following config: 

istio-namespace: istio-system
full-secrets: false
timeout (mins): 30
include: {  }
exclude: { Namespaces: kube-node-lease,kube-public,kube-system,local-path-storage }
end-time: 2023-10-23 07:58:20.105709179 -0500 CDT

Cluster endpoint: https://0.0.0.0:43387
CLI version:
version.BuildInfo{Version:"1.19.3", GitRevision:"41218ffa66c12acf77ad08b3ef07d296218d0929", GolangVersion:"go1.21.3", BuildStatus:"Clean", GitTag:"1.19.3"}

The following Istio control plane revisions/versions were found in the cluster:
Revision default:
&version.MeshInfo{
    {
        Component: "pilot",
        Revision:  "default",
        Info:      version.BuildInfo{Version:"1.17.1", GitRevision:"7d6d2adacf8dcee110a48450d537f8ad26c7225f", GolangVersion:"", BuildStatus:"Clean", GitTag:"1.17.1"},
    },
}

The following proxy revisions/versions were found in the cluster:
Revision default: Versions {1.17.1}

Fetching logs for the following containers:

default/details-v1/details-v1-7d4d9d5fcb-xv545/details
default/details-v1/details-v1-7d4d9d5fcb-xv545/istio-proxy
default/productpage-v1/productpage-v1-66756cddfd-lsmq5/istio-proxy
default/productpage-v1/productpage-v1-66756cddfd-lsmq5/productpage
default/ratings-v1/ratings-v1-85cc46b6d4-d7r57/istio-proxy
default/ratings-v1/ratings-v1-85cc46b6d4-d7r57/ratings
default/reviews-v1/reviews-v1-777df99c6d-46kt4/istio-proxy
default/reviews-v1/reviews-v1-777df99c6d-46kt4/reviews
default/reviews-v2/reviews-v2-cdd8fb88b-gp229/istio-proxy
default/reviews-v2/reviews-v2-cdd8fb88b-gp229/reviews
default/reviews-v3/reviews-v3-58b6479b-lx2xg/istio-proxy
default/reviews-v3/reviews-v3-58b6479b-lx2xg/reviews
istio-system/istio-egressgateway/istio-egressgateway-8694db4556-6sw9q/istio-proxy
istio-system/istio-ingressgateway/istio-ingressgateway-8568ffc4d4-6jwvt/istio-proxy
istio-system/istiod/istiod-76cf8b7b8b-dwstt/discovery

Fetching Istio control plane information from cluster.

• istioctl bug-report --include '"default/product*"' - grabs just the proxy information for productpage which is expected

▶ istioctl bug-report --include '"default/product*"'
Running with the following config: 

istio-namespace: istio-system
full-secrets: false
timeout (mins): 30
include: { Namespaces: default/Deployments: product* }
exclude: { Namespaces: kube-node-lease,kube-public,kube-system,local-path-storage }
end-time: 2023-10-23 08:00:50.98426119 -0500 CDT

Cluster endpoint: https://0.0.0.0:43387
CLI version:
version.BuildInfo{Version:"1.19.3", GitRevision:"41218ffa66c12acf77ad08b3ef07d296218d0929", GolangVersion:"go1.21.3", BuildStatus:"Clean", GitTag:"1.19.3"}

The following Istio control plane revisions/versions were found in the cluster:
The following proxy revisions/versions were found in the cluster:


Fetching logs for the following containers:

default/productpage-v1/productpage-v1-66756cddfd-lsmq5/istio-proxy
default/productpage-v1/productpage-v1-66756cddfd-lsmq5/productpage

Fetching Istio control plane information from cluster.

• istioctl bug-report --include '"istio-system,default/product*"' - Includes istio-system and product page pods, does not propertly grab information from istio-system namespace desipte the include looking correct include: { Namespaces: istio-system,default/Deployments: product* }

istioctl bug-report --include '"istio-system,default/product*"'

Running with the following config: 

istio-namespace: istio-system
full-secrets: false
timeout (mins): 30
include: { Namespaces: istio-system,default/Deployments: product* }
exclude: { Namespaces: kube-node-lease,kube-public,kube-system,local-path-storage }
end-time: 2023-10-23 08:02:26.659024959 -0500 CDT

Cluster endpoint: https://0.0.0.0:43387
CLI version:
version.BuildInfo{Version:"1.19.3", GitRevision:"41218ffa66c12acf77ad08b3ef07d296218d0929", GolangVersion:"go1.21.3", BuildStatus:"Clean", GitTag:"1.19.3"}

The following Istio control plane revisions/versions were found in the cluster:
The following proxy revisions/versions were found in the cluster:


Fetching logs for the following containers:

default/productpage-v1/productpage-v1-66756cddfd-lsmq5/istio-proxy
default/productpage-v1/productpage-v1-66756cddfd-lsmq5/productpage

Fetching Istio control plane information from cluster.


Fetching CNI logs from cluster.

Running Istio analyze on all namespaces and report as below:
Analysis Report:

Creating an archive at /home/nick/work/demo/istio/bug-report-logs-bug/bug-report.tar.gz.
Time used for creating the tar file is 19.341403ms.
Cleaning up temporary files in /tmp/bug-report.
Done.

• istioctl bug-report --include '"istio-system,default"' - Using namespaces and no pod filtering. Works as expected

istioctl bug-report --include '"istio-system,default"' 

Running with the following config: 

istio-namespace: istio-system
full-secrets: false
timeout (mins): 30
include: { Namespaces: istio-system,default }
exclude: { Namespaces: kube-node-lease,kube-public,kube-system,local-path-storage }
end-time: 2023-10-23 08:05:02.256065995 -0500 CDT

Cluster endpoint: https://0.0.0.0:43387
CLI version:
version.BuildInfo{Version:"1.19.3", GitRevision:"41218ffa66c12acf77ad08b3ef07d296218d0929", GolangVersion:"go1.21.3", BuildStatus:"Clean", GitTag:"1.19.3"}

The following Istio control plane revisions/versions were found in the cluster:
Revision default:
&version.MeshInfo{
    {
        Component: "pilot",
        Revision:  "default",
        Info:      version.BuildInfo{Version:"1.17.1", GitRevision:"7d6d2adacf8dcee110a48450d537f8ad26c7225f", GolangVersion:"", BuildStatus:"Clean", GitTag:"1.17.1"},
    },
}

The following proxy revisions/versions were found in the cluster:
Revision default: Versions {1.17.1}


Fetching logs for the following containers:

default/details-v1/details-v1-7d4d9d5fcb-xv545/details
default/details-v1/details-v1-7d4d9d5fcb-xv545/istio-proxy
default/productpage-v1/productpage-v1-66756cddfd-lsmq5/istio-proxy
default/productpage-v1/productpage-v1-66756cddfd-lsmq5/productpage
default/ratings-v1/ratings-v1-85cc46b6d4-d7r57/istio-proxy
default/ratings-v1/ratings-v1-85cc46b6d4-d7r57/ratings
default/reviews-v1/reviews-v1-777df99c6d-46kt4/istio-proxy
default/reviews-v1/reviews-v1-777df99c6d-46kt4/reviews
default/reviews-v2/reviews-v2-cdd8fb88b-gp229/istio-proxy
default/reviews-v2/reviews-v2-cdd8fb88b-gp229/reviews
default/reviews-v3/reviews-v3-58b6479b-lx2xg/istio-proxy
default/reviews-v3/reviews-v3-58b6479b-lx2xg/reviews
istio-system/istio-egressgateway/istio-egressgateway-8694db4556-6sw9q/istio-proxy
istio-system/istio-ingressgateway/istio-ingressgateway-8568ffc4d4-6jwvt/istio-proxy
istio-system/istiod/istiod-76cf8b7b8b-dwstt/discovery

Fetching Istio control plane information from cluster.

• istioctl bug-report --include '"istio-system/*,default/*"' - Multiple namespaces with pod filtering does not appear to work as well. it only matches the pods from istio-system

istioctl bug-report --include '"istio-system/*,default/*"'

Running with the following config: 

istio-namespace: istio-system
full-secrets: false
timeout (mins): 30
include: { Namespaces: istio-system/Deployments: *,default }
exclude: { Namespaces: kube-node-lease,kube-public,kube-system,local-path-storage }
end-time: 2023-10-23 08:12:52.858727062 -0500 CDT

Cluster endpoint: https://0.0.0.0:43387
CLI version:
version.BuildInfo{Version:"1.19.3", GitRevision:"41218ffa66c12acf77ad08b3ef07d296218d0929", GolangVersion:"go1.21.3", BuildStatus:"Clean", GitTag:"1.19.3"}

The following Istio control plane revisions/versions were found in the cluster:
Revision default:
&version.MeshInfo{
    {
        Component: "pilot",
        Revision:  "default",
        Info:      version.BuildInfo{Version:"1.17.1", GitRevision:"7d6d2adacf8dcee110a48450d537f8ad26c7225f", GolangVersion:"", BuildStatus:"Clean", GitTag:"1.17.1"},
    },
}

The following proxy revisions/versions were found in the cluster:
Revision default: Versions {1.17.1}


Fetching logs for the following containers:

istio-system/istio-egressgateway/istio-egressgateway-8694db4556-6sw9q/istio-proxy
istio-system/istio-ingressgateway/istio-ingressgateway-8568ffc4d4-6jwvt/istio-proxy
istio-system/istiod/istiod-76cf8b7b8b-dwstt/discovery

Fetching Istio control plane information from cluster.


Fetching CNI logs from cluster.

Running Istio analyze on all namespaces and report as below:
Analysis Report:

Creating an archive at /home/nick/work/demo/istio/bug-report-logs-bug/bug-report.tar.gz.
Time used for creating the tar file is 25.811298ms.
Cleaning up temporary files in /tmp/bug-report.
Done.

Steps to repeat

1. istioctl install --set profile=demo -y
2. Deploy bookinfo kubectl label namespace default istio-injection=enabled && kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml
3. Run istioctl bug-report --include '"istio-system,default/product*"' and observe that istio-system pods are not included

Version

istioctl version
client version: 1.19.3