istio: Remote secrets are not updated

Is this the right place to submit this?

  • This is not a security vulnerability or a crashing bug
  • This is not a question about how to use Istio

Bug Description

I found remote secrets are not updated in sporadic manner. Whenever it happens, restarting one of the clusters solves this, but I have no idea why this happens and how do I resolve this without restart.

$ kubectl get secret istio-remote-secret-cluster1
NAME                           TYPE     DATA   AGE
istio-remote-secret-cluster1   Opaque   1      16m

Additionally, istiod is noticed the creation of remote secret, but it has no error or related logs why this not working.

2024-03-07T07:09:04.420890Z	info	ads	Push debounce stable[10] 1 for config Secret/default/istio-remote-secret-cluster1: 100.268838ms since last change, 100.268537ms since last push, full=false
2024-03-07T07:09:04.420989Z	info	ads	XDS: Incremental Pushing ConnectedEndpoints:2 Version:2024-03-07T07:08:52Z/8

Version

$ istioctl version
client version: 1.20.2
control plane version: 1.20.2
data plane version: 1.20.2 (2 proxies)
$ kubectl version                                                           
Client Version: v1.29.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.26.11

Additional Information

Target cluster context: kind-local

Running with the following config:

istio-namespace: istio-system
full-secrets: false
timeout (mins): 30
include: {  }
exclude: { Namespaces: kube-node-lease,kube-public,kube-system,local-path-storage }
end-time: 2024-03-07 07:24:07.371986067 +0000 UTC



Cluster endpoint: https://127.0.0.1:6443
CLI version:
version.BuildInfo{Version:"1.20.2", GitRevision:"5f5d657c72d30a97cae97938de3a6831583e9f15", GolangVersion:"go1.21.5", BuildStatus:"Clean", GitTag:"1.20.2"}

The following Istio control plane revisions/versions were found in the cluster:
Revision default:
&version.MeshInfo{
    {
        Component: "pilot",
        Revision:  "default",
        Info:      version.BuildInfo{Version:"1.20.2", GitRevision:"5f5d657c72d30a97cae97938de3a6831583e9f15", GolangVersion:"", BuildStatus:"Clean", GitTag:"1.20.2"},
    },
}

The following proxy revisions/versions were found in the cluster:
Revision default: Versions {1.20.2}


Fetching logs for the following containers:

hc-telemetry/hc-telemetry-collector-k8s/hc-telemetry-collector-k8s-7d645994b7-g82gb/hc-telemetry-collector
hc-telemetry/hc-telemetry-collector-prometheus/hc-telemetry-collector-prometheus-8b56bf7c4-j4d2g/hc-telemetry-collector
istio-system/istio-eastwestgateway/istio-eastwestgateway-7cbc7b94b8-qhnxq/istio-proxy
istio-system/istio-ingressgateway/istio-ingressgateway-6548f96c45-9cw7v/istio-proxy
istio-system/istiod/istiod-776d9d457f-dc9bv/discovery

Fetching Istio control plane information from cluster.


Fetching CNI logs from cluster.

Running Istio analyze on all namespaces and report as below:
Analysis Report:
Info [IST0102] (Namespace default) The namespace is not enabled for Istio injection. Run 'kubectl label namespace default istio-injection=enabled' to enable it, or 'kubectl label namespace default istio-injection=disabled' to explicitly mark it as not needing injection.
Info [IST0102] (Namespace hc-telemetry) The namespace is not enabled for Istio injection. Run 'kubectl label namespace hc-telemetry istio-injection=enabled' to enable it, or 'kubectl label namespace hc-telemetry istio-injection=disabled' to explicitly mark it as not needing injection.
Info [IST0102] (Namespace mirrord) The namespace is not enabled for Istio injection. Run 'kubectl label namespace mirrord istio-injection=enabled' to enable it, or 'kubectl label namespace mirrord istio-injection=disabled' to explicitly mark it as not needing injection.
Info [IST0118] (Service hc-telemetry/hc-telemetry-collector-service) Port name hc-telemetry-collector-metric-port (port: 9460, targetPort: 9460) doesn't follow the naming convention of Istio port.
Creating an archive at /data/i588324/virtual-cluster-poc/bug-report.tar.gz.
Time used for creating the tar file is 61.886054ms.
Cleaning up temporary files in /tmp/bug-report.
Done.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15 (11 by maintainers)

Most upvoted comments

I think that command is supposed to emit to istio-system namespace

On Mon, Mar 11, 2024, 7:47 PM Kyungho Byoun @.***> wrote:

@howardjohn https://github.com/howardjohn I did just follow Install Multi-Primary on different networks https://istio.io/latest/docs/setup/install/multicluster/multi-primary_multi-network/#enable-endpoint-discovery. If the namespace is the problem, then it has to be modified.

— Reply to this email directly, view it on GitHub https://github.com/istio/istio/issues/49780#issuecomment-1989905098, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEYGXPPEV2TDLUOHDVA2JTYXZUCXAVCNFSM6AAAAABEKORI3KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBZHEYDKMBZHA . You are receiving this because you were mentioned.Message ID: @.***>

I think @howardjohn was proposing changes around this

PR is merged #49260. It is just a framework refactor, does change any other behavior