istio: Istio proxy container is failing to start with Failed to create directory for ./var/run/secrets/workload-spiffe-uds/socket: mkdir var/run/secrets/workload-spiffe-uds: read-only file system
Bug Description
2022-04-01T11:03:21.235838Z info JWT policy is third-party-jwt
2022-04-01T11:03:21.235843Z info using credential fetcher of JWT type in cluster.local trust domain
2022-04-01T11:03:22.256088Z info SDS socket not found. Starting Istio SDS Server
2022-04-01T11:03:22.256131Z info CA Endpoint istiod.istio-system.svc:15012, provider Citadel
2022-04-01T11:03:22.256104Z info Opening status port 15020
2022-04-01T11:03:22.256156Z info Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2022-04-01T11:03:22.256251Z info citadelclient Citadel client using custom root cert: var/run/secrets/istio/root-cert.pem
2022-04-01T11:03:22.274596Z info ads All caches have been synced up in 1.044429289s, marking server ready
2022-04-01T11:03:22.274803Z warn Failed to create directory for ./var/run/secrets/workload-spiffe-uds/socket: mkdir var/run/secrets/workload-spiffe-uds: read-only file system
2022-04-01T11:03:22.274849Z error sds Failed to set up UDS path: failed to listen on unix socket "./var/run/secrets/workload-spiffe-uds/socket": listen unix ./var/run/secrets/workload-spiffe-uds/socket: bind: no such file or directory
2022-04-01T11:03:22.274855Z info sds SDS server for workload certificates started, listening on "./var/run/secrets/workload-spiffe-uds/socket"
2022-04-01T11:03:22.274869Z info xdsproxy Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "Kubernetes"
2022-04-01T11:03:22.276579Z info sds Starting SDS grpc server
2022-04-01T11:03:22.276668Z warn Failed to create directory for ./var/run/secrets/workload-spiffe-uds/socket: mkdir var/run/secrets/workload-spiffe-uds: read-only file system
2022-04-01T11:03:22.276724Z error sds SDS grpc server for workload proxies failed to set up UDS: failed to listen on unix socket "./var/run/secrets/workload-spiffe-uds/socket": listen unix ./var/run/secrets/workload-spiffe-uds/socket: bind: no such file or directory
2022-04-01T11:03:22.277375Z info starting Http service at 127.0.0.1:15004
2022-04-01T11:03:22.278571Z info Pilot SAN: [istiod.istio-system.svc]
2022-04-01T11:03:22.280450Z info Starting proxy agent
2022-04-01T11:03:22.280480Z info Epoch 0 starting
2022-04-01T11:03:22.280500Z info Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --drain-strategy immediate --parent-shutdown-time-s 60 --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --log-format %Y-%m-%dT%T.%fZ %l envoy %n %v -l warning --component-log-level misc:error]
2022-04-01T11:03:23.277451Z warn Failed to create directory for ./var/run/secrets/workload-spiffe-uds/socket: mkdir var/run/secrets/workload-spiffe-uds: read-only file system
2022-04-01T11:03:23.277582Z error sds SDS grpc server for workload proxies failed to set up UDS: failed to listen on unix socket "./var/run/secrets/workload-spiffe-uds/socket": listen unix ./var/run/secrets/workload-spiffe-uds/socket: bind: no such file or directory
2022-04-01T11:03:23.283586Z warn ca ca request failed, starting attempt 1 in 103.583184ms
2022-04-01T11:03:23.387848Z warn ca ca request failed, starting attempt 2 in 205.386114ms
2022-04-01T11:03:23.593371Z warn ca ca request failed, starting attempt 3 in 429.688774ms
2022-04-01T11:03:24.024024Z warn ca ca request failed, starting attempt 4 in 723.617885ms
2022-04-01T11:03:25.278832Z warn Failed to create directory for ./var/run/secrets/workload-spiffe-uds/socket: mkdir var/run/secrets/workload-spiffe-uds: read-only file system
2022-04-01T11:03:25.278994Z error sds SDS grpc server for workload proxies failed to set up UDS: failed to listen on unix socket "./var/run/secrets/workload-spiffe-uds/socket": listen unix ./var/run/secrets/workload-spiffe-uds/socket: bind: no such file or directory
2022-04-01T11:03:27.808166Z info cache generated new workload certificate latency=5.533264895s ttl=23h59m59.191843153s
2022-04-01T11:03:27.808203Z info cache Root cert has changed, start rotating root cert
2022-04-01T11:03:27.808219Z info ads XDS: Incremental Pushing:0 ConnectedEndpoints:0 Version:
2022-04-01T11:03:27.808282Z info cache returned workload trust anchor from cache ttl=23h59m59.191720438s
2022-04-01T11:03:29.279726Z warn Failed to create directory for ./var/run/secrets/workload-spiffe-uds/socket: mkdir var/run/secrets/workload-spiffe-uds: read-only file system
2022-04-01T11:03:29.279814Z error sds SDS grpc server for workload proxies failed to set up UDS: failed to listen on unix socket "./var/run/secrets/workload-spiffe-uds/socket": listen unix ./var/run/secrets/workload-spiffe-uds/socket: bind: no such file or directory
2022-04-01T11:03:29.662011Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2022-04-01T11:03:37.280480Z warn Failed to create directory for ./var/run/secrets/workload-spiffe-uds/socket: mkdir var/run/secrets/workload-spiffe-uds: read-only file system
2022-04-01T11:03:37.280569Z error sds SDS grpc server for workload proxies failed to set up UDS: failed to listen on unix socket "./var/run/secrets/workload-spiffe-uds/socket": listen unix ./var/run/secrets/workload-spiffe-uds/socket: bind: no such file or directory
2022-04-01T11:03:53.281441Z warn sds SDS grpc server could not be started
2022-04-01T11:04:01.334132Z warning envoy config StreamSecrets gRPC config stream closed since 31s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: immediate connect error: No such file or directory
Version
Image: gcr.io/istio-testing/proxyv2:latest
Image ID: docker-pullable://gcr.io/istio-testing/proxyv2@sha256:c1a575c54eef3b8642ceca205c4f2551eda026d7a910e5b2330d8ab6613d366a
Additional Information
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 2
- Comments: 48 (20 by maintainers)
Commits related to this issue
- workaround for istio 38217 issue https://github.com/istio/istio/issues/38217 — committed to banzaicloud/istio-operator by LuciferInLove 2 years ago
Catching up so sorry if this is repeated - the only supported helm charts are the official helm repository (highly recommended, only one documented on istio.io) or the release tar.gz.
Pulling from git, or any other method that uses
latestis not supportedif you hit this issue you are installing in a dangerous an unsupported way
We were using the Helm Charts for istiod pointed to the tag 1.13.2. When the istio-testing was pushed to the container registry, inside the 1.13.2 helm chart there was a reference to
latestwhich caused all new pods to not be able to connect since istiod was erroring with the above error. This required us to temporarily point to the master charts since they are the only ones with the updates to support the latest changes.This error occurred due to a reference in the helm charts to latest that we have no control over. Can this be pointed to a stable tag instead of latest so it isn’t changed out from under us?
I think this is a correct statement but not intuitive if you are pointing to a tag and using a GitOps tool like Argo for helm. I think the issue of that we are pointing to the manifests that are in the Git repo and not the published charts. This might be expected but maybe there is a way to make it more clear that if you are using GitOps tooling that it won’t work as expected since there are some tasks that change the helm charts before they are published to the chart repo and that it is required to use that. I am double checking all that today to see if it clears all the issues.
yes
On Wed, Sep 7, 2022, 8:08 AM Dongfang Qu @.***> wrote:
If you are seeing this you have a mismatch between docker image versions and configmaps
@hzxuzhonghu @spstarr running into this issue as well. We are trying to upgrade from 1.13.5. We are using tetrates distroless images.
1.14.1-tetratefips-v0-distrolesshttps://istio.tetratelabs.io/could the fact these are distroless images have something to do with the “read-only file system” errors?
pulled down the Istio 1.14 manifests and used them for the install so manifests should be compatible.
any tips on how we can further debug this?
@bryankaraffa I understand how to see the error by forcing the
latestimage and test repo using the 1.13.2 helm charts.I’m wondering how people are encountering this problem without forcing the incorrect image, as it seems from above others are seeing the same error just using the 1.13.2 manifests. Maybe I am wrong, but I suspect there is something in one of the manifest files that is causing the issue (if it exists). Possibly https://github.com/istio/istio/blob/release-1.13/manifests/charts/istio-control/istio-discovery/files/gen-istio.yaml#L165 and similarly for the hub is a line further up. These lines aren’t updated to the normal release values in the pushed helm charts as far as I can tell. I’m not sure if updating those values will resolve the issue or not.
@beaudeanadams What I have to put that global value? in my application helm chart value?
or istiod application?
thank you
Anyone facing this issue, add this to your helm values, and you will use the dockerhub istio repo instead. Ensure you set ‘tag’ under global to be the same version as the helm chart (i.e. 1.13.2)
Sounds like you are running
gcr.io/istio-testing/proxyv2:latestbut didn’t pull in the new helm charts. The two are coupled