istio: Istio breaks Pod's securityContext

Bug description I have tried to deploy a StatefulSet with the following securityContext:

imagePullSecrets: []
      initContainers: []
      securityContext:
        fsGroup: 1000

in an Istio enabled namespace. When the Pod is started, I find the following on the Pod:

securityContext:
    fsGroup: 1337
  serviceAccount: default

[ ] Docs [X] Installation [ ] Networking [ ] Performance and Scalability [ ] Extensions and Telemetry [ ] Security [ ] Test and Release [X] User Experience [ ] Developer Infrastructure

Expected behavior I should be able to deploy a StatefulSet with the expected fsGroup.

Steps to reproduce the bug Deploy the following test in an Istio enabled namespace:

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: "nginx"
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      securityContext:
        fsGroup: 1000
      volumes:
      - name: configuration
        configMap:
          name: broker-config
      containers:
      - name: nginx
        image: k8s.gcr.io/nginx-slim:0.8
        command: ["sleep"]
        args: ["100000"]
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: configuration
          mountPath: /usr/share/nginx/html/broker.xml
          subPath: broker.xml
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi
---
apiVersion: v1
data:
  broker.xml: "aaaa"
kind: ConfigMap
metadata:
  name: broker-config

Then exec into the container web-0: kubectl exec -it web-0 – ‘ls -al /usr/share/nginx/html/’

drwxrwxrwx 2 root root 24 Aug 28 10:46 .
drwxr-xr-x 4 root root 30 Jul  1  2016 ..
-rw-r--r-- 1 root 1337  4 Aug 28 10:46 broker.xml

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)

client version: 1.7.0
control plane version: 1.7.0
data plane version: 1.7.0 (3 proxies)

How was Istio installed? Using Istio operator:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
 name: istiooperator
 namespace: istio-system
spec:
  profile: demo
  components:
    ingressGateways:
    - enabled: true
      k8s:
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchExpressions:
                  - key: app
                    operator: In
                    values:
                    - istio-ingressgateway
                topologyKey: kubernetes.io/hostname
              weight: 100
        service:
          ports:
          - name: tcp-activemq
            port: 61616
            targetPort: 61616
            nodePort: 32000
          - name: http2
            port: 80
            targetPort: 8080
            nodePort: 32001
          - name: https
            port: 443
            targetPort: 8443
          - name: tls
            port: 15443
            targetPort: 15443
      name: istio-ingressgateway
  values:
    sidecarInjectorWebhook:
      rewriteAppHTTPProbe: true 
    gateways:
      istio-ingressgateway:
        type: NodePort

Environment where bug was observed (cloud vendor, OS, etc)

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 1
  • Comments: 24 (8 by maintainers)

Commits related to this issue

Most upvoted comments

Hi @howardjohn ,

sorry to bother you, but I still run into this issue, even though im running on kubernetes 1.20.4 + istio 1.9.1

Istio seems to set the pods fsGroup to 1337, which blocks my container from accessing the volume.

Am I missing something here?

We have run into this issue also. From what I understand, the summary of the problem is that the Istio secret is not mounted with the appropriate permissions in k8s < 1.19 due to https://github.com/kubernetes/kubernetes/issues/57923 . Istio triggers this because it sets runAsUser and runAsGroup, and as a result, there’s a permission issue with the secret. But after reading through the comments on https://github.com/kubernetes/kubernetes/issues/57923 , it seems as though the problem reported there is that the secret is mounted with more permissions than are strictly necessary. The complaint in that issue is that the group read bit 0010 is set when the mode 0400 is requested by the container, so the effective permissions are 0440.

My understanding is that the istio sidecar container should inherit the supplemental group provided by any fsGroup setting when it sets runAsUser. Perhaps the ship has already sailed, but it seems like if fsGroup is set initially, Istio should be able to avoid overwriting fsGroup and leave runAsGroup unset and still be able to read the secret. If the code using the secret requires certain permissions, it might be necessary for it to copy the file to a new file with appropriate permissions, but it should be possible to solve this way.

The impact for us is high because we run on EKS and leverage the IAM roles assigned to ServiceAccount objects and populated by another admission controller provided by EKS. The EKS admission controller mounts the web identity tokens generated for a given IAM role with mode 0600. K8s updates this to 0440 for a read-only mount and when providing an fsGroup, but when Istio changes the fsGroup, our services are unable to read the EKS web identity token since they rely on a specific group that we set in the fsGroup setting.

We didn’t see this issue in Istio 1.5, but now that we’re trying to catch up to current, and going through the difficult 1.5 -> 1.6 transition, this impacts virtually all our services, which depend on the token having group ownership other than 1337. We’re on EKS 1.16 which does not have the fix from 1.19 . This issue leaves us between a bit of a rock and a hard place.

Any thoughts, corrections, or help here?

Hi @howardjohn, if I understand this PR https://github.com/istio/istio/pull/27367 correctly, on a Kubernetes version 1.21 cluster running Istio version 1.11.4 (installed via the operator) I should get the ENABLE_LEGACY_FSGROUP_INJECTION environment variable set to false on my istiod deployment, right? But this is not the case in my cluster. Did I missed something? Could I break something if I manually set this environment variable (by setting spec.components.pilot.k8s.env in my IstioOperator CR)?