istio: Istio breaks Pod's securityContext
Bug description
I have tried to deploy a StatefulSet with the following securityContext
:
imagePullSecrets: []
initContainers: []
securityContext:
fsGroup: 1000
in an Istio enabled namespace. When the Pod is started, I find the following on the Pod:
securityContext:
fsGroup: 1337
serviceAccount: default
[ ] Docs [X] Installation [ ] Networking [ ] Performance and Scalability [ ] Extensions and Telemetry [ ] Security [ ] Test and Release [X] User Experience [ ] Developer Infrastructure
Expected behavior
I should be able to deploy a StatefulSet with the expected fsGroup
.
Steps to reproduce the bug Deploy the following test in an Istio enabled namespace:
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app: nginx
spec:
ports:
- port: 80
name: web
clusterIP: None
selector:
app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: web
spec:
serviceName: "nginx"
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
securityContext:
fsGroup: 1000
volumes:
- name: configuration
configMap:
name: broker-config
containers:
- name: nginx
image: k8s.gcr.io/nginx-slim:0.8
command: ["sleep"]
args: ["100000"]
ports:
- containerPort: 80
name: web
volumeMounts:
- name: configuration
mountPath: /usr/share/nginx/html/broker.xml
subPath: broker.xml
- name: www
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: www
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Gi
---
apiVersion: v1
data:
broker.xml: "aaaa"
kind: ConfigMap
metadata:
name: broker-config
Then exec into the container web-0
:
kubectl exec -it web-0 – ‘ls -al /usr/share/nginx/html/’
drwxrwxrwx 2 root root 24 Aug 28 10:46 .
drwxr-xr-x 4 root root 30 Jul 1 2016 ..
-rw-r--r-- 1 root 1337 4 Aug 28 10:46 broker.xml
Version (include the output of istioctl version --remote
and kubectl version
and helm version
if you used Helm)
client version: 1.7.0
control plane version: 1.7.0
data plane version: 1.7.0 (3 proxies)
How was Istio installed? Using Istio operator:
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
name: istiooperator
namespace: istio-system
spec:
profile: demo
components:
ingressGateways:
- enabled: true
k8s:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- istio-ingressgateway
topologyKey: kubernetes.io/hostname
weight: 100
service:
ports:
- name: tcp-activemq
port: 61616
targetPort: 61616
nodePort: 32000
- name: http2
port: 80
targetPort: 8080
nodePort: 32001
- name: https
port: 443
targetPort: 8443
- name: tls
port: 15443
targetPort: 15443
name: istio-ingressgateway
values:
sidecarInjectorWebhook:
rewriteAppHTTPProbe: true
gateways:
istio-ingressgateway:
type: NodePort
Environment where bug was observed (cloud vendor, OS, etc)
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 1
- Comments: 24 (8 by maintainers)
Commits related to this issue
- Do not set fsGroup when its not needed For issues like https://github.com/istio/istio/issues/26882 On Kubernetes 1.19+ we can avoid this issue. This adds a flag to disable setting this option, and a... — committed to howardjohn/istio by howardjohn 4 years ago
- Do not set fsGroup when its not needed (#27367) * Do not set fsGroup when its not needed For issues like https://github.com/istio/istio/issues/26882 On Kubernetes 1.19+ we can avoid this issue. Thi... — committed to istio/istio by howardjohn 4 years ago
- support istio's group ID in the user namespace Without https://github.com/istio/istio/issues/26882, Istio will rewrite the fsGroup to 1337 which interferes with running builds unless we also map the ... — committed to dominodatalab/forge by steved 4 years ago
- support istio for build jobs (#60) * support istio's group ID in the user namespace Without https://github.com/istio/istio/issues/26882, Istio will rewrite the fsGroup to 1337 which interferes wi... — committed to dominodatalab/forge by steved 4 years ago
Hi @howardjohn ,
sorry to bother you, but I still run into this issue, even though im running on kubernetes 1.20.4 + istio 1.9.1
Istio seems to set the pods fsGroup to 1337, which blocks my container from accessing the volume.
Am I missing something here?
We have run into this issue also. From what I understand, the summary of the problem is that the Istio secret is not mounted with the appropriate permissions in k8s < 1.19 due to https://github.com/kubernetes/kubernetes/issues/57923 . Istio triggers this because it sets
runAsUser
andrunAsGroup
, and as a result, there’s a permission issue with the secret. But after reading through the comments on https://github.com/kubernetes/kubernetes/issues/57923 , it seems as though the problem reported there is that the secret is mounted with more permissions than are strictly necessary. The complaint in that issue is that the group read bit0010
is set when the mode0400
is requested by the container, so the effective permissions are0440
.My understanding is that the istio sidecar container should inherit the supplemental group provided by any
fsGroup
setting when it setsrunAsUser
. Perhaps the ship has already sailed, but it seems like iffsGroup
is set initially, Istio should be able to avoid overwritingfsGroup
and leaverunAsGroup
unset and still be able to read the secret. If the code using the secret requires certain permissions, it might be necessary for it to copy the file to a new file with appropriate permissions, but it should be possible to solve this way.The impact for us is high because we run on EKS and leverage the IAM roles assigned to
ServiceAccount
objects and populated by another admission controller provided by EKS. The EKS admission controller mounts the web identity tokens generated for a given IAM role with mode0600
. K8s updates this to0440
for a read-only mount and when providing anfsGroup
, but when Istio changes thefsGroup
, our services are unable to read the EKS web identity token since they rely on a specific group that we set in thefsGroup
setting.We didn’t see this issue in Istio 1.5, but now that we’re trying to catch up to current, and going through the difficult 1.5 -> 1.6 transition, this impacts virtually all our services, which depend on the token having group ownership other than
1337
. We’re on EKS 1.16 which does not have the fix from 1.19 . This issue leaves us between a bit of a rock and a hard place.Any thoughts, corrections, or help here?
Hi @howardjohn, if I understand this PR https://github.com/istio/istio/pull/27367 correctly, on a Kubernetes version 1.21 cluster running Istio version 1.11.4 (installed via the operator) I should get the
ENABLE_LEGACY_FSGROUP_INJECTION
environment variable set to false on my istiod deployment, right? But this is not the case in my cluster. Did I missed something? Could I break something if I manually set this environment variable (by settingspec.components.pilot.k8s.env
in my IstioOperator CR)?