secrets-store-csi-driver-provider-azure: secrets-store-csi-driver install fails on Azure RedHat OpenShift [ARO]
What steps did you take and what happened:
Following the secrets-store-csi-driver install docs, I hit a security issue specific to OpenShift related to the securityContext.
helm install csi-secrets-store-provider-azure csi-secrets-store-provider-azure/csi-secrets-store-provider-azure -n $target_namespace
helm ls -n $target_namespace -o yaml
helm status csi-secrets-store-provider-azure -n $target_namespace
Create key-Vault & Secret
az provider register -n Microsoft.KeyVault
az keyvault create --name $vault_name --enable-soft-delete true --location $location -g $rg_name
az keyvault show --name $vault_name
az keyvault update --name $vault_name --default-action deny -g $rg_name
kv_id=$(az keyvault show --name $vault_name -g $rg_name --query "id" --output tsv)
az keyvault secret set --name $vault_secret_name --value $vault_secret --description "CSI secret store driver - ${appName} Secret" --vault-name $vault_name
az keyvault secret list --vault-name $vault_name
az keyvault secret show --vault-name $vault_name --name $vault_secret_name --output tsv
aro_client_id=$(az aro show -n $cluster_name -g $rg_name --query 'servicePrincipalProfile.clientId' -o tsv)
Perform role assignments
az role assignment create --role Reader --assignee $aro_client_id --scope /subscriptions/$subId/resourcegroups/$rg_name/providers/Microsoft.KeyVault/vaults/$vault_name # $kv_id
az keyvault set-policy -n $vault_name --key-permissions get --spn $aro_client_id
az keyvault set-policy -n $vault_name --secret-permissions get --spn $aro_client_id
az keyvault set-policy -n $vault_name --certificate-permissions get --spn $aro_client_id
Configure & Deploy secretproviderclasses
export SUBSCRIPTION_ID=$subId
export RESOURCE_GROUP=$rg_name
export TENANT_ID=$tenantId
export KV_NAME=$vault_name
export SECRET_NAME=$vault_secret_name
envsubst < ./cnf/secrets-store-csi-provider-class.yaml > deploy/secrets-store-csi-provider-class.yaml
cat deploy/secrets-store-csi-provider-class.yaml
oc apply -f deploy/secrets-store-csi-provider-class.yaml -n $target_namespace
oc get secretproviderclasses -n $target_namespace
oc describe secretproviderclasses azure-$KV_NAME -n $target_namespace
envsubst < ./cnf/csi-demo-pod-sp.yaml > deploy/csi-demo-pod-sp.yaml
cat deploy/csi-demo-pod-sp.yaml
oc apply -f deploy/csi-demo-pod-sp.yaml -n $target_namespace
oc get po -n $target_namespace -o wide
oc get events -n $target_namespace | grep -i "Error"
oc describe pod nginx-secrets-store-inline -n $target_namespace
oc logs nginx-secrets-store-inline -n $target_namespace
Name: nginx-secrets-store-inline
Namespace: staging
Priority: 0
Node: aro-azarc-101-x7jmv-worker-westeurope1-zg27m/172.32.2.6
Start Time: Sat, 09 Jan 2021 20:54:03 +0100
Labels: <none>
Annotations: openshift.io/scc: node-exporter
Status: Pending
IP:
IPs: <none>
Containers:
nginx:
Container ID:
Image: nginx
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/mnt/secrets-store from secrets-store-inline (ro)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-jrqj9 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
secrets-store-inline:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: secrets-store.csi.k8s.io
FSType:
ReadOnly: true
VolumeAttributes: secretProviderClass=azure-kv-azarc
default-token-jrqj9:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-jrqj9
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 17m default-scheduler Successfully assigned staging/nginx-secrets-store-inline to aro-azarc-101-x7jmv-worker-westeurope1-zg27m
Warning FailedMount 78s (x7 over 15m) kubelet Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[secrets-store-inline default-token-jrqj9]: timed out waiting for the condition
Warning FailedMount 43s (x16 over 17m) kubelet MountVolume.SetUp failed for volume "secrets-store-inline" : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name secrets-store.csi.k8s.io not found in the list of registered CSI drivers
oc get events -n $target_namespace | grep -i "Error"
Warning FailedCreate daemonset/csi-secrets-store-provider-azure-secrets-store-csi-driver Error creating: pods "csi-secrets-store-provider-azure-secrets-store-csi-driver-" is forbidden: unable to validate against any security context constraint: [provider restricted: .spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.volumes[0]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.volumes[1]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.volumes[2]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.volumes[3]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.containers[0].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.containers[0].securityContext.containers[1].hostPort: Invalid value: 9808: Host ports are not allowed to be used spec.containers[1].securityContext.privileged: Invalid value: true: Privileged containers are not allowed spec.containers[1].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.containers[1].securityContext.containers[1].hostPort: Invalid value: 9808: Host ports are not allowed to be used spec.containers[2].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.containers[2].securityContext.containers[1].hostPort: Invalid value: 9808: Host ports are not allowed to be used]
63s Warning FailedCreate daemonset/csi-secrets-store-provider-azure Error creating: pods "csi-secrets-store-provider-azure-" is forbidden: unable to validate against any security context constraint: [provider restricted: .spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.volumes[0]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.volumes[1]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.containers[0].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used]
What did you expect to happen:
The install should be sucessfull following the docs
Anything else you would like to add:
Which access mode did you use to access the Azure Key Vault instance: [e.g. Service Principal, Pod Identity, User Assigned Managed Identity, System Assigned Managed Identity]
I used Service Principal
Environment:
-
Secrets Store CSI Driver version: (use the image tag): app_version: 0.0.11 chart: csi-secrets-store-provider-azure-0.0.15
-
Azure Key Vault provider version: (use the image tag):
-
Kubernetes version: (use
kubectl version
andkubectl get nodes -o wide
):
Client Version: openshift-clients-4.5.0-202006231303.p0-16-g3f6a83fb7 Server Version: 4.5.16 Kubernetes Version: v1.18.3+2fbd7c7
- Cluster type: (e.g. AKS, aks-engine, etc): Azure RedHat OpenShift [ARO]
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 24 (20 by maintainers)
@ezYakaEagle442 Sorry if I’m not clear. Those are 3 different options for install and not steps. Please let me know if a call is helpful?
From you description you installed the driver by running
You just need to add
--set grpcSupportProviders=azure
in that command. This is to tell the driver to use grpc while communicating with the driver.@ezYakaEagle442 That value is for the secrets-store-csi-driver. The helm charts in this repo have the driver charts as dep and take care of setting the
--grpc-supported-providers=azure
. However if the driver and provider are installed separately, then the--grpc-supported-providers=azure
arg needs to be added to thesecrets-store
container in the driver manually as documented here: https://github.com/Azure/secrets-store-csi-driver-provider-azure/blob/master/docs/install-yamls.md.Sample yaml config for reference: https://github.com/kubernetes-sigs/secrets-store-csi-driver/blob/master/manifest_staging/deploy/secrets-store-csi-driver.yaml#L50