secrets-store-csi-driver-provider-azure: secrets-store-csi-driver install fails on Azure RedHat OpenShift [ARO]

What steps did you take and what happened:

Following the secrets-store-csi-driver install docs, I hit a security issue specific to OpenShift related to the securityContext.

helm install csi-secrets-store-provider-azure csi-secrets-store-provider-azure/csi-secrets-store-provider-azure -n $target_namespace
helm ls -n $target_namespace -o yaml
helm status csi-secrets-store-provider-azure -n $target_namespace

Create key-Vault & Secret


az provider register -n Microsoft.KeyVault

az keyvault create --name $vault_name --enable-soft-delete true --location $location -g $rg_name
az keyvault show --name $vault_name 
az keyvault update --name $vault_name --default-action deny -g $rg_name 

kv_id=$(az keyvault show --name $vault_name -g $rg_name --query "id" --output tsv)

az keyvault secret set --name $vault_secret_name --value $vault_secret --description "CSI secret store driver - ${appName} Secret" --vault-name $vault_name
az keyvault secret list --vault-name $vault_name
az keyvault secret show --vault-name $vault_name --name $vault_secret_name --output tsv

aro_client_id=$(az aro show -n $cluster_name -g $rg_name --query 'servicePrincipalProfile.clientId' -o tsv)

Perform role assignments

az role assignment create --role Reader --assignee $aro_client_id --scope /subscriptions/$subId/resourcegroups/$rg_name/providers/Microsoft.KeyVault/vaults/$vault_name # $kv_id

az keyvault set-policy -n $vault_name --key-permissions get --spn $aro_client_id
az keyvault set-policy -n $vault_name --secret-permissions get --spn $aro_client_id
az keyvault set-policy -n $vault_name --certificate-permissions get --spn $aro_client_id

Configure & Deploy secretproviderclasses


export SUBSCRIPTION_ID=$subId
export RESOURCE_GROUP=$rg_name
export TENANT_ID=$tenantId
export KV_NAME=$vault_name
export SECRET_NAME=$vault_secret_name

envsubst < ./cnf/secrets-store-csi-provider-class.yaml > deploy/secrets-store-csi-provider-class.yaml
cat deploy/secrets-store-csi-provider-class.yaml
oc apply -f deploy/secrets-store-csi-provider-class.yaml -n $target_namespace
oc get secretproviderclasses -n $target_namespace
oc describe secretproviderclasses azure-$KV_NAME -n $target_namespace

envsubst < ./cnf/csi-demo-pod-sp.yaml > deploy/csi-demo-pod-sp.yaml
cat deploy/csi-demo-pod-sp.yaml
oc apply -f deploy/csi-demo-pod-sp.yaml -n $target_namespace

oc get po -n $target_namespace -o wide
oc get events -n $target_namespace | grep -i "Error" 
oc describe pod nginx-secrets-store-inline -n $target_namespace
oc logs nginx-secrets-store-inline -n $target_namespace

Name:         nginx-secrets-store-inline
Namespace:    staging
Priority:     0
Node:         aro-azarc-101-x7jmv-worker-westeurope1-zg27m/172.32.2.6
Start Time:   Sat, 09 Jan 2021 20:54:03 +0100
Labels:       <none>
Annotations:  openshift.io/scc: node-exporter
Status:       Pending
IP:
IPs:          <none>
Containers:
  nginx:
    Container ID:
    Image:          nginx
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /mnt/secrets-store from secrets-store-inline (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-jrqj9 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  secrets-store-inline:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            secrets-store.csi.k8s.io
    FSType:
    ReadOnly:          true
    VolumeAttributes:      secretProviderClass=azure-kv-azarc
  default-token-jrqj9:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-jrqj9
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age                 From               Message
  ----     ------       ----                ----               -------
  Normal   Scheduled    17m                 default-scheduler  Successfully assigned staging/nginx-secrets-store-inline to aro-azarc-101-x7jmv-worker-westeurope1-zg27m
  Warning  FailedMount  78s (x7 over 15m)   kubelet            Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[secrets-store-inline default-token-jrqj9]: timed out waiting for the condition
  Warning  FailedMount  43s (x16 over 17m)  kubelet            MountVolume.SetUp failed for volume "secrets-store-inline" : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name secrets-store.csi.k8s.io not found in the list of registered CSI drivers
oc get events -n $target_namespace | grep -i "Error" 
 Warning   FailedCreate   daemonset/csi-secrets-store-provider-azure-secrets-store-csi-driver   Error creating: pods "csi-secrets-store-provider-azure-secrets-store-csi-driver-" is forbidden: unable to validate against any security context constraint: [provider restricted: .spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.volumes[0]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.volumes[1]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.volumes[2]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.volumes[3]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.containers[0].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.containers[0].securityContext.containers[1].hostPort: Invalid value: 9808: Host ports are not allowed to be used spec.containers[1].securityContext.privileged: Invalid value: true: Privileged containers are not allowed spec.containers[1].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.containers[1].securityContext.containers[1].hostPort: Invalid value: 9808: Host ports are not allowed to be used spec.containers[2].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.containers[2].securityContext.containers[1].hostPort: Invalid value: 9808: Host ports are not allowed to be used]
63s         Warning   FailedCreate   daemonset/csi-secrets-store-provider-azure                            Error creating: pods "csi-secrets-store-provider-azure-" is forbidden: unable to validate against any security context constraint: [provider restricted: .spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.volumes[0]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.volumes[1]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.containers[0].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used]

What did you expect to happen:

The install should be sucessfull following the docs

Anything else you would like to add:

Which access mode did you use to access the Azure Key Vault instance: [e.g. Service Principal, Pod Identity, User Assigned Managed Identity, System Assigned Managed Identity]

I used Service Principal

Environment:

  • Secrets Store CSI Driver version: (use the image tag): app_version: 0.0.11 chart: csi-secrets-store-provider-azure-0.0.15

  • Azure Key Vault provider version: (use the image tag):

  • Kubernetes version: (use kubectl version and kubectl get nodes -o wide):

Client Version: openshift-clients-4.5.0-202006231303.p0-16-g3f6a83fb7 Server Version: 4.5.16 Kubernetes Version: v1.18.3+2fbd7c7

  • Cluster type: (e.g. AKS, aks-engine, etc): Azure RedHat OpenShift [ARO]

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 24 (20 by maintainers)

Most upvoted comments

@ezYakaEagle442 Sorry if I’m not clear. Those are 3 different options for install and not steps. Please let me know if a call is helpful?

From you description you installed the driver by running

helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver -n $target_namespace

You just need to add --set grpcSupportProviders=azure in that command. This is to tell the driver to use grpc while communicating with the driver.

@ezYakaEagle442 That value is for the secrets-store-csi-driver. The helm charts in this repo have the driver charts as dep and take care of setting the --grpc-supported-providers=azure. However if the driver and provider are installed separately, then the --grpc-supported-providers=azure arg needs to be added to the secrets-store container in the driver manually as documented here: https://github.com/Azure/secrets-store-csi-driver-provider-azure/blob/master/docs/install-yamls.md.

Sample yaml config for reference: https://github.com/kubernetes-sigs/secrets-store-csi-driver/blob/master/manifest_staging/deploy/secrets-store-csi-driver.yaml#L50