marblerun: Failed to deploy Marblerun on Kubernetes cluster on an Azure Confidential Computing Node

Issue description

Failed to deploy Marblerun on Kubernetes cluster on Azure SGX node. The coordinator fails to deploy with the error Warning FailedScheduling 30s (x9 over 7m34s) default-scheduler 0/1 nodes are available: 1 Insufficient sgx.intel.com/epc.

To reproduce

Steps to reproduce the behavior:

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
sudo apt install docker.io kubectl kubeadm kubelet build-essential autoconf gawk bison wget python3 libcurl4-openssl-dev python3-protobuf libprotobuf-c-dev protobuf-c-compiler python3-pip -y

sudo kubeadm init --apiserver-advertise-address 10.0.0.4 --pod-network-cidr=10.244.0.0/16
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

echo 'deb [arch=amd64] https://download.01.org/intel-sgx/sgx_repo/ubuntu bionic main' | sudo tee /etc/apt/sources.list.d/intel-sgx.list
wget -qO - https://download.01.org/intel-sgx/sgx_repo/ubuntu/intel-sgx-deb.key | sudo apt-key add -
echo "deb [arch=amd64] https://packages.microsoft.com/ubuntu/18.04/prod bionic main" | sudo tee /etc/apt/sources.list.d/msprod.list
wget -qO - https://packages.microsoft.com/keys/microsoft.asc | sudo apt-key add -

sudo apt update && sudo apt install -y libsgx-dcap-ql libsgx-dcap-ql-dev az-dcap-client

kubectl taint nodes --all node-role.kubernetes.io/master-

kubectl apply -k https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/sgx_nfd?ref=main
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.3.1/cert-manager.yaml
kubectl apply -k https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/operator/default?ref=main

sudo wget -O /usr/local/bin/marblerun https://github.com/edgelesssys/marblerun/releases/latest/download/marblerun
sudo chmod +x /usr/local/bin/marblerun

sudo mkdir /mnt/data
kubectl apply -f storage_class.yaml
kubectl apply -f pv.yaml

marblerun install

Expected behavior

Both the injector and the coordinator pods starting up. Only the injector pod starts up.

Environment:

  • Marblerun version: 0.3.2
  • Edgeless RT version:
  • Go version:
  • Minikube version:
  • Kubernetes version: 1.21.2

Additional info / screenshots

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15 (6 by maintainers)

Most upvoted comments

Ah, I didn’t read your output carefully enough. You’re using the sgx driver version 1.33 (probably preinstalled in your Azure VM). The Intel device plugin requires the in-tree kernel driver from 5.11 or at least 1.41:

The SGX device plugin requires Linux Kernel SGX drivers to be available. These drivers are available in Linux 5.11. The SGX DCAP out-of-tree driver v1.41 is also known to work.

They provide backward compatibility the way you solved it now.

Can I close this issue?