marblerun: Failed to deploy Marblerun on Kubernetes cluster on an Azure Confidential Computing Node
Issue description
Failed to deploy Marblerun on Kubernetes cluster on Azure SGX node. The coordinator fails to deploy with the error Warning FailedScheduling 30s (x9 over 7m34s) default-scheduler 0/1 nodes are available: 1 Insufficient sgx.intel.com/epc.
To reproduce
Steps to reproduce the behavior:
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
sudo apt install docker.io kubectl kubeadm kubelet build-essential autoconf gawk bison wget python3 libcurl4-openssl-dev python3-protobuf libprotobuf-c-dev protobuf-c-compiler python3-pip -y
sudo kubeadm init --apiserver-advertise-address 10.0.0.4 --pod-network-cidr=10.244.0.0/16
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
echo 'deb [arch=amd64] https://download.01.org/intel-sgx/sgx_repo/ubuntu bionic main' | sudo tee /etc/apt/sources.list.d/intel-sgx.list
wget -qO - https://download.01.org/intel-sgx/sgx_repo/ubuntu/intel-sgx-deb.key | sudo apt-key add -
echo "deb [arch=amd64] https://packages.microsoft.com/ubuntu/18.04/prod bionic main" | sudo tee /etc/apt/sources.list.d/msprod.list
wget -qO - https://packages.microsoft.com/keys/microsoft.asc | sudo apt-key add -
sudo apt update && sudo apt install -y libsgx-dcap-ql libsgx-dcap-ql-dev az-dcap-client
kubectl taint nodes --all node-role.kubernetes.io/master-
kubectl apply -k https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/sgx_nfd?ref=main
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.3.1/cert-manager.yaml
kubectl apply -k https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/operator/default?ref=main
sudo wget -O /usr/local/bin/marblerun https://github.com/edgelesssys/marblerun/releases/latest/download/marblerun
sudo chmod +x /usr/local/bin/marblerun
sudo mkdir /mnt/data
kubectl apply -f storage_class.yaml
kubectl apply -f pv.yaml
marblerun install
Expected behavior
Both the injector and the coordinator pods starting up. Only the injector pod starts up.
Environment:
- Marblerun version: 0.3.2
- Edgeless RT version:
- Go version:
- Minikube version:
- Kubernetes version: 1.21.2
Additional info / screenshots
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 15 (6 by maintainers)
Ah, I didn’t read your output carefully enough. You’re using the sgx driver version 1.33 (probably preinstalled in your Azure VM). The Intel device plugin requires the in-tree kernel driver from 5.11 or at least 1.41:
They provide backward compatibility the way you solved it now.
Can I close this issue?