kubevirt: VM Stuck in Pending State on macOS M1 when Running Kubevirt Using Experimental Arm64 Build

What happened: Attempting to run kubevirt using the experimental arm64 build, but the VM is stuck in a Pending state.

What you expected to happen: Pod should have status Running.

How to reproduce it (as minimally and precisely as possible):

kind create cluster

LATEST=$(curl -L https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/latest-arm64)
kubectl apply -f https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/${LATEST}/kubevirt-operator-arm64.yaml
kubectl apply -f https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/${LATEST}/kubevirt-cr-arm64.yaml

 ❯ echo $LATEST
20230706

curl -L -o virtctl https://github.com/kubevirt/kubevirt/releases/download/v1.0.0-rc.1/virtctl-v1.0.0-rc.1-darwin-arm64
chmod +x virtctl
sudo install virtctl /usr/local/bin

❯ kubectl apply -f examples/vm-cirros.yaml 
virtualmachine.kubevirt.io/vm-cirros created

❯ kubectl get vm
NAME        AGE   STATUS    READY
vm-cirros   31m   Stopped   False

❯ virtctl start vm-cirros
VM vm-cirros was scheduled to start

❯ kubectl get pods
NAME                            READY   STATUS    RESTARTS   AGE
virt-launcher-vm-cirros-b84rf   0/2     Pending   0          58s
vm/vm-cirros 
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"kubevirt.io/v1","kind":"VirtualMachine","metadata":{"annotations":{},"labels":{"kubevirt.io/vm":"vm-cirros"},"name":"vm-cirros","namespace":"default"},"spec":{"running":false,"template":{"metadata":{"labels":{"kubevirt.io/vm":"vm-cirros"}},"spec":{"domain":{"devices":{"disks":[{"disk":{"bus":"virtio"},"name":"containerdisk"},{"disk":{"bus":"virtio"},"name":"cloudinitdisk"}]},"resources":{"requests":{"memory":"128Mi"}}},"terminationGracePeriodSeconds":0,"volumes":[{"containerDisk":{"image":"registry:5000/kubevirt/cirros-container-disk-demo:devel"},"name":"containerdisk"},{"cloudInitNoCloud":{"userData":"#!/bin/sh\n\necho 'printed from cloud-init userdata'\n"},"name":"cloudinitdisk"}]}}}}
    kubevirt.io/latest-observed-api-version: v1
    kubevirt.io/storage-observed-api-version: v1
  creationTimestamp: "2023-07-06T16:44:14Z"
  finalizers:
  - kubevirt.io/virtualMachineControllerFinalize
  generation: 2
  labels:
    kubevirt.io/vm: vm-cirros
  name: vm-cirros
  namespace: default
  resourceVersion: "11554"
  uid: e6e6339d-815e-4d47-8366-b68b83d79ea7
spec:
  running: true
  template:
    metadata:
      creationTimestamp: null
      labels:
        kubevirt.io/vm: vm-cirros
    spec:
      architecture: arm64
      domain:
        devices:
          disks:
          - disk:
              bus: virtio
            name: containerdisk
          - disk:
              bus: virtio
            name: cloudinitdisk
        machine:
          type: virt
        resources:
          requests:
            memory: 128Mi
      terminationGracePeriodSeconds: 0
      volumes:
      - containerDisk:
          image: registry:5000/kubevirt/cirros-container-disk-demo:devel
        name: containerdisk
      - cloudInitNoCloud:
          userData: |
            #!/bin/sh

            echo 'printed from cloud-init userdata'
        name: cloudinitdisk
status:
  conditions:
  - lastProbeTime: "2023-07-06T17:18:32Z"
    lastTransitionTime: "2023-07-06T17:18:32Z"
    message: Guest VM is not reported as running
    reason: GuestNotRunning
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-07-06T17:18:32Z"
    message: '0/1 nodes are available: 1 Insufficient devices.kubevirt.io/kvm. preemption:
      0/1 nodes are available: 1 No preemption victims found for incoming pod..'
    reason: Unschedulable
    status: "False"
    type: PodScheduled
  created: true
  desiredGeneration: 2
  observedGeneration: 2
  printableStatus: ErrorUnschedulable
  volumeSnapshotStatuses:
  - enabled: false
    name: containerdisk
    reason: Snapshot is not supported for this volumeSource type [containerdisk[]
  - enabled: false
    name: cloudinitdisk
    reason: Snapshot is not supported for this volumeSource type [cloudinitdisk[]
vmi/vm-cirros 
apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
metadata:
  annotations:
    kubevirt.io/latest-observed-api-version: v1
    kubevirt.io/storage-observed-api-version: v1
    kubevirt.io/vm-generation: "2"
  creationTimestamp: "2023-07-06T17:18:32Z"
  finalizers:
  - kubevirt.io/virtualMachineControllerFinalize
  - foregroundDeleteVirtualMachine
  generation: 4
  labels:
    kubevirt.io/vm: vm-cirros
  name: vm-cirros
  namespace: default
  ownerReferences:
  - apiVersion: kubevirt.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: VirtualMachine
    name: vm-cirros
    uid: e6e6339d-815e-4d47-8366-b68b83d79ea7
  resourceVersion: "11553"
  uid: 896aa024-54ce-4653-9c98-fdfb7ec2cd5d
spec:
  architecture: arm64
  domain:
    cpu:
      cores: 1
      model: host-passthrough
      sockets: 1
      threads: 1
    devices:
      disks:
      - disk:
          bus: virtio
        name: containerdisk
      - disk:
          bus: virtio
        name: cloudinitdisk
      interfaces:
      - bridge: {}
        name: default
    features:
      acpi:
        enabled: true
    firmware:
      bootloader:
        efi:
          secureBoot: false
      uuid: 0d2a2043-41c0-59c3-9b17-025022203668
    machine:
      type: virt
    resources:
      requests:
        memory: 128Mi
  networks:
  - name: default
    pod: {}
  terminationGracePeriodSeconds: 0
  volumes:
  - containerDisk:
      image: registry:5000/kubevirt/cirros-container-disk-demo:devel
      imagePullPolicy: IfNotPresent
    name: containerdisk
  - cloudInitNoCloud:
      userData: |
        #!/bin/sh

        echo 'printed from cloud-init userdata'
    name: cloudinitdisk
status:
  activePods:
    b39e0ecd-6fb5-4504-8ac4-dac9273c4566: ""
  conditions:
  - lastProbeTime: "2023-07-06T17:18:32Z"
    lastTransitionTime: "2023-07-06T17:18:32Z"
    message: Guest VM is not reported as running
    reason: GuestNotRunning
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-07-06T17:18:32Z"
    message: '0/1 nodes are available: 1 Insufficient devices.kubevirt.io/kvm. preemption:
      0/1 nodes are available: 1 No preemption victims found for incoming pod..'
    reason: Unschedulable
    status: "False"
    type: PodScheduled
  currentCPUTopology:
    cores: 1
    sockets: 1
    threads: 1
  guestOSInfo: {}
  launcherContainerImageVersion: quay.io/kubevirt/virt-launcher:20230706_534bda181-arm64
  phase: Scheduling
  phaseTransitionTimestamps:
  - phase: Pending
    phaseTransitionTimestamp: "2023-07-06T17:18:32Z"
  - phase: Scheduling
    phaseTransitionTimestamp: "2023-07-06T17:18:32Z"
  qosClass: Burstable
  runtimeUser: 107
  virtualMachineRevisionName: revision-start-vm-e6e6339d-815e-4d47-8366-b68b83d79ea7-2
pod/virt-launcher-vm-cirros-b84rf 
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubectl.kubernetes.io/default-container: compute
    kubevirt.io/domain: vm-cirros
    kubevirt.io/migrationTransportUnix: "true"
    kubevirt.io/vm-generation: "2"
    post.hook.backup.velero.io/command: '["/usr/bin/virt-freezer", "--unfreeze", "--name",
      "vm-cirros", "--namespace", "default"]'
    post.hook.backup.velero.io/container: compute
    pre.hook.backup.velero.io/command: '["/usr/bin/virt-freezer", "--freeze", "--name",
      "vm-cirros", "--namespace", "default"]'
    pre.hook.backup.velero.io/container: compute
  creationTimestamp: "2023-07-06T17:18:32Z"
  generateName: virt-launcher-vm-cirros-
  labels:
    kubevirt.io: virt-launcher
    kubevirt.io/created-by: 896aa024-54ce-4653-9c98-fdfb7ec2cd5d
    kubevirt.io/vm: vm-cirros
    vm.kubevirt.io/name: vm-cirros
  name: virt-launcher-vm-cirros-b84rf
  namespace: default
  ownerReferences:
  - apiVersion: kubevirt.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: VirtualMachineInstance
    name: vm-cirros
    uid: 896aa024-54ce-4653-9c98-fdfb7ec2cd5d
  resourceVersion: "11550"
  uid: b39e0ecd-6fb5-4504-8ac4-dac9273c4566
spec:
  automountServiceAccountToken: false
  containers:
  - command:
    - /usr/bin/virt-launcher-monitor
    - --qemu-timeout
    - 354s
    - --name
    - vm-cirros
    - --uid
    - 896aa024-54ce-4653-9c98-fdfb7ec2cd5d
    - --namespace
    - default
    - --kubevirt-share-dir
    - /var/run/kubevirt
    - --ephemeral-disk-dir
    - /var/run/kubevirt-ephemeral-disks
    - --container-disk-dir
    - /var/run/kubevirt/container-disks
    - --grace-period-seconds
    - "15"
    - --hook-sidecars
    - "0"
    - --ovmf-path
    - /usr/share/AAVMF
    - --run-as-nonroot
    env:
    - name: XDG_CACHE_HOME
      value: /var/run/kubevirt-private
    - name: XDG_CONFIG_HOME
      value: /var/run/kubevirt-private
    - name: XDG_RUNTIME_DIR
      value: /var/run
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    image: quay.io/kubevirt/virt-launcher:20230706_534bda181-arm64
    imagePullPolicy: IfNotPresent
    name: compute
    resources:
      limits:
        devices.kubevirt.io/kvm: "1"
        devices.kubevirt.io/tun: "1"
        devices.kubevirt.io/vhost-net: "1"
      requests:
        cpu: 100m
        devices.kubevirt.io/kvm: "1"
        devices.kubevirt.io/tun: "1"
        devices.kubevirt.io/vhost-net: "1"
        ephemeral-storage: 50M
        memory: 510208Ki
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        add:
        - NET_BIND_SERVICE
        drop:
        - ALL
      privileged: false
      runAsGroup: 107
      runAsNonRoot: true
      runAsUser: 107
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/kubevirt-private
      name: private
    - mountPath: /var/run/kubevirt
      name: public
    - mountPath: /var/run/kubevirt-ephemeral-disks
      name: ephemeral-disks
    - mountPath: /var/run/kubevirt/container-disks
      mountPropagation: HostToContainer
      name: container-disks
    - mountPath: /var/run/libvirt
      name: libvirt-runtime
    - mountPath: /var/run/kubevirt/sockets
      name: sockets
    - mountPath: /var/run/kubevirt/hotplug-disks
      mountPropagation: HostToContainer
      name: hotplug-disks
  - args:
    - --copy-path
    - /var/run/kubevirt-ephemeral-disks/container-disk-data/896aa024-54ce-4653-9c98-fdfb7ec2cd5d/disk_0
    command:
    - /usr/bin/container-disk
    image: registry:5000/kubevirt/cirros-container-disk-demo:devel
    imagePullPolicy: IfNotPresent
    name: volumecontainerdisk
    resources:
      limits:
        cpu: 10m
        memory: 40M
      requests:
        cpu: 1m
        ephemeral-storage: 50M
        memory: 1M
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      runAsNonRoot: true
      runAsUser: 107
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/kubevirt-ephemeral-disks/container-disk-data/896aa024-54ce-4653-9c98-fdfb7ec2cd5d
      name: container-disks
    - mountPath: /usr/bin
      name: virt-bin-share-dir
  dnsPolicy: ClusterFirst
  enableServiceLinks: false
  hostname: vm-cirros
  initContainers:
  - command:
    - /usr/bin/cp
    - /usr/bin/container-disk
    - /init/usr/bin/container-disk
    env:
    - name: XDG_CACHE_HOME
      value: /var/run/kubevirt-private
    - name: XDG_CONFIG_HOME
      value: /var/run/kubevirt-private
    - name: XDG_RUNTIME_DIR
      value: /var/run
    image: quay.io/kubevirt/virt-launcher:20230706_534bda181-arm64
    imagePullPolicy: IfNotPresent
    name: container-disk-binary
    resources:
      limits:
        cpu: 100m
        memory: 40M
      requests:
        cpu: 10m
        memory: 1M
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      privileged: false
      runAsGroup: 107
      runAsNonRoot: true
      runAsUser: 107
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /init/usr/bin
      name: virt-bin-share-dir
  - args:
    - --no-op
    command:
    - /usr/bin/container-disk
    image: registry:5000/kubevirt/cirros-container-disk-demo:devel
    imagePullPolicy: IfNotPresent
    name: volumecontainerdisk-init
    resources:
      limits:
        cpu: 10m
        memory: 40M
      requests:
        cpu: 1m
        ephemeral-storage: 50M
        memory: 1M
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      runAsNonRoot: true
      runAsUser: 107
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/kubevirt-ephemeral-disks/container-disk-data/896aa024-54ce-4653-9c98-fdfb7ec2cd5d
      name: container-disks
    - mountPath: /usr/bin
      name: virt-bin-share-dir
  nodeSelector:
    kubernetes.io/arch: arm64
    kubevirt.io/schedulable: "true"
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  readinessGates:
  - conditionType: kubevirt.io/virtual-machine-unpaused
  restartPolicy: Never
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 107
    runAsGroup: 107
    runAsNonRoot: true
    runAsUser: 107
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - emptyDir: {}
    name: private
  - emptyDir: {}
    name: public
  - emptyDir: {}
    name: sockets
  - emptyDir: {}
    name: virt-bin-share-dir
  - emptyDir: {}
    name: libvirt-runtime
  - emptyDir: {}
    name: ephemeral-disks
  - emptyDir: {}
    name: container-disks
  - emptyDir: {}
    name: hotplug-disks
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-07-06T17:18:32Z"
    message: '0/1 nodes are available: 1 Insufficient devices.kubevirt.io/kvm. preemption:
      0/1 nodes are available: 1 No preemption victims found for incoming pod..'
    reason: Unschedulable
    status: "False"
    type: PodScheduled
  - lastProbeTime: "2023-07-06T17:18:32Z"
    lastTransitionTime: "2023-07-06T17:18:32Z"
    message: the virtual machine is not paused
    reason: NotPaused
    status: "True"
    type: kubevirt.io/virtual-machine-unpaused
  phase: Pending
  qosClass: Burstable

Additional context:

Environment:

  • KubeVirt version (use virtctl version):
    Server Version: version.Info{GitVersion:"v1.0.0-beta.0.525+534bda181a078a", GitCommit:"534bda181a078a90215763f849eb4eabcb8d8da2", GitTreeState:"clean", BuildDate:"2023-07-06T01:08:35Z", GoVersion:"go1.19.9", Compiler:"gc", Platform:"linux/arm64"}
  • Kubernetes version (use kubectl version):
    Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.2", GitCommit:"7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647", GitTreeState:"clean", BuildDate:"2023-05-17T14:20:07Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-15T00:38:14Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/arm64"}
  • VM or VMI specifications: https://github.com/kubevirt/kubevirt/blob/main/examples/vm-cirros.yaml
  • Cloud provider or hardware configuration: N/A
  • OS (e.g. from /etc/os-release): macOS Ventura 13.2.1 (22D68)
  • Kernel (e.g. uname -a): N/A
  • Install tools: N/A
  • Others: N/A

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Comments: 15 (2 by maintainers)

Most upvoted comments

Hey @hugeps ,Kubevirt doesn’t have a special scheduler. You need to check your default scheduler for more info.

@ByteFlyCoding in this case you need to enable emulation, please see https://github.com/kubevirt/kubevirt/blob/main/docs/software-emulation.md

+1
xpivarc on Sep 13, 2023
Read more comments on GitHub
← kubevirt: VM crash with network passt mode
kubevirt: VMI created with the pod/masquerade network on Calico cannot be accessed →