kubevirt: server error. command SyncVMI failed

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug /kind enhancement

What happened: By deploying the virtual machine, the pod has been started, but vmi has not been started, describe vmi found the following:

Status:
  Active Pods:
    3951878e-21c1-42e1-aa86-83e9df56117d:  k8s-master
  Conditions:
    Last Probe Time:       <nil>
    Last Transition Time:  <nil>
    Status:                True
    Type:                  LiveMigratable
    Last Probe Time:       <nil>
    Last Transition Time:  2021-11-04T10:00:54Z
    Message:               server error. command SyncVMI failed: "preparing host-disks failed: chown /var/run/kubevirt-private/vmi-disks/harddrive/disk.img: operation not permitted"
    Reason:                Synchronizing with the Domain failed.
    Status:                False
    Type:                  Synchronized
  Guest OS Info:
  Migration Method:  BlockMigration
  Node Name:         k8s-master
  Phase:             Scheduled
  Qos Class:         Burstable
Events:
  Type     Reason            Age              From                       Message
  ----     ------            ----             ----                       -------
  Normal   SuccessfulCreate  21s              virtualmachine-controller  Created virtual machine pod virt-launcher-windows-nfs-1-pvrnb
  Warning  SyncFailed        1s (x2 over 2s)  virt-handler, k8s-master   server error. command SyncVMI failed: "preparing host-disks failed: chown /var/run/kubevirt-private/vmi-disks/harddrive/disk.img: operation not permitted"

What you expected to happen: Both pod and vmi are running, and the virtual machine can be started. How to reproduce it (as minimally and precisely as possible):

The nfs used in the back-end uses storageClass + provisioner to automatically create and bind PV through PVC.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: winhd-nfs
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Gi
  storageClassName: nfs
---
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
  name: windows-nfs-1
spec:
  running: false
  template:
    metadata:
      labels:
        kubevirt.io/domain: windows-nfs-1
    spec:
      domain:
        cpu:
          cores: 4
        resources: 
          requests: 
            memory: 8G
          limits: 
            memory: 8G
        devices:
          disks:
          - bootOrder: 1
            cdrom:
              bus: sata
            name: cdromiso
          - bootOrder: 2
            disk:
              bus: virtio
            name: harddrive
          - cdrom:
              bus: sata
            name: virtiocontainerdisk
          interfaces: 
          - name: default 
            masquerade: {}
          rng: {}
        machine:
          type: q35
      networks: 
      - name: default
        pod: {}
      terminationGracePeriodSeconds: 0
      volumes:
      - name: cdromiso
        persistentVolumeClaim:
          claimName: windows-iso
      - name: harddrive
        persistentVolumeClaim:
          claimName: winhd-nfs
      - containerDisk:
          image: kubevirt/virtio-container-disk
          imagePullPolicy: Never
        name: virtiocontainerdisk

Anything else we need to know?:

Environment:

  • KubeVirt version (use virtctl version):
Client Version: version.Info{GitVersion:"v0.41.2", GitCommit:"29ce26e09eeed30f89f3fd5d1ad0598955de9648", GitTreeState:"clean", BuildDate:"2021-07-28T16:23:53Z", GoVersion:"go1.13.14", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{GitVersion:"v0.41.2-dirty", GitCommit:"29ce26e09eeed30f89f3fd5d1ad0598955de9648", GitTreeState:"dirty", BuildDate:"2021-07-28T16:57:02Z", GoVersion:"go1.13.14", Compiler:"gc", Platform:"linux/amd64"}
  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"d5dfb5cb416fcc32ad556b0d253307c9267b1d30", GitTreeState:"", BuildDate:"2021-07-20T15:01:11Z", GoVersion:"go1.14", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"d5dfb5cb416fcc32ad556b0d253307c9267b1d30", GitTreeState:"", BuildDate:"2020-07-30T01:54:02Z", GoVersion:"go1.14.2", Compiler:"gc", Platform:"linux/amd64"}
  • VM or VMI specifications:
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 28 (13 by maintainers)

Most upvoted comments

About it will still copy the [...] disk blocks ,I think so: It should be displayed only when the disk block is copied. Maybe my idea is wrong, so let’s see how you implemented it.

Ok, I think what may not be obvious is that you also defined a disk which is not on shared storage:

      - containerDisk:
          image: kubevirt/virtio-container-disk
          imagePullPolicy: Never

kubevirt automatically creates an overlay file and will copy it if something gets modified. So I think this may be related to the cdrom drive which is read-only by default. This is probably related to https://github.com/kubevirt/kubevirt/pull/5020. However that fix should be in 0.41 already.

To try out if it is a similar issue, can you make the cdrom writeable by setting readonly to false:

          - cdrom:
              bus: sata
              readonly: false
            name: virtiocontainerdisk

That was a workaround for this bug.

@vladikr FYI