terraform-provider-libvirt: Permission Denied

System Information

Linux distribution

 Ubuntu 18.04-lts

Terraform version

terraform -v
Terraform v0.11.11
+ provider.libvirt (unversioned)
+ provider.template v2.0.0

Provider and libvirt versions

terraform-provider-libvirt -version
terraform-provider-libvirt 0.5.1
Compiled against library: libvirt 4.0.0
Using library: libvirt 4.0.0
Running hypervisor: QEMU 2.11.1
Running against daemon: 4.0.0

If that gives you “was not built correctly”, get the Git commit hash from your local provider repository:

git describe --always --abbrev=40 --dirty

Checklist

  • Is your issue/contribution related with enabling some setting/option exposed by libvirt that the plugin does not yet support, or requires changing/extending the provider terraform schema?

    • Make sure you explain why this option is important to you, why it should be important to everyone. Describe your use-case with detail and provide examples where possible.
    • If it is a very special case, consider using the XSLT support in the provider to tweak the definition instead of opening an issue
    • Maintainers do not have expertise in every libvirt setting, so please, describe the feature and how it is used. Link to the appropriate documentation
  • Is it a bug or something that does not work as expected? Please make sure you fill the version information below:

Description of Issue/Question

Setup

(Please provide the full main.tf file for reproducing the issue (Be sure to remove sensitive information)

provider "libvirt" {
  uri = "qemu:///system"
}

#
## VM Varibles
#
variable "instance_count" {
  default = "1"
}

variable "disk_img" {
  default = "https://cloud-images.ubuntu.com/releases/xenial/release/ubuntu-16.04-server-cloudimg-amd64-disk1.img"
}

#
## Networking Varibles
#
variable "vm_network_addresses" {
  description = "Defines the network in the CIDR format"
  default = "10.0.1.0/24"
}

variable "vm_network_name" {
  description = "Defines the network name"
  default = "vm_network"
}

variable "domain_name" {
  default = "k8s.local"
}

#---- End Of Varibles ---#

resource "libvirt_volume" "os_image" {
  name   = "os_image"
  source = "${var.disk_img}"
}

#volume
resource "libvirt_volume" "volume" {
  name           = "volume-${count.index}"
  base_volume_id = "${libvirt_volume.os_image.id}"
  count = "${var.instance_count}"
}

# network

resource "libvirt_network" "vm_network" {

   name = "${var.vm_network_name}"
   addresses = ["${var.vm_network_addresses}"]
   domain    = "${var.domain_name}"

   mode = "nat"

   dhcp {
       enabled = true
   }

   dns {
     local_only = true
   }

   autostart = true
}

#Domain
resource "libvirt_domain" "domain" {
  name = "master-${count.index}"

  memory = "1024"
  vcpu = "1"

  network_interface {
    network_id   = "${libvirt_network.vm_network.id}"
    hostname = "master-${count.index}"
    wait_for_lease = true
  }

  console {
    type        = "pty"
    target_port = "0"
    target_type = "serial"
  }

  console {
    type        = "pty"
    target_type = "virtio"
    target_port = "1"
  }

  disk {
    volume_id = "${libvirt_volume.volume.id}"
  }

  count = "${var.instance_count}"
}

#############
## Outputs ##
#############

output "disk_id" {
  value = "${libvirt_volume.volume.*.id}"
}

output "network_id" {
  value = "${libvirt_network.vm_network.id}"
}

output "ip_addresses" {
  value = "${libvirt_domain.domain.*.network_interface.0.addresses}"
}

Steps to Reproduce Issue

(Include debug logs if possible and relevant).

Error: Error applying plan:

1 error(s) occurred:

* libvirt_domain.domain: 1 error(s) occurred:

* libvirt_domain.domain: Error creating libvirt domain: virError(Code=1, Domain=10, Message='internal error: process exited while connecting to monitor: 2019-01-28T02:29:14.861688Z qemu-system-x86_64: -drive file=/var/lib/libvirt/images/volume-0,format=qcow2,if=none,id=drive-virtio-disk0: Could not open '/var/lib/libvirt/images/volume-0': Permission denied')


Additional information:

Do you have SELinux or Apparmor/Firewall enabled? Some special configuration? Have you tried to reproduce the issue without them enabled?

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 1
  • Comments: 23 (1 by maintainers)

Commits related to this issue

Most upvoted comments

I fixed this problem with this:

On Ubuntu distros SELinux is enforced by qemu even if it is disabled globally, this might cause unexpected Could not open '/var/lib/libvirt/images/<FILE_NAME>': Permission denied errors. Double check that security_driver = "none" is uncommented in /etc/libvirt/qemu.conf and issue sudo systemctl restart libvirt-bin to restart the daemon.

Still running into this libvirt issue on 20.04.2 regardless of the pool used.

Slightly modified workaround is setting security_driver = "none" in /etc/libvirt/qemu.conf but followed by sudo systemctl restart libvirtd.

@gdombrov and @johnjameswhitman – That fix worked for me as well on Ubuntu 20.04.1.

@MalloZup Makes sense. Thanks for all of your work on this project!

Could very well be same as #97 see fix there, already in documentation, but very easy to miss, still an issue on ubuntu 18.04, documentation; 22f096d

If you have a problem with Could not open ‘/var/lib/libvirt/images/example.qcow2’: Permission denied’ You should add: sudo vim /etc/apparmor.d/libvirt/TEMPLATE.qemu

/var/lib/libvirt/images/**.qcow2 rwk, /var/lib/libvirt/images/**.raw rwk, /var/lib/libvirt/images/**.img rwk,

sudo systemctl restart libvirtd

I fixed this problem with this:

On Ubuntu distros SELinux is enforced by qemu even if it is disabled globally, this might cause unexpected Could not open '/var/lib/libvirt/images/<FILE_NAME>': Permission denied errors. Double check that security_driver = "none" is uncommented in /etc/libvirt/qemu.conf and issue sudo systemctl restart libvirt-bin to restart the daemon.

good

This seems like a known issue w/ apparmor’s handling of volumes from a libvirt pool (apparmor profiles require the full path to allow file-access, but virt-aa-helper doesn’t actually map the volume to its path on the filesystem). Some details:

The quick-fix for me was setting security = "none" in /etc/libvirt/qemu.conf until ubuntu/libvirt fixes the apparmor profiles.

Terraform seems to create the pool with owner/group set to 0:

root@lstor1:~# virsh pool-dumpxml lstor1-pool-root
<pool type='dir'>
  <name>lstor1-pool-root</name>
  <uuid>479e75d0-8442-4ca0-9bf9-5bf36ca4584c</uuid>
  <capacity unit='bytes'>23096088592384</capacity>
  <allocation unit='bytes'>525992448</allocation>
  <available unit='bytes'>23095562599936</available>
  <source>
  </source>
  <target>
    <path>/srv/lstor1/virt/root</path>
    <permissions>
      <mode>0755</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</pool>

However libvirtd (which writes the uploaded files, such as the cloud init, or the base ISO) runs as non-root, writing those files as libvirt-qemu:kvm:

root@lstor1:/srv/lstor1/virt/root# ls -tla
total 513473
-rw-r--r-- 1 libvirt-qemu kvm     374784 Oct 28 17:23 testvm-cloudinit.iso
drwxr-xr-x 2 root         root         5 Oct 28 17:23 .
-rw-r--r-- 1 root         root    196912 Oct 28 17:19 testvm-root
-rw-r--r-- 1 libvirt-qemu kvm  527368192 Oct 28 00:54 lstor1-focal-baseimg
drwxr-xr-x 4 root         root         4 Oct 27 21:57 ..
root@lstor1:/srv/lstor1/virt/root#

I think there may be some XSLT that can be applied to the pool definitions that will change the owner/group to the right settings, but I will need to research them.

This is still true in ubuntu 20.04, thanks to @andersla that pointed to the security_driver thing, that did the trick. That may be added back to the readme.

security

Ubuntu 20.04 with apparmor, I disable apparmor without any effect. Start apparmor and set a value security_driver = none in qemu.conf and everything right.

@MalloZup - After another day of research I have found some historical context that is interesting. However I DO AGREE… This is NOT an issue with terraform-provider-libvirt.

This looks like it may be a recurrence of an issue where QEMU is failing to generate the correct permissions for qcow2 formatted volumes. Either way this issue should be closed on your side. Would be great if we could get some traction with the QEMU team and see what they can find. Thanks again for supporting this provider for terraform!