terraform-provider-lxd: zfs.block_mode or initial.zfs.block_mode broken?

Hi,

I’ve been trying to implement this for a couple of containers that will be hosting docker swarm services to try and eliminate the IO bottleneck on ZFS, however using this device block:

  device {
    name = "root"
    type = "disk"
    properties = {
      "path"           = "/"
      "pool"           = "local"
      #"size"           = "200GiB"
      "initial.zfs.block_mode" = true
    }
  }

Results in the following errors:

╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to lxd_instance.int-dev-zvol["int-dev-zvol01"], provider "provider[\"registry.terraform.io/terraform-lxd/lxd\"]"
│ produced an unexpected new value: .device: actual set element cty.ObjectVal(map[string]cty.Value{"name":cty.StringVal("root"),
│ "properties":cty.MapVal(map[string]cty.Value{"initial.zfs.block_mode":cty.StringVal("true"), "path":cty.StringVal("/"),
│ "pool":cty.StringVal("local")}), "type":cty.StringVal("disk")}) does not correlate with any element in plan.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to lxd_instance.int-dev-zvol["int-dev-zvol01"], provider "provider[\"registry.terraform.io/terraform-lxd/lxd\"]"
│ produced an unexpected new value: .device: block set length changed from 1 to 2.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵

This does appear to create the zvol block devices as desired, however:

rpool/lxd/containers/int-dev-swarm01                                                       946M  2.08T     1.74G  legacy
rpool/lxd/containers/int-dev-swarm02                                                       947M  2.08T     1.74G  legacy
rpool/lxd/containers/int-dev-worker01                                                      947M  2.08T     1.74G  legacy
rpool/lxd/containers/int-dev-zvol01                                                       1.03G  2.08T     1.03G  -
rpool/lxd/virtual-machines/int-dev-lab04.block                                            3.76G  2.08T     4.52G  -

(The ‘legacy’ ones are normal datasets, and lab04 has a manually created, attached and functional ext4 block device)

Am I omitting something basic? Essentially I just need the provider to create zvol block devices for specific container instances rather than zfs datasets when specified.

If I manually set the pool to use zfs.block_mode this works without issue, without the disk device being specified.

Versions: lxd 5.19 lxd provider 2.3.0 terraform 1.5.3

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 28 (13 by maintainers)

Most upvoted comments

Thanks for the feedback, I agree that we should include such an example in the documentation.

Morning Din,

I can confirm with the above code plus a minor tweak (specifying target), this now works as expected.

resource "lxd_volume" "dockerzvol" {
  name  = "dockerzvol"
  pool  = "local"
  target = "host04" # <-- this must still be specified
  # Setting content_type to "filesystem" (default value), creates a filesystem.
  #
  # Setting content_type to "block", creates a block device without FS.
  # Those volumes can be attached only to virtual machines.
  content_type = "filesystem"

  config = {
    # Set zfs.block_mode to true to create block backed device.
    "zfs.block_mode" = true
    size             = "200GiB"
  }
}

# and under the instance...

  device {
    name = "dockerzvol"
    type = "disk"
    properties = {
      "path"   = "/var/lib/docker"
      "pool"   = "local"
      "source" = lxd_volume.dockerzvol.name
    }
  }

# which results in:
root@int-dev-zvol01:~# df -lhT
Filesystem                                    Type           Size  Used Avail Use% Mounted on
rpool/lxd/containers/int-dev-zvol01         zfs            4.1T  942M  4.1T   1% /
none                                          tmpfs          492K  4.0K  488K   1% /dev
tmpfs                                         tmpfs          100K     0  100K   0% /dev/lxd
/dev/zvol/rpool/lxd/custom/default_dockerzvol ext4           196G   28K  186G   1% /var/lib/docker

Thanks for all the help and guidance with this - I’d have thought this would be a reasonably common usecase/scenario enough for the docs.

I can confirm that after updating to 5.21.2 LTS, the profile works as expected 😃 thank you!

Regarding ‘container’ type volumes (or anything other than the ‘custom’ type), is it on the roadmap for our usecase to be achievable? aka, attaching a standard container with a zvol block volume mounted to /var/lib/docker, configured and deployed with this provider?

I think the issue was detected in our test suite, but here is the PR: https://github.com/canonical/lxd/pull/13656 and https://github.com/canonical/lxd/pull/13861

Have a nice weekend as well 😃

Excellent, thanks for letting me know. I’ll upgrade the cluster next week and we’ll go from there. Do you have a link to the issue, out of interest?

Have a good weekend!

Perfect. I’ll give this a try tomorrow and let you know the result. Thank you!

Sure.

lxd_profile:

resource "lxd_profile" "int-dev-zvol" {
  remote = var.lxd_remote
  name   = "int-dev-zvol"

  config = {
    "user.user-data"     = <<-EOF
      #cloud-config
      users:
        - name: ${var.lxd_user}
          no-log-init: true
          ssh-authorized_keys:
            - ${file(var.ssh_pub_key)}
      ssh_pwauth: True 
    EOF
    "boot.autostart"      = true
    "security.privileged" = true
    "security.nesting"    = true
    "security.syscalls.intercept.mknod"    = true
    "security.syscalls.intercept.setxattr" = true
    "security.syscalls.intercept.sysinfo"  = true
    "linux.kernel_modules"                 = var.lxd_kmods
    "limits.cpu"                           = "4"
    "limits.memory"                        = "4GiB"
    "limits.memory.swap"                   = false
    "limits.memory.enforce"                = "hard"
    "security.protection.delete"           = false
    #"zfs.block_mode"                       = true
  }
  
  lifecycle {
    ignore_changes = all
  }

}

lxd_instance:

resource "lxd_instance" "int-dev-zvol" {
  for_each = var.instances.zvol
  name     = each.key
  image    = var.lxd_image
  remote   = var.lxd_remote
  target   = each.value.target
  profiles = ["default", lxd_profile.int-dev-zvol.name]
  type     = each.value.type != null ? each.value.type : null

  config = {
        "user.access_interface" = each.value.type == "virtual-machine" ? var.lxd_vm_interface : var.lxd_container_interface
  }

  device {
    name = var.lxd_container_interface
    type = "nic"
    properties = {
      "nictype" = var.lxd_nic_type
      "parent"  = var.lxd_host_interface
    }
  }

  device {
    name = "root"
    type = "disk"
    properties = {
      "path"           = "/"
      "pool"           = "local"
      "size"           = "200GiB"
      "initial.zfs.block_mode" = "true"
    }
  }

  connection {
    type        = "ssh"
    user        = var.lxd_user
    private_key = file(var.ssh_private_key)
    host        = self.ipv4_address
    timeout     = var.ssh_timeout
  }

  provisioner "remote-exec" {
    inline = ["echo 'SSH connection success'"]
  }

}

So it seems this does function, but the zvol/disk device can only be defined under lxd_instance, not lxd_profile.