democratic-csi: zfs-local-dataset does not correct paths for chroot

I’m using the zfs-local-dataset driver on Nomad (following some of the instructions in #164), but when trying to spin up a job, it immediately fails with the error:

# can safely ignore this line
/usr/local/bin/mount: illegal option -- o
filesystem 'tank/nomad/data/postgresql_db_volume' cannot be mounted at '/csi/staging/postgresql/rw-file-system-single-node-writer' due to canonicalization error: No such file or directory

I dug into this some more, and found that this is an issue with the plugin, due to this line: https://github.com/democratic-csi/democratic-csi/blob/19197f9515da57f569c670d14fefb1bfd4e43460/docker/mount#L32

Nomad seems to be following spec here, as the staging_target_path passed is a valid path within the container, but democratic-csi proceeds to evaluate that path outside of the container, in the context of the host (the directory in the container is /csi/staging/postgresql/rw-file-system-single-node-writer, but the directory in the host is /var/lib/nomad/client/csi/monolith/org.democratic-csi.zfs-dataset/staging/postgresql/rw-file-system-single-node-writer)

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 15 (9 by maintainers)

Most upvoted comments

Sorry about the huge delay there, I tore down my Nomad cluster and forgot to test this. I just set it back up, and it seems to work fine with USE_HOST_MOUNT_TOOLS=0 with this job configuration:

job "zfs-storage-controller-dataset" {
  datacenters = ["dc1"]
  type        = "service"

  constraint {
    attribute = "${node.unique.id}"
    value = "7114b0d2-c3a0-2dc7-e15d-aa8a63d9ffda"
  }

  group "controller" {
    task "controller-sidecar" {
      lifecycle {
        hook = "poststart"
        sidecar = true
      }

      driver = "docker"

      env {
        BIND_TO="unix:///csi/csi.sock"
        PROXY_TO="unix:///csi/csi.sock.internal"
      }

      config {
        image = "docker.io/democraticcsi/csi-grpc-proxy:latest"
        
        privileged = true
        ipc_mode = "host"

        # This probably isn't a good idea!
        mount {
          type = "bind"
          target = "/csi"
          # Will change based on the csi_plugin.id and your data_dir!
          source = "/var/lib/nomad/client/csi/monolith/org.democratic-csi.zfs-dataset"
          readonly = false
          bind_options {
            propagation = "rshared"
          }
        }
      }
    }

    task "controller" {
      driver = "docker"

      env {
        CSI_NODE_ID = "${attr.unique.hostname}"
        USE_HOST_MOUNT_TOOLS = "0"
      }

      csi_plugin {
        # must match --csi-name arg
        id        = "org.democratic-csi.zfs-dataset"
        type      = "monolith"
        mount_dir = "/csi"
      }

      config {
        image = "democraticcsi/democratic-csi:next"

        args = [
          "--csi-version=1.5.0",
          "--csi-name=org.democratic-csi.zfs-dataset",
          "--driver-config-file=${NOMAD_TASK_DIR}/driver-config-file.yaml",
          "--log-level=debug",
          "--csi-mode=controller",
          "--csi-mode=node",
          "--server-socket=/csi/csi.sock.internal",
        ]

        privileged = true
        ipc_mode = "host"
        network_mode = "host"

        mount {
          type = "bind"
          target = "/host"
          source = "/"
          readonly = false
        }
      }

      template {
        destination = "${NOMAD_TASK_DIR}/driver-config-file.yaml"

        data = <<EOH
driver: zfs-local-dataset

zfs:
  datasetParentName: tank/nomad/data 
  detachedSnapshotsDatasetParentName: tank/nomad/snapshots

  datasetProperties:
    # key: value

  datasetEnableQuotas: true
  datasetEnableReservation: false 
  datasetPermissionsMode: "0777"
EOH
      }

      resources {
        cpu    = 250
        memory = 256
      }
    }
  }
}

The bind mount to /host is still necessary, but otherwise it seems to work great.

hatf0 on May 11, 2022

Sorry - I haven’t been able to make the time to test this change out yet. I’ll give it a run today and see how it is.

hatf0 on Apr 21, 2022