democratic-csi: Unable to mount iscsi devices in nomad deployment

Basic docs and examples started here: https://github.com/democratic-csi/democratic-csi/tree/master/docs/Nomad

Are you opening a port on the node deployment?

No, zero ports are opened there.

job "democratic-storage-node-iscsi" {
  datacenters = ["discovery"]
  type        = "system"

  group "node" {
    task "node" {
      driver = "docker"

      config {
        image = "registry.light.kow.is/docker-hub/democraticcsi/democratic-csi:latest"
        auth {
          username = "robot$readbot"
          password = "qIw1FceV5Z8YuGMYaj5O3iltjrAbrJRp"
        }

        args = [
          "--csi-version=1.2.0",
          "--csi-name=org.democratic-csi.iscsi",
          "--driver-config-file=${NOMAD_TASK_DIR}/driver-config-file.yaml",
          "--log-level=debug",
          "--csi-mode=node",
          "--server-socket=/csi-data/csi.sock",
        ]

        network_mode = "host"
        ipc_mode = "host"
        privileged = true

        mount {
          type = "bind"
          target = "/host"
          source = "/"
          readonly=false
        }
        mount {
          type = "bind"
          target = "/etc/iscsi"
          source = "/etc/iscsi"
          readonly = false
        }
        mount {
          type = "bind"
          target = "/var/lib/iscsi"
          source = "/var/lib/iscsi"
          readonly = false
        }
      }

      csi_plugin {
        id        = "truenas"
        type      = "node"
        mount_dir = "/csi-data"
      }

      template {
        destination = "${NOMAD_TASK_DIR}/driver-config-file.yaml"

        data = <<EOH
driver: freenas-iscsi
instance_id:
httpConnection:
  protocol: https
  host: 10.10.220.110
  port: 443
  # use only 1 of apiKey or username/password
  # if both are present, apiKey is preferred
  # apiKey is only available starting in TrueNAS-12
  apiKey: LOLAPIKEY
  username: root
  #password:
  allowInsecure: true
  # use apiVersion 2 for TrueNAS-12 and up (will work on 11.x in some scenarios as well)
  # leave unset for auto-detection
  apiVersion: 2
sshConnection:
  host: 10.10.220.110
  port: 22
  username: root
  # use either password or key
  privateKey: |
    LOL PRIVATE KEY LOLOLOLOL
zfs:
  # can be used to override defaults if necessary
  # the example below is useful for TrueNAS 12
  #cli:
  #  sudoEnabled: true
  #
  #  leave paths unset for auto-detection
  #  paths:
  #    zfs: /usr/local/sbin/zfs
  #    zpool: /usr/local/sbin/zpool
  #    sudo: /usr/local/bin/sudo
  #    chroot: /usr/sbin/chroot
  
  # can be used to set arbitrary values on the dataset/zvol
  # can use handlebars templates with the parameters from the storage class/CO
  # TODO: set up handlebars templates to make this far more awesome
  datasetProperties:
     "org.freenas:description": "created via democratic-csi"

  datasetParentName: vmPool/nomad/vols
  # do NOT make datasetParentName and detachedSnapshotsDatasetParentName overlap
  # they may be siblings, but neither should be nested in the other
  detachedSnapshotsDatasetParentName: vmPool/nomad/snaps
  zvolCompression: ""
  zvolDedupe: ""
  zvolEnableReservation: False
  zvolBlocksize: ""
iscsi:
  targetPortal: "10.10.220.110:3260"
  targetPortals: []
  interface:

  # MUST iensure uniqueness
  # full iqn limit is 223 bytes, plan accordingly
  # default is //template name isn't defined!
  #nameTemplate: "{ { parameters.[csi.storage.k8s.io/pvc/namespace] }}-{ { parameters.[csi.storage.k8s.io/pvc/name] }}"
  namePrefix: csi-
  nameSuffix: "-discovery"
  # add as many as needed
  targetGroups:
    # get the correct ID from the "portal" section in the UI
    - targetGroupPortalGroup: 2
      # get the correct ID from the "initiators" section in the UI
      targetGroupInitiatorGroup: 1
      # None, CHAP, or CHAP Mutual
      targetGroupAuthType: None
      # get the correct ID from the "Authorized Access" section of the UI
      # only required if using Chap
      targetGroupAuthGroup:

  extentInsecureTpc: true
  extentXenCompat: false
  extentDisablePhysicalBlocksize: true
  # 512, 1024, 2048, or 4096,
  extentBlocksize: 512
  # "" (let FreeNAS decide, currently defaults to SSD), Unknown, SSD, 5400, 7200, 10000, 15000
  extentRpm: "SSD"
  # 0-100 (0 == ignore)
  extentAvailThreshold: 0
EOH
      }

      resources {
        cpu    = 30
        memory = 150
      }
    }
  }
}

I’m going to attempt installing a single-node Nomad setup and get myself a little more familiar with the ecosystem. Maybe with a bit more knowledge I can be of more help.

I may have some updates to the ubuntu setup of the iscsi things. I need to verify it, and I’ve not had time to set up a new host node from scratch yet.

That’s fair. I’ve never had any issues and I’m aware of some very large organizations using the driver who have never reported any issues either…not that either of those is much consolation 😦

Also note the idea is common in the arena: https://www.docker.com/blog/road-to-containing-iscsi/

You could hypothetically copy the binary from the host to an isolated chroot but you’d need to copy all the linked libraries etc with it which could prove challenging. Mounting / is much easier unfortunately.

https://github.com/democratic-csi/democratic-csi/blob/master/docs/Nomad/examples/democratic-csi-iscsi-node.hcl#L35

The reason the / mount is required is because the driver actually uses the iscsiadm binary from the host (that binary must match versions with the daemon etc making it effectively impossible to just install and use directly in the driver container). This is all pretty transparent in the node driver container because I have a little shell script (https://github.com/democratic-csi/democratic-csi/blob/master/docker/iscsiadm) called iscsiadm which is simply a wrapper which invoke the real binary using chroot with…you guessed it /host as the root 😃

The basic premise with csi is that the nodes themselves handle mounting before the target container is ever created (which is why you don’t need iscsi tools in every container etc). So effectively what happens is Nomad starts the high-level process to launch a new workload and sees a volume is at play, Nomad sends a grpc request to the appropriate csi driver on the appropriate node telling it to mount the volume (stage and/or publish in csi parlance). The csi driver is instructed to mount the volume at a path designated by Nomad, after that succeeds, when the workload container actually launches Nomad automatically does a bind mount to the target location using the path previously mentioned as the designated path on the host as the source dir.

If those are the exact resources you used then they are very low and I’m not surprised if you had erratic behavior. I’m not sure what health check features are built-in to Nomad to automatically restart containers in failure scenarios but those would be good to add to the files I’m putting together for sure.

https://discuss.hashicorp.com/t/setting-up-democratic-csi-volumes-using-nomad/29077

Just to crosspost, in case someone comes across this and is wondering the same things.

Oh right, forgot about the CSC stuff so yeah, that’s why it’s being launched in that manner. Maybe it’s something to do with that? I’m not sure that port error is related or not.

OK, so having said all that, what is the current state of things? attaching etc is working (after using CSC)?

I do have an application running with a volume created via csc, registered via terraform and mounted as a file-system. It’s stayed up and running for quite a while. I haven’t had an opportunity to start/stop it so it ends up on a new node, I only have one amd64 node at the moment, and it needs amd64 arch.

I think it’s probably good, I need to redo some of the iSCSI setup on the host itself, I think ubuntu is doing some things that are incompatible with how this plugin expects iscsi to work. Like, not starting it if the /etc/iscsi/nodes folder is empty. It might be empty in this case, and things are dynamically registered, so it’s dumb. Simple overrides with systemd should solve it. I’ll write up a blog post about it. I’m also about to post in the nomad community forums about the volume configuration and see if I can’t figure out why it’s not working.

Thanks for all your help!

As an FYI the attachment mode is about how the storage is presented to the container. block-device is literally passing through a block device to your container instead of a formatted fs. That’s why it’s highly likely you want file-system. There are pretty limited use cases for block-device.

Gotcha, I appreciate that. I wasn’t sure if it had something to do with iscsi vs nfs.

That appears to be something to take up with the Nomad team. By default I don’t bind the server to any tcp port but only a unix domain socket. Maybe try to delete the container and let it restart to see if it becomes healthy?

Yeah I’m near certain that it’s because you commented or removed the targetPortals key in the config…it should be an empty array value.

In the upcoming code I handle this case gracefully but it’s not in master just yet (probably will be by end of week as I have about 4 months of work ready to land).

I should also note that I’m relatively confident Nomad has expanded csi support even further eliminating the need to manually use csc etc for volume creation and all that: https://github.com/hashicorp/nomad/issues/8212

HOLY CRAP IT WORKED.

I needed to delete that existing iqn file. It never should’ve been there. I probably set it up while trying to figure out if my iscsi stuff was working by running iscsiadm -m discovery -t st -p 10.10.220.110 That created an entry in the /etc/iscsi/nodes folder that was causing problems I think.

Deleting that, a new one was created, and the task is placed! Now to see if it’s actually working… Yeah, it’s seeing the 5GB volume that I created, and it’s working just fine now. I’ll see if I can give it a bit of soak time, and make sure things work from scratch again.

I ended up using this issue as a sorta running log of my fails, so it’s a stream of conciousness here. The bottom parts are the most recent.

Well, I’ve made sure I’m running as --net host here and --ipc host here

Unfortunately, I still get the same error message.

 failed to setup alloc: pre-run hook "csi_hook" failed: node plugin returned an internal error, check the plugin allocation logs for more information: rpc error: code = Internal desc = {"code":6,"stdout":"","stderr":"iscsiadm: Could not make /etc/iscsi/ifaces. HW/OFFLOAD iscsi may not be supported\niscsiadm: Could not read iface . Error 6\niscsiadm: Error while adding record: encountered iSCSI database failure\n"} 

Oh wait, I had to mkdir /etc/iscsi/ifaces and then I got a bit further.

failed to setup alloc: pre-run hook "csi_hook" failed: node plugin returned an internal error, check the plugin allocation logs for more information: rpc error: code = Internal desc = {"code":6,"stdout":"","stderr":"iscsiadm: Could not open /etc/iscsi/nodes/iqn.2005-10.org.freenas.ctl:csi-radarr-volume-discovery/10.10.220.110,3260: Read-only file system\niscsiadm: Error while adding record: encountered iSCSI database failure\n"} 

I just need to figure out why it’s a read-only filesystem, even though I explicitly told it not to be. The mount for /host was read-only, it’s now read-write.

next fail:

 failed to setup alloc: pre-run hook "csi_hook" failed: node plugin returned an internal error, check the plugin allocation logs for more information: rpc error: code = Internal desc = {"code":6,"stdout":"","stderr":"iscsiadm: iface iter could not stat /etc/iscsi/nodes/iqn.2005-10.org.freenas.ctl:csi-radarr-volume-discovery/10.10.220.110,3260.\niscsiadm: Could not execute operation on all records: encountered iSCSI database failure\n"} 

I don’t have a file ending in ,3260 I have one ending in ,3260,2… I think there’s something there, but I’m not sure what it should be. This leads me to believe that something’s wrong with my volume configuration… It’s missing a value

inside the /etc/iscsi/nodes/iqn.2005-10.org.freenas.ctl:csi-radarr-volume-discovery/10.10.220.110,3260,2/default there’s a node.tpgt=2 value, maybe that’s somehow missing in my volume config? It’s the Target Portal Group Tag, and it’s 2 in my case.

Somehow the file is different for what it’s looking for, vs what it found.

That error could be related to the pod container not running in the host networking namespace (ie: --net host, however you achieve that with Nomad).

Awesome! Have you ran through the node prep steps in the doc relating to iscsi? That should be a pretty good start. Note that technically multipath is not required but you are welcome to use it if it makes sense for your scenario.

That is correct. All the mounts/settings mentioned should be for the node application deployed to the cluster and not the actual workloads. Sorry that was confusing. Essentially the node application needs to run highly privileged and with a bunch of the host tools available to it so it can do the business of attaching to iscsi targets, formatting disks, etc, etc.

It shouldn’t be at csi hook time (if I understand you correctly) but rather when you deploy the node portion of the driver. All of the previous comment had to do with how the containers are deployed should be running before any workload attempts to be started.

My experience with Nomad is essentially 0 so forgive my ignorance. However the problem at hand here is that you need to mount the host / into /host inside the node container (this would be great to document how exactly this is achieved with Nomad). I would recommend browsing through this to get ideas of what’s needed for iscsi as well: https://github.com/democratic-csi/charts/blob/master/stable/democratic-csi/templates/node.yaml

From a cursory look it appears you’ll need:

• make sure you’ve installed and properly configured iscsi on your hosts (iscsiadm for example must be available directly on the host, you should have iscsid started, etc)
• / mounted to /host in the container
• /etc/iscsi mounted to /etc/iscsi in the container (bidirectional)
• /var/lib/iscsi mounted to /var/lib/iscsi in the container (bidirectional)
• the container should be launched using the host networking namespace (--net host or equivalent)
• the container should be launched using the host ipc (--ipc host or equivalent…this is for multipath specifically)

I think that should get you going on the right track.