cloudstack: Mounting NFS 4 shares of secondary storage fails on CS Hosts

ISSUE TYPE
  • Bug Report
COMPONENT NAME
Cloudstack-Agent
Secondary-Storage
NFS
CLOUDSTACK VERSION
4.15.2 (Others not tested)
CONFIGURATION

Advanced Network Sepparated storage network

OS / ENVIRONMENT
  • Completly new environment
  • Ubuntu 20.04 LTS for Hosts
  • NFS Server supporting NFS 4.1 / 4.0
  • Global Setting “secstorage.nfs.version” set to 4 or 4.1
SUMMARY

Deploying a new instance fails due to error “failed to create netfs mount” when mounting the iso / templete from secondary storage for installation of instance.

STEPS TO REPRODUCE
Cloudstack environment with NFSv 4.1 / 4 "only"
Deploy instance
EXPECTED RESULTS
- Host mounts nfs-share for providing installation media to new guest / created instance for installation
ACTUAL RESULTS
`1-09-21 22:48:11,198 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:6a9a5a89) org.libvirt.LibvirtException: internal error: Child process (/bin/mount 172.17.3.6:/volume3/secondary_storage/template/tmpl/6/216 /mnt/4ca9f81b-7a76-326f-9f2d-e64b5a5bee99 -o nodev,nosuid,noexec) unexpected exit status 32: mount.nfs: Connection timed out

2021-09-21 22:48:11,199 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:6a9a5a89) Failed to create netfs mount: 172.17.3.6:/volume3/secondary_storage/template/tmpl/6/216
org.libvirt.LibvirtException: internal error: Child process (/bin/mount 172.17.3.6:/volume3/secondary_storage/template/tmpl/6/216 /mnt/4ca9f81b-7a76-326f-9f2d-e64b5a5bee99 -o nodev,nosuid,noexec) unexpected exit status 32: mount.nfs: Connection timed out

        at org.libvirt.ErrorHandler.processError(Unknown Source)
        at org.libvirt.ErrorHandler.processError(Unknown Source)
        at org.libvirt.Connect.storagePoolCreateXML(Unknown Source)
        at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createNetfsStoragePool(LibvirtStorageAdaptor.java:255)
        at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:621)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePoolByURI(KVMStoragePoolManager.java:284)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.getPhysicalDiskFromNfsStore(LibvirtComputingResource.java:2686)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:2512)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:74)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:45)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1648)
        at com.cloud.agent.Agent.processRequest(Agent.java:661)
        at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079)
        at com.cloud.utils.nio.Task.call(Task.java:83)
        at com.cloud.utils.nio.Task.call(Task.java:29)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
2021-09-21 22:48:11,202 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:6a9a5a89) [Ljava.lang.StackTraceElement;@103a5f3d`

Executing the mount manually with -v option results in:

mount -v 172.17.3.6:/volume3/secondary_storage/template/tmpl/6/216 /test -o nodev,nosuid,noexec
mount.nfs: timeout set for Tue Sep 21 23:01:23 2021
mount.nfs: trying text-based options 'vers=4.2,addr=172.17.3.6,clientaddr=172.17.3.4'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'vers=4.1,addr=172.17.3.6,clientaddr=172.17.3.4'
mount.nfs: mount(2): No such file or directory
mount.nfs: trying text-based options 'addr=172.17.3.6'
mount.nfs: prog 100003, trying vers=3, prot=6

Manually adding option for using nfs 4

mount -v 172.17.3.6:/volume3/secondary_storage/template/tmpl/6/216 /test -o nodev,nosuid,noexec,vers=4
mount.nfs: timeout set for Tue Sep 21 23:02:50 2021
mount.nfs: trying text-based options 'vers=4,addr=172.17.3.6,clientaddr=172.17.3.4`

result is a mounted share

172.17.3.6:/volume3/secondary_storage/template/tmpl/6/216 on /test type nfs4 (rw,nosuid,nodev,noexec,relatime,vers=4.0,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.17.3.4,local_lock=none,addr=172.17.3.6)

Before joining of the host to an cloudstack cluster even with installed cloudstack-agent packages, i tested the nfs-server with the nfs-common package. the share was mountable with the “expected” results. After joining the CS cluster, the share was only accessable when explicitly giving the option vers=4 as descripted above. The SSVM is working fine currently (at least at this point i was able to upload several images)

This is for me also reproduceable with any other share (not CS related) which i provide to the server.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 17 (8 by maintainers)

Most upvoted comments

Regarding ssvm http://docs.cloudstack.apache.org/en/latest/adminguide/systemvm.html#secondary-storage-vm

According to what I know there is no nice/persistent way to set nfsv4 idmap domain for ssvm. Best corse of action now is to use nfsv3 on ssvm. All rest if configured properly will work with nfsv4 and you will get all benefits. You don’t need to change anything in ACS nfs clients on host/acs-components will negotiate latest and grates that is supported by nfs server.

Compliance can be achieved with export policy on nfs server- that is prohibiting nfsv3- but later 😃 when you have confirmed that it works as expected.