gcsfuse: gcsfuse hangs when called from systemd startup

Automatically mounting via /etc/fstab fails, so I set up a systemd mount unit that looks like this:

[Unit]
Description=GCS bucket automount
Requires=network-online.target
After=network-online.target

[Mount]
What=my-gcs-bucket
Where=/var/gcs
Type=gcsfuse-wrapper
Options=rw,user

[Install]
WantedBy=multi-user.target

gcsfuse-wrapper is a wrapper script that strips away the -n option that /sbin/mount.gcsfuse does not understand. For debugging, it also waits 15 seconds before actually calling /sbin/mount.gcsfuse to ensure that the network is actually up.

When systemd tries to mount the unit during startup, it hangs.

root@instance:~# systemctl status var-gcs.mount -l
● var-gcs.mount - GCS bucket automount
   Loaded: loaded (/etc/systemd/system/var-gcs.mount; disabled)
   Active: activating (mounting) since Thu 2016-09-08 17:37:05 UTC; 32s ago
    Where: /var/gcs
     What: my-gcs-bucket
  Control: 673 (mount)
   CGroup: /system.slice/var-gcs.mount
           ├─673 /bin/mount -n my-gcs-bucket /var-gcs -t gcsfuse-wrapper -o rw,user
           ├─674 /sbin/mount.gcsfuse-wrapper my-gcs-bucket /var/gcs -n -o rw,noexec,nosuid,nodev,user
           ├─738 sh -c '/sbin/mount.gcsfuse' '-o' 'rw' '-o' 'noexec' '-o' 'nosuid' '-o' 'nodev' '-o' 'user' 'my-gcs-bucket' '/var/gcs'
           ├─739 /sbin/mount.gcsfuse -o rw -o noexec -o nosuid -o nodev -o user my-gcs-bucket /var/gcs
           ├─742 gcsfuse -o noexec -o nosuid -o nodev -o rw my-gcs-bucket /var/gcs
           └─749 /usr/bin/gcsfuse --foreground -o noexec -o nosuid -o nodev -o rw my-gcs-bucket /var/gcs

Sep 08 17:37:05 instance systemd[1]: Mounting GCS bucket automount...
Sep 08 17:37:05 instance mount[673]: [wrapper] Sleeping for 15 seconds to wait for the network to come up...
Sep 08 17:37:21 instance mount[673]: [wrapper] Invoking gcsfuse with: '/sbin/mount.gcsfuse' '-o' 'rw' '-o' 'noexec' '-o' 'nosuid' '-o' 'nodev' '-o' 'user' 'my-gcs-bucket' '/var/gcs'
Sep 08 17:37:21 instance mount[673]: Calling gcsfuse with arguments: -o noexec -o nosuid -o nodev -o rw my-gcs-bucket /var/gcs
Sep 08 17:37:21 instance mount[673]: Using mount point: /var/gcs
Sep 08 17:37:21 instance mount[673]: Opening GCS connection...
Sep 08 17:37:21 instance mount[673]: Opening bucket...
Sep 08 17:37:21 instance mount[673]: Mounting file system...

After a while, systemd will timeout and terminate the process. If I issue a systemctl restart var-gcs.mount, it will work just fine and properly mount the bucket.

Unfortunately, I was unable to make /sbin/mount.gcsfuse accept the --debug_* options, so I cannot provide more detailed information.

What could be causing this, and how can I work around it? Or is there a better way to automatically mount the bucket on boot?

About this issue

  • Original URL
  • State: closed
  • Created 10 years ago
  • Comments: 33 (25 by maintainers)

Commits related to this issue

Most upvoted comments

I have a created a systemd service to mount a Google Cloud Storage volume via gcsfuse on boot. The trick to avoiding the “lock on boot” problem is to ensure that it runs after sys-fs-fuse-connections.mount.

It has worked for me on two Google VM instances so far. Use it at your own risk; I recommend setting a root password on the VM and enabling the serial console before rebooting, in case this service locks up your instance on boot.