cloud-sql-proxy: Running in `-fuse` mode doesn't work between containers/host

Bug Description

This could be a documentation issue, as I am unable to reference a definitive guide on using the -fuse flag for the kubernetes client. I am coming up against various errors (similar to Issue:38), wherein the image is stating the “fusermount” is not executable.

At this point, I am not sure if I am specifying the wrong flags, or if there is something greater going on. Any advise would be appreciated.

Example code

In this case, I am deploying a cloudsql proxy as a container, mounting /cloudsql/fuse as the directory in which I’d like the begin the socket path. This error also arises when using the “default” /cloudsql, I am limited to using this alternative directory due to the parent chart.

Upon deploying the below, I am seeing issues of the type:

Mounting /cloudsql/fuse...                                                                                                                            
Could not start fuse directory at "/cloudsql/fuse": cannot mount "/cloudsql/fuse": fusermount: exec: "fusermount": executable file not found in $PATH

Container yaml:

        - command:
          - /cloud_sql_proxy
          - -dir=/cloudsql/fuse
          - -fuse
          - -credential_file=/secrets/cloudsql/credentials.json
          image: gcr.io/cloudsql-docker/gce-proxy:1.17
          imagePullPolicy: IfNotPresent
          lifecycle:
            preStop:
              exec:
                command:
                - /bin/sh
                - -c
                - sleep 15
          name: cloudsql-proxy
          resources:
            limits:
              cpu: 125m
              memory: 256Mi
            requests:
              cpu: 125m
              memory: 256Mi
          securityContext:
            runAsGroup: 65532
            runAsNonRoot: true
            runAsUser: 65532
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /cloudsql/fuse
            name: userconfig-fuse
          - mountPath: /secrets/cloudsql
            name: userconfig-cloudsql-instance-credentials
            readOnly: true

How to reproduce

Add the above container to a deployment
Note the error of the container

Environment

Cloud SQL Proxy version (./cloud_sql_proxy -version): 1.17

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 23 (17 by maintainers)

Most upvoted comments

Hey @jonthegimp, thanks for the additional information.

It looks like the only piece I was missing was including bind-propagation=rshared for the volume mounts. I checked the other two options you specified (but that aren’t needed to mount) as well:

default_permissions - By default FUSE doesn’t check file access permissions, the filesystem is free to implement it’s access policy or leave it to the underlying file access mechanism (e.g. in case of network filesystems). This option enables permission checking, restricting access based on file mode. This is option is usually useful together with the allow_other mount option.
nonempty - Allows mounts over a non-empty file or directory. By default these mounts are rejected to prevent accidental covering up of data, which could for example prevent automatic backup.

I’m not sure either are particularly useful - the first might be, but it seems like the folder permissions already apply, and the per-file access can only be enabled after the link has been created (which seems to defeat the point of fuse). The second seems potentially dangerous, as it might wipe out a directory.

I opened #537 here to fix this issue. I’m still seeing that docker won’t allow the proxy to clean up the volume through the bind for some reason, but don’t know what the cause is. However this seems to be limited when running in the container and the proxy does cleanly start back up again with these changes, so I don’t think it’s a blocker.

kurtisvg on Nov 10, 2020

I don’t think there’s a concern about unmounting the wrong FUSE directory, since you can’t double-mount a directory for FUSE anyway.

I think this is my concern - that the proxy might unmount an existing FUSE directory that is still in use that or created by a different process (either another instance of the proxy or not). While obviously a configuration error, but might be non obvious behavior. However, I agree that it’s more important that the proxy make the best attempt to start up successfully in this scenario.

I think my goal here will be to restrict the unmount behavior to only when needed (preferably if a FUSE volume is already mounted, or possibly if a first attempt at unmount fails) and clarify in the flag description (and logs) that an unmount may be performed in the attempted directory.

kurtisvg on Sep 9, 2020

@Carrotman42,

To be clear: even though you can see the README, connecting to the database doesn’t work? What does ls /cloudsql/$NAME show?

Here is what I see, from the host:

❯  ls -alh /cloudsql
total 0
-r--r--r-- 0 root root 404 Aug 30  1754 README

~ 
❯  ls -alh /cloudsql/db-01
lrwxrwxrwx 0 root root 0 Aug 30  1754 /cloudsql/db-01 -> /tmp/cloudsql-proxy-tmp/db-01

❯  mysql -S /cloudsql/<project>:entity-01
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/cloudsql/<project>:db-01' (2 "No such file or directory")

Looks like it cannot find the file in the temp directory - so I mounted that location as well:

docker run -it --rm --name proxy --user=root \
  --mount type=bind,source=<keyfile>,target=/config/credentials.json \
  --mount type=bind,source=/cloudsql,target=/cloudsql,bind-propagation=rshared  \
  --mount type=bind,source=/tmp/cloudsql,target=/tmp/cloudsql,bind-propagation=rshared  \
  --device=/dev/fuse \
  --privileged \
   <image built with the fuse.Unmount commented out> \
  /cloud_sql_proxy -fuse -fuse_tmp /tmp/cloudsql -dir /cloudsql -credential_file=/config/credentials.json

I am able to see the entry in the temp dir:

❯  ls -alh /cloudsql/entity-01
lrwxrwxrwx 0 root root 0 Aug 30  1754 /cloudsql/db-01 -> /tmp/cloudsql/db-01

❯  ls -alh /tmp/cloudsql/
total 1.1M
drwxrwxr-x  2 user usergroup 4.0K Sep  9 10:05 .
drwxrwxrwt 19 root     root     1.1M Sep  9 10:06 ..
srwxrwxrwx  1 root     root        0 Sep  9 10:05 db-01
srwxrwxrwx  1 root     root        0 Sep  9 10:05 .Trash
srwxrwxrwx  1 root     root        0 Sep  9 10:05 .Trash-1001

And I am able to connect!

❯  mysql -S /cloudsql/<project>:db-01 -uroot -p
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 2677715
Server version: 5.7.14-google-log (Google)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]> exit
Bye

So, having the tmp directory shared between the containers will be necessary as well it would seem.

jonpjenkins on Sep 9, 2020

@kurtisvg,

Poking around in the code, running a docker command like you have above, I am wondering about the following lines:

from proxy/fuse/fuse.go:73

	if err := fuse.Unmount(mountdir); err != nil {
		// The error is too verbose to be useful to print out
	}

I am noticing that this will always unmount that mounted directory, even it is not mounted by fuse. If I remove those lines, then I can see the README from the host, although I cannot access the socket at /cloudsql/<project>:<instance_name>. I can still connect via the guest.

In addition, if the container shuts down uncleanly, I need to manually run a sudo fusermount -u /cloudsql before re-running the docker container.

My golang abilities are pretty rudimentary, but is there some way to have the channel (proxy.Conn) clean up that mount drive when it is shut down?

jonpjenkins on Sep 9, 2020