rclone: New --links parameter not working with mount?

I’ve created a draft PR of a code addition to have the mount command support symbolic links. No extra mount parameters are needed; this works without any extra --links command line arguments.

I still need to add documentation and tests, but it should be ready for use. For anyone who is still interested in this feature, please try out my branch and let me know if that works for you! I hope to have the PR ready for review in the next week or two.

really looking forward for this to be implemented… the only missing feature that i need is symlink support for rclone mount.

I appreciate the encouragement! I don’t have the time to dedicate to the refactor right now, but it is good to know what the appropriate direction would be if I take this up in the future.

@pasnox and I have been talking on IRC and doing some testing. As things stand now, the only issue that remains is the creation of new file symlinks on Windows mounts whenever cache mode is set to ‘full’. When cache mode is set to ‘off’, creation of new file and directory symlinks work fine as well as pre-existing file/directory symlinks. When cache mode is set to ‘full’, all preexisting symlinks work fine but creation of new file symlinks causes rclone to lock up (creation of new directory symlinks works fine). Except for this one last issue, this new feature appears to be ready to push out to the mainline branch. @pasnox said this last remaining issue is due to a workaround/hack that he cannot fix himself. I think the hack/workaround he is referencing is referenced here in the WinFSP source code: https://github.com/winfsp/winfsp/blob/6fb72555d33d228736b8156cb919edeaae4da150/src/dll/fuse/fuse_intf.c#L2293|

I think the workaround uses this function to temporarily rename a symlink file which is causing issues when using a cache mode other than ‘off’: https://github.com/winfsp/winfsp/blob/6fb72555d33d228736b8156cb919edeaae4da150/src/dll/fuse/fuse_intf.c#L224

My understanding of this could be wrong (if so, perhaps @pasnox could chime in) but ultimately, I think the only way we are going to be able to finally fix this issue and move the new feature into the mainline branch is to get some help from @ncw . @pasnox has clearly stated to me on IRC that he does not know how to fix this issue and that it is going to require help from @ncw in order to fix. Hopefully some further workaround can be implemented to make things work without having to implement changes in WinFSP because I suspect that will only cause things to stretch out further and possibly might mean the fix will never be implemented. I know @ncw is surely a very busy person but hopefully at some point, when he has time, he might be able to offer up his help or advice on fixing this problem. I’d hate to see all this hard work that @pasnox has put into this feature go to waste.

I pushed a first test PR @ https://github.com/rclone/rclone/pull/6337 Please, help me debug / test other platforms and backends.

@dinoboy197 What’s the status of your fork in regards to mount supporting symlink ?

You can grab test binaries at this address: https://beta.rclone.org/branch/pasnox-symlink/v1.60.0-beta.6493.aab476684.pasnox-symlink/

After some initial refactoring, I realized that symlink support down into the VFS layer means building symlink support for several other components - not just mount and cmount. I realized that I would need to make some fairly significant design choices regarding how to represent symlinks at that layer - should they be a File, but have a new isSymlink method, or should there be a new type that derives from Node entirely; should symlinks support propagate further down chain into the FS layer, etc. While I was aware earlier that my symlink addition for the mount command was fairly trivial and somewhat superficial, I didn’t have a proper appreciation for what all would go into proper support in the VFS layer. Not surprising considering I’m a first-time rclone contributor!

I don’t believe I have the right experience or vision to be making these kinds of changes in the VFS layer as a first-time contributor to rclone - I’d have to ask many questions and take up a considerable amount of time trying to sync with the future vision of VFS in order to come up with a solution that I thought was satisfactory. Rather, I’d feel much more comfortable if someone with more of an ownership stake in the VFS architecture and vision implemented the symlink support they’d like to see in that layer, then I’d be comfortable retrofitting my mount solution to use whatever VFS hooks were built out.

I say this not a way to suggest that my changes should go to the rclone mainline just because I’d rather not implement things in the VFS layer. Rather, I do think that VFS support is the right way to implement this and look forward to someone with more experience (both with rclone and with Golang) to drive that a bit. I’m sorry if this comes as a disappointment for anyone waiting for a full solution.

I’ll keep an eye out here, and should I become more well-versed with the codebase and VFS direction at a later date, perhaps will pick this up again!

For folks following along like myself, the PR is #4933. (thanks for working on this!)

Hey @pasnox, I just wanted to follow up on this and ask if you know whether or not the Windows implementation for this was ever fully fixed and implemented? I see that the documentation on rclone.org now includes the “–links” flag so it appears that the feature has been pushed out to the mainline branch. However, I’m not sure if that is Linux-only support or if that includes Windows too. Once again, thank you so much for all the hard work you put into implementing this. It’s so awesome to actually have this in the public build of rclone now (even if it is only for Linux… though hopefully not)!

Really, for all intents and purposes, I think the feature is working well enough on Windows to be merged into the mainline branch. The only part that is actually broken is making file symlinks on an actively mounted rclone volume. All symlinks present on local storage prior to syncing/copying work perfectly and I suspect this is all that 95% of people care about. While making file symlinks on a mounted rclone volume is certainly useful, I really don’t think it is critical. Perhaps the feature could be integrated as it is now, triggering a warning log message on Windows when someone tries to make a file symlink and then later on, when @pasnox has some time to look into the problem with a Windows VM, that issue could be fixed and the log message removed. Just an idea… I’m really anxious to see this merged into the mainline branch as well!

@pasnox: I cloned your repository at “https://github.com/pasnox/rclone.git” and ran the same tests again and here are the results:

• As far as I can tell, existing symlinks that are copied/synced to an rclone storage destination (when using “–links”) work perfectly fine now. I tried files, directories, symlinks to files/directories in the same directory, symlinks to files/directories in parent/child directories, etc and so on.
• Correct me if I’m wrong, but I believe your previous post implied that you were intending for your code changes to cause symlinks to be stored (in “.rclonelink” files) with Unix directory separators, regardless of platform, right? If that was indeed what you intended, unfortunately that is not what is actually happening. When mounting an rclone storage on Windows (without “–link”), and inspecting the contents of any “.rclonelink” file in a text editor, those paths are still using Windows directory separators (“\”). Just to be clear, these are fresh, newly created symlinks that were synced to the rclone storage with this new rclone build. Here is an example path from one of those files: subdir\test2.txt
• In regards to creating new symlinks on an actively mounted rclone storage on Windows (which was mounted with “–links”), only the creation of directory symlinks work. Creating directory symlinks using “mklink /D [symlink] [target]” works perfectly fine, regardless of whether the target is in the same directory or a parent/child directory. File symlinks however, do not work (“mklink [symlink] [target]”). The following error is returned whenever attempting to create any file symlink: “Cannot create a file when that file already exists.”

Please let me know if there is any more information I can provide or anything else you would like for me to test. Otherwise, I will patiently wait for your next build to test. Thanks @pasnox!

Hi Sorry for jumping in, but are binaries usable? Will changes be merged into mainline rclone?

Latest builds can be found here https://beta.rclone.org/branch/pasnox-symlink/ And yes, of course, goal is to mainstream the changes.

If somebody could test it on windows that would be helpful by the way 😃 I can only test macOS / Linux

@pasnox - nice one 😃 Look forward to seeing the results. Have a nice vacation!

On Sun, 13 Feb 2022, Filipe Azevedo wrote:

Hey, I’m trying to try to fix this feature request as it is a very needed feature for me. Unfortunately, I comes from C/C++, and Go is just shadow for me, so i will have to minimal learn it as I need. Would you wanna give some inputs on what/where would be to be patched in the vfs ?

In case the question was addressed to me: I’m not at all familiar with the rclone source code (and indeed I no longer have access to a cloud storage service on which to test anything). Sorry.

I think this would have to be a new flag for mount, say --use-links or something like that which inteprets the .rclonelink files and turns them back into symlinks.

Therefore you would be able to mount your drive remote, and have the symlinks appear as symlinks.

Is that what is required do you think?