rclone: rclone copy --metadata not working for folders

I’m willing to pledge $100 for full metadata preservation*. Unfortunately I don’t have the clout to get my employer to sponsor it (I don’t think they use rclone). @ncw do you use something like Bountysource?

* by full metadata what I mean is that

user@A ~$ rclone copy important_docs S2:remote_archive
user@B ~$ rclone copy S2:remote_archive important_docs

should have the exact same result as

user@A ~$ rsync -aHAX important_docs B:/home/user/important_docs

where A and B are both linux systems.

I would also accept non-preservation of hardlinks (i.e., rsync without the -H option). But directory mtime and permissions are a must!

This is for my own usage. Unfortunately I’m not a rich man, have zero Golang experience and not much free time. If there is some sort of crowdfunding/bounty setup I’d be willing to chip in up to say $200.

I’m currently thinking about metadata for directories.

I had a quick thought about this. Would it be of interest if I could write a small command that copies dir modtimes from one remote to another (for backends that support it)? It would be intended as more of a short-term stop-gap to tide us over while we wait hopefully for full dir metadata support, which would be a much more complicated project.

Many backends (including local and drive) already support listing directory modtime, like:

rclone lsf --dirs-only -R --format "pt" src_remote:path

So the command I’m imagining would do basically the equivalent of the above command, then loop through all the results and do something like err = dir.SetModTime(ctx, t) for each corresponding dir on the dest. (Probably with a --modify-window check first, and maybe also support for --update.)

The problem with this idea is that directories don’t support SetModTime at the moment. This is something which will need to be added to backends to support it.

An alternate approach might be to tack it on to the end of copy/sync as an optional extra step, very much like --create-empty-src-dirs.

That is my ultimate plan 😃

My use case is that my Google Drive currently has all the correct modtimes of my folders, and when I use rclone to sync them to a new computer, I want to be able to follow that with a command to copy over all the folder modtimes from drive to local. Modtime is the main piece of dir metadata I care about (especially because of macOS “packages” which are really dirs), so in my opinion, a command like this would deliver a lot of value for relatively little effort.

Is this something you’d want?

It is a very useful thing. However it is not at all a straight forward bit of work since the assumption that rclone can create and destroy directories whenever it wants is baked into every backend.

Some backends will need the modtime of directories updating if you write files in them, but I imagine some won’t.

I had a quick thought about this. Would it be of interest if I could write a small command that copies dir modtimes from one remote to another (for backends that support it)? It would be intended as more of a short-term stop-gap to tide us over while we wait hopefully for full dir metadata support, which would be a much more complicated project.

Many backends (including local and drive) already support listing directory modtime, like:

rclone lsf --dirs-only -R --format "pt" src_remote:path

So the command I’m imagining would do basically the equivalent of the above command, then loop through all the results and do something like err = dir.SetModTime(ctx, t) for each corresponding dir on the dest. (Probably with a --modify-window check first, and maybe also support for --update.)

An alternate approach might be to tack it on to the end of copy/sync as an optional extra step, very much like --create-empty-src-dirs.

My use case is that my Google Drive currently has all the correct modtimes of my folders, and when I use rclone to sync them to a new computer, I want to be able to follow that with a command to copy over all the folder modtimes from drive to local. Modtime is the main piece of dir metadata I care about (especially because of macOS “packages” which are really dirs), so in my opinion, a command like this would deliver a lot of value for relatively little effort.

Is this something you’d want?

I’m added simple example add metadata for local backend https://github.com/rclone/rclone/compare/master...lxShaDoWxl:rclone:master

I think for file level metadata, we can use --metadata option with rclone

Yes, but not for folders, so in the example above rclone and rsync would give different outputs.