Q: best method for copying large # of files

What is the command to use multithreading in rsync?

When you google it, a lot of people say “just use rclone”. :wink:

I did find this though:

Why? Hard disc have the best performance during sequential read/write. If you copy files multithreaded, you are loosing the best sequence. In the worst case mulithreaded copy means random read/write. Which is the worst case performance scenario for a hard disc. Isnt that so? Am I missing something here?

EDIT:
I was doing some testing with rclone. Nice tool. BUT: It does not preserve timestamps for directories. When I sync my .gnome folder to /tmp, rclone just creates a new timestamp:

 2022-10-19@13:45 gnome-rclone
 2020-09-14@11:43 gnome-rsync

rsync preserves the timestamp.

This is a rclone limitation which is documented:

That is basically a show stopper for local backup/restore operations.

2 Likes

–metadata is supposed to preserve metadata when syncing objects. I haven’t tried it with folders outside of SFTP because I’ve been more concerned with the objects themselves. But it DOES preserve them over SFTP which is what I’m using it for right now.

Also, sequential transfers only work well for certain size files and up. If you’re dealing with a lot of small, sparse files, they’re going to get loaded and sent before they can even saturate your link. So you want to be pushing more than one file at a time. Also it tends to be unlikely that files are stored sequentially even on a spinning disk, and pretty much never on a RAID array or SSD (doing so in those cases would actually hurt performance so the controllers actively avoid it).

There are situations where I’d use rsync instead. Mostly for larger files where I can get a benefit from only transferring the deltas. If I have to transfer a ton of small files with no deltas involved, rclone will do it better.

There are several tools for synchronization and backup.listed herw that.you may find useful if haven’t seen it already
https://wiki.archlinux.org/title/Synchronization_and_backup_programs

I tried that, but it does not preserve timestamps on folders.

And the man page says:

Rclone supports preserving all the available metadata on files (not directories) when using the --metadata or -M flag.

For me that is a clear no go. For a local backup/restore procedure I would want a 100 % identical copy of all files, links AND folders.

1 Like

Interesting. I’ll have to try that locally. Too bad that doesn’t work the way you’re looking for, but this is why we have multiple options. Right tool for the right job and all.