Comment by cachius

17 hours ago

rclone --multi-thread-streams allows transfers in parallel, like robocopy /MT

You can also run multiple instances of rsync, the problem seems how to efficiently divide the set of files.

9 comments

cachius

Reply
pama  16 hours ago

Sometimes find (with desired maxdepth) piped to gnu-parallel rsync is fine.

SoftTalker  16 hours ago

robocopy! Wow, blast from the past. Used to use it all the time when I worked in a Windows shop.

  • bob1029  16 hours ago

    I am using robocopy right now on a project. The /MIR option is extremely useful for incrementally maintaining copies of large local directories.

adolph  14 hours ago

My go-to for fast and easy parallelization is xargs -P.

  find a-bunch-of-files | xargs -P 10 do-something-with-a-file

       -P max-procs
       --max-procs=max-procs
              Run up to max-procs processes at a time; the default is 1.
              If max-procs is 0, xargs will run as many processes as
              possible at a time.

  • akdev1l  13 hours ago

    note that one should use -print0 and -0 for safety

    • adolph  12 hours ago

      Thanks! I've been using the -F{} do-something-tofile "{}" approach which is also handy for times in which the input is one pram among others. -0 is much faster.

      Edit: Looks like when doing file-by-file -F{} is still needed:

        # find tmp -type f | xargs -0 ls
  ls: cannot access 'tmp/b file.md'$'\n''tmp/a file.md'$'\n''tmp/c file.md'$'\n': No such file or directory

      2 replies →