Month: October 2004

If you ever had to copy a file from one machine to another over a network, chances are you know more than one way to do it by now. Especially if you were doing it on some Unix machine. If you are still struggling, here is a short list to get you started:

• Send the file as attachment via email.
• Copy it over the web using http/https or ftp/ftps protocols.
• Use secure copy (scp) provided by many secure shell (ssh) implementations.

Copying a file from one place to another (even remote) is easy. It gets slightly more complex when you need something a bit more smart than just a copy operation. Here is a short list of requirements you might have to your copy process:

• Secure (encrypted) transaction.
• Transfer of only those parts of file which have changed since last copy.
• Support for resuming the transfer if connection was broken, instead of resending the whole thing.
• Limit the bandwidth consumed by the copy process.

If you ever had to satisfy any or all of these requirements in a single solution, you might have scratched your head more than once. Or, at least, spent some quality time on the web looking for the right tool. I know, because I did. In order to improve someone else’s chances of finding such a tool faster, I’ll describe it here.

The tools itself is called rsync. As far as I know, it comes with most Unix boxes and for sure with most Linux installations. rsync has a number of options and a very flexible functionality. Just by itself it can satisfy all of the requirements above, except for secure encrypted transaction. And the beauty of the rsync is that it can be easily integrated with other tools to satisfy even more requirements. In order to provide encrypted transfer it can be easily used with secure shell (ssh) implementation, such as OpenSSH.

Enough of theory, let’s go for an examlpe.

Here is one way of using rsync which satisfies all the requirements above.

rsync -e ssh --bwlimit=50 /path/to/local/file user@remote_host:/path/to/remote/dir

Now, let’s see how it works. rsync itself can transfer only the changed parts of the file. If there is no prior copy of the file at the remote host, than the whole thing will be transfered. rsync by itself supports resuming of transfer. So, if connection between your two machines went down for a while, you can just rerun the same rsync command and it will continue from where it stopped. rsync supports bandwidth limitations. In the example above, I used --bwlimit=50 which will result in rsync transferring at speeds of 50 KiloBytes (not bits) per second or less. The encryption part is done by secure shell and rsync is told to use it by -e ssh argument. As you could probably guessed, in the example above, /path/to/local/file should point to the file that you want to transfer. user should be a username to use when identifying at remote_host. If there is a need for password input, you will be prompted before the transfer. /path/to/remote/dir should obviously point to the directory at remote machine where the file must be stored.

Read the manual for rsync and ssh and you will discover a number of other useful options that will make your life easier. To close on a nice note, I’ll suggest you use two other parameters to rsync: -r which tells rsync to use recursive mode, in case you want to transfer a directory structure and not a single file; and --progress which tells rsync to show what it is doing in real-time.