{"id":8009,"date":"2004-10-06T02:36:13","date_gmt":"2004-10-06T00:36:13","guid":{"rendered":"https:\/\/mamchenkov.net\/wordpress\/2004\/10\/06\/copying-files-to-remote-destinations-with-bells-and-whistles\/"},"modified":"2005-04-20T12:47:54","modified_gmt":"2005-04-20T09:47:54","slug":"copying-files-to-remote-destinations-with-bells-and-whistles","status":"publish","type":"post","link":"https:\/\/mamchenkov.net\/wordpress\/2004\/10\/06\/copying-files-to-remote-destinations-with-bells-and-whistles\/","title":{"rendered":"Copying files to remote destinations.  With bells and whistles."},"content":{"rendered":"<!-- google_ad_section_start -->\n<p><a href=\"\/wordpress\/wp-content\/20041006-rsync_logo.jpg\"><img decoding=\"async\" class=\"alignright\" src=\"\/wordpress\/wp-content\/thumb-20041006-rsync_logo.jpg\" alt=\"\"\/><\/a>If you ever had to copy a file from one machine to another over a network, chances are you know more than one way to do it by now.  Especially if you were doing it on some Unix machine.  If you are still struggling, here is a short list to get you started:<\/p>\n<ul>\n<li>Send the file as attachment via email.<\/li>\n<li>Copy it over the web using http\/https or ftp\/ftps protocols.<\/li>\n<li>Use secure copy (scp) provided by many secure shell (ssh) implementations.<\/li>\n<\/ul>\n<p>Copying a file from one place to another (even remote) is easy.  It gets slightly more complex when you need something a bit more smart than just a copy operation.   Here is a short list of requirements you might have to your copy process:<\/p>\n<ul>\n<li>Secure (encrypted) transaction.<\/li>\n<li>Transfer of only those parts of file which have changed since last copy.<\/li>\n<li>Support for resuming the transfer if connection was broken, instead of resending the whole thing.\n\t<\/li>\n<li>Limit the bandwidth consumed by the copy process.<\/li>\n<\/ul>\n<p>If you ever had to satisfy any or all of these requirements in a single solution, you might have scratched your head more than once.  Or, at least, spent some quality time on the web looking for the right tool.  I know, because I did.  In order to improve someone else&#8217;s chances of finding such a tool faster, I&#8217;ll describe it here.<\/p>\n<p>The tools itself is called <a href=\"http:\/\/samba.anu.edu.au\/rsync\/\">rsync<\/a>.  As far as I know, it comes with most Unix boxes and for sure with most Linux installations.  rsync has a number of options and a very flexible functionality.  Just by itself it can satisfy all of the requirements above, except for secure encrypted transaction.  And the beauty of the rsync is that it can be easily integrated with other tools to satisfy even more requirements.  In order to provide encrypted transfer it can be easily used with secure shell (ssh) implementation, such as <a href=\"http:\/\/www.openssh.com\/\">OpenSSH<\/a>.<\/p>\n<p>Enough of theory, let&#8217;s go for an examlpe.<\/p>\n<p>Here is one way of using rsync which satisfies all the requirements above.<\/p>\n<pre>rsync -e ssh --bwlimit=50 \/path\/to\/local\/file user@remote_host:\/path\/to\/remote\/dir<\/pre>\n<p>Now, let&#8217;s see how it works.  rsync itself can transfer only the changed parts of the file.  If there is no prior copy of the file at the remote host, than the whole thing will be transfered.  rsync by itself supports resuming of transfer.  So, if connection between your two machines went down for a while, you can just rerun the same rsync command and it will continue from where it stopped.  rsync supports bandwidth limitations.  In the example above, I used <code>--bwlimit=50<\/code> which will result in rsync transferring at speeds of 50 KiloBytes (not bits) per second or less.  The encryption part is done by secure shell and rsync is told to use it by <code>-e ssh<\/code> argument.  As you could probably guessed, in the example above, <code>\/path\/to\/local\/file<\/code> should point to the file that you want to transfer.  <code>user<\/code> should be a username to use when identifying at <code>remote_host<\/code>.  If there is a need for password input, you will be prompted before the transfer.  <code>\/path\/to\/remote\/dir<\/code> should obviously point to the directory at remote machine where the file must be stored.<\/p>\n<p>Read the manual for rsync and ssh and you will discover a number of other useful options that will make your life easier.  To close on a nice note, I&#8217;ll suggest you use two other parameters to rsync: <code>-r<\/code> which tells rsync to use recursive mode, in case you want to transfer a directory structure and not a single file; and <code>--progress<\/code> which tells rsync to show what it is doing in real-time.<\/p>\n<!-- google_ad_section_end -->\n","protected":false},"excerpt":{"rendered":"<!-- google_ad_section_start -->\n<p>If you ever had to copy a file from one machine to another over a network, chances are you know more than one way to do it by now. Especially if you were doing it on some Unix machine. If you are still struggling, here is a short list to get you started: Send the &hellip; <a href=\"https:\/\/mamchenkov.net\/wordpress\/2004\/10\/06\/copying-files-to-remote-destinations-with-bells-and-whistles\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Copying files to remote destinations.  With bells and whistles.<\/span><\/a><\/p>\n<!-- google_ad_section_end -->\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"_links_to":"","_links_to_target":""},"categories":[1,18,62],"tags":[1960],"keyring_services":[],"class_list":["post-8009","post","type-post","status-publish","format-standard","hentry","category-general","category-programming","category-technology","tag-command-line"],"aioseo_notices":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":12267,"url":"https:\/\/mamchenkov.net\/wordpress\/2010\/03\/19\/copy-ssh-key-to-remote-machine\/","url_meta":{"origin":8009,"position":0},"title":"Copy SSH key to remote machine","author":"Leonid Mamchenkov","date":"March 19, 2010","format":false,"excerpt":"Those of us who use secure shell (SSH) for logging in to remote machines, already know about key authentication, which is so much easier and sometimes more secure than password authentication.\u00a0 We also know that in order to make it work you need to: generate a pair of keys with\u2026","rel":"","context":"In &quot;All&quot;","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":29153,"url":"https:\/\/mamchenkov.net\/wordpress\/2018\/12\/28\/ssh-examples-tips-tunnels\/","url_meta":{"origin":8009,"position":1},"title":"SSH Examples, Tips &#038; Tunnels","author":"Leonid Mamchenkov","date":"December 28, 2018","format":false,"excerpt":"\"SSH Examples, Tips & Tunnels\" is a nice collection of tips and examples for Secure Shell (ssh) users. It covers a variety of scenarios from simple remote connections, to file copying, to tunnels and jump hosts.","rel":"","context":"In &quot;All&quot;","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":7373,"url":"https:\/\/mamchenkov.net\/wordpress\/2004\/03\/23\/how-can-i-patch-a-script-on-a-linux-system\/","url_meta":{"origin":8009,"position":2},"title":"How can I patch a script on a Linux system?","author":"Leonid Mamchenkov","date":"March 23, 2004","format":false,"excerpt":"\"linux how to patch sample script\" Well, patching is simple. Consider the following scenario: we have a script file named script.pl to which we need to make changes, save them in the patch. We start by copying the original file to the working file with the command cp script.pl script-my.pl\u2026","rel":"","context":"In &quot;All&quot;","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":14979,"url":"https:\/\/mamchenkov.net\/wordpress\/2011\/05\/28\/vim-tips-of-the-day\/","url_meta":{"origin":8009,"position":3},"title":"Vim tips of the day","author":"Leonid Mamchenkov","date":"May 28, 2011","format":false,"excerpt":"I'm using Vim text editor for more than decade now. \u00a0And I still can't say I really know Vim. \u00a0I'm used to it and my fingers remember the commands. \u00a0And for those commands that I don't remember, I've found some way of working around and got used to it too.\u2026","rel":"","context":"In &quot;All&quot;","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":17655,"url":"https:\/\/mamchenkov.net\/wordpress\/2013\/01\/09\/git-separating-folder-into-different-repository-with-history\/","url_meta":{"origin":8009,"position":4},"title":"Git : separating folder into different repository, with history","author":"Leonid Mamchenkov","date":"January 9, 2013","format":false,"excerpt":"First things first. \u00a0If you don't use git for version control yet, stop right now and go plan your migration. \u00a0You'll thank me later. \u00a0Now. \u00a0A few days ago I had a tricky problem. \u00a0A chunk of code that was initially all over the project has been refactored into a\u2026","rel":"","context":"In &quot;All&quot;","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":15999,"url":"https:\/\/mamchenkov.net\/wordpress\/2011\/12\/13\/day-in-brief-2011-12-13\/","url_meta":{"origin":8009,"position":5},"title":"Day in brief &#8211; 2011-12-13","author":"Leonid Mamchenkov","date":"December 13, 2011","format":false,"excerpt":"Just upgraded two blogs to WordPress 3.3. Cool stuff, needs exploring. http:\/\/t.co\/Gkhe4bN4 # New note : ZionEyez | A Social Media Company http:\/\/t.co\/0q77VyAL # New note : NAS Charts - File Copy Write Performance - SmallNetBuilder http:\/\/t.co\/rmGzVQZK # I will count every mobile developer who I told about #PhoneGap as\u2026","rel":"","context":"In &quot;All&quot;","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_sharing_enabled":true,"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/posts\/8009","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/comments?post=8009"}],"version-history":[{"count":0,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/posts\/8009\/revisions"}],"wp:attachment":[{"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/media?parent=8009"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/categories?post=8009"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/tags?post=8009"},{"taxonomy":"keyring_services","embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/keyring_services?post=8009"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}