Posted 2011/04/25
Fast SSH file copy with tar
I have been reinstalling one of our research clusters and had to copy around 200GB of data over the network. The data was mostly tiny files and I half remembered an old trick using tar to speed up the transfer. After some googling I refined it down to this:
$ tar cz dir_name | ssh -c blowfish user@server tar xz -C "/path/to/stash"
According to the documentation using the blowfish cypher increases the speed further.