Challenge: “A Million Little Pieces” – Part 3, SSH

This post is the second in a series; see here for part 1 and here for part 2.

This time, I will use the Secure Shell Protocol, better known as SSH, to make a secure connection or “tunnel.” SSH is often included with many versions of Linux, and typically works seamlessly within it. Though it is not included by default with Windows, it is a relatively simple matter to install and adapt for Windows-based systems.

For the Windows Server 2019 computer, I used a program called Cygwin to create a Linux-like environment, which includes an SSH server to ‘listen’ for a SSH connection request. Cygwin is not exactly a Linux virtual machine, but functions very much like one for our purposes.

For the Win10 machine, I again used Windows Subsystem for Linux (WSL).


Method One: Rsync

  1. Once the SSH connection was established, I ran the following command:
time rsync -rv --no-p --no-g --no-o --no-perms --omit-dir-times admin@server.ip.address:/path/to/files/ /mnt/c/local/path/to/files/

The “time” command reports how long the “rsync” command took, and most of the other switches tell rsync to ignore metadata (file ownership, permissions, etc.) The result:

M1 Final result: 482 minutes


Side Test: Why is Rsync so slow?

Rsync had once again performed very poorly, which gave me pause, because many people that know much more than me swear by it, so I thought there must be something wrong with my process.

My theory is that Rsync’s slowness has something to do with transferring files over the internet connection, so I decided to eliminate this variable and see if it reduced the disparity between the rsync and xcopy methods.

1) I used rsync to copy files from one directory to another directory on the same computer:
Time Taken: 4minutes, 20sec

2) Next, I used “xcopy” to copy files those same files:
Time Taken: 40 seconds!

Both methods run much faster, but there is still a large gap between their speeds. Perhaps rsync was slowed down by running WSL on the Win10 machine, or by running cygwin on the remote server, or maybe another factor. Either way, I can conclude…

Side Test Final Result: It was NOT the Internet Connection that Slowed Down Rsync


Method Two: Use SCP

SCP is a secure copy protocol, specifically for SSH connections and a very simple command:

time scp -r admin@server.ip.address:/path/to/files/ /mnt/c/local/path/to/files/

M2 Final Result: 2594 Minutes…Yikes!


Well, using SSH and Taking 8 Hours to Complete, the “Winner” Is…

Method One: Rsync

Lessons Learned:

  • Again, using the “host” system and native utilities is much faster than using any kind of virtual machine
  • The internet connection is not the source of Rsync’s peculiar slowness
  • SSH is still great for simple, command-based connections to a remote machine
  • Next up: Using the newer Wireguard transfer method…

Related Posts

Leave a Reply

Your email address will not be published. Required fields are marked *