- 26 Feb 2015
There are many, many ways to do remote file transfer between two computers; a few common ones discussed here along with a few specific tutorials on seperate pages. These instructions are assuming you are connecting to the HPCC and wish to transfer files to or from the HPCC to another machine. Also most of these instructions will be written for Linux based systems (specifically Scientific Linux) with extra instructions on how to either extend it to OS X or Windows.
SSH Based Protocol (SCP, SFTP)
The most common way to do file transfer between two machines is to use some method involving SSH including (but not limited to) SCP and SFTP. The first method of using these programs is to use the terminal. These were tested on Linux but the terminal commands should work verbatim on OS X (the Windows terminal however is quite different). The first is Secure Copy (SCP), and it simply takes two arguments: the source and destination of the file to transfer in that order. When transferring from or to a remote host you need to provide the address of the remote host in addition to the path of the file. The SCP command for copying a file from a remote host to your locale machine would then look something like this:
user@host $ scp firstname.lastname@example.org:/path/to/file /path/to/local/destination
Note the colon between the path name of the file and the address of the remote host. For using SCP to copy a file from your local machine to the HPCC it would look like this:
user@host $ scp /path/to/local/file email@example.com:/path/to/destination
SCP can only be used for transferring files between two computers; it can not create directories or move files around on a given machine. SCP transfers are also non-interruptable, i.e. if they are stopped before the transfer is complete, it must be restarted from the beginning. However all things being equal, SCP is generally faster than SFTP due to using a different algorithm for transfer.
Secure File Transfer Program (SFTP) is the next command for transferring files between computers. It has both an interactive mode and a non-interactive mode for transferring files. The non-interactive mode functions very similar to SCP, it takes two arguments: the source and destination of the file to transfer in that order. To use SFTP to copy a file from a remote host to your local machine it would look something like this:
user@host $ sftp firstname.lastname@example.org:/path/to/file /path/to/local/destination
The main difference in using SFTP from SCP is the availability of the interactive mode of SFTP. To start an interactive session of SFTP, run the command with the address of the remote host you wish to connect to. For example to start an interactive session of SFTP on the HPCC the command would look like:
user@host $ sftp email@example.com
Upon successful completion your terminal prompt should turn into the SFTP prompt and you are then ready to run SFTP commands to access files, directories, etc. on the remote host. In this interactive session you can copy files to and from the remote host, create or remove diretories, change file permissions, create symbolic links, and more. To get help on any of the SFTP commands, type ? (or
) when in an interactive SFTP session. SFTP can be used more as a remote file system when run in its interactive mode. SFTP also has the advantage that its transfers can be interrupted; you can pause a transfer of a file and resume it later without having to restart the entire transfer.
One can also use SCP or SFTP to transfer files using a GUI based program rather than just using the terminal. Many different programs exist that implement SCP and SFTP for different systems. The two I would suggest are FileZilla
(Linux, OS X, Windows) or WinSCP
(Windows), and I will give some brief instructions on how to use FileZilla.
Filezilla can be installed almost certainly through the package manager of your system (on Linux) or easily installed from packages from their website. After FileZilla is installed, it just needs the address of the host, your username and password, and the port of the connection for transfer. Once a successful connection has been made, you can use FileZilla to browse directories of both systems, transfer files both ways, etc. For connection to the HPCC, the Filezilla settings are as follows:
- Hostname: hpcc.msu.edu
- Username: (your username)
- Password: (your password)
- Port: 22
On Linux systems one can connect to a remote host directly through your file manager of choice, allowing remote file access through the file browser. The two examples I will give are for Nautilus (the Gnome default) and Thunar (the Xfce default) file managers. In Nautilus in Location Options (the gear icon) select "Connect to server..." and use sftp://hpcc.msu.edu as the server address (for connecting to the HPCC). Nautilus will then ask for your username and password (which are your HPCC username and password) and then connect to the server. Thunar functions much in the same way except that you enter the server address, sftp://hpcc.msu.edu directly into the address bar. Thunar will then ask for your username and password and connect to the server.
There are a few specialized methods for doing file transfer that are useful for the T2K group. These methods have their own wiki page and more detailed instructions on how to use them. They can be found here: