- 26 Feb 2015
Using iRODS for file transfer
This page contains instructions on how to install and use iRODS to facilitate file transfers. These instructions are specifically tailored for accessing the QMUL server using the T2K_ASG_Reader account and mostly copied from http://www.t2k.org/asg/oagroup/gadatastorage/index_html
. The commands and install were performed on an install of Scientific Linux 7 (SL7), which is a Red Hat 7 based system, using a BASH shell. Adjust your commands accordingly if you are using a different system / shell. If you are working on the HPCC
, iRODS is already installed for T2K group use, so you can skip to the section on how to use iRODS.
To install iRODS you need only one extra program (that is probably already installed) called
. Once you install
you can download and install iRODS. To download the correct version of iRODS (v3.2) do the following:
user@host $ git clone https://github.com/irods/irods-legacy.git
user@host $ cd irods-legacy
user@host $ git checkout tags/3.2
Once the download completes, change directories to iRODS and run the irodssetup script. It will ask you to build an iRODS server, answer NO, then continue with the installation.
user@host $ ./irodssetup
After the script runs iRODS should be properly installed on your machine. Finally for convenience, you should add the iRODS commands to your
$PATH variable. The iRODS commands are located in
The .irodsEnv File
The iRODS software also needs a special configuration file that tells it where and how to connect to the server you wish to copy files to or from. This file is known as the
file, which will be placed in the
directory. If you are working on the HPCC
then this step has been automated for you -- run the
script located in
and it will setup iRODS and pick the default .irodsEnv
file. If you need a different
, keep reading.
First thing is to create the
directory for the
file. Which ever
file you wish to use should be placed in this directory with the name .irodsEnv
exactly (including the leading period). The default
file for the T2K_ASG_Reader account can be found here: http://www.t2k.org/asg/oagroup/gadatastorage/.irodsEnv or it (along with others) can also be found in
/mnt/research/T2K/usr/irods-legacy/iRODS/irodsEnv/ on the HPCC.
To run iRODS you first must initialize it to setup the proper connection to the server. These instructions will use the example of the T2K_ASG_Reader account, but can be used with any account, you just need to use the correct
file and password. To initialize the connection run:
user@host $ iinit
Note the double 'i' for the command. You should then be prompted for your password. If you are using the T2K_ASG_Reader account then the password can be found here: http://www.t2k.org/asg/oagroup/gadatastorage/index_html
. Once the password is accepted you can then copy files from the iRODS server using the iRODS commands. Also, if you are using iRODS on the HPCC, you must be using the Gateway node in order for the connection to work. If you are on a development node, the connection will be refused by the firewall.
The iRODS commands are essentially the same as the corresponding unix commands with an 'i' prepended to the command (like the
command above). The most common commands are summerized below:
ils : it will show you the files in the collections
ils -l : it will show you the files using a long listing format (also the time when they were uploaded will be shown)
icd : it will allow you to move among the collections
iput : it will allow you to upload a file (iput <filename> <collectionwheretocopy>) [only for users with a personal account]. Use -f to overwrite the file if already exists
iput -r : it will allow you to upload a directory (iput -r <directory> <collectionwheretocopy>) [only for users with a personal account]. Use -f to overwrite the directory if already exists.
iget : it will allow you to get a files (iget <fullfilename> <yourUnixdirectory>). Use -f to overwrite the file if already exists
iget -r : it will allow you to download a collection (ie directory). Use -f to overwrite the directory if already exists.
irm -f : to delete a file. Please note -f should be added to force the removal of the file, otherwise it is just copied to the trash area. (irm -f <filename>)
irm -rf : to delete a collection. Please note -f should be added to force the removal of the directory, otherwise it is just copied to the trash area. (irm -fr <collectionname>)
The commands function pretty much as expected to their unix equivalent. For example, if I wanted to download a file I would use iget
user@host $ iget /QMULZone1/home/asg/asgRun123/banff/postfit/postfit_banff_v7_all_params.root ~/Downloads
Which would download the file
directory. The path names work just like normal unix; you can either specify the full path name like I did above, or you can specify a relative path name, but you need to be in the proper directory for the relative path name.