-- AndrewCudd - 26 Feb 2015

Using iRODS for file transfer

This page contains instructions on how to install and use iRODS to facilitate file transfers. These instructions are specifically tailored for accessing the QMUL server using the T2K_ASG_Reader account and mostly copied from http://www.t2k.org/asg/oagroup/gadatastorage/index_html. The commands and install were performed on an install of Scientific Linux 7 (SL7), which is a Red Hat 7 based system, using a BASH shell. Adjust your commands accordingly if you are using a different system / shell. If you are working on the HPCC, iRODS is already installed for T2K group use, so you can skip to the section on how to use iRODS.

Installing iRODS

To install iRODS you need only one extra program (that is probably already installed) called git. Once you install git you can download and install iRODS. To download the correct version of iRODS (v3.2) do the following:

user@host $ git clone https://github.com/irods/irods-legacy.git
user@host $ cd irods-legacy
user@host $ git checkout tags/3.2

Once the download completes, change directories to iRODS and run the irodssetup script. It will ask you to build an iRODS server, answer NO, then continue with the installation.

user@host $ ./irodssetup

After the script runs iRODS should be properly installed on your machine. Finally for convenience, you should add the iRODS commands to your $PATH variable. The iRODS commands are located in /path/to/iRODS/clients/icommands/bin/.


The .irodsEnv File

The iRODS software also needs a special configuration file that tells it where and how to connect to the server you wish to copy files to or from. This file is known as the .irodsEnv file, which will be placed in the $HOME/.irods/ directory. If you are working on the HPCC then this step has been automated for you -- run the nd280setup.sh script located in /mnt/research/T2K/nd280software/ and it will setup iRODS and pick the default .irodsEnv file. If you need a different .irodsEnv, keep reading.

First thing is to create the $HOME/.irods/ directory for the .irodsEnv file. Which ever .irodsEnv file you wish to use should be placed in this directory with the name .irodsEnv exactly (including the leading period). The default .irodsEnv file for the T2K_ASG_Reader account can be found here: http://www.t2k.org/asg/oagroup/gadatastorage/.irodsEnv or it (along with others) can also be found in /mnt/research/T2K/usr/irods-legacy/iRODS/irodsEnv/ on the HPCC.

Running iRODS

To run iRODS you first must initialize it to setup the proper connection to the server. These instructions will use the example of the T2K_ASG_Reader account, but can be used with any account, you just need to use the correct .irodsEnv file and password. To initialize the connection run:

user@host $ iinit

Note the double 'i' for the command. You should then be prompted for your password. If you are using the T2K_ASG_Reader account then the password can be found here: http://www.t2k.org/asg/oagroup/gadatastorage/index_html. Once the password is accepted you can then copy files from the iRODS server using the iRODS commands. Also, if you are using iRODS on the HPCC, you must be using the Gateway node in order for the connection to work. If you are on a development node, the connection will be refused by the firewall.

The iRODS commands are essentially the same as the corresponding unix commands with an 'i' prepended to the command (like the iinit command above). The most common commands are summerized below:
  • ils : it will show you the files in the collections
  • ils -l : it will show you the files using a long listing format (also the time when they were uploaded will be shown)
  • icd : it will allow you to move among the collections
  • iput : it will allow you to upload a file (iput <filename> <collectionwheretocopy>) [only for users with a personal account]. Use -f to overwrite the file if already exists
  • iput -r : it will allow you to upload a directory (iput -r <directory> <collectionwheretocopy>) [only for users with a personal account]. Use -f to overwrite the directory if already exists.
  • iget : it will allow you to get a files (iget <fullfilename> <yourUnixdirectory>). Use -f to overwrite the file if already exists
  • iget -r : it will allow you to download a collection (ie directory). Use -f to overwrite the directory if already exists.
  • irm -f : to delete a file. Please note -f should be added to force the removal of the file, otherwise it is just copied to the trash area. (irm -f <filename>)
  • irm -rf : to delete a collection. Please note -f should be added to force the removal of the directory, otherwise it is just copied to the trash area. (irm -fr <collectionname>)
The commands function pretty much as expected to their unix equivalent. For example, if I wanted to download a file I would use iget as follows:

user@host $ iget /QMULZone1/home/asg/asgRun123/banff/postfit/postfit_banff_v7_all_params.root ~/Downloads

Which would download the file postfit_banff_v7_all_params.root into my Downloads/ directory. The path names work just like normal unix; you can either specify the full path name like I did above, or you can specify a relative path name, but you need to be in the proper directory for the relative path name.


Topic revision: r4 - 08 May 2015, AndrewCudd
 

This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback