-- AndrewCudd - 03 Feb 2015

Using Globus / GRID for file transfer

This page contains instructions on how to use Globus to facilitate file transfers using the GRID and GRID-FTP. These instructions are specifically tailored for accessing the t2ksrm.nd280.org server at TRIUMF. The commands and install were performed on an install of Scientific Linux 7 (SL7), which is a Red Hat 7 based system, using a BASH shell. Adjust your commands accordingly if you are using a different system / shell.

Installing globus-url-copy (Globus Toolkit)

To install globus-url-copy and the corresponding commands you need to install the Globus Toolkit. Install at least version GT-5.2.3 or higher; these instructions were performed using version 5.2.3. The Globus documenation for installing the Globus Toolkit v5.2.3 is located here: http://toolkit.globus.org/toolkit/docs/5.2/5.2.3/admin/install/. The instructions below are more or less directly from the documentation. If you are working on the HPCC (currently only working on the gateway node), Globus software is already installed and just needs to be loaded using the following commands:

user@host $ module load GNU
user@host $ module load GLOBUS

Then you can skip the instructions on installing the Globus Toolkit.

You can try to install the toolkit using a deb or rpm package using the corresponding commands or you can install it from source. These instructions will continue by installing the Globus Toolkit from source.
  • Download the Globus Toolkit and extract the tarball
user@host $ tar -zxvf gt5.2.3-all-source-installer.tar.gz
  • Create the directory you wish to install the Globus Toolkit; by default it will install into /usr/local/globus-5.2.3
    • Wherever you choose to install it, make sure you have write permissions to that directory. For example, to install into /usr/local/globus-5.2.3 you need to have root permissions.
    • The documentation suggests to create a user named globus to own the directory; I did not do that and simply installed it as root.
  • Before installing the Globus Toolkit, make sure you have the necessary dependencies installed. The list can be found in the documentation, and if make fails it will tell you which packages you are missing. Simply install the missing package and try again.
  • cd to the Globus Tooklit source directory and run the following commands to install the entire Globus Toolkit:
user@host $ ./configure --prefix=/Your/Install/Directory
user@host $ make
user@host $ make install
  • If you do not wish to install the entire toolkit and want only the bare minimum for globus-url-copy, instead of make run:
user@host $ make gridftp
user@host $ make install

That should now give you a working installation of the Globus Toolkit or a subset of the toolkit. Finally for convenience, you should add the globus commands to your $PATH, like so:

user@host $ export PATH=$PATH:/path/to/globus/install/bin

Installing User GRID Certificate

Accessing the t2ksrm.nd280.org server using GRID-FTP requires a set of security certificates and keys. Once you have the certificate in the form of a PKCS12 (.p12) file you can install the corresponding certificate and key for globus to use. The certificate and key should be installed into the $HOME/.globus directory using the names below.

user@host $ openssl pkcs12 -in YourCert.p12 -clcerts -nokeys -out $HOME/.globus/usercert.pem
user@host $ openssl pkcs12 -in YourCert.p12 -nocerts -out $HOME/.globus/userkey.pem

Finally, the usercert.pem and the userkey.pem need to be read/write only by the owner of the file (i.e. you) otherwise you will get an error when you try to use them later. To change this, run:

user@host $ chmod 600 $HOME/.globus/usercert.pem
user@host $ chmod 600 $HOME/.globus/userkey.pem

Installing CA Certificates

Next a set of CA certificates needs to be installed (assuming it is not installed already) for another part of the secure transfer. These instructions follow almost verbatim from here: https://wiki.egi.eu/wiki/EGI_IGTF_Release. You will (maybe) need your user GRID certificate to be installed in your browser to access that link. Then you have a few options on how to go about installing the CA certificates. I personally installed the packages through YUM, but you can also install specific RPM's or unpack the certificates from source. Installing through YUM would be best if possible since YUM will do everything for you and then updating the certificates later takes only a single YUM command.

To install the CA certificates via YUM:
  • Add the following repo file to your /etc/yum.repos.d/ directory. I called it egi-trustanchors.repo
  • And then, assuming you are installing the EGI trust anchors for the first time, run:
user@host $ yum update
user@host $ yum install ca-policy-egi-core 

That will install basically every CA certificate you will ever need, however it may install them in a directory that Globus is not looking in.

To install the certificates via tar-ball, do the following:
  • Download the certificate.tar files that you need. For accessing t2ksrm.nd280.org you will need the ca_GridCanada.tar.gz file (grab the latest version).
  • Exatract the tarball and move the all of the certificate files to your certificate directory (more on that in a moment).
  • The file structure should look something like this after running ls -l
lrwxrwxrwx. 1 cuddandr T2K   14 Feb 4 09:54 5d674a88.0 -> GridCanada.pem
lrwxrwxrwx. 1 cuddandr T2K   21 Feb 4 09:54 5d674a88.namespaces -> GridCanada.namespaces
lrwxrwxrwx. 1 cuddandr T2K   25 Feb 4 09:54 5d674a88.signing_policy -> GridCanada.signing_policy
lrwxrwxrwx. 1 cuddandr T2K   14 Feb 4 09:54 bffbd7d0.0 -> GridCanada.pem
lrwxrwxrwx. 1 cuddandr T2K   21 Feb 4 09:54 bffbd7d0.namespaces -> GridCanada.namespaces
lrwxrwxrwx. 1 cuddandr T2K   25 Feb 4 09:54 bffbd7d0.signing_policy -> GridCanada.signing_policy
-rw-r--r--. 1 cuddandr T2K   40 Nov 25 08:49 GridCanada.crl_url
-rw-r--r--. 1 cuddandr T2K  408 Nov 25 08:49 GridCanada.info
-rw-r--r--. 1 cuddandr T2K  455 Nov 25 08:49 GridCanada.namespaces
-rw-r--r--. 1 cuddandr T2K 1521 Nov 25 08:49 GridCanada.pem
-rw-r--r--. 1 cuddandr T2K  248 Nov 25 08:49 GridCanada.signing_policy

Specifically note the symbolic links that point from the "random" strings to the GridCanada files. If these symbolic links do not exist, they must be created.

Now Globus has several different directories it can search for these CA certificates and if it does not find the certificates where it looked then the transfer will fail. YUM defaults to installing them into /etc/grid-security/certificates and if you extracted them from a tarball they could be anywhere. If you extracted them from source then you should¹ place them into the $HOME/.globus/certificates directory (which you need to create). If the certificates are somewhere else, then we will simply link the $HOME/.globus/certificates directory to where the certificates are instead of copying everything (which should also work if you want to copy everything) like so:
user@host $ ln -s /path/to/certificates/ $HOME/.globus/certificates

On the HPCC the CA certificates for use with Globus reside in the /mnt/research/T2K/usr/etc/grid-security/certificates/ directory. Once the link has been created (or the certificates have been copied) the CA certificates are now ready for use with Globus.

¹ You can also put them in the /etc/grid-security/certificates/ directory and hope that Globus looks there. On my system Globus only looked in $HOME/.globus/certificates/ which is why I created a soft-link as above.

Running globus-url-copy

Now that all the security certificates are installed, we can move on to trying to run globus-url-copy. First before we try to use globus-url-copy we need to set up a proxy that tells the server who we are using our user certificates. This is done by running grid-proxy-init with optional arguments. Running grid-proxy-init by itself will generate a valid proxy for twelve hours and look for your usercert.pem and userkey.pem in its default location (which should be $HOME/.globus/). You can change the amount of time the proxy is valid by using the -valid flag with a time specified in HH:MM format. So nominally the command will look as follows:

user@host $ grid-proxy-init -valid 24:00

Which will create a proxy valid for the next twenty-four hours.

If the command complains about not finding your usercert.pem / userkey.pem, or you wish to use a different set than your default, then you can use the -cert and -key flags with the location of the respective files:

user@host $ grid-proxy-init -valid 24:00 -cert /path/to/cert.pem -key /path/to/key.pem

Lastly, the -help flag will give a small summary of the various -flags of the command.

Now that the proxy is set up, we can finally run globus-url-copy to facilitate file transfer. The official documenation of the comamnd may be found here: http://toolkit.globus.org/toolkit/docs/5.2/5.2.3/appendices/commands/#globus-url-copy. The command has many flags to alter its function, but it will get the job done with no flags passed to it. The basic usage looks as follows:

user@host $ globus-url-copy <source_url> <destination_url>

The <source_url> is the file you want to grab off of the server, e.g. gsiftp://t2ksrm.nd280.org/pnfs/nd280.org/data/nd280data/production005/F/mcp/neut/2010-11-air/magnet/beamc/anal/oa_nt_beam_90300000-0000_6rk2dnkozo5s_anal_000_prod005magnet201011airc-bsdv01.root, and the <destination_url> is where you want the file to go and what you want to name it. The destination can be written in normal unix path names (i.e. without the preceeding file://) and can use relative paths as well.

For grabbing files specifically off of the t2ksrm.nd280.org server using GRID-FTP you must prepend your <source_url> with gsiftp:// and that will then download the file using the correct protocol. Finally I would suggest running globus-url-copy with the -vb flag, which will display some useful information such as bytes transferred and the speed. Again you can run it with the -help command for a short description of all of the flags, or look at the documentation. So the full command to run would look something like this:

user@host $ globus-url-copy -vb gsiftp://t2ksrm.nd280.org/pnfs/nd280.org/data/nd280data/production005/F/mcp/neut/2010-11-air/magnet/beamc/anal/oa_nt_beam_90300000-0000_6rk2dnkozo5s_anal_000_prod005magnet201011airc-bsdv01.root /mnt/research/T2K/oaAnalysisFiles/oaFile1.root

Which would download the file oa_nt_beam_90300000-0000_6rk2dnkozo5s_anal_000_prod005magnet201011airc-bsdv01.root from the t2ksrm.nd280.org server at TRIUMF to /mnt/research/T2K/oaAnalysisFiles/ and name it oaFile1.root

Topic revision: r7 - 26 Feb 2015, AndrewCudd

This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback