- 03 Feb 2015
Using Globus / GRID for file transfer
This page contains instructions on how to use Globus to facilitate file transfers using the GRID and GRID-FTP. These instructions are specifically tailored for accessing the t2ksrm.nd280.org server at TRIUMF. The commands and install were performed on an install of Scientific Linux 7 (SL7), which is a Red Hat 7 based system, using a BASH shell. Adjust your commands accordingly if you are using a different system / shell.
Installing globus-url-copy (Globus Toolkit)
To install globus-url-copy and the corresponding commands you need to install the Globus Toolkit. Install at least version GT-5.2.3 or higher; these instructions were performed using version 5.2.3. The Globus documenation for installing the Globus Toolkit v5.2.3 is located here: http://toolkit.globus.org/toolkit/docs/5.2/5.2.3/admin/install/
. The instructions below are more or less directly from the documentation. If you are working on the HPCC (currently only working on the gateway node), Globus software is already installed and just needs to be loaded using the following commands:
user@host $ module load GNU
user@host $ module load GLOBUS
Then you can skip the instructions on installing the Globus Toolkit.
You can try to install the toolkit using a deb or rpm package using the corresponding commands or you can install it from source. These instructions will continue by installing the Globus Toolkit from source.
- Download the Globus Toolkit and extract the tarball
user@host $ tar -zxvf gt5.2.3-all-source-installer.tar.gz
- Create the directory you wish to install the Globus Toolkit; by default it will install into
- Wherever you choose to install it, make sure you have write permissions to that directory. For example, to install into
/usr/local/globus-5.2.3 you need to have root permissions.
- The documentation suggests to create a user named globus to own the directory; I did not do that and simply installed it as root.
- Before installing the Globus Toolkit, make sure you have the necessary dependencies installed. The list can be found in the documentation, and if
make fails it will tell you which packages you are missing. Simply install the missing package and try again.
- cd to the Globus Tooklit source directory and run the following commands to install the entire Globus Toolkit:
user@host $ ./configure --prefix=/Your/Install/Directory
user@host $ make
user@host $ make install
- If you do not wish to install the entire toolkit and want only the bare minimum for globus-url-copy, instead of make run:
user@host $ make gridftp
user@host $ make install
That should now give you a working installation of the Globus Toolkit or a subset of the toolkit. Finally for convenience, you should add the globus commands to your
, like so:
user@host $ export PATH=$PATH:/path/to/globus/install/bin
Installing User GRID Certificate
Accessing the t2ksrm.nd280.org server using GRID-FTP requires a set of security certificates and keys. Once you have the certificate in the form of a PKCS12 (.p12) file you can install the corresponding certificate and key for globus to use. The certificate and key should be installed into the
directory using the names below.
user@host $ openssl pkcs12 -in YourCert.p12 -clcerts -nokeys -out $HOME/.globus/usercert.pem
user@host $ openssl pkcs12 -in YourCert.p12 -nocerts -out $HOME/.globus/userkey.pem
and the userkey.pem
need to be read/write only by the owner of the file (i.e. you) otherwise you will get an error when you try to use them later. To change this, run:
user@host $ chmod 600 $HOME/.globus/usercert.pem
user@host $ chmod 600 $HOME/.globus/userkey.pem
Installing CA Certificates
Next a set of CA certificates needs to be installed (assuming it is not installed already) for another part of the secure transfer. These instructions follow almost verbatim from here: https://wiki.egi.eu/wiki/EGI_IGTF_Release
. You will (maybe) need your user GRID certificate to be installed in your browser to access that link. Then you have a few options on how to go about installing the CA certificates. I personally installed the packages through YUM, but you can also install specific RPM's or unpack the certificates from source. Installing through YUM would be best if possible since YUM will do everything for you and then updating the certificates later takes only a single YUM command.
To install the CA certificates via YUM:
- Add the following repo file to your
/etc/yum.repos.d/ directory. I called it egi-trustanchors.repo
- And then, assuming you are installing the EGI trust anchors for the first time, run:
user@host $ yum update
user@host $ yum install ca-policy-egi-core
That will install basically every CA certificate you will ever need, however it may install them in a directory that Globus is not looking in.
To install the certificates via tar-ball, do the following:
- Download the certificate.tar files that you need. For accessing t2ksrm.nd280.org you will need the ca_GridCanada.tar.gz file (grab the latest version).
- Exatract the tarball and move the all of the certificate files to your certificate directory (more on that in a moment).
- The file structure should look something like this after running
lrwxrwxrwx. 1 cuddandr T2K 14 Feb 4 09:54 5d674a88.0 -> GridCanada.pem
lrwxrwxrwx. 1 cuddandr T2K 21 Feb 4 09:54 5d674a88.namespaces -> GridCanada.namespaces
lrwxrwxrwx. 1 cuddandr T2K 25 Feb 4 09:54 5d674a88.signing_policy -> GridCanada.signing_policy
lrwxrwxrwx. 1 cuddandr T2K 14 Feb 4 09:54 bffbd7d0.0 -> GridCanada.pem
lrwxrwxrwx. 1 cuddandr T2K 21 Feb 4 09:54 bffbd7d0.namespaces -> GridCanada.namespaces
lrwxrwxrwx. 1 cuddandr T2K 25 Feb 4 09:54 bffbd7d0.signing_policy -> GridCanada.signing_policy
-rw-r--r--. 1 cuddandr T2K 40 Nov 25 08:49 GridCanada.crl_url
-rw-r--r--. 1 cuddandr T2K 408 Nov 25 08:49 GridCanada.info
-rw-r--r--. 1 cuddandr T2K 455 Nov 25 08:49 GridCanada.namespaces
-rw-r--r--. 1 cuddandr T2K 1521 Nov 25 08:49 GridCanada.pem
-rw-r--r--. 1 cuddandr T2K 248 Nov 25 08:49 GridCanada.signing_policy
Specifically note the symbolic links that point from the "random" strings to the GridCanada files. If these symbolic links do not exist, they must be created.
Now Globus has several different directories it can search for these CA certificates and if it does not find the certificates where it looked then the transfer will fail. YUM defaults to installing them into
and if you extracted them from a tarball they could be anywhere. If you extracted them from source then you should\xB9 place them into the
directory (which you need to create). If the certificates are somewhere else, then we will simply link the
directory to where the certificates are instead of copying everything (which should also work if you want to copy everything) like so:
user@host $ ln -s /path/to/certificates/ $HOME/.globus/certificates
On the HPCC
the CA certificates for use with Globus reside in the
directory. Once the link has been created (or the certificates have been copied) the CA certificates are now ready for use with Globus.
\xB9 You can also put them in the /etc/grid-security/certificates/
directory and hope that Globus looks there. On my system Globus only looked in $HOME/.globus/certificates/
which is why I created a soft-link as above.
Now that all the security certificates are installed, we can move on to trying to run
. First before we try to use
we need to set up a proxy that tells the server who we are using our user certificates. This is done by running
with optional arguments. Running
by itself will generate a valid proxy for twelve hours and look for your
in its default location (which should be
). You can change the amount of time the proxy is valid by using the
flag with a time specified in
format. So nominally the command will look as follows:
user@host $ grid-proxy-init -valid 24:00
Which will create a proxy valid for the next twenty-four hours.
If the command complains about not finding your
, or you wish to use a different set than your default, then you can use the
flags with the location of the respective files:
user@host $ grid-proxy-init -valid 24:00 -cert /path/to/cert.pem -key /path/to/key.pem
flag will give a small summary of the various -flags of the command.
Now that the proxy is set up, we can finally run
to facilitate file transfer. The official documenation of the comamnd may be found here: http://toolkit.globus.org/toolkit/docs/5.2/5.2.3/appendices/commands/#globus-url-copy
. The command has many flags to alter its function, but it will get the job done with no flags passed to it. The basic usage looks as follows:
user@host $ globus-url-copy <source_url> <destination_url>
is the file you want to grab off of the server, e.g. gsiftp://t2ksrm.nd280.org/pnfs/nd280.org/data/nd280data/production005/F/mcp/neut/2010-11-air/magnet/beamc/anal/oa_nt_beam_90300000-0000_6rk2dnkozo5s_anal_000_prod005magnet201011airc-bsdv01.root, and the
is where you want the file to go and what you want to name it. The destination can be written in normal unix path names (i.e. without the preceeding file://) and can use relative paths as well.
For grabbing files specifically off of the t2ksrm.nd280.org server using GRID-FTP you must prepend your <source_url>
and that will then download the file using the correct protocol. Finally I would suggest running
flag, which will display some useful information such as bytes transferred and the speed. Again you can run it with the
command for a short description of all of the flags, or look at the documentation. So the full command to run would look something like this:
user@host $ globus-url-copy -vb gsiftp://t2ksrm.nd280.org/pnfs/nd280.org/data/nd280data/production005/F/mcp/neut/2010-11-air/magnet/beamc/anal/oa_nt_beam_90300000-0000_6rk2dnkozo5s_anal_000_prod005magnet201011airc-bsdv01.root /mnt/research/T2K/oaAnalysisFiles/oaFile1.root
Which would download the file
from the t2ksrm.nd280.org server at TRIUMF to
and name it