You are here: Foswiki>T2K Web>HPCC (15 Sep 2015, AndrewCudd)EditAttach
-- AndrewCudd - 26 Feb 2015

High Performance Computing Cluster (HPCC)

The High Performance Computing Cluster (HPCC) is the computing cluster that the T2K group uses at MSU. It consists of a series of computing nodes that run Red Hat Enterprise Linux release 6.3 on various scales of hardware. The HPCC is split in to three main components: the Gateway Node, the Development Nodes, and the Computing Nodes.

The Gateway Node is the node that you access when you first SSH in to the HPCC. It is the access point for the development nodes and is the only node in the system that has access to the outside world. The Gateway node is for monitoring and checking the status of jobs and for transferring data to the HPCC. The Gateway node is the only point in the system where file transfer utilities such as globus-url-copy, dccp, and iRODS work since they are not blocked by the firewall. Gateway nodes sometimes run slower than the dev nodes (below) as well.

The Development Nodes ('dev nodes') are a set of nodes meant for developing code and general, relatively light, use. There are six main types of development nodes which parallel the different types of Computing Nodes the HPCC has available. The Development nodes are where you will usually work on the HPCC, however the Development nodes have some restrictions on what can be run. Jobs that run longer than two hours may be killed without warning and jobs using excessive resources may also be killed without warning (although I am not sure what constitutes excessive resources).

The Computing Nodes are the nodes that do the heavy computing for the cluster. Jobs that get submitted to the queue are then sent to a Computing Node once the resources requested become available. The HPCC has several different classes of Computing Nodes depending on the resources needed for the job, such as large amounts of RAM or GPU enabled cluster nodes. More information on the available hardware can be found here:

Accessing the HPCC

The HPCC is normally accessed using the SSH protocol using your favorite means of SSH connection. The address for SSH and just about any other connection (e.g. SFTP) is, which if logging in to the HPCC will take you to the Gateway Node. From the Gateway Node you then can SSH in to which ever Development Node you wish, or instead use the Gateway Node for your work. Normally for SSH connection you are going to want X11 Forwarding enabled, which can be enabled with the -X flag for SSH using the terminal.

user@host ssh -X

File System

The HPCC file system is exactly what you would expect for a Linux based system, however it is probably arranged slightly differently from a personal Linux system. Your /home directory is located at /mnt/home/your_user_name so the only difference is to remember the /mnt that precedes the /home directory. The T2K group space is located at /mnt/research/T2K (note the capital T2K). For transferring files between the HPCC and another machine the File Transfer page has more details on various ways to transfer files.

HPCC Backups

The HPCC automatically creates backups of all reseach and home directories of its users. Backups are created hourly for the past 24 hours, daily for the past week, and then weekly for the past month. These backups are stored in a special hidden directory on the HPCC and can be accessed like any other directory once you know where to go.

The backups for home directories are stored in the following directory: ~/.zfs/snapshot/

The backups for research directories are stored in the following directroy: /mnt/research/groupname/.zfs/snapshot/

Inside the snapshot directory are then directories which contain the backed up files for a certain timestamp.

Checking the status of the HPCC

Sometimes, the cluster is shut down for various reasons (security, cooling, etc). Read the email digests for information about outages. One can also see the current status here:, or from their home page (though sometimes it isn't updated until after the problem is resolved). You can also submit a ticket / contact the HPCC through their contact form which can be found here:

Links / Resources

The HPCC has a wiki page with various tutorials, unstructions, and information pages which can be found here: Below are links to various pages of tutorials and information related to T2K use of the HPCC:
Topic revision: r10 - 15 Sep 2015, AndrewCudd

This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback