8. Cluster Guidelines

PICLas has been tested on numerous different Linux- and CPU-based clusters and is continuously been developed to run efficiently on high-performance systems.

8.1. Simulating at HLRS

Unfortunately, the GitHub and GitLab servers are not available on machines at the HLRS, such as the Hawk, due to restricted internet access. The workaround is to use ssh tunneling and proxy jump or remote forwarding to access the repositories.

8.1.1. Cloning with the SSH protocol

Two methods for checking out the code are described in the following.

8.1.1.1. Method 1 (Proxy Jump)

To clone a repository from, e.g., github.com on a HLRS system, the ssh proxy jump must first be set up. Simply connect to the system via ssh and add the following lines to the ssh configuration file under ~/.ssh/config on the cluster

Host github.com
   HostName   github.com
   ProxyJump  RemoteHost

where the RemoteHost has internet access and can be accessed via ssh from the HLRS system. It is also defined

Host RemoteHost
   User         username
   HostName     hostname.something.com
   ForwardX11   yes

Then simply clone the repository via

git clone git@github.com:mygroup/myproject.git

and all the ssh traffic via github.com is automatically re-routed over the RemoteHost.

8.1.1.2. Method 2 (Remote Forwarding)

You can use the SSH protocol to clone the repository. You have to connect to the cluster with the RemoteForward option

ssh -R 7777:github.com:22 username@hazelhen.hww.de

To avoid using the above command every time, you can add the following to your .ssh/config file:

host hlrs
   hostname        hazelhen.hww.de
   user            username
   RemoteForward   7777 github.com:22

and login with ssh hlrs. Now you can clone the repository when logged onto the cluster by

git clone ssh://git@localhost:7777/piclas/piclas.git

Note that if you experience problems when connecting, e.g., when the warning

Warning: remote port forwarding failed for listen port 7777

is issued, you have to choose a different port, e.g., 1827 (or any other 4-digit number) and re-connect. If the code has already been cloned using the original port, the number of the port must be changed in ./git/config to the new number for git fetch/pull operations, which would then look like

[remote "origin"]
  url = ssh://git@localhost:1827/piclas/piclas.git
  fetch = +refs/heads/*:refs/remotes/origin/*

8.1.2. Compiling and executing PICLas

For building on the hazelhen cluster, certain modules have to be loaded and included in the .bashrc or .profile:

module unload PrgEnv-cray
module load PrgEnv-intel
module load cray-hdf5-parallel

An example submit script for the test queue is then

#!/bin/bash
#PBS -N Testcase
#PBS -M email@university.de
#PBS -m abe
#PBS -l nodes=4:ppn=24
#PBS -l walltime=00:24:59
#PBS -q test

# number of cores per node
nodecores=24

# switch to submit dir
cd $PBS_O_WORKDIR

# get number of total cores
ncores=$(cat $PBS_NODEFILE | wc -l)

module unload craype-hugepages16M

# restart
aprun -n $ncores -N $nodecores -j 1 ./piclas parameter.ini DSMC.ini restart.h5 1>log 2>log.err

More information on using the queue system can be found in the HLRS wiki.

Section last updated: 27.03.2019