When we started thinking about simulating forest landscapes with a crazy high (i.e. single tree) resolution and were bouncing ideas how to accomplish this back and forth, we always tried to gauge whether we’d end up with useful tool for all practical purposes; practical here means to have a tool that can be used with standard PCs (no extra expensive hardware required) and in a time frame of minutes to hours (so you don’t have to wait weeks until a simulation is finished). Quite frequently, when we liked an idea but were not so sure about the “practical” impacts, we’d resort to the Moore’s law argument which goes like: “it might not be practical today, but in a couple of years computers will be much faster and will have much more memory…”. Actually, we *did* see some progress in the field of office computer hardware over the last years (and, by using multithreading and 64 bit architecture, we also try to facilitate these trends as far as possible). However, the big computer-power revolution happened in the supercomputer-area. The Top-500 list of supercomputers lists (surprise) the 500 fastest computer clusters in the world since the early 1990s. And the computer power grew by amazing 5 to 6 orders of magnitude in these 20 years. Since the machinery used by those supercomputers is not radically different from the stuff that we were used to (i.e., Intel or AMD processors, Linux operating system), it has always been tempting to tap into that resource. Enter the VSC:
The VSC is a supercomputer situated at the technical university Vienna and jointly operated by a bunch of Austrian universities. The first cluster (VSC-1) was built in 2009 and featured about 4000 processor cores – in 2009 it reached the rank 156 in the top-500 list of supercomputers. In the meanwhile, the cluster was extended (VSC-2, completed 2011, 21000 cores, awesome #56 in the top-500 list 2011), and currently plans to build VSC-3 are under way. Fortunately, the University of Natural Resources and Life Sciences (BOKU) is part of the VSC-consortium, which allows us to apply for using the computer resources.
Currently we work on an EU-funded project (www.fundiveurope.eu) that tries to boost biodiversity research in Europe. Our task within the project is to estimate the effect of biodiversity on the provisioning of ecosystem services in disturbed forest landscapes. To do this, we simulate a large number of different artificial landscapes characterized by different species mixtures and disturbance regimes – all in all a large number of simulations (about 4500). When we realized that our office PCs would have to crunch numbers for month, we seriously started to explore the supercomputer option.
To actually create a “cluster version” of iLand for the VSC-1 was extremely easy, because the required components were already more or less in place:
From a user’s perspective, the “cluster” is just like your typical Linux shell; you have a home directory and have access to all the available software (such as Matlab, R, Mathematica, …). Qt and subversion was already there, so compiling iLand for with the (recommended) Intel compiler (icpc) was not very complicated: you “checkout” the source from the SVN-repository and build with a command such as:
qmake iland.pro –spec linux-icc-64 CONFIG+=release make
the “linux-icc-64” tells Qt to user the mkspecs of the icpc-compiler, and all the rest happens automagically behind the scenes – awesome! Some minor changes later (very convenient with SVN) we had a running iLand executable and were able to start What really helped a lot in the whole process was the IT staff of the VSC: very friendly and helpful, thank you very much, guys! The cluster facilitates the “sun grid engine” (SGE) system for managing computer jobs. iLand does not use “MPI”-style parallelization (i.e. the execution of one program at the same on multiple nodes). Therefore, we use the cluster as if it was simply a collection of individual PCs (nodes): a single simulation consists of copying the input data to the target node, running the simulation on the node, and copying back the results to the home-directory. The whole process is managed by the SGE; what we had to specify is a “sge”-script:
#$ -V
#$ -cwd
#$ -N iLand
#$ -t 1-2864:1
#$ -pe smp 8
#$ -l h_rt=02:00:00
# N: name
# t: the number of tasks 1-x:stepsize
# pe: one node for each execution
# export OMP_NUM_THREADS: execute with 8 threads.
# start with qsub run.missing.sge
#!/bin/sh
export OMP_NUM_THREADS=8
echo "copy data to the shared memory of the node /dev/shm"
mkdir /dev/shm/iland_project
mkdir /dev/shm/iland_project/log
mkdir /dev/shm/iland_project/output
cp -r $HOME/iland/projects/FUNDIV/* /dev/shm/iland_project
MISSID=${SGE_TASK_ID}
### the $ is the argument (i.e. the ID)
### get the 2nd and the 3rd value (comma separated)
VARSPECIES=`grep ^$MISSID, /dev/shm/iland_project/design/master_sim_table_miss.csv | cut -d "," -f 12`
VARDRI=`grep ^$MISSID, /dev/shm/iland_project/design/master_sim_table_miss.csv | cut -d "," -f 13`
VARTYPE=`grep ^$MISSID, /dev/shm/iland_project/design/master_sim_table_miss.csv | cut -d "," -f 14`
ID=`grep ^$MISSID, /dev/shm/iland_project/design/master_sim_table_miss.csv | cut -d "," -f 2`
echo "ID $ID species: $VARSPECIES and DRI: $VARDRI type: $VARTYPE missing#: $MISSID"
echo "now run the model, for 500 years"
$HOME/iland/build/ilandc/ilandc /dev/shm/iland_project/project_Hain_sim.xml 500 user.generic_disturbance.return_interval=$VARDRI user.generic_disturbance.type=$VARTYPE model.settings.seedDispersal.externalSeedBackgroundInput="$VARSPECIES"
echo "copy back the results to data${ID}"
mkdir $HOME/jobs/results/data${ID}
mv /dev/shm/iland_project/output/* $HOME/jobs/results/data${ID}
mv /dev/shm/iland_project/log/log.txt $HOME/jobs/results/data${ID}/
echo "done."
The script combines some Linux shell magic (“grep” et al) and command line options of ilandc to extract the parameters of each simulation from a “master”-table that contains the layout of the full exercise.
The scheduling of jobs on the cluster is a bit unpredictable: sometimes not a single job is executed for hours, then, as if in a hurry, fifty simulations are run in parallel. A simulation on one node takes about the same time as on an office PC (the VSC-1 uses Intel quadcore CPUs (X5550), but the trick is the number of nodes. The general workload of the cluster seemed to decline considerably around Christmas, providing a good slot for the iLand simulations: the full simulation set took about 2.5 days, peaking at 300 simulations per hour – I guess that at this point almost 50% of the cluster was simulating forest landscapes ;)