Running SDM calculations on XSEDE Comet

Post date: Apr 5, 2020 5:29:38 PM

File transfer from local machine to XSEDE Comet

Transferring of files to Comet from local machine and vice versa can be done using "rsync" or using the "Globus Transfer" utilities

Using rsync:

To rsync files and directories, do the following:

rsync -avz <somefolder> <USER>@comet.sdsc.edu:/oasis/projects/rut147/<USER>/

Using Globus:

For more information about how to use Globus to set up a local "Endpoint" for file transfers to Comet, please visit :

https://www.globus.org/data-transfer

Storage Resource on Comet:

The storage/simulation volume is located at

Running SDM Calculations on Comet

Comet uses a queuing system to launch any jobs on its GPU nodes. The queuing system is based on "SLURM" and uses .slurm script to launch jobs.

To run jobs on Comet, each complex directory needs a .slurm script. An example .slurm script is given here:

#!/bin/bash

#SBATCH -J 6vww-1

#SBATCH -p gpu

#SBATCH --gres=gpu:4

#SBATCH -N 1 # This is nodes, not cores

#SBATCH -n 1 # one process per node so we get one entry per node

#SBATCH -t 18:00:00 # Max time allotted for job

#SBATCH -A TG-MCB150001

echo "Number of nodes: $SLURM_NNODES"

echo "Nodelist: $SLURM_NODELIST"

echo "Number of tasks: $SLURM_NTASKS"

echo "Tasks per node: $SLURM_TASKS_PER_NODE"

jobname=6vww-1-congored

scontrol show hostname $SLURM_NODELIST > .slurm_nodes

awk '{ for(i=0;i<4;i++)print $1 ","0":"i",1,centos-OpenCL,,/tmp"}' < .slurm_nodes > nodefile

cd ..

rsync -av --exclude='slurm*.out' ${jobname} /scratch/$USER/$SLURM_JOBID/

cd /scratch/$USER/$SLURM_JOBID/${jobname}

for i in `seq 1 10` ; do

  ./runopenmm ~/software/async_re-openmm/bedamtempt_async_re.py ${jobname}_asyncre.cntl

  rsync -av * $SLURM_SUBMIT_DIR/

done

Change the jobname in the slurm file to suit your job. If I have many ligands I put a placeholder for the -J and jobname entries:

#SBATCH -J <job>

jobname=<job>

and then use sed -i "/<job>/${complex}/g" ${complex}/${complex}.slurm in a for loop running over the directories in the complex/ directory

To launch the slurm script do for example: sbatch <jobname>.slurm

This is how the .slurm script works:

The slurm script runs the simulation in 10 batches.

1. It firsts rsync's the simulation directory to the /scratch/ disk of the node,

2. It then runs the batch,

3. rsync's back to the working directory, and

4. runs the next batch and so on.

So, for example, if you want the asyncre job to run for 960 minutes, the asyncre.cntl file should specify 96 minutes as the WALL_TIME.

Free Energy Analysis on Comet

Prepare your R environment (first time only):

mkdir ~/R/x86_64-pc-linux-gnu-library

cp -r ~u15684/R/x86_64-pc-linux-gnu-library/3.6 ~/R/x86_64-pc-linux-gnu-library/

Load the R package:

Navigate to the scripts/ directory of the SDM project. For example:

cd /oasis/projects/nsf/rut147/<username>/6vww-1/scripts/

Edit the

setup-setting.sh

file to point it to the top-level SDM project directory (first entry, in this example it would be /oasis/projects/nsf/rut147/<username>/6vww-1) and the number of samples to discard for equilibration (last entry), and then run the analyze script:

bash ./analyze.sh