12.14 PWFA¶

12.14 PWFA

Overview¶

The PWFA experiment rents computing resources from the public computing platform of the Slurm cluster to complete its computing tasks. Available resources are as follows:

Resource Type	Resource	Description
Login node	pvm057	Dedicated login node for PWFA
Slurm	Compute nodes of the spub partition	spub[001-016] and gpu036, with the highest usage priority
home directory	/ihepfs/pwfa/user/{username}	Dedicated storage
working directory	/ihepfs/pwfa/data/	Dedicated storage

12.14.1 A Brief Introduction of the Slurm Cluster¶

12.14.1.1 register¶

See application for your cluster user name to get registered in the cluster.

12.14.1.2 Log in¶

ssh command is used to log in the cluster, <user_name>stands for your real user name.

# lxlogin: shared login nodes
ssh <user_name>@lxlogin.ihep.ac.cn

# pvm057 : dedicated login nodes for group pwfa
ssh <user_name>@pvm057.ihep.ac.cn

12.14.1.3 Cluster usage¶

There are two ways to use cluster
Interactive mode : Log into the node pvm057 and debug your program.
Job mode : submit slurm jobs with command sbatch and run your fine-tuned program.
- Jobs can be submitted to partition pwfadebug and spub.
More detailed usage can be found in the Slurm User Manual

12.14.2 Available computing resources¶

The PWFA experiment is renting computing services of the spub computing platform.
Available computing resources are in the Slurm cluster and can be classified into three catagories.

Partition	Node	Function	Description
login	pvm057	Dedicated login node	Log in with command `ssh` , interactive mode
pwfadebug	pvm058	Dedicated worker node	Submit jobs with command `sbatch`, job mode
spub	spub[001-016],gpu036	Shared worker nodes	Submit jobs with `sbatch`command, job mode

12.14.3 Accessable storage¶

/ihepfs is used as data or home directory for the PWFA experiment, details can be found in the following table.
<user_name> stands for your cluster user name

Directory	Functionality	quota limit
`/ihepfs/pwfa/user/<user_name>`	home directory	100GB，200,000 files
`/ihepfs/pwfa/data/<user_name>`	data directory	1TB，200,000 files

12.14.4 Installed software¶

Software	Installation path	Code source
fbpic	/cvmfs/slurm.ihep.ac.cn/centos7.9/anaconda3/envs/fbpic	github fbpic
QuickPIC	/cvmfs/pwfa.ihep.ac.cn/software/QuickPIC-OpenSource-dev/source/qpic.e	github QuickPIC

12.14.5 Slurm job samples¶

Users of the PWFA expriments can submit to partition pwfadebug and spub
To submit jobs, some job parameters including partition, account and qos must be provided.

partition	account	qos
pwfadebug	pwfa	spubpwfa
spub	pwfa	spubpwfa

12.14.5.1 QuickPIC¶

The following sample job script qpic_sample_job.sh show how to submit a QuickPIC job to partition spub.

$ cat qpic_sample_job.sh
#! /bin/bash

#SBATCH --partition=spub
#SBATCH --account=pwfa
#SBATCH --qos=spubpwfa
#SBATCH --ntasks=32
#SBATCH --mem-per-cpu=4GB
#SBATCH --job-name=quickPIC

module use /cvmfs/slurm.ihep.ac.cn/alma9/modulefiles
module load quickpic/develop
export UCX_NET_DEVICES=mlx5_0:1
export OMP_NUM_THREADS=1
echo $SLURM_NTASKS
mpirun -np $SLURM_NTASKS /cvmfs/slurm.ihep.ac.cn/alma9/software-gcc11/QuickPIC-OpenSource-dev/source/qpic.e

Explanation:

module use /cvmfs/slurm.ihep.ac.cn/alma9/modulefiles and module load quickpic/develop are used to load the software environment required to run QuickPIC

UCX_NET_DEVICES and OMP_NUM_THREADS set the network device and thread environment variables required to run QuickPIC

qpic.e is the excecutable program of QuickPIC, the defauld input file is qpinput.json which should be put in the current directory to submit jobs.

Details about qpinput.json can be found with the link of Input File for QuickPIC .

To submit QuickPIC job, type the following command

$ sbatch qpic_sample_job.sh

12.14.5.2 fbpic¶

The following job script fbpic_cpu.sh shows how to submit a fbpic job to partition spub.

$ cat fbpic_cpu.sh
#! /bin/bash

#=====================================================
# FBPIC CPU job on one spub CPU node
#
# Usage:
#   sbatch fbpic_cpu.sh
#   sbatch fbpic_cpu.sh fbpic_script.py
#
# Tune --cpus-per-task and THREADS together if you do
# not want to use all 56 CPU cores on the node.
#=====================================================

#SBATCH --partition=spub
#SBATCH --account=pwfa
#SBATCH --qos=spubpwfa
#SBATCH --job-name=fbpic_cpu
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=56
#SBATCH --mem=240G
#SBATCH --output=fbpic_cpu_%j.out
#SBATCH --error=fbpic_cpu_%j.err

set -euo pipefail

source /ihepfs/pwfa/data/public/env_FBPIC.sh

export NUMBA_CACHE_DIR=/ihepfs/pwfa/data/${USER}/cache
mkdir -p "${NUMBA_CACHE_DIR}"

# FBPIC CPU multi-threading. Keep these equal to --cpus-per-task.
THREADS="${SLURM_CPUS_PER_TASK:-56}"
export MKL_NUM_THREADS="${THREADS}"
export NUMBA_NUM_THREADS="${THREADS}"
export OMP_NUM_THREADS="${THREADS}"
export OPENBLAS_NUM_THREADS="${THREADS}"
export NUMEXPR_NUM_THREADS="${THREADS}"

# Keep all worker threads on the CPU cores allocated by Slurm.
export OMP_PROC_BIND=close
export OMP_PLACES=cores

FBPIC_SCRIPT="${1:-fb-in.py}"

echo "Job ID: ${SLURM_JOB_ID}"
echo "Node list: ${SLURM_JOB_NODELIST}"
echo "FBPIC script: ${FBPIC_SCRIPT}"
echo "CPU threads: ${THREADS}"
echo "Memory requested: ${SLURM_MEM_PER_NODE:-240G}"

srun --cpu-bind=cores python "${FBPIC_SCRIPT}"

Explanation:

This script runs an fbpic CPU job on one spub CPU node. Submit it with sbatch fbpic_cpu.sh [fbpic_script.py]; if fbpic_script.py is not specified, fb-in.py is run by default.

If you do not want to use all 56 CPU cores on the node, adjust --cpus-per-task and the script's THREADS variable together.

source /ihepfs/pwfa/data/public/env_FBPIC.sh is used to load the environment required to run fbpic.

NUMBA_CACHE_DIR sets numba's cache directory, avoiding cache conflicts when multiple users share the same installation.

MKL_NUM_THREADS, NUMBA_NUM_THREADS, OMP_NUM_THREADS, OPENBLAS_NUM_THREADS, and NUMEXPR_NUM_THREADS set the number of threads used by each of fbpic's computing libraries; keep these equal to --cpus-per-task.

OMP_PROC_BIND and OMP_PLACES bind worker threads to the CPU cores allocated by Slurm, avoiding thread migration that would hurt performance.

To submit the job, run the following command

$ sbatch fbpic_cpu.sh

🤖 AI 助手