12.9 HEPS¶

Overview¶

Resource type	Resource amount	Notes
Login nodes
HTCondor
Slurm
Home directory
Work directory

12.9.1 New Users¶

Apply for a computing cluster account.

Refer to the account application chapter. After approval from the HEPS experiment contact, users can obtain an AFS account.

Initialize the computing environment.

After logging in for the first time, run:

$ ln -s /sharefs/heps/data/opt/config/bash_profile ~/.bash_profile
$ source ~/.bash_profile

12.9.2 Computing Resources¶

HEPS computing resources include the following nodes:

Node	Logical CPU cores per node	Node type	Scheduler	Notes
accap002.ihep.ac.cn		28	Login node
accap001.ihep.ac.cn - accap009.ihep.ac.cn	56	Compute node	Slurm	Excluding accap002
accap010.ihep.ac.cn - accap015.ihep.ac.cn	72	Compute node	Slurm
accap016.ihep.ac.cn - accap017.ihep.ac.cn	88	Compute node	Slurm
accap019.ihep.ac.cn - accap053.ihep.ac.cn	36	Compute node	Slurm	SSH login is not allowed

After the 2025 summer system upgrade, all compute servers were upgraded to ALMA-9. Users need to reset the login configuration:

$ rm ~/.bash_profile
$ ln -s /sharefs/heps/data/opt/config/bash_profile ~/.bash_profile

Log out and log in again for the setting to take effect.

For VNC access to accap002, disable screen blanking after login: Applications -> System Tools -> Settings -> Power -> Blank Screen Never.

12.9.3 Storage Directories¶

Directory	Quota / file limit	Quota command
/afs/ihep.ac.cn/users/a-z/username	500 MB / none	`fs listquota /afs/ihep.ac.cn/users/a-z/username`
/workfs2/heps/username	5 GB / 50,000 files	`lfs quota -u username /workfs2`
/scratchfs/heps/username	500 GB / none; data kept for two weeks	`lfs quota -u username /scratchfs`
/sharefs/heps/user/username	3 TB / none	`lfs quota -u username /sharefs`

Some users may have personal directories under /workfs2/csns/username, /workfs2/others/username, /scratchfs/csns/username, or /scratchfs/others/username.

Usage recommendations:

Keep only important personal configuration files under /afs.
/workfs2 can be read and written on login nodes, but Slurm jobs on compute nodes cannot create or modify files there.
Use /sharefs for important results and data that need to be read and written by login and compute nodes.
Use /scratchfs for temporary large files and intermediate results.
Back up important results to your personal computer in time.

12.9.4 Programs and Examples¶

HEPS computing programs are installed under /sharefs/heps/data/APcodes and /sharefs/heps/data/opt/. See /sharefs/heps/data/APcodes/README.md for a detailed program list.

12.9.4.1 Run Parallel Programs Through Slurm¶

Submit jobs after logging in to lxlogin.ihep.ac.cn or accap002.ihep.ac.cn. The recommended number of parallel cores is a multiple of 36.

Example sbatch.sh:

#! /bin/bash
#SBATCH --partition=heps
#SBATCH --account=heps
#SBATCH --job-name=DA
#SBATCH --qos=regular
#SBATCH --output=slurm-%j.txt
#SBATCH --ntasks=36

mpiexec -np 36 Pelegant parTrack2.ele

Submit the job:

$ sbatch sbatch.sh

For debugging, use the hepsdebug queue. It allows jobs with no more than 36 tasks and a maximum run time of 30 minutes:

#SBATCH --qos=hepsdebug

Useful commands:

$ sinfo -p heps
$ squeue
$ scancel jobid

12.9.4.3 Run Licensed MATLAB¶

HEPS MATLAB is installed under /afs/ihep.ac.cn/soft/hepsap/matlab. Log in to accap002.ihep.ac.cn by SSH or VNC and run:

$ matlab

The installed MATLAB supports parallel computing with up to 512 parallel cores. MATLAB parallel jobs do not need Slurm; request parallel cores inside MATLAB:

>> parpool(512)

12.9.4.4 Run Licensed Mathematica¶

HEPS Mathematica is installed under /sharefs/heps/data/opt/Mathematica/12.0:

$ /sharefs/heps/data/opt/Mathematica/12.0/bin/mathematica

The license supports up to 17 concurrent Mathematica main programs and up to 136 cores for parallel computing. Examples are available under:

/sharefs/heps/data/opt/examples/MathematicaExamples/

For single-node parallel jobs on accap002, add the following before the parallel section:

LaunchKernels[20]

For cross-node parallel jobs, submit resources through Slurm. Non-interactive Wolfram Language scripts can be run with:

math -script test.wls

Interactive jobs can first allocate resources:

salloc -p heps --qos=regular --ntasks=72 -t 01:00:00

Then launch Mathematica on accap002 and run the parallel computation.

12.9.5 FAQ¶

12.9.5.1 Cannot Write to the AFS Personal Directory¶

Users need to renew tokens about every 10 hours:

$ echo 'password' | kint username
$ aklog -d

You can also create a scheduled task:

$ sh /sharefs/heps/data/opt/config/generate_crontab_klog.sh username password

If the AFS directory is full, check quota and disk usage:

$ fs listquota /afs/ihep.ac.cn/users/a-z/username
$ du -h /afs/ihep.ac.cn/users/a-z/username

MATLAB users should clear cache files regularly:

$ rm ~/.felix

12.9.5.2 Cannot Log In by SSH from a Linux Host¶

If SSH reports Host key verification failed, remove the lines containing the target host, such as accap002, from ~/.ssh/known_hosts.

12.9.5.3 MATLAB Plots Do Not Display Correctly on accap002¶

Start MATLAB with software OpenGL:

$ matlab -softwareopengl

🤖 AI 助手