12.12 HERD

Overview

Service Type	Resource	Notes
Login node	lxlogin.ihep.ac.cn
Computing resources	804 cores; idle or remote resources can be shared	HTCondor: HepJob
VNC and related services	herdtb01	Use containers
Home directory	/afs/ihep.ac.cn/users/	500 MB
Working directories	/herdfs/data, /herdfs/user	See 12.12.2

12.12.1 Account Application

• Apply for a computing cluster account.

Refer to Chapter 3. After approval from the HERD experiment computing contact person, the user can obtain an AFS account.

12.12.2 Storage Directories

Available storage directories are divided into two types: public storage provided by the computing cluster and HERD project-owned storage. Public storage includes the personal user directory under AFS, with a 500 MB quota.

HERD project-owned storage has two parts:

Storage Path	Quota	Features	Usage Notes
/herdfs/data	Dynamically controlled according to project needs	Working directory, large capacity	User directories are created under USER; do not create directories directly under data without permission
/herdfs/user		User directory	Stores personal files

12.12.3 Job Management

HERD uses the HepJob toolset for job management. HepJob wraps native HTCondor commands and is optimized for the IHEP cluster. Users should use HepJob instead of native HTCondor commands.

Environment Setup

Add the HepJob command directory to PATH:

export PATH=/cvmfs/common.ihep.ac.cn/software/hepjob/bin:$PATH

Job Submission: hep_sub

hep_sub -g herd job.sh

Common parameters:

Parameter	Description
-g herd	Use computing resources of the HERD group
-o <file>	Standard output file; default is job_name.out
-e <file>	Standard error file; default is job_name.err
-n <N>	Submit N jobs in a batch
-mem <MB>	Request memory in MB
-np <N>	Request number of cores, used for MPI jobs
-os <ver>	Specify operating system version, such as SL7

Note: The job script, such as job.sh, must have executable permission. Use chmod +x job.sh to grant permission.

Job Query: hep_q

# View all jobs of the current user
hep_q

# View jobs of a specified user
hep_q -u <username>

# View a specified job ID
hep_q -i 3745232.1

# View running jobs
hep_q -run

Output fields: JOBID is the job ID, OWNER is the owner, SUBMITTED is the submission time, RUN_TIME is the runtime, ST is the status (I = idle, R = running), PRI is the priority, SIZE is virtual memory usage, and CMD is the job program name.

Job Deletion: hep_rm

# Delete a specified job
hep_rm 3745232.1

# Delete a whole job cluster
hep_rm 3745232

# Delete multiple jobs at the same time
hep_rm 3745232 3745233.0

# Delete all jobs of the current user
hep_rm -a

# Force deletion of jobs in X state
hep_rm -a -forcex

All HepJob commands support -h for help, for example hep_sub -h.

Use Local Resources

Submit jobs to the IHEP local cluster, which is the default method:

hep_sub -g herd job.sh

If the -rmt parameter is not specified, jobs are scheduled by default to IHEP local HTCondor compute nodes (herdws001-012, 804 cores in total).

Use Dongguan Resources

Submit jobs to the Dongguan site by specifying the remote site with -rmt:

hep_sub -g herd -rmt job.sh

For query and deletion commands, add the -rmt parameter to the default commands in the same way.

The Dongguan site provides additional computing resources and is suitable when the local cluster is busy or when larger-scale computing is needed.

12.12.4 Grid Authentication

HERD uses INDIGO IAM for grid authentication management: https://herd-iam.ihep.ac.cn.

IAM supports OIDC/WLCG Token, SAML/eduGain, IHEP-SSO, and X.509 certificate plus VOMS proxy authentication.

Register an IAM Account

1. Visit https://herd-iam.ihep.ac.cn.
2. Choose a login method. IHEP-SSO or eduGain is recommended.
3. Fill in the registration form and wait for administrator approval.
4. After approval, log in to IAM to obtain a Token or VOMS proxy.

Obtain Token / VOMS Proxy

• WLCG Token: IAM supports WLCG JWT Profile by default and can generate it directly from the web interface.
• VOMS Proxy: VOMS AA service address is herd-iam.ihep.ac.cn:8443, and the VO name is herd.

12.12.5 Grid Data Management

HERD grid data management is based on Rucio: https://herd-rucio.ihep.ac.cn.

Rucio is used for HERD dataset registration, distribution, replication, and lifecycle management. It is currently open to production users.

Environment Setup

source /cvmfs/herd.ihep.ac.cn/grid/env.sh

Common Operations

# List available RSEs (storage endpoints)
rucio list-rses

# List datasets
rucio list-dids --filter type=DATASET

# Download data
rucio download <dataset>

Storage Endpoints

HERD grid storage endpoints are provided by collaborating sites. Administrators should contact zhangxuantong@ihep.ac.cn to register SEs in Rucio.

12.12.6 Software and Environment

HERD offline software HERDOS has been deployed on CVMFS. It includes a complete toolchain for detector simulation (Geant4), digitization, reconstruction, and analysis.

Use Through CVMFS

# CentOS 7 / AlmaLinux 9
source /cvmfs/herd.ihep.ac.cn/HERDOS/centos7_amd64_gcc850/Release/v00-10/ExternalLibs/bashrc.sh

Version	Environment Command
v00-10	source /cvmfs/herd.ihep.ac.cn/HERDOS/centos7_amd64_gcc850/Release/v00-10/setup.sh
v00-06	source /cvmfs/herd.ihep.ac.cn/HERDOS/SLC7/Pre-release/v00-06/setup.sh
v00-05	source /cvmfs/herd.ihep.ac.cn/HERDOS/SLC7/Pre-release/v00-05/setup.sh

For detailed release notes, see /cvmfs/herd.ihep.ac.cn/HERDOS/README.

Build from Source

git clone git@code.ihep.ac.cn:herdos/offline.git
cd offline
git checkout v00-10
./build_RelWithDebInfo.sh
source install/setup.sh

12.12.7 Other Management Services

HERD provides the following auxiliary management services on the herdtb01 node. These services are deployed in containers.

Other Services

Service	Description
VNC	Remote graphical desktop for ROOT/Geant4 visualization and related use cases

To add more services, contact the HERD computing administrator.

12.12.8 Service Support

HERD computing administrators:

Administrator	Email
Xiaowei Jiang	jiangxw@ihep.ac.cn
Xuantong Zhang	zhangxuantong@ihep.ac.cn

🤖 AI 助手

💬 快速提问 ×

有问题？输入后跳转到对话页继续交流

发送 →