9.1.1 HTCondor Jobs¶

9.1.1.1 Overview¶

HTCondor is an open-source batch processing system widely used in high-energy physics computing. It supports large-scale computing clusters, provides high flexibility, and is especially suitable for high-throughput job scheduling. The Computing Center has migrated most local resources to HTCondor. Based on the actual cluster environment, the Computing Center has configured and optimized HTCondor and deployed the HepJob toolset to help users manage jobs. HepJob wraps optimizations for the IHEP cluster. Unless there are special requirements, users should use HepJob instead of native HTCondor commands.

9.1.1.2 HepJob Toolset¶

1) Environment Setup and Job Preparation¶

All HepJob commands are located in the following directory. It is recommended that you add this directory to your PATH.

Bash users:

$ export PATH=/cvmfs/common.ihep.ac.cn/software/hepjob/bin:$PATH

Tcsh users:

$ setenv PATH /cvmfs/common.ihep.ac.cn/software/hepjob/bin:$PATH

Job files submitted to the HTCondor computing cluster must have executable permission. Use the following commands to check and grant file permissions.

Check file permissions:

$ /bin/ls -l job.sh
-rw-r--r-- 1 jiangxw u07 85 Aug 29 18:23 job.sh

In this example, job.sh does not have executable permission. Run the following command to grant executable permission:

$ /bin/chmod +x job.sh

Check again. The extra x in the first column indicates executable permission.

$ /bin/ls -l job.sh
-rwxr-xr-x 1 jiangxw u07 85 Aug 29 18:23 job.sh

2) Job Submission¶

Command format:

hep_sub [-h] [-g {physics,juno,dybrun,dyw,u07,offlinerun,pku,longq}]
             [-p {virtual,local,ali}] [-u {vanilla,grid,docker}] [-o OUT]
             [-e ERROR] [-n NUMBER] [-os OPERATINGSYSTEM]
             [-t {atlasbm,hxmtbm,wljMC}] [-prio PRIORITY]
             [-np NUMBERPROCESS] [-argu ARGUMENTS [ARGUMENTS ...]]
             [-dir DIRECTORY] [-mem MEMORY] [-quiet] [-part PARTITION]
             [-name NAME] [-slurm] [-site SITENAME] [-jf JOBFILE]
             [-tf TRANSFERFILE] [-wn WORKNODE] [-wt WALLTIME]
             jobscript

Common parameters:

jobscript: Job script name. It can be an absolute or relative path.

$ hep_sub job.sh

-g: User group used by the job. If not set, the user's primary group is used. For example, to use the juno group:

$ hep_sub -g juno job.sh

-p: Resource pool. Two resource pools are currently provided: local physical machine resources and virtual virtual machine resources. If not specified, local is used by default.

$ hep_sub -p virtual job.sh

-u: Job type. If not specified, the default is vanilla. Most local-cluster users do not need to specify this parameter. To run grid jobs, specify -u grid.

$ hep_sub -u grid job.sh

-o: Standard output file path. If not specified, a standard output file named job_name.out is generated in the current directory.
-e: Standard error file path. If not specified, a standard error file named job_name.err is generated in the current directory.
-l: Job log file path. If not specified, no log file is generated. The log file records job runtime status and is usually not meaningful for ordinary users.
-os: Required operating system version. If not specified, the default is SL6. For example, to use SL7 resources:

$ hep_sub -os SL7 job.sh

-np: Number of CPU cores to request. This is needed for MPI jobs.
-argu: Arguments passed to the job script. For example, if the job program argument is 1:

$ hep_sub -argu 1 job.sh

-mem: Required memory size in MB. For example, to request more than 3 GB of memory:

$ hep_sub -mem 3000 job.sh

If only -mem is specified, the job is scheduled to machines with the largest memory.

-site: Remote site where the job runs.
Convenient templates: In addition to basic parameters, hep_sub provides job templates for different experiments to simplify job submission. Currently, the cepcmpi template is provided, and more experiment templates will be added gradually. Use -t to specify the template. For example, to submit a CEPC MPI job:

$ hep_sub -t cepcmpi job.sh

Note

Using other parameters together with -t may affect how the -t template is applied.

3) Query Job Status¶

Command format:

hep_q [-h] [-u [USER]] [-i ID] [-run] [-p {virtual,local,ali}]
      [-t {atlasbm,hxmtbm,wljMC}] [-st STARTTIME]
      [-stat {run,idle,other} [{run,idle,other} ...]] [-slurm]

Common parameters:

-u: Query jobs of a specified user. The default is the current user.

$ hep_q -u <username>

Running hep_q -u without a username outputs information for the current user's jobs.

-i: Query a specified cluster ID or complete job ID. A job system job ID consists of two parts: clusterid and processid. clusterid is the job cluster number, and processid is the internal number in the job cluster. A complete job ID has the format clusterid.processid, such as 3745232.1.

Query job 3745232.1:

$ hep_q -i 3745232.1

Query all jobs in job cluster 3745232:

$ hep_q -i 3745232

-run: Query running jobs. Without this option, all jobs are output. Example output:

JOBID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD
5519.0 user1 04/11 15:11 0+00:00:00 I 0 0.0 myjob.sh
3656.0 user2 04/11 17:12 0+00:00:00 I 0 0.1 job.sh

JOBID is the job ID, OWNER is the job owner, SUBMITTED is the submission time, RUN_TIME is the runtime, ST is the current status, PRI is the priority, SIZE is virtual memory usage, and CMD is the job program name.

-t: Select a specific template. For example, cepcmpi users need to add this parameter when querying jobs.

$ hep_q -i 3745232 -t cepcmpi

4) Delete Jobs¶

Command format:

hep_rm [-h] [-a] [-forcex] [-p {virtual,physical}] [-site SITENAME]
       [-t {atlasbm,hxmtbm,wljMC}] [-name NAME] [-slurm]
       [jobs [jobs ...]]

Common parameters:

jobs: Job IDs to delete. Multiple job IDs can be specified at the same time.

$ hep_rm 3745232 3745233.0

This command deletes all jobs in job cluster 3745232 and the job 3745233.0.

-a: Delete all jobs of the current user.

$ hep_rm -a

-t: Select a specific template. For example, cepcmpi users need to add this parameter when deleting jobs.

$ hep_rm 3745232 -t cepcmpi

-forcex: Force deletion of jobs in X state. This parameter is used only for jobs in X state. Such jobs are usually in deletion state because node failures prevent compute nodes and the scheduler from releasing connections normally. You can specify a job ID or use it together with -a. For example, if job 3745232 is in X state:

$ hep_rm 3745232 -forcex

5) Other¶

All HepJob commands support -h and --help to view command descriptions.

9.1.1.3 Job and Resource Information¶

1) Wall Time and Job Limits¶

On a login node, run the following command to view wall-time limits by group. For example, to view the limits of the juno group:

$ hep_clus -g juno --walltime

Experiment	Short job limit (hours)	Normal job limit (hours)	Mid job limit (hours) and resource usage limit
BES	<0.5	<40	<100:10%
JUNO	<0.5	<20	<100:10%
DYW	<0.5	<10	<100:10%
CEPC	<0.5	<10	<100:10%
ATLAS	<0.5	<10	<100:10%
CMS	<0.5	<10	<100:10%
HXMT	<0.5	<14	<100:10%
GECAM	<0.5	<24	<100:10%
LHCb	<0.5	<100
LHAASO	<0.5	<15	<100:10%

For experiments without mid-job settings, the default resource usage limit for submitted mid jobs is 1.

9.1.1.4 Usage Tips¶

1) Using Macros in Job Submission¶

Macros are provided by the scheduling system. Users can define rules during job submission. Currently, only basic variable passing is supported. Variable passing can be used for jobscript, -o, -e, and -argu. The syntax consists of % and {}:

%{macro_name}

Macros are useful for batch job submission. See Section 2 below for details. Macros can also be used to include job IDs, group names, and similar values in job file names. More features will be added later.

2) Batch Job Submission¶

Batch job submission submits multiple jobs at once and supports submitting a large number of jobs. Use -n to specify the number of jobs to submit in one batch.

$ hep_sub job.sh."%{ProcId}" -n 5000

%{ClusterId} is the job cluster ID. %{ProcId} is the internal ID in the job cluster. It starts from 0 and increases by 1 (0,1,2,3,...). Therefore, %{ProcId} can be used to match jobs with their output or to customize output names. Job scripts can use suffixes, prefixes, or substrings corresponding to %{ProcId} to distinguish jobs.

You can also pass %{ProcId} as an argument to the job script and use it to map other files, such as input data files:

$ hep_sub job.sh -argu "%{ProcId}" -n 5000

Note: When submitting batch jobs, ensure that the job script exists and has executable permission. Avoid accidentally submitting an excessively large number of jobs, because heavy resource usage affects your own job priority.

Example 1:

Prepare executable job scripts as follows:

real_job_20191204_0.sh
real_job_20191204_10.sh
real_job_20191204_1.sh
real_job_20191204_2.sh
real_job_20191204_3.sh
real_job_20191204_4.sh
real_job_20191204_5.sh
real_job_20191204_6.sh
real_job_20191204_7.sh
real_job_20191204_8.sh
real_job_20191204_9.sh

Because the script names all follow the format real_job_20191204_*.sh and the key characters are numbers increasing from 0, submit them with:

$ hep_sub real_job_20191204_"%{ProcId}".sh -n 11

Example 2:

If the job script names do not contain increasing numbers starting from 0, but still follow a pattern, for example:

real_job_20191201.sh
real_job_20191202.sh
real_job_20191203.sh
real_job_20191204.sh
real_job_20191205.sh
real_job_20191206.sh
real_job_20191207.sh
...
real_job_20191230.sh
real_job_20191231.sh

Prepare an additional parent job script and pass %{ProcId} to it. The parent script maps the process ID to the actual files. For the scripts above, whose names end with dates, prepare real_job_parent.sh:

#!/bin/bash

# get procid from command line
procid=$1

# map 0,1,2,...,30 to 1,2,3,...,31
sub_name_number=`expr $procid + 1`

# format 1,2,3,...,31 to 01,02,03,...,31
sub_name=`printf "%02d\n" $sub_name_number`

# run the real job script by the formatted file name
bash real_job_201912"${sub_name}".sh

Submit the jobs with:

$ hep_sub real_job_parent.sh -argu "%{ProcId}" -n 31

3) Get Basic Job Information in a Job Program¶

Job ID

The job ID can be obtained through the _CONDOR_IHEP_JOB_ID environment variable. This variable is valid only while the job is running. Example in bash:

#!/bin/bash
JobId=$_CONDOR_IHEP_JOB_ID

Execution node

The execution node can be obtained through _CONDOR_IHEP_REMOTE_HOST:

#!/bin/bash
ExecWorkNode=$_CONDOR_IHEP_REMOTE_HOST

Job submission time

The submission time can be obtained through _CONDOR_IHEP_SUBMISSION_TIME:

#!/bin/bash
SubmissionTime=$_CONDOR_IHEP_SUBMISSION_TIME

9.1.1.5 Recommendations and Notes for Experiments¶

1) BESIII¶

For standard BOSS jobs, use the simplified boss.condor command:

$ boss.condor joboptions.txt

To specify job memory:

$ boss.condor -mem 5000 joboptions.txt

To use multiple CPU cores:

$ boss.condor -cpu 2 joboptions.txt

To adjust the maximum job runtime:

$ boss.condor -wt mid joboptions.txt

The following job types are provided: test, short, mid, long, and special.

To submit a batch of jobs with consecutive job numbers, use -n:

# Suppose user jobs job0, job1, and job2 exist.
# Use -n and "%{ProcId}" for batch submission.

$ boss.condor -n 3 job%"{ProcId}"

# "{ProcId}" replaces the number starting from 0 in the job name.
# Add % before it. This submits job0, job1, and job2 at once.

The job number must start from 0.

boss.condor also supports the "one-stop" job service, packaging SIM and REC into one overall job:

$ boss.condor sim_joboption.txt rec_joboption.txt ana_joboption.txt

One submission completes three jobs.

The "one-stop" service also supports batch submission:

# Suppose user jobs sim0, rec0, ana0, sim1, rec1, and ana1 exist.

$ boss.condor -n 2 sim%"{ProcId}" rec%"{ProcId}" ana%"{ProcId}"

# This syntax submits two user jobs:
# the first job is sim0, rec0, ana0;
# the second job is sim1, rec1, ana1.

BESIII users should set the user group to physics when submitting other jobs:

$ hep_sub -g physics job.sh

For job query and deletion, see the previous sections.

2) CEPC MPI Jobs¶

Example MPI job submission:

$ hep_sub /cefs/CEPCMPI/usernem/mpijob.sh -g cepcmpi -t cepcmpi -np 20

Here, -t specifies the cepcmpi experiment job template, and -np is the requested number of cores. The full path of the job script must be provided.

For job query and deletion, see the previous sections.

3) COMET Jobs¶

For job submission, query, and deletion, refer to the BESIII instructions.

4) Other Experiments Without Special Instructions¶

Refer to the previous sections.

FAQ¶

See FAQ.

🤖 AI 助手