12.6 CMS¶

Overview¶

Resource Type	Capacity	Notes
Login Node	lxlogin.ihep.ac.cn	General-purpose login node
HTCondor	T2: 1728 cores / T3: 836 cores	T2 shares endpoint with ATLAS, HTCondor-CE v24
Slurm	1×A100 + V100 shared pool + RTX 5090 nodes	cmsgpu group, see 7.6.7
home Directory	/afs/ihep.ac.cn/users/	500MB, see 7.6.3
Working Directory	T2 EOS: 1.7 PB / T3 publicfs: 849.1 TB	cceos.ihep.ac.cn, see 7.6.3/7.6.5

Login:

ssh lxlogin.ihep.ac.cn

Grid Authentication:

Since 2023, CMS has been migrating from X.509 certificates to WLCG Token authentication. CERN CMS IAM: https://cms-auth.web.cern.ch
IHEP T2 HTTP/XRootD/HTCondor all support Token authentication. GSI (voms-proxy-init) is being phased out.

12.6.2 Tier3 Local Computing¶

CMS local computing resources: 836 cores, managed by HTCondor. Access via lxlogin.

HTCondor Job Submission:

condor_submit submit -group cms

T3 Site Production (2025 data):

Jobs completed: ~11,192,024 (-8.0% YoY)
CPU hours: ~3,788,453 (+141.0% YoY)

12.6.3 Tier3 Local Storage¶

Public Data Area¶

/publicfs/cms

Total capacity: 849.1 TB (Lustre filesystem), ~97% used, nearly full. Availability >99%.

Check usage:

lfs quota -u <afs_username> /publicfs

Public Software Area¶

/afs/ihep.ac.cn/soft/CMS/
/cvmfs/cms.cern.ch/ (maintained by CERN)

Personal Storage Areas¶

Directory	Quota	Features	Notes
`/afs/ihep.ac.cn/users/<username>`	500MB	Backed up, readable from all nodes	Config files, scripts
`/workfs/cms/<username>`	5GB / 50,000 files	Backed up, read-only on compute nodes	Important personal files
`/scratchfs/cms/<username>`	500GB / 2-week retention	Read/write from all nodes	Temporary files

12.6.4 Tier2 Grid Computing¶

System: HTCondor-CE + Batch HTCondor, upgraded to HTCondor v24.

CreamCE, mentioned in older documentation, was fully decommissioned in 2023. All resources now use HTCondor-CE.

Resources: 1728 cores (48.6k HS23), sharing the same CE endpoint with ATLAS.

T2 Site Production (2025 data):

Jobs completed: ~2,933,904 (-46.3% YoY, due to shift toward analysis job types)
CPU hours: ~3,139,908 (+4.8% YoY)
Data uploaded: 158.0 TB (-51.5% YoY)
Data downloaded: 779.7 TB (-28.5% YoY)

Site Availability/Reliability: Ranked 15^th among all 49 CMS T2 sites globally (Jul 2025 – Jul 2026).

Maintenance and Procurement Project (approved 2023, completed Dec 2024):

Resource Type	Before	After	Multiplier
T2 CPU	420 cores	1728 cores	3.8x
T2 Storage	673.5 TB	1.7 PB	2.5x

12.6.5 Tier2 Grid Storage¶

System: EOS storage (shared instance with ATLAS, Belle2, Biomend, OPS).

In May 2023, T2 storage was migrated from DPM to EOS. dCache/SRM has been decommissioned. The old address ccsrm.ihep.ac.cn has been replaced by cceos.ihep.ac.cn.

Resources: 1.7 PB total

Central managed data: ~540 TB
User data area: ~350 TB capacity, ~216.3 TB currently used (61.8%)

T2 EOS Access:

EOS XRootD endpoint: root://cceos.ihep.ac.cn:1094/

EOS WebDAV endpoint: https://cceos.ihep.ac.cn:9000/eos/ihep/cms/

Common commands:

# Grid tools (recommended)
gfal-ls root://cceos.ihep.ac.cn:1094//store/user/
gfal-ls https://cceos.ihep.ac.cn:9000/eos/ihep/cms/store/user/

# xrdfs access
xrdfs root://cceos.ihep.ac.cn:1094/ ls /store/

# davix-ls access
davix-ls https://cceos.ihep.ac.cn:9000/eos/ihep/cms/store/

# EOS client (set environment variable first)
export EOS_MGM_URL=root://cceos.ihep.ac.cn:1094
eos ls /store/

User Space Quota Policy (updated 2026):

Condition	Quota
Usage < 9 TB	No quota (default)
Usage exceeds 9 TB	9 TB quota automatically applied at 08:00 next morning
Over-quota	Email notification sent, please clean up
Persistent over-quota ~6 months	Oldest files cleaned first

12.6.6 Grid Job Submission¶

CMSSW Environment Setup¶

export VO_CMS_SW_DIR=/cvmfs/cms.cern.ch/
source $VO_CMS_SW_DIR/cmsset_default.sh

# List installed CMSSW versions
scramv1 list -c CMSSW

# First time: create local CMSSW directory
scramv1 project CMSSW CMSSW_X_Y_Z

# Initialize runtime environment
cd CMSSW_X_Y_Z/src/
eval `scramv1 runtime -sh`

CRAB Job Submission¶

source /cvmfs/cms.cern.ch/crab/crab.(c)sh

# Create and submit jobs
crab submit [-c <CRAB-configuration-file>]

# Check job status
crab status [-d] <CRAB-project-directory>

# Retrieve logs and results
crab getlog [-d] <CRAB-project-directory>

More details: https://twiki.cern.ch/twiki/bin/view/CMSPublic/WorkBookCRAB3Tutorial

T2/T3 Bidirectional Data Access (new in 2025)¶

T3 jobs accessing T2 data: Add certificate options in hep_sub to enable T3 jobs to access T2 EOS storage via XRootD protocol.
```
hep_sub --in <your voms proxy> --cert <your job>
```
T2 jobs accessing T3 data: After linking your CERN certificate at http://ccs.ihep.ac.cn/jeeplus/a, T2 jobs can access T3 data via XRootD proxy.

12.6.7 GPU Resources¶

CMS Priority GPU¶

Item	Description
Resource	1×NVIDIA A100 GPU, priority access for cmsgpu group
Management	Slurm shared pool (includes V100 GPUs), QOS: `cmsnormal`
Suitable for	Scientific/HPC workloads requiring higher numerical precision
Apply	Request cmsgpu group approval at https://ink.ihep.ac.cn/dashed
Submit	`#SBATCH --qos=cmsnormal`

IHEP-AI Shared GPU Platform¶

Item	Description
Resource	Slurm cluster: 2×8-GPU RTX 5090 nodes; Kubernetes cluster: 1×8-GPU RTX 5090 node
Management	hepai group
Suitable for	AI training, large model workloads (mixed precision)
Apply	Request hepai group approval
Platform	https://ai.ihep.ac.cn

Slurm documentation: http://afsapply.ihep.ac.cn/cchelp/zh/local-cluster/jobs/slurm/

12.6.8 Notes¶

/publicfs/cms is ~97% full. Please remove unused files promptly to free space.
/workfs is read-only on compute nodes. Jobs writing to this directory will fail; use /scratchfs for job output.
Keep files per directory under 1000; create subdirectories for large file collections.
T2 EOS user storage exceeding 9 TB will be automatically restricted. Watch for quota alert emails in your CERN inbox.
X.509 grid certificates are being replaced by WLCG Tokens. New users should use Token authentication directly.
Contact site coordinator for questions: zhangxuantong@ihep.ac.cn.

🤖 AI 助手