10.3 T2/T3 Cross-site x509 Access

10.3 T2/T3 Cross-site x509 Access¶

Within the IHEP WLCG grid infrastructure, files can be transferred bidirectionally between T2 (Tier-2) and T3 (Tier-3) storage using x509 grid certificates. The following uses the CMS experiment as an example.

Prerequisites¶

A valid grid certificate (see 8.1 Grid Certificate)
Grid certificate DN registered on the INK Dashboard under the X509 Certificate tab to complete the certificate→user mapping (required by the T3 xrootd proxy besproxy to resolve the certificate to a cluster user)
Joined the relevant VO (see 8.2 Join VO)
A valid x509 proxy certificate (typically named x509up_<uid>)

Environment Configuration¶

Set the following x509 environment variables in your job script. The CVMFS paths differ slightly per experiment; use the CMS example as a reference:

export X509_USER_PROXY=${TMPDIR}/x509up_uid    # proxy cert path (on the worker node)
export XrdSecPROTOCOL=gsi,unix                  # xrootd auth protocol
export X509_VOMS_DIR=/cvmfs/cms.cern.ch/grid/etc/grid-security/vomsdir
export X509_CERT_DIR=/cvmfs/cms.cern.ch/grid/etc/grid-security/certificates
export X509_VOMSES=/cvmfs/cms.cern.ch/grid/etc/vomses

Reference paths for other experiments:

Experiment	Certificate / CVMFS path prefix
CMS	`/cvmfs/cms.cern.ch/grid/etc/grid-security/` (`certificates`, `vomsdir`) and `/cvmfs/cms.cern.ch/grid/etc/vomses`
ATLAS	Corresponding directories under `/cvmfs/atlas.cern.ch/repo/sw/`
LHCb	Corresponding directories under `/cvmfs/lhcb.cern.ch/etc/grid-security/`

Job Submission¶

Use the -cert x509 flag of hep_sub to submit jobs requiring certificate authentication, and use -in to specify the proxy certificate file:

hep_sub job.sh -in /path/to/x509up_uid -cert x509

Parameters:

-cert x509 : Enables x509 certificate authentication mode. hep_sub writes +HepJob_CertType = "x509" into the HTCondor ClassAd, allowing the job system to configure the certificate environment accordingly.
-in /path/to/x509up_uid : Specifies the proxy certificate file path. hep_sub transfers it to the worker node's $TMPDIR along with the job.

T3 → T2 File Access¶

Access T2 EOS storage (cceos.ihep.ac.cn) from a T3 site (besproxy01.ihep.ac.cn, corresponding to /publicfs/cms/ public storage):

# List T2 EOS directory
xrdfs root://cceos.ihep.ac.cn ls /eos/ihep/cms/store/user/<username>/

# Upload from T3 worker to T2 EOS
xrdcp localfile root://cceos.ihep.ac.cn//eos/ihep/cms/store/user/<username>/

# Download from T2 EOS to T3 worker
xrdcp root://cceos.ihep.ac.cn//eos/ihep/cms/store/user/<username>/remotefile .

T2 → T3 File Access¶

Access T3 public storage (besproxy01.ihep.ac.cn) from a T2 site:

# List T3 public storage directory
xrdfs root://besproxy01.ihep.ac.cn:2094/ ls /publicfs/cms/user/<username>/

# Upload from T2 worker to T3
xrdcp localfile root://besproxy01.ihep.ac.cn:2094//publicfs/cms/user/<username>/

# Download from T3 to T2 worker
xrdcp root://besproxy01.ihep.ac.cn:2094//publicfs/cms/user/<username>/remotefile .

Using gfal Tools¶

In addition to xrootd clients, you can use gfal-* commands. Ensure gfal is in PATH (e.g., add /cvmfs/grid.cern.ch/alma9-ui-test/usr/bin):

export PATH=$PATH:/cvmfs/grid.cern.ch/alma9-ui-test/usr/bin

# List directory
gfal-ls root://cceos.ihep.ac.cn//eos/ihep/cms/store/user/<username>/

# Copy file
gfal-copy localfile root://cceos.ihep.ac.cn//eos/ihep/cms/store/user/<username>/

Verifying Certificate Status¶

Before submitting a job, verify the x509 proxy certificate is valid on the login node:

grid-proxy-info          # View basic proxy certificate information
voms-proxy-info --all    # View VOMS extended information (VO, roles, etc.)

If the proxy certificate is about to expire, regenerate it:

grid-proxy-init -valid 72:0 -out /path/to/x509up_uid

🤖 AI 助手