8.3 Lustre File Storage¶
8.3.1 Overview¶
LUSTRE is an open-source, scalable, high-performance, object-oriented parallel file system that is widely used on supercomputing platforms. More than 70% of the world's supercomputers use the LUSTRE file system. LUSTRE consists of metadata servers (MDS), object storage servers (OSS), and clients. Through metadata servers, LUSTRE combines storage devices on all object storage servers into a unified storage space and provides it to clients for mounting. It is fully compatible with POSIX semantics. LUSTRE is highly scalable, can easily manage tens of PB of storage, and provides very high performance, with aggregate bandwidth reaching tens or even hundreds of GB per second. Therefore, it is widely used in high-performance computing. However, due to metadata server limitations, LUSTRE has no advantage in managing massive numbers of small files, and it does not support erasure coding or replicas. Therefore, the LUSTRE file system is mainly used to store massive experiment data. LUSTRE is currently the Computing Center's main disk file system.
LUSTRE is the largest-scale file system used in the high-performance computing platform. It provides massive storage services for BES, DYB, JUNO, and other experiments. Current Lustre file system information is listed in Table 6.
| Total Space (TB) | Purpose | User Quota | Backup Status | |
|---|---|---|---|---|
| /bes3fs | 2500 | BESIII production data | Raw data is backed up to tape; others are not backed up | |
| /besfs4 | 2500 | BESIII production data | Raw data is backed up to tape; others are not backed up | |
| /besfs5 | 2400 | BESIII production data, BESIII group data, user data | 50 GB per user under the users directory; group directory quotas vary by group |
Raw data is backed up to tape; others are not backed up |
| /besfs7 | 3600 | BESIII production data | Raw data is backed up to tape; others are not backed up | |
| /besfs8 | 3300 | BESIII production data | Raw data is backed up to tape; others are not backed up | |
| /besfs9 | 4000 | BESIII production data | Raw data is backed up to tape; others are not backed up | |
| /besfs10 | 4100 | BESIII production data | Raw data is backed up to tape; others are not backed up | |
| /publicfs | 3000 | ATLAS, CMS, LHCb, and UCAS sub-pool shared data disk | 5 TB and 300,000 files per user | Resources are isolated by group; no backup |
| /sharefs | 1500 | ALICPT, BES, HBKG, HEPS, and MBH sub-pool shared data disk | Resources are isolated by group; no backup | |
| /scratchfs | 572 | User temporary files | 500 GB and 200,000 files per user | No backup |
| /scratchfs2 | 1300 | User temporary files, gradually replacing /scratchfs |
500 GB and 200,000 files per user | No backup |
| /workfs2 | 22 | User personal files, recommended for important files or results | 5 GB and 50,000 files per user | Full backup |
| /cefs | 2500 | CEPC experiment data and user data | No backup | |
| /junofs | 3100 | JUNO experiment data | 500 GB and 300,000 files per user | No backup |
| /lustre/juno26 | 2600 | JUNO experiment data | 500 GB and 300,000 files per user | No backup |
| /lustre/junoflash | 88 | JUNO experiment data | No backup | |
| /dybfs2 | 1400 | DYB experiment data and user data | 1 TB and 300,000 files per user | Raw data is backed up to tape; others are not backed up |
| /gecamfs | 2200 | GECAM experiment data | No backup | |
| /hxmtfs | 1300 | HXMT experiment data | 500 GB and 50,000 files per user home directory; 5 TB and 1,000,000 files per user data directory | No backup |
| /hxmtfs2 | 2200 | HXMT experiment data | No backup | |
| /hpcfs | 1800 | Slurm GPU application data | No backup | |
| /lhaasofs | 610 | LHAASO user data | 200 GB and 500,000 files per user | No backup |
| /epfs | 610 | EP experiment data and user data | No backup | |
| /extpfs | 610 | EXTP experiment data and user data | No backup | |
| /herdfs | 1700 | HERD experiment data and user data | 200 GB and 300,000 files per user | No backup |
| /hepsfs | 645 | BSRF experiment data and user data | No backup | |
| /hepsdatafs | 29000 | HEPSBL central storage experiment data | No backup | |
| /lustrefs/atlas | 635 | ATLAS experiment data and user data | No backup | |
| /lustrefs/alicpt | 1300 | ALICPT experiment data and user data | No backup | |
| /aifs | 194 | IHEPAI artificial intelligence application data and user data | 500 GB and 500,000 files per user home directory | No backup |
| /sdcfs | 1700 | SDC submitted data and user data | Submitted data is backed up; others are not backed up |
Table 6. Lustre file system information.
- The Lustre file system fully supports Linux POSIX semantics. Users can access Lustre with Linux file system commands and APIs.
- To use user ACLs and extended attributes, submit a request to ics@ihep.ac.cn.
/workfsis dedicated to interactive use. It is readable and writable on login nodes, but users cannot create or modify files under/workfsfrom compute nodes. In other words, user jobs cannot write to or modify files under/workfswhile running.
8.3.2 Common Commands¶
Check User Resource Quota¶
Command format:
Example:
Example output:
quotas for user zhangsan (uid XXXX):
Filesystem kbytes quota limit grace files quota limit grace
/publicfs 338626804 524288000 524288000 - 232010 300000 300100
(used space) (soft capacity quota) (hard capacity quota) (used files) (soft file quota)
Check Quota Usage of Directories Under /besfs5/user and /besfs5/group¶
- In a specific directory, run
lsattr -pon any regular file (not a directory) to view the project ID of that directory.
This shows that the project ID of the directory is 1097.
- Run
lfs quota -p 1097 /besfs5to view quota usage of that directory.
# lfs quota -p 1097 -h /besfs5
Disk quotas for prj 1097 (pid 1097):
Filesystem used quota limit grace files quota limit grace
/besfs5 1.362T* 50G 50G - 293249 0 0 -
Check Space Usage of Each Disk Pool in a Lustre File System¶
Command format:
The -p parameter is followed by the disk pool name. For example, the ATLAS group has acfs.atlaspool, the CMS group has acfs.cmspool, and the LHCb group has acfs.lhcbpool.
Examples:
Example output:
Set File Striping to Improve Concurrent Access Performance for a Single File¶
Command format:
n is the stripe count.
Example:
Files under the directory inherit the directory striping attribute. To specify striping for a single file, create an empty file first with touch, and then set its striping attribute.
Set Directory ACLs¶
For example, the directory /besfs4/wanglutest is owned by root:root and has permission 750; user wanglu (60007:600) has no access to this directory.
-bash-4.2$ ll /besfs4 |grep wanglutest
drwxr-x---+ 2 root root 4096 May 11 15:24 wanglutest
-bash-4.2$ cd /besfs4/wanglutest
-bash: cd: /besfs4/wanglutest: Permission denied
Grant user wanglu permission on this directory:
Check wanglu's permissions:
-bash-4.2$ getfacl /besfs4/wanglutest
getfacl: Removing leading '/' from absolute path names
# file: besfs4/wanglutest
# owner: root
# group: root
user::rwx
user:wanglu:rwx
group::r-x
mask::rwx
other::---
The user can now read and write in the directory.
-bash-4.2$ pwd
/afs/ihep.ac.cn/users/w/wanglu
-bash-4.2$ cd /besfs4/wanglutest
-bash-4.2$ cp /etc/hosts .
-bash-4.2$ ls -lh
total 3.1M
-rw-r--r-- 1 wanglu u07 3.0M May 11 15:22 f1
-rw-r--r-- 1 wanglu u07 292 May 11 15:38 hosts
Delete a user's ACL permission:
Notes:
- Only the owner of a directory can modify ACL permissions on that directory.
- After ACL permissions are set,
ls -lshows a+after the permission field. At this point, the original Linux permission rules no longer fully apply. - To add an ACL for a group, use
setfacl -m group:xxx /xxx/xxx.
To restore the original Linux permission display and clear the + after ls -l, use setfacl -b.
For complete ACL syntax, see:
http://wiki.lustre.org/images/5/57/PosixAccessControlInLinux.pdf
Or refer to Linux man pages:
FAQ¶
See FAQ.
有问题？输入后跳转到对话页继续交流