Storage Service Overview¶

Computing Center provides various levels of file storage services for each experimental group and individual users, including AFS File Storage, EOS File Storage, LUSTRE File Storage, and Tape Storage.

If you need to share files or directories in your personal directory for collaboration, read Directory and File Access Authorization first. Personal directories are private by default. Grant access only to the required user or group, share only the required directory, and prefer read-only permissions.

AFS is a distributed file system composed of data servers and file servers, providing users with unified storage space. AFS is based on Kerberos authentication and has strong security. The OpenAFS system deployed by the Computing Center at IHEP is mainly used for user identity authentication and personal file secure storage, including source code, programs, documents and other files. It is backed up regularly to ensure data security. AFS also provides storage for public software libraries of each experiment group, which are published as read-only to ensure consistency of the software version on all nodes. The advantages of AFS are high data security and strong privacy, suitable for storing personal data. However, the storage space is small and access speed is slow. [warning]Moreover, due to the restriction of AFS's security mechanism, if you submit jobs in PBS or other job management system, you will not be able to get the job results in AFS directory.

LUSTRE is an open source, scalable parallel file system widely used in supercomputing platforms. More than 70% of the TOP500 supercomputers use LUSTRE file systems globally. LUSTRE consists of metadata servers (MDS), object storage servers (OSS) and clients. Through the metadata server, LUSTRE combines all object storage servers' storage targets into a unified storage space for clients to mount, fully compatible with POSIX semantics. LUSTRE has strong scalability and can easily manage PB-EB level of storage space, providing tens or even hundreds GB bytes of aggregated bandwidth, widely used in high-performance computing. Due to the limitations of LUSTRE's own design, its performance for "massive small files" scenarios is poor. Therefore, LUSTRE is mainly used to manage massive experimental data storage. LUSTRE is currently the most important disk file system in the Computing Center.

Tape storage is composed of tape drives, tape slots and robotic arms. Unlike disks, tapes are sequential access devices that must be loaded from the slot to the tape drive before they can be read or written from the head. The Computing Center has deployed a hierarchical storage management software called CASTOR to virtualize the tapes in the tape library into a transparent space that can be randomly accessed, similar to an FTP service. Data import (upload) or export (download) can be achieved through command line or API without considering the sequential access characteristics and complex loading/unloading process of magnetic material. Generally speaking, magnetic tapes have large storage capacity, low price, long preservation time and green energy saving advantages, and are widely used in high-energy physics fields. However, magnetic tape libraries are mechanical equipment with slow access speed and high failure rate, so they are mainly used for backup and long-term preservation of raw data.

Basic file operations¶

Switch current directory¶

$cd target_path

For example:

$cd /home

will change the current directory to /home

Create a file¶

$touch FileName

For example:

$touch myfile

Creates an empty file named myfile in the current directory.

Copy a file¶

$cp source_file target_file

For example:

$ cp /home/sourcefile /tmp/destinationfile

Copys the sourcefile in /home to /tmp, and renames it as destinationfile.

Migrate a file¶

$mv source_file target_file

For example:

$ mv /home/sourcefile /tmp/destinationfile

Moves the sourcefile in /home to /tmp, and renames it as destinationfile.

Delete a file¶

$rm filename

For example:

$ rm /home/file

Deletes the file /home/file

check the usage of a directory¶

du -sh directory_path

For example:

$ du -sh /home/

Displays the total disk space used by all files in /home and the number of files.

List the detailed attributes of a file¶

$ls -lh file_path

For example:

$ ls -l h /home/file

Displays the detailed attributes of /home/file.

List all files in a directory recursively¶

$ls -lRh directory_path

For example:

$ ls -lRh /home

Lists all files under directory_path, including subdirectories, and displays their detailed attributes.

Info

For most storage systems, if you just want to list all filenames in a directory, using the ls command without -l option will get results faster.

🤖 AI 助手