Frequently Asked Questions¶

14.1 How to Get Help¶

If you encounter problems when using the computing platform, you can get help through the following channels.

a. Telephone support during working hours: 88236855

b. Email support: helpdesk@ihep.ac.cn ihep_computing_service@ihep.ac.cn

c. Web helpdesk: http://helpdesk.ihep.ac.cn

When contacting administrators, please provide as much information as possible, including your account, login node, job ID, error messages, time of occurrence, related paths, job scripts, and log files. Please also keep the problem scene unchanged whenever possible so that administrators can investigate it more efficiently.

1. After logging in to lxlogin, I cannot edit files and see errors such as .Xauthority does not exist, unauthorized, or Permission denied.

This type of problem is usually related to an AFS token, home directory quota, or an abnormal .Xauthority file. You can first run:

-bash-4.2$ kinit user
Password for user@IHEPKRB5:
-bash-4.2$ aklog -d

Replace user with your own computing platform account. After the commands complete successfully, try editing or deleting the file again.

If the .Xauthority error still appears, remove the file, log out, and log in again:

-bash-4.2$ rm -f ~/.Xauthority
-bash-4.2$ exit

It is also recommended to check whether the home directory quota has been used up:

fs listquota ~

Hidden files or hidden directories may also occupy a large amount of space:

ls -a ~

If you can no longer log in and cannot run the commands above, please submit your account, login node, and complete error message through Helpdesk so that administrators can assist with the check.

2. I forgot my password. How can I reset it?

Visit https://newlogin.ihep.ac.cn or https://login.ihep.ac.cn, log in with the registered email address of your cluster account, and change your password. After changing the password, use the new password to log in to the computing environment.

If unified authentication works but lxlogin still reports an incorrect password, log in to ccsinfo.ihep.ac.cn to check the status of your computing account, account expiration date, and password expiration date. If the issue persists, send the account issue to ihep_computing_service@ihep.ac.cn.

3. My password is correct, but I still cannot log in to lxlogin.

Please check the following items first:

a. Whether your computing platform account has been applied for and approved.

b. Whether your account or password has expired. You can check this at ccsinfo.ihep.ac.cn.

c. Whether you are using the correct login address, for example:

ssh user@lxlogin.ihep.ac.cn

d. Whether you are always logging in to a specific abnormal node. It is recommended to use the domain name lxlogin.ihep.ac.cn so that the system can assign a login node automatically.

If you still cannot log in, send your account, login command, error message, and the last time you logged in successfully to ihep_computing_service@ihep.ac.cn.

4. SSH authentication succeeds, but the shell hangs or no command prompt appears.

Common causes include problems with the home directory, startup scripts, quota, or a referenced filesystem. For example, if .bashrc or .bash_profile references a faulty disk or an inaccessible path, the login process may hang after authentication.

If you can still access your directory through SFTP, with help from another account in the same group, or by other means, you can temporarily rename the related startup files and try logging in again. If both SSH and SFTP hang, submit a ticket with your account, login node, where the session hangs, and any startup scripts recently modified.

5. My account has expired. What should I do?

If you can still log in to ccsinfo.ihep.ac.cn, check the account expiration date on the account information page and apply for an extension according to the page instructions.

If you cannot handle it through self-service, send the expired account information and extension request to your experiment group leader, and copy ihep_computing_service@ihep.ac.cn. After the group leader approves the extension, the account administrator will extend the account.

6. My computing platform account application has not been approved for a long time. What should I do?

First confirm whether the experiment group, supervisor, or responsible person selected during application is correct, and contact the corresponding experiment group administrator to check the approval progress.

When submitting a ticket, provide your unified authentication account, application time, requested experiment group or resource group, and page screenshots so that administrators can identify which approval step is pending.

7. How do I apply for or change experiment group, subgroup, or Slurm account permissions?

Log in to ccsinfo.ihep.ac.cn to apply for the corresponding subgroup or resource group, such as BES, JUNO, LHAASO, LHCb, ATLAS, CEPC, or a GPU resource group. After joining the corresponding group, you will usually obtain the related directory, software environment, or Slurm account permissions.

If you cannot find the target group, the group application is stuck, or you are not sure which group to apply for, contact the experiment group leader for confirmation before submitting a ticket.

8. How should my computing platform account be handled after I change my institution, email address, or unified authentication account?

If your bound email address, unified authentication account, or identity changes, it may affect computing platform login, notification delivery, or account extension workflows. Please submit a ticket and describe your original unified authentication account, computing platform account, new bound email address or new unified authentication account, and the computing platform permissions that need to be retained or migrated.

9. Can I log in directly to dedicated nodes such as beslogin or ailogin?

Different login nodes and computing resources have different security policies and access scopes. In general, use lxlogin.ihep.ac.cn as the public entry point first. Some dedicated nodes require direct login through their own domain names or require the corresponding experiment group or resource group permissions.

If you cannot log in to a dedicated node, provide the target node, login command, account, error message, and experiment group. Do not use a fixed login node as your only long-term entry point.

10. Set an alias for the login node.

You can edit ~/.ssh/config and set an SSH alias.

Host lx
    Hostname lxlogin.ihep.ac.cn
    User user
    Port 22

Then you can log in to the cluster with:

[user@localhost ~]$ ssh lx

Replace user with your own computing platform account.

14.3 Job Issues¶

My job has been queued for a long time and still cannot run.

a) For users who have recently run a large number of jobs, the scheduler recalculates and lowers their priority in real time to ensure fairness among users.

b) The IHEP computing cluster usually runs under high load. During job peaks, resource shortages, or when high-priority public service jobs are running, jobs may wait in the queue for a long time.

c) Special jobs, such as long-running jobs, large-memory jobs, or many GPU jobs, have more limited available resources and may wait longer.

d) If the queue reason shows MaxGRESPerAccount, ReqNodeNotAvail, PartitionNodeLimit, or similar messages, check the reason in squeue to determine whether it is caused by group resource limits, node maintenance, or parameter limits.

e) If the above reasons do not explain the issue, contact administrators and provide the JobID, submission time, partition, account, qos, and submission script.
Slurm reports an account, partition, or qos mismatch when I submit a job. What should I do?

The --partition, --account, and --qos values in a Slurm job script must match the resource groups for which you are authorized. If the error says Invalid account or account/partition combination specified, it usually means that you have not joined the corresponding resource group or that the script parameters are incorrect.

Log in to ccsinfo.ihep.ac.cn to apply for the corresponding resource group or Slurm account. After approval, modify the job script according to the user manual and resubmit the job.
How do I request a GPU in a Slurm job? Why is no GPU allocated after I enter a GPU partition?

Selecting a GPU partition alone may not allocate a GPU. You must explicitly request GPU resources in the job script. For example:
```
#SBATCH --partition=gpu
#SBATCH --account=higgsgpu
#SBATCH --qos=debug
#SBATCH --gres=gpu:v100:1
```
Here, --gres=gpu:v100:1 requests one v100 GPU card. Use the actual parameters required by your experiment group resources and the user manual.
If there is no dedicated CPU partition, can I submit non-GPU jobs to a GPU partition?

Some experiment groups or resource pools may not have a separate CPU partition. If you are authorized to use a GPU partition and the job does not need a GPU, you may submit a normal CPU job without the --gres option. However, you must still follow the limits of that partition and resource group, including limits on job count, CPU, memory, and runtime.
The job status is Hold, Held, CG, or FAILED. What does it mean?

Common causes include incorrect job script parameters, no write permission on the output directory, exhausted storage quota, filesystem problems, high scheduler load, node problems, or remaining processes when the job exits.

For an HTCondor job, use the following command to check the hold reason for a specific JobID:
```
hep_q -i JobID -hold
```
Or:
```
hep_q -u user -hold
```
For a Slurm job, check the job status and exit code:
```
sacct -j JobID
```
To cancel a job, use:
```
scancel JobID
```
If you are unsure about the reason shown by the command, keep the failed job and logs, and contact administrators.
Job commands respond slowly, or report that they cannot connect to the job scheduling service. What should I do?

When the scheduler is under high load, commands such as hep_q, squeue, and hep_sub may respond slowly, and connection failures to the scheduling service may occasionally occur. Retry later and avoid frequent repeated queries in a short time.

If the issue continues for a long time, provide the command, error message, account, login node, and occurrence time, and contact administrators to check the scheduler service status.
Can I run computing programs directly on login nodes?

Login nodes are mainly used for editing files, preparing environments, debugging small programs, and submitting jobs. They are not suitable for long-running computing tasks or heavy I/O tasks. Large computations, batch tasks, large-memory tasks, and GPU tasks should be submitted to compute nodes through the job system.

If a script runs slowly on a login node or affects interactive use, submit it as a job and check whether the running directory is on an appropriate storage path.
Where should job output files and logs be written?

Job output should be written to a directory that is writable from compute nodes, has enough quota, and is appropriate for the experiment group. Do not write large output files, temporary files, or logs to the home/AFS directory. If the output directory has no write permission, the filesystem is read-only, the quota is full, or the file count limit is reached, jobs may enter Hold, FAILED, or remain stuck for a long time.

Before submitting jobs, confirm that the output path exists, that you have write permission, that the quota is sufficient, and that many jobs are not writing to the same small file at the same time.
What information should I provide when I need administrator help with a job problem?

Please provide as much job information as possible, including but not limited to JobID, submission time, error symptoms and messages, job logs, running path, submission script, partition, account, qos, and input/output directories. Keep the problem scene whenever possible. The more information administrators have, the easier it is to locate the problem.
Commands such as hep_sub or hep_q report command not found. What should I do?

This usually means the hepjob command path has not been added to your environment variables. Try:
```
export PATH=$PATH:/cvmfs/common.ihep.ac.cn/software/hepjob/bin/
```
You can also run the commands by absolute path. If the commands still cannot be used, provide the login node, current shell, echo $PATH output, and the complete error message.

14.4 File Storage Issues¶

1. What should different directories on the computing platform be used for?

The AFS/home directory is suitable for personal source code, scripts, configuration files, and a small number of important files. Its default space is small and it is not suitable for large datasets or large job output.

Experimental data, analysis results, and large files should be placed in experiment group directories or public storage for which you have permission, such as /publicfs, /workfs2, /besfs5, /lustrefs, /hpcfs, and /scratchfs2. A large number of small files can affect the performance of parallel filesystems. It is recommended to archive them or use a more suitable directory.

2. How do I confirm which directories I can use?

First check your experiment group and subgroup information, and then confirm the default directories according to the user manual or experiment group instructions. Different experiment groups have different data directories and public directories. If entering a directory reports Permission denied, you may not have joined the corresponding subgroup, or the directory may not belong to your experiment group.

If you are not sure which directory to use, provide your account, experiment group, target directory, intended use, and error message in the ticket.

3. My directory suddenly cannot be written to. What happened?

Personal directories and group public directories usually have maximum space and file count limits. When space or file count exceeds the limit, errors such as Disk quota exceeded, No space left on device, inability to create files, or inability to edit files may occur.

Check the following items:

a. Whether the user's personal space quota has reached the limit.

b. Whether the user's personal file count has reached the limit.

c. Whether the directory or project directory has reached its quota.

d. Whether the experiment group public directory or the entire filesystem is full.

Check the quota of the filesystem where the error occurs. Do not check only home or AFS if the error is reported in another path.

4. The capacity is not full. Why still cannot I write files?

In addition to space quota, the file count limit may also be reached. Large numbers of small files, conda environments, package caches, log files, and job intermediate files can quickly consume the file count quota.

If the file count limit is reached, writing may fail or Disk quota exceeded may appear even when the used capacity is not full. Clean unnecessary small files or package large numbers of small files into tar archives before keeping them.

5. How do I check my storage quota usage?

For AFS storage, use:

fs listquota ~

Or:

fs listquota /afs/ihep.ac.cn/users/u/user

For Lustre-like directories, you can usually use:

lfs quota -u user -h /publicfs
lfs quota -u user -h /scratchfs2
lfs quota -u user -h /lustrefs/juno26

Some spaces limited by directory or project require directory quota query commands, for example:

lfs_dir /aifs/user/data/user

Use the exact command required by the corresponding filesystem documentation.

6. I need more storage space or a project directory. How should I apply?

First confirm whether your experiment group has already assigned you a default directory. If you really need quota expansion or a new project directory, explain the purpose, expected capacity, number of files, and usage period to your supervisor, project leader, or experiment group administrator.

Most experiment group directory expansions require approval from the responsible person before administrators can process them. Creating a new directory alone usually does not automatically increase the quota.

7. Can public directories such as /scratchfs2 or /publicfs be temporarily expanded?

Public directories usually serve multiple experiment groups or users and may not support temporary expansion for individual users. For example, if the file count limit on /scratchfs2 is reached, it is recommended to clean data, package large numbers of small files, or apply to use a more suitable dedicated directory of the experiment group.

If expansion is truly needed, describe the business need, expected usage, number of files, and usage period, and ask the experiment group leader to confirm it.

8. I accidentally deleted files. Can they be recovered?

Recovery depends on whether the directory has a backup or recycle mechanism. Some AFS/home directories may be recoverable from backup or recovery directories, and some directories may have a recycle bin.

However, Helpdesk records show multiple cases where files deleted from directories such as /publicfs, /scratchfs, and /besfs5/groups could not be recovered. After accidental deletion, stop writing to the directory immediately and submit the path, deletion time, deletion command, and range of files to be recovered. Whether recovery is possible depends on the actual backup policy of that directory.

9. Which directories are backed up? How do I restore files from backup?

Refer to the backup service section of the user manual. Directories with backups can be restored according to the backup service process. Data not listed in the backup list is usually not recoverable.

When submitting a restore request, provide your account, full path, time of deletion or abnormal event, desired recovery point, and the file or directory scope.

10. Filesystem access is slow, ls hangs, or job I/O is stuck. What should I do?

Possible causes include the filesystem approaching capacity limits, high metadata server pressure, hardware failure, or scripts accessing an abnormal path. For example, if .bashrc references a faulty storage path, login may hang.

First confirm whether the slowness is concentrated on a specific path, such as /hpcfs, /besfs5, or /publicfs, and avoid repeatedly accessing it heavily or resubmitting jobs. When contacting administrators, provide your account, login node, stuck path, command, time of occurrence, and whether jobs are affected.

14.5 Software Environment and Other Issues¶

How do I use Module to load software environments?

Add the modulefile path first, then view and load software. For example:
```
module use /cvmfs/slurm.ihep.ac.cn/alma9/modulefiles
module avail
module load software/version
```
If a job reports missing Python packages, CUDA, PyTorch, or other software, confirm that the interactive environment and the job script load the same module environment.
CVMFS mount is abnormal or reports Transport endpoint is not connected. What should I do?

If accessing /cvmfs/... reports a mount problem, missing software environment, I/O error, or Transport endpoint is not connected, the CVMFS mount on the node may be abnormal.

You can first switch to another login node or retry later. If the problem persists, submit a ticket with the node name, CVMFS path, command output, and required software environment.
A job reports missing Python packages or software libraries. What should I do?

First confirm whether the required software has already been provided by the experiment group or the computing platform, and load the corresponding module according to the manual. If experiment-related public software libraries need to be installed, it is not recommended to install them in your personal home directory. Confirm with the experiment leader and place them in an experiment group public software directory or CVMFS environment.
A GPU job reports a CUDA, PyTorch, or driver version mismatch. What should I do?

Confirm the driver and CUDA versions supported by the current GPU nodes, and then choose matching PyTorch, CUDA module, or container environment. If newer software requires a higher driver or CUDA version, it may need to wait for a system-wide upgrade and cannot be solved by a single user inside a job.
The software environment works on an interactive node but not after submitting a job. What should I do?

Check whether the job script fully loads the modules, environment variables, and software paths used in the interactive environment. Environment already loaded in an interactive shell will not automatically appear in a batch job. It is recommended to write module use, module load, source, and other environment initialization commands into the job script.
I want to apply for new computing resources or adjust resource configuration. What should I do?

First confirm the requirement with your supervisor, project leader, or experiment group administrator, including resource type, usage period, expected job scale, CPU/GPU/memory needs, and data storage needs. After confirmation, the responsible person should apply or communicate with the computing center according to the experiment group process.

🤖 AI 助手

Frequently Asked Questions¶

14.1 How to Get Help¶

14.2 Login and Account Issues¶

14.3 Job Issues¶

14.4 File Storage Issues¶

14.5 Software Environment and Other Issues¶