===== Usefull admin tips for the EEMCS-HPC CLuster =====

==== Users/Accounts ====

A **User** is a single person, an **Account** is a group of users, or even a group of groups.

=== Activating Users ===
  * (hpc-storage3) Add the user to the NIS database, the user will be added to the default cluster account **ctit**

In order to use additional slurm scripts, load the following module :
<code bash>
module load slurm/utils
</code>

  * (hpc-head1/2) move the user to the correct cluster account (read research or student group)

for this you can use the following script :
<code bash>
sacctmgr-move-user <groupname> <username>
</code>
or 
<code bash>
sacctmgr-move-user ctit <groupname> <username>
</code>

  * (ad.utwente.nl) add them to the mailing list : EEMCS-Hpc-Cluster-Users (EEMCS-IDS-Cluster-Users)
  * send the requesting user en explanation how to get started.

=== Adjust the Quality of Service or QOS. ===
Every account already wil have a default QOS defined.

In some cases a different QOS is required, to Modifying Quality of service use the following command :
<code bash>
# default QOS
sudo sacctmgr modify user where name=<username> set DefaultQOS=<defaultQOSname>
# additional QOS (temporary)
sudo sacctmgr modify user where name=<username> set QOS+=<additionalQOSname>
# remove additional QOS (temporary)
sudo sacctmgr modify user where name=<username> set QOS-=<additionalQOSname>
</code>

=== Creating new Accounts ===
Creating new accounts (read reasearch groups) : 
<code bash>
sudo sacctmgr create account name=<groupname> parent=<faculty> fairshare=1
</code>

The fair share factor is the amount of investment in K€ !

=== dump userdatabase ===

In order to recover or get an overview of all the activated account an **dump configuration file** can be generated using following command : 
<code bash>
sudo sacctmgr-dump
</code>
This will create a ctit_<date>.cfg file containing all the accounts/users and their priority factor structure.

==== Cleanup /local ====

If the **/local** folder gets clogged, check what jobs are running on that specific node :
<code bash>
squeue --nodelist=<node-name>
</code>

Remove all unrelated folder within the /local folder.

==== GPU monitoring ====

GPU's can be monitored using the tools supplied in the module nvidia/nvtop :
<code bash>
module load nvidia/nvtop
</code>

To monitor the GPU's on a specific node, use one of the following commands :
<code bash>
nvidia-smi-node <nodename>
nvtop-node <nodename>
</code>

To show the jobs assigned gpu, use the following command :
<code bash>
scontrol show job <jobid> -d
</code>
//Note : look for GRES=gpu(IDX:...) //

To show the history of a job, use the following command :
<code bash>
sacct --format=User,Account,Partition,State%25,ExitCode,AllocTRES%60 -j <jobid>
</code>
//Node : more parameters can be monitored, see : sacct --helpformat //

==== Utilization ====
Utilization Cpu reports can be generated using one of the following commands :
<code bash>
 sreport cluster AccountUtilization cluster=ctit -P start=1/1/21 end=12/31/21 > Utilisation_2021
 sreport cluster AccountUtilizationByUser cluster=ctit account=<account_name> start=2020-03-25 end=2020-03-25
 sreport cluster AccountUtilizationByUser cluster=ctit user=<user_name> start=2020-03-25 end=2020-03-25
</code>

Utilization for Gpus can be generated using one of the following commands :

<code bash>
 sreport cluster AccountUtilization cluster=ctit -P --tres="gres/gpu" start=1/1/21 end=12/31/21 > Utilisation_2021
 sreport cluster AccountUtilizationByUser cluster=ctit --tres="gres/gpu" account=<account_name> start=2020-03-25 end=2020-03-25
 sreport cluster AccountUtilizationByUser cluster=ctit --tres="gres/gpu" user=<user_name> start=2020-03-25 end=2020-03-25
</code>

==== Maintenance ====

=== create maintenance reservation ===
To create a maintenance reservation use the following command : 
<code bash>
scontrol create reservation starttime=2022-03-23T8:00:00 duration=480 user=root flags=maint,ignore_jobs nodes=ALL
</code>

=== terminate running jobs ===
<code bash>
squeue -ho %A -t R | xargs -n 1 scancel 
</code>

=== stopping slurm daemons ===
<code bash>
scontrol shutdown
</code>

=== undraining a node ===
<code bash>
sudo scontrol update NodeName=<node_name> State=DOWN Reason="undraining"
sudo scontrol update NodeName=<node_name> State=RESUME
</code>

=== system serial number ===
<code bash>
sudo dmidecode -s system-serial-number
</code>

==== Powersaving ====

To keep the power usage at a lower level, compute nodes not being used will powerdown after a certain amount of time.
These definitions are located in the //slurm.conf// file. See the //SuspendTime// for the actual time.
To disable this functionality change to following line:
<code bash>
#SuspendExcParts=debug
</code>
to:
<code bash>
SuspendExcParts=debug
</code>

To powerup a node temporary execute the following command:
<code bash>
sudo scontrol update node=<nodelist> state=POWER_UP
</code>
or powerdown with :
<code bash>
sudo scontrol update node=<nodelist> state=POWER_DOWN
</code>