ClusterSetup

Two network adapters are used for each VM.

Each node uses a fixed IP address.

Example SSH access:

ssh -p 2221 hpcuser@localhost

I’m using Manjaro (Arch) as my base system.

Download Rocky Linux Minimal ISO:

https://rockylinux.org/download

Choose:

• x86_64
• Minimal ISO

The minimal installation keeps the environment lightweight and avoids unnecessary packages.

Open VirtualBox and create a new virtual machine.

Choose:

Minimal Install

Launch NetworkManager TUI:

sudo nmtui

Navigate:

Edit a connection

Select:

enp0s8

Set:

DO NOT modify NAT interface.

ip addr

Expected:

From host machine:

ssh -p 2221 hpcadmin@localhost

If the connection works, port forwarding is configured correctly.

Check timezone:

timedatectl

Set timezone if needed:

sudo timedatectl set-timezone Asia/Kolkata

Edit:

sudo vim /etc/hosts

Add:

10.10.10.1   master master.local
10.10.10.2   login login.local
10.10.10.11  compute01 compute01.local
10.10.10.12  compute02 compute02.local
10.10.10.13  compute03 compute03.local

This allows hostname-based communication between nodes.

For initial setup and testing:

sudo systemctl disable --now firewalld

NOTE:

Firewall rules can be configured properly later after cluster setup is complete.

Check status:

getenforce

Temporarily disable enforcing:

sudo setenforce 0

Persist configuration:

sudo vim /etc/selinux/config

Modify:

SELINUX=permissive

This helps avoid permission-related issues during initial cluster setup.

hostnamectl

Expected:

Static hostname: master.local

If hostname is not updated(showing previous), reboot the machine.

At this point, the master node setup is complete and ready for cloning or further cluster configuration.

Power off VM:

sudo poweroff

Create snapshot in VirtualBox:

clean-base-master

Create FULL CLONES.

IMPORTANT: Enable:

Reinitialize the MAC address of all network cards

IMPORTANT: Each cloned Linux machine must have unique machine-id.

cat /etc/machine-id

Ensure every node has unique machine-id.

Launch:

sudo nmtui

Edit internal adapter:

enp0s8

ip route

Expected:

default via 10.0.2.2 dev enp0s3
10.10.10.0/24 dev enp0s8

From host:

IMPORTANT:

NFS shared storage requires same UID/GID across all nodes.

Verify user identity on all nodes:

id hpcadmin

Expected:

uid=1000(hpcadmin) gid=1000(hpcadmin)

All nodes must match.

Purpose:

• MPI communication
• Slurm execution
• cluster automation
• remote management

SSH trust is generated ONLY from login node.

This mimics real HPC architecture:

User -> Login Node -> Compute Nodes

SSH into login node from host:

ssh -p 2222 hpcadmin@localhost

Generate SSH key:

ssh-keygen

Press Enter for all prompts.

Generated files:

~/.ssh/id_rsa
~/.ssh/id_rsa.pub

Passwordless SSH becomes essential for:

Purpose:

Provide shared home directories across all cluster nodes.

This is a core HPC architecture requirement.

Without shared storage:

• source code differs per node
• binaries differ
• scripts unavailable across nodes
• MPI execution becomes difficult

Real HPC systems commonly use:

• NFS
• Lustre
• BeeGFS
• GPFS

This lab begins with NFS.

Install on ALL nodes:

sudo dnf install -y nfs-utils

SSH into master node.

sudo systemctl enable --now nfs-server

Verify:

systemctl status nfs-server

Expected:

active (running)

Edit exports file:

sudo vim /etc/exports

Add:

/home 10.10.10.0/24(rw,sync,no_root_squash)

sudo exportfs -v

Expected export:

/home 10.10.10.0/24(...)

Perform on:

• login
• compute01
• compute02
• compute03

sudo mv /home /home.local

Purpose:

Preserve original local home directory.

df -h

Expected:

master:/home

mounted on:

/home

From login node:

touch ~/shared_test

From compute01:

ls ~

Expected:

shared_test

This confirms successful shared filesystem operation.

Without this configuration:

• NFS mount disappears after reboot

On ALL client nodes:

sudo vim /etc/fstab

Add:

master:/home   /home   nfs   defaults   0 0

IMPORTANT:

Always validate fstab before rebooting.

Run:

sudo mount -a

If no errors appear:

• fstab configuration is correct

Purpose:

Validate persistent cluster infrastructure after reboot.

This is important real-world HPC administration practice.

On login node:

df -h

Expected:

master:/home

mounted automatically.

mount | grep home

Expected:

master:/home on /home type nfs

ls ~

Expected:

• previously created shared files still visible

Originally the cluster was created using:

Rocky Linux 10

Problem encountered:

No match for argument: slurm

Reason:

• Rocky Linux 10 HPC ecosystem still immature
• Slurm packages unavailable in standard repos
• OpenHPC EL10 support incomplete

Resolution:

• cluster rebuilt using Rocky Linux 9 Minimal

Important HPC lesson:

Production HPC clusters usually avoid bleeding-edge enterprise releases because:

• scheduler ecosystems lag behind
• HPC repos may not exist
• compatibility becomes unstable

Install on ALL nodes:

sudo dnf install -y epel-release

Required on Rocky Linux 9 and 10.

Install on ALL nodes:

sudo crb enable

Install on ALL nodes:

sudo dnf install -y \
munge munge-libs \
slurm slurm-slurmd slurm-slurmctld

Problem:

id: 'slurm': no such user

Reason:

Some Slurm packages on Rocky Linux do not automatically create the slurm user.

Resolution:

Create manually on ALL nodes.

sudo useradd -r -g slurm -s /sbin/nologin slurm

Explanation:

id slurm

Expected:

uid=...
gid=...

Cluster daemons should not run permanently as root.

Therefore:

• Munge runs as munge user
• Slurm runs as slurm user

This is standard enterprise daemon security practice.

Purpose:

Create secure trust relationship between all cluster nodes.

IMPORTANT:

All nodes MUST use the same:

/etc/munge/munge.key

This key acts as the cluster-wide shared secret.

ONLY on master node:

sudo /usr/sbin/create-munge-key

ls -l /etc/munge/munge.key

Expected:

-rw-------

owned by:

munge munge

Problem:

Munge key accidentally generated independently on all nodes.

Result:

Nodes could not authenticate each other.

Important HPC lesson:

Munge works using a:

shared trust model

All nodes MUST share identical munge.key.

Problem encountered:

Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password)

Reason:

Using:

sudo scp ...

attempted root SSH login.

Rocky Linux disables direct root SSH login by default.

Resolution:

Use normal user SCP.

On master:

sudo rm -f /tmp/munge.key

Perform on:

• login
• compute01
• compute02
• compute03

On ALL nodes:

sudo systemctl enable --now munge

Problem:

munge: Error: Failed to access "/var/run/munge/munge.socket.2"

Reason:

Munge service never started on master.

Resolution:

sudo systemctl enable --now munge

Important lesson:

Difference between:

systemctl status munge

Expected:

active (running)

From master:

munge -n | ssh compute01 unmunge

munge -n | ssh login unmunge

Expected:

STATUS: Success

Munge introduced:

• cluster-wide authentication
• distributed trust model
• shared secret authentication
• daemon security
• inter-node trust validation

Purpose:

Enable centralized resource scheduling.

IMPORTANT:

/etc/slurm/slurm.conf

must be IDENTICAL on ALL nodes.

Problem:

sinfo
-> localhost

Reason:

Default example slurm.conf remained active.

Resolution:

Replace configuration with custom cluster topology.

Important HPC lesson:

Incorrect topology configuration is one of the most common Slurm admin problems.

On master:

sudo vim /etc/slurm/slurm.conf

Replace ENTIRE file with:

ClusterName=hpccluster
SlurmctldHost=master

MpiDefault=none
ProctrackType=proctrack/cgroup
ReturnToService=2

SlurmctldPidFile=/run/slurmctld.pid
SlurmdPidFile=/run/slurmd.pid

SlurmdSpoolDir=/var/spool/slurmd
StateSaveLocation=/var/spool/slurmctld

SlurmUser=slurm

SwitchType=switch/none
TaskPlugin=task/affinity

SchedulerType=sched/backfill

SelectType=select/cons_tres
SelectTypeParameters=CR_Core

AuthType=auth/munge

NodeName=compute01 CPUs=2 State=UNKNOWN
NodeName=compute02 CPUs=2 State=UNKNOWN
NodeName=compute03 CPUs=2 State=UNKNOWN

PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP

Problem:

sudo: a terminal is required to read the password

Resolution:

Use:

sudo -S

Important Linux administration concept:

sudo over non-interactive SSH requires stdin password handling.

ls -ld /var/spool/slurmctld

Expected:

slurm slurm

On ALL nodes:

sudo mv /tmp/slurm.conf /etc/slurm/slurm.conf

ONLY on master:

sudo systemctl enable --now slurmctld

systemctl status slurmctld

Expected:

active (running)

On:

• compute01
• compute02
• compute03

sudo systemctl enable --now slurmd

systemctl status slurmd

Expected:

active (running)

On master:

sudo scontrol reconfigure

Problem:

Nodes entered:

DRAIN

after CPU topology changes.

Reason:

Slurm detected mismatch between:

• configured CPUs
• detected hardware topology

Resolution:

Clear stale slurmd state.

On compute nodes:

sudo systemctl stop slurmd

On master:

sudo scontrol update NodeName=compute01 State=RESUME

sudo scontrol update NodeName=compute02 State=RESUME

sudo scontrol update NodeName=compute03 State=RESUME

sinfo

Expected:

PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
debug* up infinite 3 idle compute[01-03]

scontrol show nodes

This displays:

• CPU information
• memory
• node states
• scheduler topology
• resource allocation

srun -N 3 hostname

Expected:

compute01.local
compute02.local
compute03.local

srun -N 3 -n 6 hostname

This verifies:

• task distribution
• multi-node scheduling
• CPU allocation

srun -N 1 -n 2 hostname

Expected:

compute01.local
compute01.local

Earlier:

srun -N 1 -n 2 hostname

failed because nodes initially had:

CPUs=1

After increasing VM CPUs and updating slurm.conf:

CPUs=2

the scheduler successfully allocated 2 tasks on a single node.

Important HPC lesson:

Slurm strictly enforces:

• CPU limits
• task allocation
• topology constraints
• resource consistency

Initially:

• each VM had a 30 GB local disk
• /home was exported from master over NFS

Concern encountered:

Shared /home only had ~30 GB available.
Spack and HPC software stacks can become very large.

Important HPC storage concept:

Real HPC clusters usually separate:

Therefore a dedicated:

/apps

filesystem was created.

This is significantly closer to real HPC architecture.

Each VM still has its own independent local storage.

Example:

When:

master:/home

was mounted via NFS:

• compute node local /home became hidden
• but local disks still exist

Only the exported filesystem becomes shared.

Therefore:

/apps

was created as a dedicated shared software layer.

A second virtual disk was attached ONLY to:

master

Purpose:

• centralized software storage
• shared application filesystem
• Spack software stack

Disk size selected:

60 GB

Disk type:

VDI (Dynamically Allocated)

After booting master:

lsblk

Expected:

sda    30G
sdb    60G

Important:

Only:

sdb

was modified.

sudo fdisk /dev/sdb

Inside fdisk:

Create GPT label:

g

Create partition:

n

Accept defaults.

Write changes:

w

lsblk

Expected:

sdb
└── sdb1

sudo mkfs.xfs /dev/sdb1

Why XFS?

• default enterprise filesystem
• highly scalable
• excellent for large filesystems
• commonly used in HPC

sudo blkid /dev/sdb1

Example:

UUID="xxxxxxxx-xxxx-xxxx"

Edit:

sudo vim /etc/fstab

Add:

UUID=YOUR_UUID_HERE   /apps   xfs   defaults   0 0

Important Linux administration concept:

UUIDs are preferred over:

/dev/sdb1

because device names can change after reboot.

df -h

Expected:

/apps

mounted with approximately:

60 GB

mkdir -p /apps/{spack,modulefiles,software,sources}

Resulting structure:

/apps
├── spack
├── modulefiles
├── software
└── sources

Purpose of directories:

This is similar to production HPC software layouts.

Edit exports file:

sudo vim /etc/exports

Add:

/apps 10.10.10.0/24(rw,sync,no_root_squash)

showmount -e localhost

Expected:

/apps
/home

On:

• login
• compute01
• compute02
• compute03

Create:

sudo mkdir -p /apps

Edit:

sudo vim /etc/fstab

Add:

master:/apps   /apps   nfs   defaults,_netdev   0 0

df -h

Expected:

master:/apps

mounted on all nodes.

On master:

touch /apps/testfile

On compute node:

ls /apps

Expected:

testfile

This verifies:

• shared software filesystem
• NFS functionality
• cluster-wide software visibility

Current cluster storage architecture:

This resembles real HPC environments significantly more closely.

Instead of:

dnf install openmpi

a centralized HPC software stack was created.

Benefits:

• software installed once
• accessible cluster-wide
• compiler hierarchy support
• reproducible builds
• module-based software environments
• centralized software management

This approach is significantly closer to real HPC centers.

In production HPC environments:

• operating system packages are separated
• scientific software stack is separated
• compiler stacks are separated
• MPI stacks are separated

This separation improves:

• portability
• reproducibility
• scalability
• maintenance

The cluster uses:

/apps

as shared application storage.

Because:

/apps

is mounted cluster-wide through NFS:

• software installed once becomes visible everywhere
• compute nodes automatically access installed software
• users share common software stack
• recompilation is unnecessary on compute nodes

This is one of the MOST important HPC architecture concepts.

Installed manually on ALL nodes:

sudo dnf groupinstall -y "Development Tools"

Purpose:

Provides:

• gcc
• g++
• gfortran
• make
• linker tools
• build utilities
• development environment

Important note:

System compilers are still required even when using Spack.

Reason:

Spack builds software from source code and therefore still requires:

• assembler
• linker
• basic compiler toolchain

Run on:

master

Move into shared applications directory:

cd /apps

Verify:

pwd

Expected:

/apps

Clone Spack:

git clone https://github.com/spack/spack.git

Expected:

/apps/spack

This creates centralized package manager installation.

cd spack

Verify:

pwd

Expected:

/apps/spack

Initialize Spack:

source share/spack/setup-env.sh

Purpose:

Adds:

• spack command
• shell integration
• environment variables
• package management environment

to current shell session.

Without initialization:

• spack command will not work
• spack load will fail
• software environments will not activate

source

does NOT execute script normally.

Instead:

it modifies current shell environment

This is extremely important in:

• Spack
• modules
• MPI environments
• compiler environments

To avoid sourcing manually every session:

Edit:

vim ~/.bashrc

Add:

source /apps/spack/share/spack/setup-env.sh

Reload shell:

source ~/.bashrc

Verify:

spack --version

Expected:

• Spack version information

Clone repository:

git clone https://github.com/spack/spack-packages.git

Purpose:

Adds additional package definitions.

spack repo set --destination "$(pwd)/spack-packages" builtin

Purpose:

Configure additional repository support.

This allows:

• package metadata
• package recipes
• software definitions

to become visible to Spack.

spack info gcc

Expected:

Detailed GCC package information.

This verifies:

• Spack environment working
• package repository functional
• package metadata accessible

Spack operates as:

source-based HPC package manager

Unlike:

Production HPC systems commonly separate:

• OS packages
• scientific software stacks

exactly like this setup.

When installing software using Spack:

software is compiled from source

Advantages:

• architecture optimization
• compiler flexibility
• dependency control
• reproducibility
• variant support

Disadvantages:

• longer installation time

Before installing software:

spack compiler find

Purpose:

Detect system compilers available for building software.

Verify detected compilers:

spack compilers

Expected:

• compiler list

Install GCC version:

spack install gcc@13.4.0

Purpose:

Create dedicated HPC compiler stack.

This may take considerable time because:

• GCC itself is compiled from source
• dependencies are built
• optimization occurs

Production HPC systems frequently maintain multiple compiler versions simultaneously.

Example:

gcc@11
gcc@12
gcc@13
intel
nvhpc
clang

Reasons:

• application compatibility
• compiler ABI differences
• reproducibility
• benchmark consistency

List installed packages:

spack find

Expected example:

==> 1 installed package
gcc@13.4.0

spack load gcc@13.4.0

Purpose:

Activate compiler environment.

Verify compiler path:

which gcc

Expected:

• Spack GCC path

Verify version:

gcc --version

Expected:

• GCC 13.4.0

When software is loaded:

• PATH changes
• LD_LIBRARY_PATH changes
• MANPATH changes

This allows applications to locate:

• executables
• shared libraries
• runtime dependencies

automatically.

After installing GCC:

spack compiler find

Purpose:

Allow Spack to use newly installed GCC for future package builds.

Verify:

spack compilers

Expected:

• new GCC compiler entry

Install MPI runtime:

spack install openmpi

Purpose:

Create shared MPI runtime stack.

This installation includes:

• MPI runtime
• MPI compiler wrappers
• MPI libraries
• distributed runtime environment

MPI stacks are tightly coupled with compiler toolchains.

Example:

openmpi built using gcc@13

This is important because:

• ABI compatibility matters
• runtime compatibility matters
• mixed compiler environments may fail

Show dependency tree:

spack find -d

Purpose:

Displays:

• dependency hierarchy
• compiler hierarchy
• runtime dependencies

Example:

openmpi
   ^gcc@13.4.0

Every Spack installation contains:

• package version
• compiler information
• dependency tree
• unique installation hash

Example:

openmpi@5.0.7%gcc@13.4.0

Meaning:

Sometimes multiple builds of same version exist because of:

• different compilers
• different variants
• different dependencies

Spack assigns:

unique hashes

View hashes:

spack find -l

Expected:

abc1234 gcc@13.4.0
xyz5678 openmpi@5.0.7

Load using hash:

spack load /xyz5678

This is extremely common in production HPC systems.

spack load openmpi

Verify:

which mpicc

Expected:

• Spack OpenMPI wrapper path

Verify MPI runtime:

mpirun --version

Expected:

• OpenMPI version information

mpicc

is NOT a compiler itself.

It is a wrapper that automatically injects:

• MPI headers
• MPI libraries
• runtime linker flags

around GCC.

This is a critical HPC build concept.

Display loaded packages:

spack find --loaded

Unload OpenMPI:

spack unload openmpi

Unload all software:

spack unload --all

Verify:

spack find --loaded

Expected:

• no loaded packages

Create source file:

vim mpi_sum.c

Purpose:

• validate MPI runtime
• validate multinode execution
• validate scheduler-aware execution
• compare serial vs parallel runtime
• demonstrate reduction operations
• calculate speedup
• calculate efficiency

Paste:

#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

#define N 1000000000

double serial_sum()
{
    double sum = 0.0;

    for(long long int i = 0; i < N; i++)
    {
        sum += i;
    }

    return sum;
}

int main(int argc, char *argv[])
{
    int rank, size;

    long long int start_index;
    long long int end_index;
    long long int chunk;

    double local_sum = 0.0;
    double global_sum = 0.0;

    double serial_start;
    double serial_end;

    double parallel_start;
    double parallel_end;

    MPI_Init(&argc, &argv);

    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    if(rank == 0)
    {
        serial_start = MPI_Wtime();

        double serial_result = serial_sum();

        serial_end = MPI_Wtime();

        printf("\n");
        printf("========== SERIAL COMPUTATION ==========\n");
        printf("Sum = %.0f\n", serial_result);
        printf("Time taken = %f seconds\n",
               serial_end - serial_start);
    }

    MPI_Barrier(MPI_COMM_WORLD);

    chunk = N / size;

    start_index = rank * chunk;

    if(rank == size - 1)
        end_index = N;
    else
        end_index = start_index + chunk;

    parallel_start = MPI_Wtime();

    for(long long int i = start_index;
        i < end_index;
        i++)
    {
        local_sum += i;
    }

    MPI_Reduce(&local_sum,
               &global_sum,
               1,
               MPI_DOUBLE,
               MPI_SUM,
               0,
               MPI_COMM_WORLD);

    parallel_end = MPI_Wtime();

    if(rank == 0)
    {
        double parallel_time =
            parallel_end - parallel_start;

        double serial_time =
            serial_end - serial_start;

        printf("\n");
        printf("========== PARALLEL COMPUTATION ==========\n");
        printf("Processes used = %d\n", size);
        printf("Sum = %.0f\n", global_sum);
        printf("Time taken = %f seconds\n",
               parallel_time);

        printf("\n");
        printf("========== PERFORMANCE ==========\n");
        printf("Speedup = %f\n",
               serial_time / parallel_time);

        printf("Efficiency = %f\n",
               (serial_time / parallel_time) / size);
    }

    MPI_Finalize();

    return 0;
}

This program uses:

MPI_Reduce()

Purpose:

Combine partial sums from all MPI ranks into:

single global result

Flow:

local sums
    ↓
MPI_Reduce
    ↓
global sum

This is one of the MOST important MPI collective communication operations.

Serial execution:

single process computes entire workload

Parallel execution:

multiple MPI ranks divide workload

Example:

Advantages:

• reduced execution time
• distributed workload
• scalable computation

Initialize Spack:

source /apps/spack/share/spack/setup-env.sh

Load compiler:

spack load gcc@13.4.0

Load MPI runtime:

spack load openmpi

Verify:

which mpicc

Expected:

• MPI compiler wrapper path

Compile:

mpicc mpi_sum.c -o mpi_sum

Verify executable:

ls

Expected:

mpi_sum

MPI programs are compiled using:

mpicc

instead of:

gcc

because MPI wrapper automatically injects:

• MPI headers
• MPI libraries
• linker flags

Run interactively:

mpirun -np 4 ./mpi_sum

Expected:

========== SERIAL COMPUTATION ==========
...

========== PARALLEL COMPUTATION ==========
...

========== PERFORMANCE ==========
...

This validates:

• MPI runtime
• MPI communication
• reduction operations
• distributed execution

Because:

• source code
• executable
• software stack

exist on shared filesystem:

/apps
or
/home

compute nodes automatically access:

• compiled binaries
• runtime libraries
• MPI stack

without recompilation.

This is one of the major HPC architecture advantages.

Create batch script:

vim submit.sh

Paste:

#!/bin/bash
#SBATCH -N 3
#SBATCH --ntasks-per-node=1
#SBATCH --job-name=test
#SBATCH --output=%J.out
#SBATCH --error=%J.err
#SBATCH --time=01:00:00
##SBATCH --nodelist=compute03,compute01

source /apps/spack/share/spack/setup-env.sh

spack load openmpi

mpirun -np $SLURM_NTASKS ./$1

Make executable:

chmod +x submit.sh

Inside batch script:

source /apps/spack/share/spack/setup-env.sh

initializes Spack environment.

spack load openmpi

loads MPI runtime stack.

Without these:

• mpirun may fail
• MPI libraries may not load
• runtime linker errors may occur

$SLURM_NTASKS

contains:

total MPI process count allocated by Slurm

Example:

Therefore:

mpirun -np $SLURM_NTASKS

automatically launches correct number of MPI ranks.

Submit job:

sbatch submit.sh mpi_sum

Expected:

Submitted batch job 101

This validates:

• Slurm scheduler
• distributed allocation
• multinode MPI execution
• scheduler-aware runtime launch

Check user jobs:

squeue --me

Expected:

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)

Show detailed job info:

scontrol show job <jobid>

Example:

scontrol show job 101

Displays:

• allocated nodes
• CPUs
• runtime
• scheduling information
• execution directory
• output locations

Check cluster nodes:

sinfo

Expected:

debug* up infinite 3 idle compute[01-03]

After completion:

ls

Expected:

101.out
101.err

View output:

cat 101.out

Expected:

• serial runtime
• parallel runtime
• speedup
• efficiency

The program calculates:

Ideal values:

Real systems experience:

• communication overhead
• synchronization cost
• MPI latency
• reduction overhead

Cancel job:

scancel <jobid>

Example:

scancel 101

Cancel all user jobs:

scancel -u $USER

Launch interactive shell:

srun --pty bash

Distributed hostname test:

srun -N 3 -n 3 hostname

Expected:

• hostnames from allocated nodes

This validates:

• scheduler allocation
• distributed execution
• runtime communication

Typical HPC workflow:

Load software
    ↓
Compile application
    ↓
Prepare Slurm script
    ↓
Submit job
    ↓
Monitor execution
    ↓
Analyze outputs

Production HPC systems usually separate:

• operating system packages
• scientific software stack
• compiler stack
• MPI runtime stack
• scheduler integration

exactly like this setup.

This architecture provides:

• reproducibility
• scalability
• centralized administration
• software consistency
• runtime portability

Cluster roles:

Important concept:

Only master runs LDAP SERVER (slapd).

Login and compute nodes run LDAP CLIENTS (nslcd).

Before starting, understand the layers.

Authentication flow:

SSH Login
   ↓
PAM
   ↓
nslcd
   ↓
LDAP server

Run ONLY on:

master

Install LDAP packages:

sudo dnf install -y \
openldap \
openldap-servers \
openldap-clients

Package explanation:

Run ONLY on:

master

Enable and start slapd:

sudo systemctl enable --now slapd

Verify:

systemctl status slapd

Expected:

active (running)

Run ONLY on:

master

Generate secure LDAP admin password hash:

slappasswd

Enter desired password.

Example:

cluster123

Example output:

{SSHA}xxxxxxxxxxxxxxxx

Important:

Copy this hash carefully.

It will be used in LDAP database configuration.

Run ONLY on:

master

Create LDAP database configuration file:

vim ~/db.ldif

Paste:

dn: olcDatabase={2}mdb,cn=config
changetype: modify
replace: olcSuffix
olcSuffix: dc=hpc,dc=local

dn: olcDatabase={2}mdb,cn=config
changetype: modify
replace: olcRootDN
olcRootDN: cn=Manager,dc=hpc,dc=local

dn: olcDatabase={2}mdb,cn=config
changetype: modify
replace: olcRootPW
olcRootPW: PASTE_HASH_HERE

Replace:

PASTE_HASH_HERE

with the hash generated using:

slappasswd

LDAP uses a hierarchical tree called:

DIT -- Directory Information Tree

Example structure:

dc=hpc,dc=local
 ├── ou=People
 └── ou=Groups

Meaning of terms:

Run ONLY on:

master

Apply configuration:

sudo ldapmodify -Y EXTERNAL -H ldapi:/// \
-f ~/db.ldif

Expected:

modifying entry

Schemas define:

• user attributes
• group attributes
• UNIX account structure

Run ONLY on:

master

Import cosine schema:

sudo ldapadd -Y EXTERNAL -H ldapi:/// \
-f /etc/openldap/schema/cosine.ldif

Import nis schema:

sudo ldapadd -Y EXTERNAL -H ldapi:/// \
-f /etc/openldap/schema/nis.ldif

Import inetorgperson schema:

sudo ldapadd -Y EXTERNAL -H ldapi:/// \
-f /etc/openldap/schema/inetorgperson.ldif

Schema explanation:

Run ONLY on:

master

Create base LDAP tree:

vim ~/base.ldif

Paste:

dn: dc=hpc,dc=local
objectClass: top
objectClass: dcObject
objectClass: organization
o: HPC Cluster
dc: hpc

dn: ou=People,dc=hpc,dc=local
objectClass: organizationalUnit
ou: People

dn: ou=Groups,dc=hpc,dc=local
objectClass: organizationalUnit
ou: Groups

Apply tree:

ldapadd -x \
-D "cn=Manager,dc=hpc,dc=local" \
-W \
-f ~/base.ldif

Enter LDAP admin password.

Run ONLY on:

master

Verify tree:

ldapsearch -x -b "dc=hpc,dc=local"

Expected:

• dc=hpc,dc=local
• ou=People
• ou=Groups

Run ONLY on:

master

Create group file:

vim ~/group.ldif

Paste:

dn: cn=hpcusers,ou=Groups,dc=hpc,dc=local
objectClass: posixGroup
cn: hpcusers
gidNumber: 10000

Add group:

ldapadd -x \
-D "cn=Manager,dc=hpc,dc=local" \
-W \
-f ~/group.ldif

Run ONLY on:

master

Generate password hash:

slappasswd

Example password:

test123

Copy generated hash carefully.

Run ONLY on:

master

Create user file:

vim ~/user.ldif

Paste:

dn: uid=testUser,ou=People,dc=hpc,dc=local
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: shadowAccount
cn: Test User
sn: User
uid: testUser
uidNumber: 10001
gidNumber: 10000
homeDirectory: /home/testUser
loginShell: /bin/bash
userPassword: PASTE_HASH_HERE

Replace:

PASTE_HASH_HERE

with actual hash from:

slappasswd

Add user:

ldapadd -x \
-D "cn=Manager,dc=hpc,dc=local" \
-W \
-f ~/user.ldif

Run ONLY on:

master

Verify user:

ldapsearch -x \
-b "dc=hpc,dc=local" uid=testUser

Expected:

• LDAP user entry

Now begin client-side integration.

Run ONLY on:

login

Install LDAP client packages:

sudo dnf install -y \
openldap-clients \
nss-pam-ldapd

Run ONLY on:

login

Create configuration:

sudo vim /etc/nslcd.conf

Paste:

uid nslcd
gid ldap

uri ldap://master
base dc=hpc,dc=local

Explanation:

Secure configuration:

sudo chmod 600 /etc/nslcd.conf

Run ONLY on:

login

Enable service:

sudo systemctl enable --now nslcd

Verify:

systemctl status nslcd

Expected:

active (running)

Run ONLY on:

login

Test LDAP communication:

ldapsearch -x \
-H ldap://master \
-b "dc=hpc,dc=local" uid=testUser

Expected:

• LDAP user entry

This verifies:

• login node can communicate with master LDAP server

Run ONLY on:

login

Edit:

sudo vim /etc/nsswitch.conf

Modify lines:

passwd: files ldap
group: files ldap
shadow: files ldap

Important:

This tells Linux:
check local files first, then LDAP.

Run ONLY on:

login

Restart service:

sudo systemctl restart nslcd

Run ONLY on:

login

Test centralized user lookup:

getent passwd testUser

Expected:

testUser:x:10001:10000:Test User:/home/testUser:/bin/bash

Important concept:

This proves Linux itself recognizes LDAP users.

This is one of the most important LDAP verification steps.

Now we must configure: master with EXACT same setup (master should be able to recognise the user).

Run ONLY on:

login

Install package:

sudo dnf install -y oddjob-mkhomedir

Enable service:

sudo systemctl enable --now oddjobd

Purpose:

Automatically creates user home directory
during first successful login.

Run ONLY on:

login

Edit:

sudo vim /etc/pam.d/system-auth

Add LDAP lines in appropriate sections.

Add in auth section:

auth        sufficient    pam_ldap.so use_first_pass

Add in account section:

account     sufficient    pam_ldap.so

Add in password section:

password    sufficient    pam_ldap.so use_authtok

Add in session section:

session     optional      pam_ldap.so
session     optional      pam_mkhomedir.so

Important:

Do NOT remove pam_unix.so lines.

Run ONLY on:

login

Edit:

sudo vim /etc/pam.d/password-auth

Add SAME LDAP PAM lines.

Reason:

Different login methods use different PAM stacks.

Run ONLY on:

login

Restart services:

sudo systemctl restart nslcd
sudo systemctl restart oddjobd

Run ONLY on:

login

Test login:

su - testUser

Enter LDAP password.

Expected:

Creating directory '/home/testUser'

Successful login indicates:

• LDAP authentication working
• PAM integration working
• automatic home directory creation working

Run ONLY on:

login

After login:

pwd

Expected:

/home/testUser

Cluster now contains:

Verify LDAP tree:

ldapsearch -x -b "dc=hpc,dc=local"

Verify LDAP user:

ldapsearch -x -b "dc=hpc,dc=local" uid=testUser

Verify LDAP authentication:

ldapwhoami -x \
-D "uid=testUser,ou=People,dc=hpc,dc=local" \
-W \
-H ldap://master

Verify Linux user resolution:

getent passwd testUser

Verify nslcd service:

systemctl status nslcd

Master node:

• provides centralized identity database

Login/compute nodes:

• authenticate users
• resolve LDAP users
• create user sessions

This separation is fundamental in distributed authentication systems.

Example:

cd ~/ldap-scripts
ls

Expected:

createUser.sh

Create file:

vim createUser.sh

Paste the complete script exactly as shown below:

#!/bin/bash 

LDAP_ADMIN="cn=Manager,dc=hpc,dc=local"
BASE_DN="dc=hpc,dc=local"

USERNAME=$1
UIDNUM=$2
GIDNUM=$3

if [ $# -ne 3 ]; then
    echo "Usage: $0 <username> <uid> <gid>"
    exit 1
fi

# Check if user already exists
USER_EXISTS=$(ldapsearch -x \
-b "ou=People,${BASE_DN}" "(uid=${USERNAME})" dn \
| grep "^dn:")

if [ ! -z "$USER_EXISTS" ]; then
    echo "ERROR: User ${USERNAME} already exists!"
    exit 1
fi

# Check if group already exists
GROUP_EXISTS=$(ldapsearch -x \
-b "ou=Groups,${BASE_DN}" "(cn=${USERNAME})" dn \
| grep "^dn:")

if [ ! -z "$GROUP_EXISTS" ]; then
    echo "ERROR: Group ${USERNAME} already exists!"
    exit 1
fi

PASSWORD="${USERNAME}@@123"

PASSWORD_HASH=$(slappasswd -s "$PASSWORD")

echo
echo "=================================="
echo "Creating LDAP User"
echo "=================================="
echo "Username : ${USERNAME}"
echo "UID      : ${UIDNUM}"
echo "GID      : ${GIDNUM}"
echo "=================================="

# Create private group LDIF
cat > /tmp/${USERNAME}_group.ldif <<EOF
dn: cn=${USERNAME},ou=Groups,${BASE_DN}
objectClass: posixGroup
cn: ${USERNAME}
gidNumber: ${GIDNUM}
EOF

# Create user LDIF
cat > /tmp/${USERNAME}.ldif <<EOF
dn: uid=${USERNAME},ou=People,${BASE_DN}
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: shadowAccount
cn: ${USERNAME}
sn: ${USERNAME}
uid: ${USERNAME}
uidNumber: ${UIDNUM}
gidNumber: ${GIDNUM}
homeDirectory: /home/${USERNAME}
loginShell: /bin/bash
userPassword: ${PASSWORD_HASH}
EOF

echo
echo "Adding LDAP group..."

ldapadd -x \
-D "${LDAP_ADMIN}" \
-W \
-f /tmp/${USERNAME}_group.ldif

if [ $? -ne 0 ]; then
    echo "ERROR: Failed to create LDAP group!"
    exit 1
fi

echo
echo "Adding LDAP user..."

ldapadd -x \
-D "${LDAP_ADMIN}" \
-W \
-f /tmp/${USERNAME}.ldif

if [ $? -ne 0 ]; then
    echo "ERROR: Failed to create LDAP user!"
    exit 1
fi

echo
echo "=================================="
echo "LDAP User Created Successfully"
echo "=================================="
echo "Username : ${USERNAME}"
echo "Password : ${PASSWORD}"
echo "UID      : ${UIDNUM}"
echo "GID      : ${GIDNUM}"
echo "=================================="

Before using the script:

chmod +x createUser.sh

The script requires THREE arguments:

Syntax:

./createUser.sh <username> <uid> <gid>

Example:

./createUser.sh student01 11001 11001

Explanation:

IMPORTANT:

The script creates a private group automatically.

Therefore username and group name become identical.

Example:

The script automatically creates password using:

username@@123

Example:

IMPORTANT:

Users should change password after first login.

The script performs following operations automatically:

Example:

./createUser.sh student02 11002 11002

Expected output:

==================================
Creating LDAP User
==================================
Username : student02
UID      : 11002
GID      : 11002
==================================

LDAP admin password will be requested.

After successful creation:

==================================
LDAP User Created Successfully
==================================
Username : student02
Password : student02@@123
UID      : 11002
GID      : 11002
==================================

Verify user using:

getent passwd student02

or:

id student02

Expected:

uid=11002(student02)

Verify LDAP entry:

ldapsearch -x \
-b "dc=hpc,dc=local" uid=student02

Test login:

su - student02

or:

ssh student02@login

Expected:

successful login

IMPORTANT:

UID and GID values must remain unique cluster-wide.

Avoid:

• duplicate UID
• duplicate GID

because duplicate IDs can create:

• ownership conflicts
• wrong usernames
• incorrect file ownership
• Slurm identity problems

Example of BAD configuration:

This can cause:

whoami returning wrong username

Always use unique UID/GID values.

In HPC clusters:

• NFS
• Slurm
• MPI
• LDAP
• SSH

all depend on:

consistent unique UID/GID mappings

across the entire cluster.

This is one of the MOST important concepts in centralized HPC authentication.

Example:

cd ~/ldap-scripts
ls

Expected:

deleteUser.sh

Create file:

vim deleteUser.sh

Paste the complete script exactly as shown below:

#!/bin/bash 

LDAP_ADMIN="cn=Manager,dc=hpc,dc=local"
BASE_DN="dc=hpc,dc=local"

USERNAME=$1

if [ $# -ne 1 ]; then
    echo "Usage: $0 <username>"
    exit 1
fi

echo
echo "=================================="
echo "Deleting LDAP User"
echo "=================================="
echo "Username : ${USERNAME}"
echo "=================================="

echo
echo "Deleting LDAP user entry..."

ldapdelete -x \
-D "${LDAP_ADMIN}" \
-W \
"uid=${USERNAME},ou=People,${BASE_DN}"

if [ $? -ne 0 ]; then
    echo "ERROR: Failed to delete LDAP user!"
    exit 1
fi

echo
echo "LDAP user deleted successfully."

# Ask about deleting group
echo
read -p "Delete LDAP group '${USERNAME}'? (y/n): " DELETE_GROUP

case $DELETE_GROUP in
    y|Y)

        ldapdelete -x \
        -D "${LDAP_ADMIN}" \
        -W \
        "cn=${USERNAME},ou=Groups,${BASE_DN}"

        if [ $? -eq 0 ]; then
            echo "LDAP group deleted successfully."
        else
            echo "WARNING: Failed to delete LDAP group."
        fi
        ;;

    n|N)
        echo "Skipping LDAP group deletion."
        ;;

    *)
        echo "Invalid option. Skipping group deletion."
        ;;
esac

# Ask about deleting home directory
echo
read -p "Delete home directory '/home/${USERNAME}'? (y/n): " DELETE_HOME

case $DELETE_HOME in
    y|Y)

        if [ -d "/home/${USERNAME}" ]; then
            sudo rm -rf "/home/${USERNAME}"

            if [ $? -eq 0 ]; then
                echo "Home directory deleted successfully."
            else
                echo "WARNING: Failed to delete home directory."
            fi
        else
            echo "Home directory does not exist."
        fi
        ;;

    n|N)
        echo "Skipping home directory deletion."
        ;;

    *)
        echo "Invalid option. Skipping home deletion."
        ;;
esac

echo
echo "=================================="
echo "User cleanup completed"
echo "=================================="

Before using the script:

chmod +x deleteUser.sh

The script requires ONE argument:

Syntax:

./deleteUser.sh <username>

Example:

./deleteUser.sh student02

The script performs following operations automatically:

Example:

./deleteUser.sh student02

Expected output:

==================================
Deleting LDAP User
==================================
Username : student02
==================================

LDAP admin password will be requested.

After successful deletion:

LDAP user deleted successfully.

The script then asks:

Delete LDAP group 'student02'? (y/n):

If:

y

is selected, corresponding LDAP group will also be removed.

Next prompt:

Delete home directory '/home/student02'? (y/n):

If:

y

is selected, user home directory will also be deleted.

IMPORTANT:

Do NOT delete shared groups accidentally.

Example:

hpcusers

may contain multiple users.

Deleting shared groups can create:

• broken group mappings
• permission problems
• Slurm identity issues

The script therefore asks confirmation before deleting groups.

Deleting home directories permanently removes:

• source codes
• job scripts
• outputs
• datasets
• SSH keys
• configuration files

IMPORTANT:

Home directory deletion is irreversible.

Always verify before deleting.

Verify deletion:

getent passwd student02

Expected:

no output

Verify LDAP entry removal:

ldapsearch -x \
-b "dc=hpc,dc=local" uid=student02

Expected:

no entries returned

Verify login failure:

su - student02

Expected:

user does not exist

In centralized HPC environments:

• user lifecycle management
• account cleanup
• UID/GID consistency
• group management

are critical administrative tasks.

Improper deletion can create:

• orphaned files
• broken permissions
• Slurm scheduling issues
• incorrect ownership mappings

Therefore automated and controlled cleanup procedures are highly important in production HPC systems.

Example:

cd ~/ldap-scripts
ls

Expected:

changePassword.sh

Create file:

vim changePassword.sh

Paste the complete script exactly as shown below:

#!/bin/bash

LDAP_ADMIN="cn=Manager,dc=hpc,dc=local"
BASE_DN="dc=hpc,dc=local"

USERNAME=$1

if [ $# -ne 1 ]; then
    echo "Usage: $0 <username>"
    exit 1
fi

# Check if user exists
USER_EXISTS=$(ldapsearch -x \
-b "ou=People,${BASE_DN}" "(uid=${USERNAME})" dn \
| grep "^dn:")

if [ -z "$USER_EXISTS" ]; then
    echo "ERROR: User ${USERNAME} does not exist!"
    exit 1
fi

echo
echo "=================================="
echo "LDAP Password Change"
echo "=================================="
echo "Username : ${USERNAME}"
echo "=================================="

# Take password securely
echo
read -s -p "Enter new password: " PASSWORD1
echo

read -s -p "Confirm new password: " PASSWORD2
echo

# Check password match
if [ "$PASSWORD1" != "$PASSWORD2" ]; then
    echo
    echo "ERROR: Passwords do not match!"
    exit 1
fi

# Generate LDAP password hash
PASSWORD_HASH=$(slappasswd -s "$PASSWORD1")

# Create LDIF file
cat > /tmp/${USERNAME}_password.ldif <<EOF
dn: uid=${USERNAME},ou=People,${BASE_DN}
changetype: modify
replace: userPassword
userPassword: ${PASSWORD_HASH}
EOF

echo
echo "Updating LDAP password..."

ldapmodify -x \
-D "${LDAP_ADMIN}" \
-W \
-f /tmp/${USERNAME}_password.ldif

if [ $? -ne 0 ]; then
    echo
    echo "ERROR: Failed to change password!"
    exit 1
fi

echo
echo "=================================="
echo "Password updated successfully"
echo "=================================="
echo "Username : ${USERNAME}"
echo "=================================="

Before using the script:

chmod +x changePassword.sh

The script requires ONE argument:

Syntax:

./changePassword.sh <username>

Example:

./changePassword.sh student01

The script performs following operations automatically:

Example:

./changePassword.sh student01

Expected output:

==================================
LDAP Password Change
==================================
Username : student01
==================================

The script asks:

Enter new password:

Then:

Confirm new password:

LDAP admin password will also be requested.

After successful update:

==================================
Password updated successfully
==================================
Username : student01
==================================

Test login using updated password:

su - student01

or:

ssh student01@login

Expected:

successful login using new password

Because LDAP authentication is centralized, the updated password works automatically on:

• login node
• compute nodes
• master node

IMPORTANT:

LDAP stores password hashes, not plain-text passwords.

The script automatically generates secure LDAP-compatible password hashes using:

slappasswd

This improves security by avoiding direct plain-text password storage.

Advanced Slurm administration should always be performed incrementally.

Instead of enabling many advanced features simultaneously, only ONE feature was introduced and validated at a time.

This approach helps:

• isolate issues properly
• simplify debugging
• understand scheduler behavior
• maintain cluster stability

This is how production HPC administrators commonly manage scheduler configuration changes.

The cluster was upgraded gradually in the following order:

Goal:

Teach Slurm to understand:

• CPU resources
• memory resources
• consumable resources
• schedulable resources

without enabling strict runtime enforcement.

This creates a much safer and more stable intermediate scheduler state.

Initial scheduler behavior:

• node-aware scheduling
• CPU-aware scheduling

BUT:

Slurm did NOT properly understand actual node memory capacity.

This can be verified using:

scontrol show node compute01

Without:

RealMemory

configured, Slurm cannot properly perform memory-aware scheduling.

Actual node memory was checked using:

ssh compute01 free -m
ssh compute02 free -m
ssh compute03 free -m

Observed memory:

~1700 MB

IMPORTANT:

Production HPC clusters NEVER allocate 100% physical memory to jobs.

Reason:

The operating system itself requires memory for:

• kernel
• SSH sessions
• Slurm daemons
• filesystem cache
• system services
• background tasks

A portion of memory must always remain reserved for stable node operation.

Node configuration was updated inside:

sudo vim /etc/slurm/slurm.conf

Original configuration:

NodeName=compute01 CPUs=2 State=UNKNOWN
NodeName=compute02 CPUs=2 State=UNKNOWN
NodeName=compute03 CPUs=2 State=UNKNOWN

Updated configuration:

NodeName=compute01 CPUs=2 RealMemory=1200 State=UNKNOWN
NodeName=compute02 CPUs=2 RealMemory=1200 State=UNKNOWN
NodeName=compute03 CPUs=2 RealMemory=1200 State=UNKNOWN

RealMemory=1200

does NOT represent total physical memory.

Instead, it defines:

schedulable memory available to jobs

Example:

This reserved memory helps maintain:

• system stability
• scheduler reliability
• daemon responsiveness
• SSH availability

during heavy workloads.

After configuring:

RealMemory

Slurm now understands:

• CPU resources
• schedulable memory
• consumable resources

instead of only allocating entire nodes.

This is a major scheduler upgrade.

The scheduler transitioned from:

simple node allocation

to:

resource-aware scheduling

The following important scheduler options were enabled:

SchedulerType=sched/backfill

SelectType=select/cons_tres
SelectTypeParameters=CR_Core

TaskPlugin=task/affinity

Updated configuration was distributed cluster-wide.

Copy configuration:

scp /etc/slurm/slurm.conf login:/tmp/
scp /etc/slurm/slurm.conf compute01:/tmp/
scp /etc/slurm/slurm.conf compute02:/tmp/
scp /etc/slurm/slurm.conf compute03:/tmp/

Move configuration into place:

sudo mv /tmp/slurm.conf /etc/slurm/

Compute nodes:

ssh compute01 sudo -S mv /tmp/slurm.conf /etc/slurm/
ssh compute02 sudo -S mv /tmp/slurm.conf /etc/slurm/
ssh compute03 sudo -S mv /tmp/slurm.conf /etc/slurm/

Restart controller on master node:

sudo systemctl restart slurmctld

Restart compute daemons:

ssh compute01 sudo -S systemctl restart slurmd
ssh compute02 sudo -S systemctl restart slurmd
ssh compute03 sudo -S systemctl restart slurmd

Purpose:

Reload updated scheduler configuration across all nodes.

Verify node state:

sinfo

Expected:

compute[01-03] idle

Verify scheduler memory recognition:

scontrol show node compute01

Expected:

RealMemory=1200

This validates:

• scheduler configuration loaded correctly
• memory-aware scheduling active
• stable cluster operation

At this stage:

memory-aware scheduling

was enabled.

BUT:

strict kernel-level memory enforcement

was NOT enabled.

This distinction is extremely important.

Current behavior:

#SBATCH --mem=500M

does:

• reserve memory in scheduler accounting
• affect scheduling decisions
• prevent scheduler over-allocation

BUT does NOT:

• hard-limit memory usage
• kill memory-overusing jobs
• enforce kernel-level restrictions

because:

TaskPlugin=task/cgroup

was intentionally NOT enabled.

Earlier experiments with:

TaskPlugin=task/cgroup

introduced runtime instability.

Observed issues included:

• batch runtime failures
• node drain states
• unstable job execution
• scheduler inconsistencies

Reason:

Modern Rocky Linux 9 systems use:

cgroup v2

which often requires significantly more careful Slurm configuration and kernel-level tuning.

Instead of introducing unstable runtime enforcement early, the cluster was intentionally kept in a stable intermediate state using:

TaskPlugin=task/affinity

This approach provides:

• stable runtime execution
• CPU affinity support
• resource-aware scheduling
• simplified administration

without destabilizing scheduler behavior.

The cluster still uses:

ProctrackType=proctrack/cgroup

This is generally safe because:

• lightweight process tracking only
• no strict runtime enforcement
• no aggressive kernel-level isolation

This differs significantly from:

TaskPlugin=task/cgroup

which performs actual resource enforcement and isolation.

Keeping:

proctrack/cgroup

while avoiding:

task/cgroup

provides a much safer and more stable educational HPC environment.

After enabling resource-aware scheduling:

• sbatch jobs remained stable
• MPI jobs executed correctly
• multinode execution continued working
• no node drain issues occurred
• no runtime instability observed

This confirmed that the scheduler upgrade was successful.

The cluster successfully evolved from:

basic educational scheduler

into:

resource-aware production-like scheduler

while maintaining:

• stable runtime execution
• multinode MPI support
• scheduler reliability
• simplified administration

This forms the foundation required for future advanced features such as:

• CPU affinity and task binding
• QoS
• fairshare
• accounting
• scheduling policies
• advanced resource management

Advanced Slurm administration should always be performed incrementally.

Instead of enabling multiple advanced features simultaneously, only one feature was introduced and tested at a time.

This helps isolate issues and understand scheduler behavior properly.

Cluster components already working:

This provided a stable foundation before introducing advanced scheduler features.

Goal:

Enable Slurm to understand and schedule cluster resources more accurately.

This phase introduced:

• memory-aware scheduling
• consumable resource scheduling
• core-aware scheduling
• scheduler backfill support

without enabling strict kernel-level resource enforcement.

Goal:

Understand and validate:

• CPU affinity
• process placement
• task binding
• scheduler-controlled CPU allocation

This phase connects directly with:

• parallel performance
• cache locality
• MPI optimization
• OpenMP optimization

CPU topology information was inspected using:

ssh compute01 lscpu

Important fields:

This information becomes important for:

• task placement
• thread pinning
• NUMA optimization
• cache locality

Program used to validate CPU placement:

#define _GNU_SOURCE

#include <stdio.h>
#include <sched.h>
#include <unistd.h>

int main() {

    int cpu = sched_getcpu();

    printf("Process running on CPU: %d\n", cpu);

    sleep(20);

    return 0;
}

Compile:

gcc affinity.c -o affinity

Purpose:

The program asks Linux:

Which CPU core am I running on?

This helps visualize scheduler-controlled CPU placement.

Slurm batch script:

#!/bin/bash
#SBATCH -N 2
#SBATCH --ntasks-per-node=1
#SBATCH --job-name=affinity
#SBATCH --output=%j.out
#SBATCH --error=%j.err
#SBATCH --time=00:01:00

srun ./affinity

Submit:

sbatch affinity.sh

Observed behavior:

Process running on CPU: 0
Process running on CPU: 0

IMPORTANT:

CPU numbering is local to each node.

Meaning:

• compute01 has CPU0
• compute02 also has CPU0

Each node maintains independent CPU numbering.

This is an important distributed-system concept.

Modified batch script:

#!/bin/bash
#SBATCH -N 2
#SBATCH --ntasks-per-node=2
#SBATCH --job-name=affinity
#SBATCH --output=%j.out
#SBATCH --error=%j.err
#SBATCH --time=00:01:00

srun ./affinity

Observed behavior:

Process running on CPU: 0
Process running on CPU: 1
Process running on CPU: 1
Process running on CPU: 0

This validates:

• task distribution across cores
• scheduler-controlled placement
• multi-core resource allocation

Each node distributed tasks across available CPUs.

Explicit CPU binding script:

#!/bin/bash
#SBATCH -N 1
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=1
#SBATCH --job-name=bindtest
#SBATCH --output=%j.out
#SBATCH --error=%j.err
#SBATCH --time=00:01:00

srun --cpu-bind=cores ./affinity

Submit:

sbatch bind.sh

Observed behavior:

Process running on CPU: 0
Process running on CPU: 1

This validates:

• explicit CPU pinning
• deterministic CPU placement
• Slurm affinity control
• Linux scheduler cooperation

Goal:

Transform Slurm from:

basic scheduler

into:

persistent accounting-aware scheduler

This phase introduced:

• MariaDB backend
• Slurm accounting daemon
• persistent job history
• resource usage tracking
• scheduler accounting infrastructure

Performed ONLY on master node.

Install packages:

sudo dnf install -y mariadb-server mariadb

Enable and start service:

sudo systemctl enable --now mariadb

Verify:

sudo systemctl status mariadb

Expected:

active (running)

Run:

sudo mysql_secure_installation

Recommended configuration:

Enter MariaDB:

sudo mysql -u root -p

Create database:

CREATE DATABASE slurm_acct_db;

Create Slurm database user:

CREATE USER 'slurm'@'localhost' IDENTIFIED BY 'slurmdbpass';

Grant permissions:

GRANT ALL PRIVILEGES ON slurm_acct_db.* TO 'slurm'@'localhost';

Apply changes:

FLUSH PRIVILEGES;

Exit:

EXIT;

Install:

sudo dnf install -y slurm-slurmdbd

Purpose:

Installs:

slurmdbd

which connects:

• Slurm scheduler
• accounting database
• resource tracking infrastructure

Create configuration:

sudo vim /etc/slurm/slurmdbd.conf

Configuration:

AuthType=auth/munge

DbdHost=master
DbdPort=6819

SlurmUser=slurm

DebugLevel=info
LogFile=/var/log/slurmdbd.log
PidFile=/run/slurmdbd.pid

StorageType=accounting_storage/mysql
StorageHost=localhost
StoragePort=3306
StorageUser=slurm
StoragePass=slurmdbpass
StorageLoc=slurm_acct_db

Modify:

sudo vim /etc/slurm/slurm.conf

Add:

AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageHost=master
AccountingStoragePort=6819
JobAcctGatherType=jobacct_gather/linux

Purpose:

Enable:

• persistent job accounting
• resource tracking
• historical scheduler data

Restart accounting daemon on master:

sudo systemctl enable --now slurmdbd

Verify:

sudo systemctl status slurmdbd

Expected:

active (running)

Restart Slurm controller:

sudo systemctl restart slurmctld

Restart compute daemons:

ssh compute01 sudo -S systemctl restart slurmd
ssh compute02 sudo -S systemctl restart slurmd
ssh compute03 sudo -S systemctl restart slurmd

Check cluster:

sinfo

Expected:

compute[01-03] idle

Check accounting cluster registration:

sacctmgr list cluster

Expected:

hpccluster

Submit MPI job:

sbatch submit.sh a.out

Check scheduler queue:

squeue --me

Check accounting history:

sacct

Observed output:

JobID           JobName  Partition    AllocCPUS      State
22                 test      debug             4  COMPLETED
22.batch          batch                       2  COMPLETED
22.0              prted                       2  COMPLETED

This validates:

• persistent accounting
• resource tracking
• MPI runtime tracking
• scheduler database integration

Accounting now tracks internal execution hierarchy:

This provides deeper visibility into scheduler behavior and runtime execution structure.

The cluster now supports:

The cluster has now evolved from a basic educational setup into a much more realistic HPC infrastructure environment.

Before this phase:

Slurm could:

• schedule resources
• track jobs
• track users
• record accounting information

BUT it could NOT yet:

• enforce user policies
• restrict CPU usage
• limit runtime
• prevent resource abuse

This phase enables:

policy-aware scheduling

similar to real production HPC systems.

Slurm scheduling hierarchy:

Cluster
 └── Account
      └── User
           └── QoS

Production HPC systems commonly organize users using:

• projects
• research groups
• departments
• labs

called:

Accounts

Users are then associated with:

• accounts
• QoS policies
• scheduler limits

Before enabling QoS enforcement, the following components were already working:

This provided the required foundation for policy enforcement.

QoS stands for:

Quality of Service

QoS controls:

• maximum CPUs
• maximum jobs
• walltime limits
• scheduler priorities
• resource restrictions
• policy enforcement

Production HPC clusters heavily rely on QoS to maintain fairness and prevent abuse.

QoS does NOT directly enforce kernel-level runtime isolation.

Instead, it performs:

scheduler-side policy enforcement

Meaning:

Slurm scheduler decides:

• whether jobs may start
• whether jobs remain pending
• whether limits are exceeded

WITHOUT requiring:

task/cgroup

or strict kernel enforcement.

This keeps the cluster significantly more stable while still enabling realistic scheduler policies.

Before creating accounts and policies, verify cluster registration.

Run on:

• login node
• or master node

sacctmgr list cluster

Expected:

hpccluster

Purpose:

Verifies:

• accounting database connectivity
• slurmdbd functionality
• cluster accounting registration

Accounts represent:

• research groups
• projects
• departments
• user groups

Create a general account.

Run on master node:

sudo sacctmgr add account general

Verify:

sacctmgr list account

Expected:

general

Linux users and Slurm accounts are NOT the same thing.

Linux users provide:

authentication identity

Slurm accounts provide:

scheduler policy grouping

This distinction is extremely important in production HPC systems.

LDAP users already existed in Linux.

However, Slurm accounting ALSO requires user associations.

Example user:

rishi

Run on master node:

sudo sacctmgr add user rishi account=general

Add additional users if needed:

sudo sacctmgr add user hpcadmin account=general

Verify associations:

sacctmgr list user

Expected:

Users associated with:

general

Create QoS policy named:

normal

Run on master node:

sudo sacctmgr add qos normal

Verify:

sacctmgr list qos

Expected:

normal

The following limits were configured.

Connect QoS policy to account.

Run:

sudo sacctmgr modify account general set qos=normal

Purpose:

Users associated with:

general

now inherit:

normal

QoS policies.

Verify:

• user associations
• account associations
• QoS assignments

Run:

sacctmgr show assoc

Purpose:

Displays:

• users
• accounts
• partitions
• qos assignments

Initially, QoS appeared to exist correctly:

• QoS visible
• associations visible
• accounting working
• job tracking working

BUT:

policy enforcement was NOT working

Jobs exceeding configured CPU limits still executed successfully.

Example:

MaxTRESPerUser=cpu=4

BUT:

6 CPU jobs still executed

This revealed an extremely important missing scheduler configuration.

The following line was missing from:

/etc/slurm/slurm.conf

Missing configuration:

AccountingStorageEnforce=limits,qos

Without this line:

• accounting works
• sacct works
• qos appears correctly
• associations appear correctly

BUT:

scheduler policies are NOT enforced

This is one of the most common beginner Slurm configuration mistakes.

Edit configuration on master node:

sudo vim /etc/slurm/slurm.conf

Add:

AccountingStorageEnforce=limits,qos

Recommended placement:

Near existing accounting configuration:

AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageHost=master
AccountingStoragePort=6819
JobAcctGatherType=jobacct_gather/linux

This line enables actual scheduler-side enforcement for:

• qos limits
• account limits
• user policies
• scheduler restrictions

Without:

AccountingStorageEnforce

QoS becomes informational only.

Copy updated configuration from master node:

scp /etc/slurm/slurm.conf login:/tmp/
scp /etc/slurm/slurm.conf compute01:/tmp/
scp /etc/slurm/slurm.conf compute02:/tmp/
scp /etc/slurm/slurm.conf compute03:/tmp/

Move configuration into place on login node:

sudo mv /tmp/slurm.conf /etc/slurm/

Move configuration into place on compute nodes:

ssh compute01 sudo -S mv /tmp/slurm.conf /etc/slurm/
ssh compute02 sudo -S mv /tmp/slurm.conf /etc/slurm/
ssh compute03 sudo -S mv /tmp/slurm.conf /etc/slurm/

Restart controller on master node:

sudo systemctl restart slurmctld

Restart compute daemons:

ssh compute01 sudo -S systemctl restart slurmd
ssh compute02 sudo -S systemctl restart slurmd
ssh compute03 sudo -S systemctl restart slurmd

Purpose:

Reload updated scheduler policy configuration.

Run:

sinfo

Expected:

compute[01-03] idle

This validates:

• scheduler stability
• successful configuration reload
• healthy compute nodes

Job requesting:

4 CPUs

using:

#SBATCH -N 2
#SBATCH --ntasks-per-node=2

executed successfully.

Observed output:

SLURM_NTASKS=4
SLURM_JOB_NUM_NODES=2
SLURM_CPUS_ON_NODE=2

This validates:

• multinode allocation
• scheduler variable propagation
• QoS-aware MPI execution
• proper CPU accounting

MPI jobs remained fully functional while QoS policies were enforced.

This validates successful integration between:

• Slurm scheduler
• QoS policies
• multinode scheduling
• MPI runtime
• resource accounting

Job script modified:

#SBATCH --time=02:00:00

Observed scheduler behavior:

PENDING
Reason=QOSMaxWallDurationPerJobLimit

This validates:

• runtime limit enforcement
• scheduler-side walltime policy control

QoS-restricted jobs are usually:

queued and pending

NOT:

immediately rejected

This is standard production HPC scheduler behavior.

Scheduler explains pending reason using:

squeue --me

or:

scontrol show job <jobid>

This greatly improves scheduler transparency.

View configured QoS settings:

sacctmgr show qos normal

Observed configuration:

MaxWall=01:00:00
MaxTRESPU=cpu=4
MaxJobsPU=2

This validates:

• correct QoS policy configuration
• scheduler policy persistence
• accounting database integration

The cluster now supports multiple scheduler layers simultaneously.

This architecture is significantly closer to real production HPC clusters.

The cluster now supports:

The cluster has now evolved into a significantly more realistic multi-user HPC scheduling environment.

At the time of validation, the cluster already supported:

This validation phase ensures all components function together correctly.

Run on login node:

sinfo

Expected:

PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
debug*       up   infinite      3   idle compute[01-03]