HPC Infrastructure
Operations for
Senior Engineers

Exploration before instruction.

You SSH into a live cluster before you're told what you're looking at. Examine running processes, read the scheduler configuration, inspect mounted filesystems, list network interfaces. No instructions — guided exploration followed by a debrief.

You leave this module able to read a cluster's topology and understand its components without requiring documentation.

Build admin empathy by being the user first.

Before you administer anything, you submit jobs. Serial jobs, multi-threaded jobs, massively parallel MPI jobs. You deliberately submit a job requesting 32 cores that only uses 4 — and read the output to identify the waste. Every operational decision later in the course will be informed by this perspective.

You leave this module able to submit, monitor, and interpret real HPC jobs — and recognise what a poorly-specified job looks like from the scheduler's side.

The most transferable module for datacenter veterans — familiar concepts, radically different implementation.

You know enterprise NAS. This module starts by showing you precisely why it fails at HPC — and what replaces it. Parallel filesystem architecture, stripe tuning, quota management, scratch lifecycle. You'll benchmark storage under parallel write load, tune stripe counts, and simulate a scratch-full failure — diagnosing and recovering a cluster where a job is failing because a disk has silently filled.

You leave this module able to choose the right storage tier for a workload, configure parallel filesystem stripes, and diagnose I/O bottlenecks before they become incidents.

Understanding when the network is the bottleneck — and proving it with numbers.

Why standard Ethernet is insufficient for tightly-coupled parallel workloads. High-speed fabric architecture — concepts, health monitoring, and verification. You'll benchmark interconnect performance, verify your jobs are actually using the high-speed fabric and not silently falling back to standard networking, and diagnose a job running ten times slower than expected because of exactly that failure. The module closes with a direct mapping onto GPU collective communications in large-scale compute workloads.

You leave this module able to verify network fabric usage, benchmark interconnect performance, and identify network-related job slowdowns.

The operational heart of HPC administration. Most of your working day lives here.

SLURM architecture, partition design, quality of service configuration, fair-share accounting. Node states and what each means operationally. Job dependencies, job arrays, preemption, checkpointing. Every topic is scenario-driven — a researcher reports their job has been pending for six hours; you diagnose it using the scheduler's own tools. Root causes are intentionally varied across participants.

You leave this module able to diagnose pending jobs systematically, configure fair-share policies, and perform node maintenance without disrupting running workloads.

The invisible infrastructure that makes or breaks researcher productivity.

Why environment modules exist. How to write them. SPACK for dependency-aware package management. Singularity containers — why Docker doesn't run on HPC clusters, and how to convert Docker images to run under the job scheduler. You'll pull a PyTorch image, convert it, and submit it as a cluster job. Directly applicable to AI infrastructure container workflows.

You leave this module able to maintain software stacks on a shared cluster, support users with module issues, and run containerised workloads under a job scheduler.

Framed as a user lifecycle exercise — onboarding and offboarding drives every topic naturally.

Linux account management at scale, LDAP integration, SSH key policies, filesystem security on shared parallel filesystems. The lab is a complete user lifecycle — onboard an account from scratch, then offboard it in the correct sequence: suspend the account, handle running jobs, archive the home directory, remove scheduler associations. Everything in the right order.

You leave this module able to manage the complete user lifecycle on an HPC cluster and understand the filesystem security model for shared environments.

Build a dashboard you would actually use — not a toy example.

HPC monitoring philosophy: job-level metrics matter as much as node-level metrics. You'll deploy a scheduler metrics exporter and node exporters on the shared cluster, build a working Grafana dashboard covering cluster utilisation, queue depth, job wait time, and node health — and configure two production-grade alerts: storage reaching capacity, and a node stuck in drain state. Then generate real cluster load and watch the metrics respond.

You leave this module able to build and maintain cluster observability and configure proactive alerting before users have to complain.

No new labs. A mapping session that makes everything you've learned immediately portable.

This module does not teach AI infrastructure. It shows you exactly where the HPC knowledge from this course sits inside the large-scale compute stack — and names precisely what is new territory from here. GPU collective communications, model checkpoint storage patterns, inference infrastructure, orchestration for ML workloads. You'll know what you know, what you don't yet know, and what to learn next.

You leave this module able to walk into an AI infrastructure role or course already understanding the infrastructure layer — with zero vocabulary confusion.