Deploy an HPC cluster in AWS with aws-parallelcluster and running a SLURM job
Parallel Cluster configuration file
See Cluster Configuration File official AWS docs.
For my HPC - Project (Deep Packet Inspection) i used the following configuration file:
Region: eu-north-1
Image:
Os: ubuntu2204 # Optimized for modern compilers & MPI runtimes
HeadNode:
InstanceType: c6i.2xlarge # Manage jobs without paying for massive idle hardware
Networking:
SubnetId: subnet-xxxx # Put your specific VPC subnet ID here
ElasticIp: true
Ssh:
KeyName: hpc_id # The EC2 key pair used to log in via terminal
Scheduling:
Scheduler: slurm # Deploys the Slurm workload manager natively
SlurmQueues:
- Name: compute-fleet
ComputeResources:
- Name: hpc-nodes
InstanceType: c6in.2xlarge
MinCount: 0 # Dynamically scales down to 0 to save money when idle
MaxCount: 8 # scales up to 8 nodes under load
Efa:
Enabled: false
Networking:
SubnetIds:
- subnet-xxxx # Private subnet, different from the previous one, routes via NAT gateway
PlacementGroup:
Enabled: true # Forces nodes physically close together in the data center
SharedStorage:
- Name: SharedEBS
StorageType: Ebs
MountDir: /shared
EbsSettings:
VolumeType: gp3
Size: 50 Getting started
Context: the following are the steps + problem solving process i did to run my HPC - Project - Deep Packet Inspection with pattern matching on a HPC cluster on AWS.
Prerequisites
Ensure on AWS you already know the following:
- a subnet id for the workers with NAT gateway and an elastic ip allocated
- a subnet id for the head + an elastic id allocated to the head node (required for ssh connect)
- a key pair (EC2)
- the region e.g. eu-north-1
$REGION = "eu-north-1"
$VPC_ID = "vpc-0b809620dcbd9da69"
$PUBLIC_SUBNET_ID = "subnet-0f798c95e50812c9f" # existing subnet for head node
$AZ = "eu-north-1a" # must match the public subnet's AZ
$PRIVATE_CIDR = "172.31.80.0/24" # must not overlap existing subnetsFind an existing vpc:
aws ec2 describe-vpcs --region eu-north-1 --query "Vpcs[].VpcId" --output textCreate a private subnet for nodes:
aws ec2 create-subnet --region $REGION --vpc-id $VPC_ID --cidr-block $PRIVATE_CIDR --availability-zone $AZ --tag-specifications "ResourceType=subnet,Tags=[{Key=Name,Value=hpc-compute-private-subnet}]"Ensure the AZ is the same as the subnet used for public nodes. Save the returned subnetId and add it to the configuration file.
Get the subnet id
aws ec2 describe-subnets --region eu-north-1 --filters "Name=tag:Name,Values=hpc-compute-private-subnet" --query "Subnets[].SubnetId" --output textNext, allocate an Elastic IP address for the NAT Gateway
aws ec2 allocate-address --region $REGION --domain vpc --tag-specifications "ResourceType=elastic-ip,Tags=[{Key=Name,Value=hpc-nat-eip}]"Save the returned AllocationId or run:
aws ec2 describe-addresses --region eu-north-1 --query "Addresses[].[AllocationId,PublicIp,AssociationId]" --output tableThen create the NAT Gateway (must sit in public environment). VERY IMPORTANT NOTE: when you make the gateway be seure to use the public subnet id. It is very easy to understand why:
- we have a private subnet without access to internet (no public ipv4) and a public subnet
- after creating the private subnet, we will also create a route table that routes all the traffic (0.0.0.0) to the gateway
- If we put the gateway in the private subnet, the traffic will simply route to the network itself.
- What instead we want is to route the traffic to the subnet with internet access: the public one.
aws ec2 create-nat-gateway --region $REGION --subnet-id $PUBLIC_SUBNET_ID --allocation-id <allocation-id-from-step-3> --tag-specifications "ResourceType=natgateway,Tags=[{Key=Name,Value=hpc-nat-gateway}]"Save the returned NatGatewayId. Wait for it to become available.
You can run:
aws ec2 describe-nat-gateways --region $REGION --nat-gateway-ids <nat-gateway-id> --query "NatGateways[0].State"to see if it is “available”.
Next, create a route table for the private subnet
aws ec2 create-route-table --region $REGION --vpc-id $VPC_ID --tag-specifications "ResourceType=route-table,Tags=[{Key=Name,Value=hpc-compute-private-rt}]"Save the return RouteTableId. Next step is to add the NAT Gateway route, we want to route all the traffic to the NAT: 0.0.0.0/0 -> NAT. In this way the private node can access internet. This ensure that the cloud-init configuration doesn’t fail and the workers node is properly set. Run the following command:
aws ec2 create-route --region $REGION --route-table-id <route-table-id> --destination-cidr-block 0.0.0.0/0 --nat-gateway-id <nat-gateway-id>Associate the private subnet with this route table:
aws ec2 associate-route-table --region $REGION --subnet-id <private-subnet-id> --route-table-id <route-table-id>Then verify the route table
aws ec2 describe-route-tables --region $REGION --route-table-ids <route-table-id> --query "RouteTables[0].Routes"It should show:
172.31.0.0/16 → local
0.0.0.0/0 → nat-xxxxxxxxFinally, update the configuration file (.yaml) with the new subnet id. You can run:
aws ec2 describe-subnets --region eu-north-1 --output table --query "Subnets[*].{SubnetID: SubnetId, Name: Tags[?Key=='Name'].Value | [0]}"To query all subnets with their id and name.
Now you should have all the prerequisites and you can create the cluster, in short the command is:
pcluster create-cluster --cluster-name hpc-cluster --cluster-configuration cluster-config.yaml --region eu-north-1If you follow all the steps explained in “Prerequisites”, you can skip some stuff on the next section.
Starting the cluster
Create a virtual environment. Don’t use newer version (see explanation there), instead use:
py -3.11 -m venv pcluster-env
.\pcluster-env\Scripts\Activate.ps1
python --version # ensure the correct version
pip install aws-parallelcluster
pcluster version #ensure it worksTo start the cluster:
pcluster create-cluster --cluster-name hpc-cluster --cluster-configuration cluster-config.yamlYou should see this output (or similar):
{
"cluster": {
"clusterName": "hpc-cluster",
"cloudformationStackStatus": "CREATE_IN_PROGRESS",
"cloudformationStackArn": "arn:aws:cloudformation:eu-north-1:22........024:stack/hpc-cluster/b76.........e7",
"region": "eu-north-1",
"version": "3.15.1",
"clusterStatus": "CREATE_IN_PROGRESS",
"scheduler": {
"type": "slurm"
}
},ParallelCluster clusters don’t have their own dedicated page in the standard, because they’re built from several underlying services pieced together. However you can see:
- A stack called clusterName (e.g. hpc-cluster) on CloudFormation → Stacks
- Head and workers nodes on EC2 → Instances
- EBS
You may want to swap EBS with FSx, but you need to allocate alteast 1200 GB for a FSx instance and ofc it will cost a lot more.
Describe the cluster:
pcluster describe-cluster --cluster-name hpc-cluster --region eu-north-1Wait until status goes from in progress to done.
Be sure that you can connect to the head note through SSH. Then run the following command:
scp -v -i ~\.ssh\hpc_id.pem -r . ubuntu@PUBLIC_IPV4:/shared/Be sure benchmark.slurm is uploaded, so you can run:
sbatch benchmark.slurmSurprisly, i created the slurm file on Windows, and it gave me the following error:
sbatch: error: Batch script contains DOS line breaks (\r\n)
sbatch: error: instead of expected UNIX line breaks (\n).So i first had to run:
dos2unix benchmark.slurmAfter running sbatch benchmark.slurm i got: Submitted batch job 1
two useful commands are squeue, which gave me output:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1 compute-f benchmar ubuntu CF 0:46 4 compute-fleet-dy-hpc-nodes-[1-4]and sinfo:
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
compute-fleet* up infinite 4 mix# compute-fleet-dy-hpc-nodes-[1-4]In squeue, CF means CompletingFuture/Configuring, this is expected behavior as on AWS it requires a few minutes to start up the instances.
By running:
sinfo -N
NODELIST NODES PARTITION STATE
compute-fleet-dy-hpc-nodes-1 1 compute-fleet* mix#
compute-fleet-dy-hpc-nodes-2 1 compute-fleet* mix#
compute-fleet-dy-hpc-nodes-3 1 compute-fleet* mix#
compute-fleet-dy-hpc-nodes-4 1 compute-fleet* mix#
You can see more informations about the computing nodes.
You can also have more details by looking for logs:
sudo tail -50 /var/log/parallelcluster/slurm_resume.log
After 13 minutes the nodes were still not up, so i run ssh to log into the first node ssh ompute-fleet-dy-hpc-nodes-1 and checked:
sudo tail -100 /var/log/cloud-init-output.logThe log stopped at:
+ cinc-client --local-mode --config /etc/chef/client.rb --log_level info --logfile /var/log/chef-client.log --force-formatter --no-color --chef-zero-port 8889 --json-attributes /etc/chef/dna.json --override-runlist aws-parallelcluster-entrypoints::initwhich is the program that configure SLURM on workers nodes.
However the program was still alive, in fact ps aux | grep cinc-client:
root 958 0.4 3.4 931436 134720 ? Sl 16:48 0:04 /opt/cinc/embedded/bin/ruby --disable-gems /bin/cinc-client --local-mode --config /etc/chef/client.rb --log_level info --logfile /var/log/chef-client.log --force-formatter --no-color --chef-zero-port 8889 --json-attributes /etc/chef/dna.json --override-runlist aws-parallelcluster-entrypoints::init
ubuntu 1461 0.0 0.0 7008 2536 pts/0 S+ 17:05 0:00 grep --color=auto cinc-clientOur compute nodes couldn’t finish bootstrapping because they were stuck in a retry loop trying to reach AWS APIs (like DynamoDB) to register themselves with the cluster, and diagnosis showed they had no outbound internet path — the route table pointed to an Internet Gateway, but IGWs only work for instances with a public IP, which our compute nodes didn’t have. AWS’s best-practice architecture for this is a two-subnet design: a public subnet for the head node (which needs a public/Elastic IP for direct SSH access) and a separate private subnet for compute nodes (which don’t need direct internet exposure, only outbound access). The private subnet routes its outbound traffic through a NAT Gateway sitting in the public subnet, giving compute nodes internet/API access without exposing them individually. We created that new private subnet, launched a NAT Gateway with its own Elastic IP, and pointed the private subnet’s route table at the NAT Gateway instead of the IGW directly — which resolved the node registration failures.
Output of the job can be seen in a “.out” file in the directory where you run the job program, e.g. in my case:
cat benchmark_p_4.outAfter these fix, the job worked succesfully.
Cleaning / Deleting everything
To delete the cluster once you’re finished:
pcluster delete-cluster --cluster-name hpc-cluster --region eu-north-1Check for EC2 instances leftover:
aws ec2 describe-instances --region eu-north-1 --filters "Name=tag:parallelcluster:cluster-name,Values=hpc-cluster" --query "Reservations[].Instances[].[InstanceId,State.Name]" --output table------------------------------------------
| DescribeInstances |
+----------------------+-----------------+
| i-0bcfc7a273a980d7c | terminated |
| i-0fc66369b1bf1740e | terminated |
| i-077a2928c47d3b1c9 | terminated |
| i-0a380eef314559455 | terminated |
| i-0d0770e4db545992a | terminated |
| i-03f9f4fbc2dc73f3e | shutting-down |
| i-0eec37153e369d622 | shutting-down |
| i-04f08c603182c67ef | shutting-down |
| i-0af69e75e9fca9304 | shutting-down |
+----------------------+-----------------+If anything shows besides terminated, you can run:
aws ec2 terminate-instances --region eu-north-1 --instance-ids <instance-id>
Now we have to find and delete the NAT Gateway. It will return something like nat-xxxxxx. Delete it since it has an ongoing cost, run:
aws ec2 delete-nat-gateway --region eu-north-1 --nat-gateway-id nat-xxxxxUnattached Elastic IPs also incur charges, so don’t skip this.
aws ec2 describe-addresses --region eu-north-1 --query "Addresses[].[AllocationId,PublicIp,AssociationId]" --output tableRun:
aws ec2 release-address --region eu-north-1 --allocation-id <allocation-id>Now it’s the route table turn:
aws ec2 describe-route-tables --region eu-north-1 --filters "Name=tag:Name,Values=hpc-compute-private-rt" --query "RouteTables[].RouteTableId" --output textto delete the route table run:
aws ec2 delete-route-table --region eu-north-1 --route-table-id <route-table-id>If you get an error like:
aws: [ERROR]: An error occurred (DependencyViolation) when calling the DeleteRouteTable operation: The routeTable 'rtb-0e9e2ff986f1d3466' has dependencies and cannot be deleted.Then delete first the subnet. First look for subnets with:
aws ec2 describe-subnets --region eu-north-1 --filters "Name=tag:Name,Values=hpc-compute-private-subnet" --query "Subnets[].SubnetId" --output textThen delete them with:
aws ec2 delete-subnet --region eu-north-1 --subnet-id subnet-xxxxxxRunning multiple jobs with multiple nodes
One interesting thing i noticed is that by default SLURM seems to prioritize jobs that can be solved with the currently available nodes. So for example if i have 10 jobs and maximum of 8 nodes, if say the first job requires 1, the second 2, the third 8 and the fourth 8, the fifth 3. Then it will run the 1,2 and 3.
Here is a log from a recent run:
ubuntu@ip-172-31-69-91:/shared$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
compute-fleet* up infinite 2 idle% compute-fleet-dy-hpc-nodes-[3-4]
compute-fleet* up infinite 3 alloc# compute-fleet-dy-hpc-nodes-[1-2,5]
compute-fleet* up infinite 3 down# compute-fleet-dy-hpc-nodes-[6-8]
ubuntu@ip-172-31-69-91:/shared$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
5 compute-f dpi_benc ubuntu CF 0:51 1 compute-fleet-dy-hpc-nodes-5
2 compute-f dpi_benc ubuntu CF 0:54 1 compute-fleet-dy-hpc-nodes-2
1 compute-f dpi_benc ubuntu CF 0:57 1 compute-fleet-dy-hpc-nodes-1
9 compute-f dpi_benc ubuntu PD 0:00 1 (Resources)
10 compute-f dpi_benc ubuntu PD 0:00 1 (Priority)
11 compute-f dpi_benc ubuntu PD 0:00 1 (Priority)
12 compute-f dpi_benc ubuntu PD 0:00 1 (Priority)