{"id":1924,"date":"2024-12-06T12:59:00","date_gmt":"2024-12-06T12:59:00","guid":{"rendered":"https:\/\/www.nicktailor.com\/?p=1924"},"modified":"2024-12-06T13:46:26","modified_gmt":"2024-12-06T13:46:26","slug":"setting-up-slurmctld-on-ubuntu-22-04-with-troubleshooting","status":"publish","type":"post","link":"https:\/\/nicktailor.com\/tech-blog\/setting-up-slurmctld-on-ubuntu-22-04-with-troubleshooting\/","title":{"rendered":"How to setup HPC-Slurm Controller Node"},"content":{"rendered":"

Refer to Key Components for HPC Cluster Setup<\/a><\/span>; for which pieces you need to setup.<\/strong><\/p>\n

This guide provides step-by-step instructions for setting up the Slurm controller daemon (`slurmctld`) on Ubuntu 22.04.<\/strong> It also includes common errors encountered during the setup process and how to resolve them.<\/p>\n

Step 1: Install Prerequisites<\/h2>\n

To begin, install the required dependencies for Slurm and its components:<\/p>\n

sudo apt update && sudo apt upgrade -y
\nsudo apt install -y munge libmunge-dev libmunge2 build-essential man-db mariadb-server mariadb-client libmariadb-dev python3 python3-pip chrony<\/p>\n

Step 2: Configure Munge (Authentication for slurm)<\/h2>\n

Munge is required for authentication within the Slurm cluster.<\/p>\n

1. Generate a Munge key on the controller node:
\nsudo create-munge-key<\/p>\n

2. Copy the key to all compute nodes:
\nscp \/etc\/munge\/munge.key user@node:\/etc\/munge\/<\/p>\n

3. Start the Munge service:
\nsudo systemctl enable –now munge<\/p>\n

Step 3: Install Slurm<\/h2>\n

1. Download and compile Slurm:
\nwget https:\/\/download.schedmd.com\/slurm\/slurm-23.02.4.tar.bz2
\ntar -xvjf slurm-23.02.4.tar.bz2
\ncd slurm-23.02.4
\n.\/configure –prefix=\/usr\/local\/slurm –sysconfdir=\/etc\/slurm
\nmake -j$(nproc)
\nsudo make install<\/p>\n

2. Create necessary directories and set permissions:
\nsudo mkdir -p \/etc\/slurm \/var\/spool\/slurm \/var\/log\/slurm
\nsudo chown slurm: \/var\/spool\/slurm \/var\/log\/slurm<\/p>\n

3. Add the Slurm user:
\nsudo useradd -m slurm<\/p>\n

Step 4: Configure Slurm; more complex configs contact Nick Tailor<\/h2>\n

1. Generate a basic `slurm.conf` using the configurator tool at
\nhttps:\/\/slurm.schedmd.com\/configurator.html. Save the configuration to `\/etc\/slurm\/slurm.conf`.<\/p>\n

# Basic Slurm Configuration<\/p>\n

ClusterName=my_cluster<\/p>\n

ControlMachine=slurmctld # Replace with your control node’s hostname<\/p>\n

# BackupController=backup-slurmctld # Uncomment and replace if you have a backup controller<\/p>\n

.<\/span><\/span><\/p>\n

# Authentication<\/p>\n

AuthType=auth\/munge<\/p>\n

CryptoType=crypto\/munge<\/p>\n

.<\/span><\/span><\/p>\n

# Logging<\/p>\n

SlurmdLogFile=\/var\/log\/slurm\/slurmd.log<\/p>\n

SlurmctldLogFile=\/var\/log\/slurm\/slurmctld.log<\/p>\n

SlurmctldDebug=info<\/p>\n

SlurmdDebug=info<\/p>\n

.<\/span><\/span><\/p>\n

# Slurm User<\/p>\n

SlurmUser=slurm<\/p>\n

StateSaveLocation=\/var\/spool\/slurm<\/p>\n

SlurmdSpoolDir=\/var\/spool\/slurmd<\/p>\n

.<\/span><\/span><\/p>\n

# Scheduler<\/p>\n

SchedulerType=sched\/backfill<\/p>\n

SchedulerParameters=bf_continue<\/p>\n

.<\/span><\/span><\/p>\n

# Accounting<\/p>\n

AccountingStorageType=accounting_storage\/none<\/p>\n

JobAcctGatherType=jobacct_gather\/linux<\/p>\n

.<\/span><\/span><\/p>\n

# Compute Nodes<\/p>\n

NodeName=node[1-2] CPUs=4 RealMemory=8192 State=UNKNOWN<\/p>\n

PartitionName=debug Nodes=node[1-2] Default=YES MaxTime=INFINITE State=UP<\/p>\n

2. Distribute `slurm.conf` to all compute nodes:
\nscp \/etc\/slurm\/slurm.conf user@node:\/etc\/slurm\/<\/p>\n

3. Restart Slurm services:
\nsudo systemctl restart slurmctld
\nsudo systemctl restart slurmd<\/p>\n

Troubleshooting Common Errors<\/h2>\n

.<\/span><\/span><\/p>\n

root@slrmcltd:~# tail \/var\/log\/slurm\/slurmctld.log<\/p>\n

[2024-12-06T11:57:25.428] error: High latency for 1000 calls to gettimeofday(): 20012 microseconds<\/p>\n

[2024-12-06T11:57:25.431] fatal: mkdir(\/var\/spool\/slurm): Permission denied<\/p>\n

[2024-12-06T11:58:34.862] error: High latency for 1000 calls to gettimeofday(): 20029 microseconds<\/p>\n

[2024-12-06T11:58:34.864] fatal: mkdir(\/var\/spool\/slurm): Permission denied<\/p>\n

[2024-12-06T11:59:38.843] error: High latency for 1000 calls to gettimeofday(): 18842 microseconds<\/p>\n

[2024-12-06T11:59:38.847] fatal: mkdir(\/var\/spool\/slurm): Permission denied<\/p>\n

Error: Permission Denied for \/var\/spool\/slurm<\/h3>\n

This error occurs when the `slurm` user does not have the correct permissions to access the directory.<\/p>\n

Fix:<\/span>
\nsudo<\/span> mkdir<\/span> -p \/var\/spool\/<\/span>slurm<\/span>
\nsudo<\/span> chown<\/span> -R <\/span>slurm<\/span>: \/var\/spool\/<\/span>slurm<\/span>
\nsudo<\/span> chmod<\/span> -R 755 \/var\/spool\/<\/span>slurm<\/span><\/p>\n

Error: Temporary Failure in Name Resolution<\/h3>\n

Slurm could not resolve the hostname `slurmctld`. This can be fixed by updating `\/etc\/hosts`:<\/p>\n

1. Edit `\/etc\/hosts` and add the following:<\/span>
\n127.0.0.1 <\/span>slurmctld<\/span>
\n192.168.20.8 <\/span>slurmctld<\/span><\/p>\n

2. Verify the hostname matches `ControlMachine` in `\/etc\/slurm\/slurm.conf`.<\/p>\n

3. Restart networking and test hostname resolution:
\nsudo systemctl restart systemd-networkd
\nping slurmctld<\/p>\n

Error: High Latency for gettimeofday()<\/h3>\n

Dec 06 11:57:25 slrmcltd.home systemd[1]: Started Slurm controller daemon.<\/p>\n

Dec 06 11:57:25 slrmcltd.home slurmctld[2619]: slurmctld: error: High latency for 1000 calls to gettimeofday(): 20012 microseconds<\/p>\n

Dec 06 11:57:25 slrmcltd.home systemd[1]: slurmctld.service: Main process exited, code=exited, status=1\/FAILURE<\/p>\n

Dec 06 11:57:25 slrmcltd.home systemd[1]: slurmctld.service: Failed with result ‘exit-code’.<\/p>\n

This warning typically indicates timing issues in the system.<\/p>\n

Fixes:<\/span>
\n1. Install and configure `<\/span>chrony<\/span>` for time synchronization:<\/span>
\n<\/span>sudo<\/span> apt install <\/span>chrony<\/span>
\n<\/span>sudo<\/span> systemctl<\/span> enable –now <\/span>chrony<\/span>
\n\u00a0\u00a0\u00a0<\/span>chronyc<\/span> tracking<\/span>
\n<\/span>timedatectl<\/span>
\n2. For virtualized environments, optimize the clocksource:
\nsudo echo tsc > \/sys\/devices\/system\/clocksource\/clocksource0\/current_clocksource<\/p>\n

3. Disable high-precision timing in `slurm.conf` (optional):
\nHighPrecisionTimer=NO
\nsudo systemctl restart slurmctld<\/p>\n

Step 5: Verify and Test the Setup<\/h2>\n

1. Validate the configuration:
\nscontrol reconfigure
\n– no errors mean its working. If this doesn\u2019t work check the connection between nodes
\nupdate your \/etc\/hosts to have the hosts all listed across the all machines and nodes.<\/p>\n

2. Check node and partition status:
\nsinfo<\/p>\n

root@slrmcltd:\/etc\/slurm# sinfo<\/p>\n

PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<\/p>\n

debug* up infinite 1 idle* node1<\/p>\n

3. Monitor logs for errors:
\nsudo tail -f \/var\/log\/slurm\/slurmctld.log<\/p>\n

.<\/span><\/span><\/p>\n

Written By: Nick Tailor<\/p>\n","protected":false},"excerpt":{"rendered":"

Refer to Key Components for HPC Cluster Setup; for which pieces you need to setup. This guide provides step-by-step instructions for setting up the Slurm controller daemon (`slurmctld`) on Ubuntu 22.04. It also includes common errors encountered during the setup process and how to resolve them. Step 1: Install Prerequisites To begin, install the required dependencies for Slurm and itsRead More …<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[143],"tags":[],"class_list":["post-1924","post","type-post","status-publish","format-standard","hentry","category-hpc"],"_links":{"self":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts\/1924","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/comments?post=1924"}],"version-history":[{"count":9,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts\/1924\/revisions"}],"predecessor-version":[{"id":1941,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts\/1924\/revisions\/1941"}],"wp:attachment":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/media?parent=1924"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/categories?post=1924"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/tags?post=1924"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}