Slurm - XWiki

1

SLURM is the Simple Linux Utility for Resource Management and is an open source, fault-tolerant, and highly scalable cluster management and job scheduling system for large and small Linux clusters.

2

3

Slurm is fully integrated in our system. You do not need set any environment variables.

== Partitions ==

A partition is a subset of the cluster, a bundle of compute nodes with the same characteristics.

9

10

Based on access restrictions our cluster is divided in different partitions. 'sinfo' will only show partitions you are allowed to use. Using 'sinfo -a' shows all partitons.

11

12

A partition is selected by '-p PARTITIONNAME'.

13

14

{| border="1" align="center"

15

|-

16

! scope="col" | '''Partition'''

17

! scope="col" | '''No. Nodes'''

18

! scope="col" | '''Cores/M'''

19

! scope="col" | '''Tot. Cores'''

20

! scope="col" | '''RAM/GB'''

21

! scope="col" | '''CPU'''

22

! scope="col" | '''Remark/Restriction'''

|-

| itp

| 10

| 20

| 200

| 64

| Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz

|

|-

| dfg-big

| 3

| 32

| 96

| 128

| 8-Core AMD Opteron(tm) Processor 6128

| Group Valenti

|-

| dfg-big

| 3

| 48

| 144

| 128/256

| 12-Core AMD Opteron(tm) Processor 6168

| Group Valenti

|-

| dfg-big

| 4

| 64

| 256

| 128/256

| 16-Core AMD Opteron(tm) Processor 6272

| Group Valenti

|-

| dfg-big

| 4

| 48

| 192

| 128/256

| 12-Core AMD Opteron(tm) Processor 6344

| Group Valenti

|-

| fplo

| 2

| 12

| 24

| 256

| Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz

| Group Valenti

|-

| fplo

| 4

| 16

| 32

| 256

| Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz

| Group Valenti

|-

| dfg-xeon

| 5

| 16

| 32

| 128

| Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz

| Group Valenti

|-

| dfg-xeon

| 7

| 20

| 140

| 128

| Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz

| Group Valenti

|-

| iboga

| 44

| 20

| 880

| 64

| Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz

| Group Rezzolla

|-

| dreama

| 1

| 40

| 40

| 1024

| Intel(R) Xeon(R) CPU E7-4820 v3 @ 1.90GHz

| Group Rezzolla

|-

| barcelona

| 8

| 40

| 320

| 192

| Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz

| Group Valenti

|}

Most nodes are for exclusive use by their corresponding owners. The itp and itp-big nodes are for common usage. Except for 'fplo' and 'dfg-big' nodes, all machines are connected with Infiniband for all traffic (IP and internode communitcation - MPI)

123

124

== Submitting Jobs ==

125

126

In most cases you want to submit a non interactive job to be executed in our cluster.

127

128

This is very simple for serial (1 CPU) jobs:

129

130

sbatch -p PARTITION jobscript.sh

131

132

where jobscript.sh is a shell script with your job commands.

133

134

Running '''openMPI''' jobs is not much more complictated:

135

136

sbatch -p PARTITION -n X jobscript.sh

137

138

where X is the number of desired MPI processes. Launch the job in the jobscript with:

139

140

mpirun YOUREXECUTABLE

141

142

You don't have to worry about the number of processes or specific nodes. Both slurm and openmpi know

143

about each other.

144

145

If you want '''infiniband''' for your MPI job (which is usually a good idea, if not running on the same node), you have to request the feature infiniband:

146

147

sbatch -p dfg -C infiniband -n X jobscript.sh

148

149

Note: Infiniband is not available for 'fplo' and 'dfg-big'.

150

151

Running '''SMP jobs''' (multiple threads, not necessary mpi). Running MPI jobs on a single node is recommended for the

152

dfg-big nodes. This are big host with up to 64 cpu's per node, but 'slow' gigabit network connection. Launch SMP jobs with

153

154

sbatch -p PARTITION -N 1 -n X jobscript.sh

155

156

=== Differences in network the network connection ===

157

158

The new v3 dfg-xeon nodes are equipped with 10 GB network. This is faster (trough put) and has lower latency then gigabit ethernet, but is not is not as fast as the DDR infinband network. The 10 GB network is used for MPI and I/O. Infiniband is only use for MPI.

159

160

== Defining Resource limits ==

161

162

By default each job allocates 2 GB memory and a run time of 3 days. More resources can be requested by

--mem-per-cpu=<MB>

where <MB> is the memory in megabytes. The virtual memory limit is 2.5 times of the requested real memory limit.

167

168

The memory limit is not a hard limit. When exceeding the limit, your memory will be swapped out. Only when using more the 150% of the limit your job will be killed. So be conservative, to keep enough room for other jobs. Requested memory is blocked from the use by other jobs.

-t or --time=<time>

where time can be set in the format "days-hours". See man page for more formats.

173

174

== Memory Management ==

175

176

In Slurm you specify only one parameter, which is the limit for your real memory usage and drives the decision where your job is started. The virtual memory of your job maybe 2.5 times of your requested memory. You can exceed your memory limit by 20%. But this will be swap space instead of real memory. This prevents crashing if you memory limit is a little to tight.

177

178

== Inline Arguments ==

179

180

sbatch arguments can be written in the jobfile:

<pre>

#! /bin/bash