Slurm, round up num CPUs

[albertz]

Fix #229.

Example: You have rqmt["cpu"]==1. Without this change, you get SLURM_CPUS_PER_TASK=1. You also by default have SLURM_JOB_NUM_NODES=1. Due to the Slurm hyper-threading logic, it might round-up the num CPUs, i.e. you get NumCPUs=2. This results in SLURM_TASKS_PER_NODE=2.

Without the additional srun that was introduced in #212, Slurm will not really handle the SLURM_TASKS_PER_NODE, and this problem was not noticed. However, with the srun (which should always be fine and follows standard Slurm practice), this is a problem now.

This PR fixes this by avoiding that SLURM_CPUS_PER_TASK > NumCPUs.

Test this yourself: Create slurm-test-script.sh with content (example taken from the sbatch man page):

#!/bin/sh
#SBATCH --time=1
srun hostname |sort

Run sbatch --cpus-per-task=1 slurm-test-script.sh. Then check the output log file.

[michelwi]

This is on the i6 cluster.. but will it break things on the apptek cluster? Or other clusters where sisyphus might be used?
@curufinwe do we use hyper-threading?

[albertz]

This is on the i6 cluster.. but will it break things on the apptek cluster? Or other clusters where sisyphus might be used? @curufinwe do we use hyper-threading?

I don't see how this can break anything? In the worst case, you allocate one more CPU than what would be necessary?

[NeoLegends]

Agree with Albert, even if we don‘t hyperthrrad (which we likely do?) CPUs don‘t cost much fairshare normally.

[michelwi]

I don't see how this can break anything?

If we have a CPU cluster without hyper-threading and mostly use single-cpu tasks, then this updates would reduce the number of Jobs that can run in parallel by a factor of 2, would it not?
I would call this "a serious problem" if not "broken".

[michelwi]

# Use an even number of CPUs, because when we have SLURM_CPUS_PER_TASK=1 (or any uneven num),
# but Slurm rounds the NumCPUs up to account for hyper-threading (NumCPUs=2),
# this will lead to SLURM_NTASKS=2, i.e. then this job would be executed twice.

Just out of curiosity:
what would be the behavior for another uneven number? (e.g. SLURM_CPUS_PER_TASK=3)
if rounded up to 4, then both SLURM_NTASKS=2 (needs 6 cores) and SLURM_NTASKS=1 (needs 3 cores) is not correct.

[albertz]

Hm, I tested this slurm-test-script.sh above for other numbers of --cpus-per-task, and it seems, for all numbers >=2, I get only one task, it only runs it once? I don't fully understand this yet. This needs some further investigation.

[albertz]

I don't really understand it so far. Currently, it seems to me like using max(..., 2) instead of _round_up_to_even(...) would already work, but I'm not really sure that's correct.

[albertz]

As #231 is merged now, this here is obsolete.