From 0ad8340af82c102bc379181893d4a23d965175e1 Mon Sep 17 00:00:00 2001 From: roversch Date: Tue, 13 Aug 2024 10:09:10 +0200 Subject: [PATCH] Add instructions for consistent benchmarks --- README.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/README.md b/README.md index 67226b39..9f967b77 100644 --- a/README.md +++ b/README.md @@ -95,6 +95,16 @@ make bench_result/image/dgemm_performance.png bench_result/image/dgemm_performan You are welcome to contribute by organizing the existing benchmark, writing more benchmarks, and writing scripts to visualize the results. +### Making benchmarks give more consistent results + +So far, we haven't been able to make the benchmarks give always the same results on the same processor. However, what we tried so far is : + +- isolating CPU 11 by adding 'nohz_full=5,11 isolcpus=domain,managed_irq,5,11 irqaffinity=0-4,6-10' to the boot parameters (see https://manuel.bernhardt.io/posts/2023-11-16-core-pinning/) +- using taskset -c 11 to then run the benchmark only on that core +- use performance governor (see https://google.github.io/benchmark/reducing_variance.html) +- turn off Intel Boost (see https://llvm.org/docs/Benchmarking.html) +- use benchmark repetitions, random interleaving, warmup time (these are options of Google benchmark) + ## Docker image To automate building and running benchmarks, a [Dockerfile](Dockerfile) is provided. Use the following commands to build a Docker image and run benchmarks in a Docker container: ```bash