Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add instructions for consistent benchmarks #26

Merged
merged 1 commit into from
Sep 11, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,16 @@ make bench_result/image/dgemm_performance.png bench_result/image/dgemm_performan

You are welcome to contribute by organizing the existing benchmark, writing more benchmarks, and writing scripts to visualize the results.

### Making benchmarks give more consistent results

So far, we haven't been able to make the benchmarks give always the same results on the same processor. However, what we tried so far is :

- isolating CPU 11 by adding 'nohz_full=5,11 isolcpus=domain,managed_irq,5,11 irqaffinity=0-4,6-10' to the boot parameters (see https://manuel.bernhardt.io/posts/2023-11-16-core-pinning/)
- using taskset -c 11 to then run the benchmark only on that core
- use performance governor (see https://google.github.io/benchmark/reducing_variance.html)
- turn off Intel Boost (see https://llvm.org/docs/Benchmarking.html)
- use benchmark repetitions, random interleaving, warmup time (these are options of Google benchmark)

## Docker image
To automate building and running benchmarks, a [Dockerfile](Dockerfile) is provided. Use the following commands to build a Docker image and run benchmarks in a Docker container:
```bash
Expand Down
Loading