Skip to content

Commit

Permalink
Hoist rich directory rendering, add note on benchmark defaults
Browse files Browse the repository at this point in the history
This is to make it a little clearer what the lookup process for a fixture looks like.
  • Loading branch information
nicholasjng committed Dec 3, 2024
1 parent e361428 commit d581d59
Showing 1 changed file with 15 additions and 8 deletions.
23 changes: 15 additions & 8 deletions docs/cli/fixtures.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Instead, nnbench borrows a bit of pytest's fixture concept to source parameters
Suppose you have a benchmark defined in a single file, `metrics.py`:

```python
# benchmarks.py
# metrics.py
import nnbench


Expand All @@ -20,15 +20,18 @@ def accuracy(model, data):
```

To supply `model` and `data` to the benchmark, define both values as return values of similarly named functions in a `conf.py` file in the same directory.
The layout of your benchmark directory should look like this:

```python
# Benchmark directory layout:
# 📂 benchmarks
# ┣━━ conf.py
# ┣━━ metrics.py
# ┣━━ ...
# ---------------------------------------------------
```commandline
📂 benchmarks
┣━━ conf.py
┣━━ metrics.py
┣━━ ...
```

Inside your `conf.py` file, you might define your values as shown below. Note that currently, all fixtures must be raw Python callables, and must match input values of benchmarks exactly.

```python
# benchmarks/conf.py
def model():
return MyModel()
Expand All @@ -44,6 +47,10 @@ Then, nnbench will discover and auto-use these values when running this benchmar
$ nnbench run benchmarks.py
```

!!! Warning
Benchmarks with default values for their arguments will unconditionally use those defaults over potential fixtures.
That is, for a benchmark `def add(a: int, b: int = 1)`, only the named parameter `a` will be resolved.

## Fixtures with inputs

Like in pytest, fixtures can consume inputs. However, in nnbench, fixtures can consume other inputs by name only within the same module scope, i.e. members within the same `conf.py`.
Expand Down

0 comments on commit d581d59

Please sign in to comment.