You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I would to offer at least a partial parametrization of benchmarks via fixtures, meaning that we during parameter hydration in the main benchmark loop, we resolve fixture values and place them inside the parameters.
Fixtures should also be sourced from a Python config file like conftest.py. For simplicity, we can do the first iteration with a single, global file.
Similarly, scopes do not have to be supported immediately, maybe a session-wide scope works for now.
Now, running with nnbench benchmark.py should make nnbench go into conf.py, instantiate the fixture values, and add them to the parameters.
Requirements:
Add a fixture decorator wrapping a callable, import into top-level package as nnbench.fixture.
In the benchmark loop, import from hard-coded path, either conf.py if the input path to nnbench run <path> is a file, or dir/conf.py if the input is a directory.
Give an example on advanced configuration in fixtures, maybe via envvars.
Document the fixture facility.
The text was updated successfully, but these errors were encountered:
Instead of the current solution of parametrization via default arguments:
I would to offer at least a partial parametrization of benchmarks via fixtures, meaning that we during parameter hydration in the main benchmark loop, we resolve fixture values and place them inside the parameters.
Fixtures should also be sourced from a Python config file like conftest.py. For simplicity, we can do the first iteration with a single, global file.
Similarly, scopes do not have to be supported immediately, maybe a session-wide scope works for now.
Design:
Now, running with
nnbench benchmark.py
should make nnbench go into conf.py, instantiate the fixture values, and add them to the parameters.Requirements:
nnbench.fixture
.conf.py
if the input path tonnbench run <path>
is a file, ordir/conf.py
if the input is a directory.The text was updated successfully, but these errors were encountered: