Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentation regarding how Criterion does its measurements #95

Open
paulolieuthier opened this issue Jan 16, 2016 · 2 comments
Open

Comments

@paulolieuthier
Copy link

This is really important for people writing papers about Haskell who use Criterion to perform measurements. I'm helping a team that is using a modified version of Criterion to measure the energy consuption of benchmarks (we plan to file a pull request when it's done).

It's mainly to explain the statistical reasoning behind the runBenchmark function from Criterion.Measurement.

@rrnewton
Copy link
Member

rrnewton commented Jun 8, 2016

That sounds great. If you have suggestions or a PR, they would be welcome. We're trying to make a new wave of changes to this library, so now is a good time.

At the very least we could link some of the good blog posts that explain the linear regression methodology.

One thing I would like to clarify to users is the difference between taking the "Mean" and taking the "OLS regression". I have found that people are often not clear on that and it's not self explanatory.

Likewise the "y" and "iters" labels are hard to remember and not super accessible.

@Blaisorblade
Copy link

This is really important for people writing papers about Haskell who use Criterion to perform measurements.

+💯

One thing I would like to clarify to users is the difference between taking the "Mean" and taking the "OLS regression". I have found that people are often not clear on that and it's not self explanatory.

Also + 💯 . I have some clue about linear regression, but that makes sense if I'm plotting a dependent variable against an independent variable—which are the variables here? The dependent one should be runtime, but what's the independent variable?

Never got a clue until today. #122 suggests it's the number of iterations in a "single" measurement, something I've never seen in benchmarking frameworks. (Also, the paper cited in #122 seems to question the approach).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants