Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Where can I find benchmark results? #7

Open
marcinz opened this issue Jan 31, 2022 · 6 comments
Open

Where can I find benchmark results? #7

marcinz opened this issue Jan 31, 2022 · 6 comments

Comments

@marcinz
Copy link

marcinz commented Jan 31, 2022

I am wondering if the benchmarks in this repo (or some other cupy benchmarks) are run regularly somewhere. If they are, is there a place where I can find the results?

@emcastillo
Copy link
Member

They aren't 😄

@kmaehashi
Copy link
Member

This is in our TODO list for a long time... How can we make it public? I'm thinking of:

  • For every release, run a benchmark against the wheel. Persist the raw benchmark results and rendered HTML to GCS.
  • For every pull-requests, run the benchmark against the code, then compare with the raw benchmark results in GCS, then persist rendered HTML to GCS.

@leofang
Copy link
Member

leofang commented Feb 16, 2022

@kmaehashi FYI: https://github.com/spcl/npbench

@jakirkham
Copy link
Member

Interesting hadn't heard of that one before.

Another popular one is airspeed velocity (asv), which NumPy & SciPy both use to run and show benchmarks. Here's NumPy's benchmark results.

@jakirkham
Copy link
Member

cc @pentschev (in case you have thoughts here)

@pentschev
Copy link
Member

I agree ASV is potentially the best candidate for a library like CuPy as it will provide a historic view on a per-commit basis, making it really useful to catch regressions too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants