Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Port HTML report to chart.js #229

Merged
merged 1 commit into from
Nov 14, 2020
Merged

Conversation

jonascarpay
Copy link

@jonascarpay jonascarpay commented Nov 6, 2020

Originally intended to close #213, has turned into a rewrite using chart.js.

fibber.html
report.html

Feedback, criticism, feature requests welcome!

To do

  • Report/template

    • better error bars
    • Colophon
    • Explanation
    • Outlier info
    • test with different data sets
    • (optional?) legends
    • strut 0-width bars in overview
    • fix tooltip on scatter plot
    • (collapsible) table of contents?
    • general code cleanup
    • git history cleanup
    • make titles links
    • things are too bold
    • link style more subtle
    • overview fixed height, responsive width
    • stack KDE and scatter at small widths
    • make tick display more robust
    • move example chart to www/report.html
    • match precision with old report
    • change cursor when hovering y-axis
    • change sort order
      • interface?
      • report order
      • alphabetical order
      • lacitebahpla order
      • time order
  • Haskell+packaging

    • Drop js-flot, js-jquery
    • Include chart.js
      • make js-chart?
      • EMBED flag
    • split out custom JS+CSS?

@considerate @cartazio

@RyanGlScott
Copy link
Member

Hooray! It looks like this is still a draft, so let me know when this is ready for a full review.

@jonascarpay
Copy link
Author

jonascarpay commented Nov 12, 2020

This is ready for review. The fact that it's one commit is a bit unfortunate, but since most work happened in large files that have since disappeared there is not really a nice way to make it more atomic.

@RyanGlScott
Copy link
Member

Awesome! I'll do a more thorough review of this patch over the weekend, but a quick before-and-after comparison makes me notice two things right off the bat:

  • The legends appear to be missing after your changes. Is this intentional?
  • After your changes, it is no longer possible to manually zoom in/out or pan horizontally on the overview chart. I'm guessing this is because this functionality has been replaced by the ability to click on an individual benchmark and have the chart automatically zoom in/out accordingly? If so, perhaps that's a reasonable justification for removing this functionality, although I wonder if certain people relied on this.

Some other random thoughts:

  • There are some not-so-obviously discoverable features in the new chart engine, such as the ability to toggle between linear and log scales by clicking on the x-axis. I wonder if it would be worth describing these features in prose somewhere on the HTML page itself?
  • I'm also not clear on what some of the new drop-down menu options (e.g., colex) mean. These might be worth spelling out.
  • I'm not sure if it's just my web browser, but trying to hover my mouse over the yellow trendlines in the per-benchmark reports feels a lot more finicky after your changes, in the sense that the timing information in a tooltip won't appear unless my mouse is directly over the line. In contrast, with today's criterion, your mouse doesn't have to be directly over the trendline in order for the tooltip to appear; as long as it is close enough to the trendline, the tooltip will surface. Is it possible to increase the "hitbox" of the trendline so that it mimics the behavior of today's criterion?

@jonascarpay
Copy link
Author

jonascarpay commented Nov 13, 2020

After your changes, it is no longer possible to manually zoom in/out or pan horizontally on the overview chart.

Yeah, zooming was left out intentionally. I figured the primary use case was to see bars that got dwarfed by others, which I felt the click-to-focus was a better interface for. It's possible to do zooming, but it also requires a disproportionate amount of code to get right. What do you think?

There are some not-so-obviously discoverable features in the new chart engine, such as the ability to toggle between linear and log scales by clicking on the x-axis.

Makes sense, I added a note.

I'm also not clear on what some of the new drop-down menu options (e.g., colex) mean. These might be worth spelling out.

Both lexical and colex sort by group, with lexical the head is most significant and with colex the tail is most significant (i.e. big-endian vs little-endian). I suppose "by group" would be a better name for lexical, do you know a good, self-evident name for colex?

trying to hover my mouse over the yellow trendlines in the per-benchmark reports feels a lot more finicky after your changes

414911b expanded the radius, do you think it's OK now?

@considerate
Copy link
Contributor

considerate commented Nov 13, 2020

@RyanGlScott Note that report.html hasn’t been regenerated with the updated radius for the hit boxes yet so to test the tooltips you would have to generate a new report.

@considerate
Copy link
Contributor

@jonascarpay @RyanGlScott I added titles to the kernel density estimates and scatter plots to provide the same amount of context as the previous version for those plots.

I think it might somewhat redundant and would personally like to keep the output as free from noise as possible. What do you think?

@jonascarpay jonascarpay marked this pull request as draft November 13, 2020 11:51
Copy link
Member

@RyanGlScott RyanGlScott left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for looking into the legend issue. I'm now reminded of the existence of #207, however, where people specifically complained that legends were making their charts unreadable. In light of this, I'm wondering if I should take back my previous suggestion to reinstate the titles. Or, at the very least, allow one to toggle whether the legend appears or not with a mouse click. What do you think?

Yeah, zooming was left out intentionally. I figured the primary use case was to see bars that got dwarfed by others, which I felt the click-to-focus was a better interface for. It's possible to do zooming, but it also requires a disproportionate amount of code to get right. What do you think?

Upon further thought, I agree with your assessment that click-to-zoom is a better interface for this.

Both lexical and colex sort by group, with lexical the head is most significant and with colex the tail is most significant (i.e. big-endian vs little-endian). I suppose "by group" would be a better name for lexical, do you know a good, self-evident name for colex?

Ah, "colex" stands for "colexical"! That's illuminating, especially since I couldn't even find what colex meant in a search engine. At the very least, I would recommend expanding "colex" to "colexical". I would also recommend spelling out what each of these options mean in the "want to understand this report?" section.

414911b expanded the radius, do you think it's OK now?

Much better, thanks!

I added titles to the kernel density estimates and scatter plots to provide the same amount of context as the previous version for those plots.

I think it might somewhat redundant and would personally like to keep the output as free from noise as possible. What do you think?

Personally, I'm a fan of the titles and labels, since it makes the information more self-contained. On the other hand, the y-axis labels do cause the actual charts themselves to shrink a fair bit in order to accommodate the new text, so I can definitely understand your reservations there. Ultimately, I think this comes down to a judgment call about optimizing the amount of information displayed versus the size of the chart. Given that the axis labels are implied by the titles themselves, I think I would be fine with omitting them.

templates/default.tpl Outdated Show resolved Hide resolved
templates/default.tpl Outdated Show resolved Hide resolved
templates/criterion.js Outdated Show resolved Hide resolved
@RyanGlScott
Copy link
Member

I've timed out for now, but this is looking very promising. More review to come later!

@cartazio
Copy link

Is there a preview link that works on mobile? :))

@jonascarpay
Copy link
Author

jonascarpay commented Nov 13, 2020

Alright, that should do it for the feedback up to now.

@cartazio Refined Github gives me this preview link, I think that should work. Most new features work best on reports with a bigger spread of orders of magnitude though, but it'll do. Also, it does look like on mobile part of the overview chart gets eaten, will look into that.

@jonascarpay jonascarpay marked this pull request as ready for review November 13, 2020 15:16
@cartazio
Copy link

cartazio commented Nov 13, 2020 via email

@jonascarpay
Copy link
Author

One good thing to test is having an example plot where there’s a huge
range of magnitudes of values.

This is the benchmark suite we've used for that purpose before

One issue with the previous was some sort of huge algorithmic penalty for actually rendering some of the plots.

Hmm, I don't remember there being any such algorithm in the old code. Do you remember how that issue would manifest?

Was that part of the issues with the old flot code?

No, the issue was a choice between a) the stable version that didn't support things like log axes and b) the new version that was unstable in every sense of the word

@considerate
Copy link
Contributor

One issue with the previous was some sort of huge algorithmic penalty for
actually rendering some of the plots. Was that part of the issues with the
old flot code?

Without running any proper benchmarks (ironic, isn't it?) generating the same charts took much longer with flot than with chart.js. That said, it still takes a non-trivial amount of time to render the benchmark link posted above on mobile.

templates/default.tpl Outdated Show resolved Hide resolved
templates/default.tpl Outdated Show resolved Hide resolved
Criterion/Report.hs Outdated Show resolved Hide resolved
Criterion/Report.hs Show resolved Hide resolved
templates/criterion.js Show resolved Hide resolved
templates/default.tpl Outdated Show resolved Hide resolved
templates/default.tpl Outdated Show resolved Hide resolved
@considerate
Copy link
Contributor

considerate commented Nov 14, 2020

colexical order sorts groups in descending alphabetical order

I'm not sure I agree that this description is correct, the groups are sorted in ascending alphabetical order but we split the report name on '/' and compare the two right-most group elements first with (the default) ascending alphabetical string comparison, then if equal the group elements to the left of those are compared until one of the elements are unequal. If all elements are equal the shorter array is sorted before the longer one.

This image provides a visualization of the lexical and colexical orderings. The right-most 1s are placed before the right-most 5s in colexical order.

@RyanGlScott
Copy link
Member

This image provides a visualization of the lexical and colexical orderings. The right-most 1s are placed before the right-most 5s in colexical order.

OK. My attempts at succinctly characterizing this have failed, so perhaps it's best that I stop trying. :)

In that case, I'm content with the current description, but include a link to the Wikipedia article on lexicographic ordering you mentioned above, since that would be valuable context to have.

@RyanGlScott
Copy link
Member

One last request: do you mind writing up a description of the changes from this patch in the changelog.md file? There have been quite a few changes (both in terms of new functionality and UI differences), and it would be good to advertise them prominently. Thanks!

@considerate
Copy link
Contributor

One last request: do you mind writing up a description of the changes from this patch in the changelog.md file? There have been quite a few changes (both in terms of new functionality and UI differences), and it would be good to advertise them prominently. Thanks!

Yes, of course, I'll co-ordinate with @jonascarpay and describe the changes in the changelog either today or tomorrow.

@jonascarpay
Copy link
Author

Alright, that's the Changelog. Do you want me to squash the history?

@RyanGlScott
Copy link
Member

Alright, that's the Changelog. Do you want me to squash the history?

Please do. And thanks again for all of your combined work on this!

3249df7 add chart.js thing
a806dfd Output table
424d0f7 WIP
4b72393 Add colophon and styling
ec94eba Properly format scatter tooltip
60fdba8 Add outliers
d51ffeb Add grokularation
9a8b08e Minimum bar width
0ca856b Remove stray console logs
73d573b Implement animating error bars
80da94a Replace annotation plugin with custom function
9be269c Fix groups and regression table
1c1b66a Make overview ticks robust
56a9375 Render 3 significant digits
f4afc21 Update overview tick rendering
2886337 Add endpoint tick on logarithmic scale if within one order
efd3b6b Fix hover on y-axis
c543d8d Split chart.html into separate JS and CSS files
3f6d9a7 Update to use js-chart
c14274c Remove backup file
0674754 Remove unused jQuery plugin
930da09 Update www/ with new reports
d29acdd Flatten templates/js/
21bebd8 Add groups field to report
fda1377 Make drawErrorBar scope
b65e7db Comment on criterion.js functions
9e719c8 Re-render report.html
2ad7928 cleanup
414911b Expand hover hit radius for points
543778c Add explanatory note about interactivity
d6d5ce9 Clip error bars to chart area
0547c42 Use chart.js helper to get chart position
ed2a680 Add legend that hides groups
7142270 Constrain legend box width
2eec914 Add titles and axes labels
64d9076 Don't display labels
910573b Fix "a" in outlier information
5c26f2c Fix wording
ce402c4 Cater explanation depending on screen or print
8e501fe Make legend toggle-able
8da2146 Controls explanation and legend chevron
cd2dd68 Avoid page breaks inside single report
7fc20ac Update grokularation with no-prints and scatter plot CI explanation
fe49774 Colophon link color in printed version
fc8690e Re-generate report.html and fibber.html
2d00dcd Render overview labels over bars on small screens
1ba261e Base error bars on bar index
271bbcf Remove `report` from JS context
446e2a6 Link to Wikipedia for (co)lexical order
198f1ff Update changelog for new HTML reports
@jonascarpay
Copy link
Author

Done, thank you as well!

@cartazio
Copy link

cartazio commented Nov 14, 2020 via email

@RyanGlScott RyanGlScott merged commit 50eadc5 into haskell:master Nov 14, 2020
@RyanGlScott
Copy link
Member

I've uploaded criterion-1.5.8.0 to Hackage with these changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Logarithmic scale when plotting results
4 participants