Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci(l2): automate flamegraph tests #1366

Open
wants to merge 34 commits into
base: main
Choose a base branch
from
Open

Conversation

dsocolobsky
Copy link
Contributor

@dsocolobsky dsocolobsky commented Dec 2, 2024

Motivation

We want to generate flamegraphs both for ethrex and for reth on each push to main, and then be able to view them in github pages or similar.

You can see the flamegraphs at https://lambdaclass.github.io/ethrex/

Description

  • Github workflow for generating the flamegraphs and deploying the .svg files to Github Pages
  • The ethrex_l2CLI was modified to be able to create a default config non-interactively with ethrex_l2 config create default --default since the interactive CLI was not working in Github CI.
  • ethrex_l2 test load was modified so that it now retries when it can't connect to the server instead of failing directly, this was needed to run the test in Github CI.
  • test_data/genesis-load-test.json was modified to add balance to a test account, since the tests were failing sometimes otherwise.
  • You can run make flamegraph locally to run the same scripts locally to generate the Ethrex flamegraphs (not the Reth ones).

Notes

  • I had to run the perf record and then inferno-collapse-perf plus inferno-flamegraph separately because of a bug in perf with Github CI where it would output garbage to the stdout instead of the actual perf data to a perf.data file.

Things that we should fix/check

  • The calculated elapsed time is sometimes very short, so maybe we're calculating it wrong.
  • Make sure the flamegraphs are being correctly generated, that the symbols are resolved properly. We saw some weird things where some symbols were duplicated.
  • We are also calling perf script with the --no-inline option, since it takes a very long time without it, maybe we're not getting the proper data because of that.

Other possible improvements

  • Reth takes a long time to compile in the CI, perhaps it can be cached or uploaded/downloaded as a binary since we only use it to run the perf test.
  • Check if more things can be cached.
  • Output should be more quiet since the CI logs are running very long and it's hard to debug.
  • Maybe the web can be improved, We can generate .png files as preview since the .svg look cropped there.
  • The Slack message can be improved by including a preview.
  • scripts/flamegraph.sh and .github/scripts/flamegraph_watcher.sh and very similar and they could be merged into one, however I hardcoded some CI-paths in the latter one.

@dsocolobsky dsocolobsky force-pushed the automate-perf-tests branch 5 times, most recently from 4b9b05a to 45b74ad Compare December 2, 2024 14:03
@dsocolobsky dsocolobsky force-pushed the automate-perf-tests branch 3 times, most recently from 8df6f77 to 56dccda Compare December 2, 2024 15:49
@dsocolobsky dsocolobsky force-pushed the automate-perf-tests branch 17 times, most recently from d6742e6 to 2d9e8fd Compare December 2, 2024 20:29
@@ -0,0 +1,209 @@
name: Daily Flamegraph Reporter
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Not sure if it's daily though, I forgot I had named it like that but I think it's supposed to run in each merge to master; I will ask Javi/Ivan.

* More quiet output in flamegraph commands
* Remove some debug prints/ls/etc.
* Wait more time for reth to compile before testing
@dsocolobsky dsocolobsky force-pushed the automate-perf-tests branch 2 times, most recently from fc0c167 to d79edc9 Compare December 6, 2024 17:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants