Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add more dict benches #977

Merged
merged 8 commits into from
Dec 19, 2024
Merged

Add more dict benches #977

merged 8 commits into from
Dec 19, 2024

Conversation

JulianGCalderon
Copy link
Contributor

@JulianGCalderon JulianGCalderon commented Dec 16, 2024

  • Adds two more benchmark programs:
    • dict_insert: Inserts a lot of distinct values on a dictionary
    • dict_snapshot: Takes a snapshot of a filled dictionary a lot of times
  • Refactor the benches.rs to make it easier to add new benches, by calling the compare function.
  • I found that some of the older benches were not sending the correct number of parameters, so I fixed that also.

Checklist

  • Linked to Github Issue
  • Unit tests added
  • Integration tests added.
  • This change requires new documentation.
    • Documentation has been added/updated.

Copy link

github-actions bot commented Dec 16, 2024

Benchmark results Main vs HEAD.

Command Mean [s] Min [s] Max [s] Relative
base factorial_2M.cairo (JIT) 4.124 ± 0.053 4.053 4.243 1.00
base factorial_2M.cairo (AOT) 4.295 ± 0.057 4.239 4.418 1.04 ± 0.02
Command Mean [s] Min [s] Max [s] Relative
:--- ---: ---: ---: ---:
head factorial_2M.cairo (JIT) 4.022 ± 0.020 3.973 4.045 1.00
head factorial_2M.cairo (AOT) 4.225 ± 0.027 4.186 4.280 1.05 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
base fib_2M.cairo (JIT) 3.608 ± 0.040 3.546 3.678 1.00
base fib_2M.cairo (AOT) 3.790 ± 0.018 3.759 3.818 1.05 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
:--- ---: ---: ---: ---:
head fib_2M.cairo (JIT) 3.573 ± 0.023 3.529 3.596 1.00
head fib_2M.cairo (AOT) 3.776 ± 0.015 3.754 3.807 1.06 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
base linear_search.cairo (JIT) 3.800 ± 0.026 3.762 3.841 1.00
base linear_search.cairo (AOT) 3.906 ± 0.025 3.872 3.953 1.03 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
:--- ---: ---: ---: ---:
head linear_search.cairo (JIT) 3.760 ± 0.017 3.738 3.780 1.00
head linear_search.cairo (AOT) 3.870 ± 0.030 3.819 3.917 1.03 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
base logistic_map.cairo (JIT) 3.925 ± 0.036 3.878 3.973 1.00
base logistic_map.cairo (AOT) 4.008 ± 0.020 3.973 4.041 1.02 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
:--- ---: ---: ---: ---:
head logistic_map.cairo (JIT) 3.870 ± 0.027 3.822 3.905 1.00
head logistic_map.cairo (AOT) 3.934 ± 0.018 3.910 3.973 1.02 ± 0.01

@codecov-commenter
Copy link

codecov-commenter commented Dec 16, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.76%. Comparing base (e47a42e) to head (fa43b91).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #977   +/-   ##
=======================================
  Coverage   80.76%   80.76%           
=======================================
  Files         107      107           
  Lines       29256    29256           
=======================================
  Hits        23628    23628           
  Misses       5628     5628           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link

github-actions bot commented Dec 17, 2024

Benchmarking results

Benchmark for program dict_insert

Open benchmarks
Command Mean [s] Min [s] Max [s] Relative
Cairo-vm (Rust, Cairo 1) 21.019 ± 0.155 20.811 21.370 86.95 ± 1.61
cairo-native (embedded AOT) 4.197 ± 0.051 4.124 4.289 17.36 ± 0.36
cairo-native (embedded JIT using LLVM's ORC Engine) 3.970 ± 0.050 3.880 4.025 16.42 ± 0.35
cairo-native (standalone AOT with -march=native) 0.242 ± 0.004 0.235 0.248 1.00

Benchmark for program dict_snapshot

Open benchmarks
Command Mean [s] Min [s] Max [s] Relative
Cairo-vm (Rust, Cairo 1) 5.615 ± 0.016 5.601 5.645 17.73 ± 0.24
cairo-native (embedded AOT) 4.073 ± 0.024 4.045 4.113 12.86 ± 0.19
cairo-native (embedded JIT using LLVM's ORC Engine) 3.981 ± 0.039 3.932 4.049 12.57 ± 0.21
cairo-native (standalone AOT with -march=native) 0.317 ± 0.004 0.314 0.325 1.00

Benchmark for program factorial_2M

Open benchmarks
Command Mean [s] Min [s] Max [s] Relative
Cairo-vm (Rust, Cairo 1) 14.182 ± 0.094 14.009 14.347 10.51 ± 0.07
cairo-native (embedded AOT) 4.283 ± 0.050 4.211 4.369 3.17 ± 0.04
cairo-native (embedded JIT using LLVM's ORC Engine) 4.013 ± 0.038 3.955 4.068 2.97 ± 0.03
cairo-native (standalone AOT with -march=native) 1.350 ± 0.002 1.347 1.355 1.00

Benchmark for program fib_2M

Open benchmarks
Command Mean [s] Min [s] Max [s] Relative
Cairo-vm (Rust, Cairo 1) 14.622 ± 0.137 14.374 14.860 186.94 ± 1.86
cairo-native (embedded AOT) 3.808 ± 0.073 3.702 3.942 48.69 ± 0.95
cairo-native (embedded JIT using LLVM's ORC Engine) 3.845 ± 0.047 3.795 3.944 49.15 ± 0.63
cairo-native (standalone AOT with -march=native) 0.078 ± 0.000 0.078 0.079 1.00

Benchmark for program linear_search

Open benchmarks
Command Mean [s] Min [s] Max [s] Relative
Cairo-vm (Rust, Cairo 1) 5.882 ± 0.064 5.786 5.985 2943.88 ± 70.23
cairo-native (embedded AOT) 4.083 ± 0.045 4.009 4.124 2043.52 ± 48.82
cairo-native (embedded JIT using LLVM's ORC Engine) 3.903 ± 0.031 3.864 3.946 1953.75 ± 44.30
cairo-native (standalone AOT with -march=native) 0.002 ± 0.000 0.002 0.002 1.00

Benchmark for program logistic_map

Open benchmarks
Command Mean [s] Min [s] Max [s] Relative
Cairo-vm (Rust, Cairo 1) 5.666 ± 0.052 5.574 5.717 23.35 ± 0.22
cairo-native (embedded AOT) 4.057 ± 0.048 3.998 4.150 16.72 ± 0.20
cairo-native (embedded JIT using LLVM's ORC Engine) 3.991 ± 0.056 3.918 4.075 16.45 ± 0.23
cairo-native (standalone AOT with -march=native) 0.243 ± 0.000 0.242 0.244 1.00

programs/benches/dict_snapshot.c Show resolved Hide resolved
azteca1998
azteca1998 previously approved these changes Dec 18, 2024
edg-l
edg-l previously approved these changes Dec 18, 2024
@edg-l edg-l added this pull request to the merge queue Dec 18, 2024
@edg-l edg-l removed this pull request from the merge queue due to a manual request Dec 18, 2024
@JulianGCalderon JulianGCalderon marked this pull request as draft December 18, 2024 14:32
@JulianGCalderon JulianGCalderon marked this pull request as ready for review December 18, 2024 17:54
azteca1998
azteca1998 previously approved these changes Dec 18, 2024
edg-l
edg-l previously approved these changes Dec 18, 2024
@edg-l edg-l added this pull request to the merge queue Dec 18, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to a conflict with the base branch Dec 18, 2024
@gabrielbosio gabrielbosio added this pull request to the merge queue Dec 19, 2024
Merged via the queue into main with commit f53c947 Dec 19, 2024
32 checks passed
@gabrielbosio gabrielbosio deleted the dict-benches branch December 19, 2024 16:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants