Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merkle performance improvements #233

Closed
wants to merge 27 commits into from
Closed

Merkle performance improvements #233

wants to merge 27 commits into from

Commits on Apr 21, 2024

  1. attempt

    Cherry-Picked from OffchainLabs/nitro@flatmerkleapril16
    rauljordan authored and eljobe committed Apr 21, 2024
    Configuration menu
    Copy the full SHA
    a149ebd View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    dec2e82 View commit details
    Browse the repository at this point in the history
  3. Add the total time for each step-size.

    It can be easy to look at small average step and hash times and miss
    that the total time is what we're really trying to reduce.
    eljobe committed Apr 21, 2024
    Configuration menu
    Copy the full SHA
    3c30265 View commit details
    Browse the repository at this point in the history
  4. Add some doc comments and tests to merkle.rs

    I definitely had some incorrect assumptions about this data structure
    which made it more difficult to learn. So, I'm documenting how it
    works and adding some tests.
    
    The simple_merkle test is currently failing because the `set` method
    doesn't allow setting an index larger than the largest currently
    set leaf's index.
    
    There is some debate as to whether or not this is the correct
    behavior. To run the test, use:
    
    ```
    $> cargo test -- --include-ignored
    ```
    eljobe committed Apr 21, 2024
    Configuration menu
    Copy the full SHA
    51069f5 View commit details
    Browse the repository at this point in the history
  5. Allow callers to extend the Merkle Tree by adding leaves.

    At this point, the new root hash is eagerly calculated after each call to `extend`.
    eljobe committed Apr 21, 2024
    Configuration menu
    Copy the full SHA
    2ecc4f5 View commit details
    Browse the repository at this point in the history
  6. Extend the memory merkle instead of clearing it.

    If this happened frequently, it should really improve the perfomance
    of the machine.  However, it looks like it doesn't happen at all with
    the benchmark inputs.
    eljobe committed Apr 21, 2024
    Configuration menu
    Copy the full SHA
    05b157f View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    965a202 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    477e49c View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2024

  1. Fix the implementation of extend.

    Previously, it could hit an index out of bounds if the new leafs
    caused any parent layer to grow beyond its current size.
    eljobe committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    cb1d10c View commit details
    Browse the repository at this point in the history
  2. Add criterion benchmark for a big merkle tree.

    Hopefully, this will allow us to compare this branch's implementation
    of a merkle tree to the one on merkle-perf-a.
    eljobe committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    e644b89 View commit details
    Browse the repository at this point in the history

Commits on Apr 23, 2024

  1. Include the creation of a fresh tree for each iteration.

    The previous implementation was growing the same layers and
    dirty_indices arrays because the clone isn't deep (I guess.)
    eljobe committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    e8868a8 View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2024

  1. Add some profiling and instrumentation code.

    There are a few different things going on in this commit.
    
    1. I've added some counters for when methods get called on the Merkle
    tree.
    
    2. I've added integration with gperftools for profiling specific areas
    of the code.
    eljobe committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    1f52875 View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2024

  1. Make features work

    This allows me to profile CPU and Heap independently, and to enable
    and disable the call counters independently.
    eljobe committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    976cb36 View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2024

  1. Add a benchmark for new_advanced.

    This part of the code is obviously slow.
    Let's see if we can improve it.
    eljobe committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    5e14a2b View commit details
    Browse the repository at this point in the history
  2. Actually set the cached merkle back on the instance.

    This is why there were all those unexpected "new_advanced" calls on
    the memory merkle. The resizes were actually setting self.merkle
    back to None.
    eljobe committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    2f2e173 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    fc16ec3 View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2024

  1. Update the logic for expand to include upper layers.

    There was a bug where expanding the lowest layer and calling set on
    all of the new elements was not sufficient to grow the upper layers.
    
    This commit also fixes a warning about the package-level profile
    override being ineffective.
    eljobe committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    1a52847 View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2024

  1. Remove the bold submodule.

    I don't think it's being used.
    eljobe committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    72bc92f View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    6f7cafa View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    6a3c245 View commit details
    Browse the repository at this point in the history
  4. Make clippy happy.

    eljobe committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    36578d7 View commit details
    Browse the repository at this point in the history
  5. Add the fake stuff for the benchmarks to the Dockerfile.

    I have no idea why this is needed. But, it makes `make docker`
    successful again.
    eljobe committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    97d93d5 View commit details
    Browse the repository at this point in the history
  6. Make clippy even happier.

    eljobe committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    e3d7cf4 View commit details
    Browse the repository at this point in the history
  7. Cargo fmt changes only.

    eljobe committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    521a7d8 View commit details
    Browse the repository at this point in the history

Commits on May 1, 2024

  1. Fix right-shifting to zero.

    eljobe committed May 1, 2024
    Configuration menu
    Copy the full SHA
    cf50743 View commit details
    Browse the repository at this point in the history
  2. Turn off always_merkelize for now.

    The system tests are timing out because the implementation is still
    too slow for large steps with lots of store and resize memory calls.
    eljobe committed May 1, 2024
    Configuration menu
    Copy the full SHA
    cc71b32 View commit details
    Browse the repository at this point in the history

Commits on May 2, 2024

  1. Configuration menu
    Copy the full SHA
    77866ec View commit details
    Browse the repository at this point in the history