Hierarchical state diffs #5978

michaelsproul · 2024-06-21T07:31:24Z

Proposed Changes

This PR contains the freezer DB changes from the main tree-states branch, bringing the small & efficient archive node database to stable Lighthouse 🎉

Additional Info

(From @dapplion): Some resources to get up to speed with HDiff concepts:

dapplion · 2024-06-21T08:53:02Z

I get a compilation error from xdelta3 trying to compile this branch:

EDIT: Fixed (cropped previous text because it's long)

michaelsproul · 2024-06-21T08:58:09Z

looks like it might be a Mac thing

I had to fork xdelta3 already, we should move it under sigp and patch it

dapplion · 2024-06-21T10:04:03Z

TODOs

EDIT: completed

dapplion · 2024-06-21T10:05:49Z

TODO ~ Some metrics that we should add:

Track store metrics by DB column #6041
- byte size of diff_buffer_cache_size
- elements in diff_buffer_cache_size
- counter for replayed blocks
- counter for diff_buffer_cache hits / misses
- timer for storage time
- timer for load times
Track store metrics by DB column #6041
- counter for diffs loaded / stored
- counter for snapshots loaded / stored

beacon_node/store/src/hot_cold_store.rs

* Add store cache size metrics * Add compress timer metrics * Add diff apply compute timer metrics * Add diff buffer cache hit metrics * Add hdiff buffer load times * Add blocks replayed metric * Move metrics to store * Future proof some metrics --------- Co-authored-by: Michael Sproul <[email protected]>

beacon_node/store/src/hdiff.rs

jimmygchen

Looks really good, thanks for addressing the comments! 🎉

I don't fully understand all the database details, but I've reviewed all the changes and I don't see any obvious issues.

I believe this is good to go, and happy to help optimise further if we need to.

dapplion

Testing this we noted much worse performance of the binary patch library than we expected originally. The BeaconState has grown significantly since Michael did the original research of what diff algorithm to use. Despite Electra activating in some months there's no guarantee that the network will consolidate, so rampant state growth is a concern.

Currently it takes 1.3 sec to compute a diff on average on a healthy holesky node.

		State size (MB)
% Supply staked	Validators (M)	0% bloat	100% bloat	200% bloat
100	3.75	543.75	1087.5	1631.25
80	3	435	870	1305
60	2.25	326.25	652.5	978.75
40	1.5	217.5	435	652.5
30	1.125	163.125	326.25	489.375
20	0.75	108.75	217.5	326.25

There's a possibility of the BeaconState growing close to 1GB. I run some benchmarks on our diff library, and the time to compute a diff is (random mutations of total size diff bytes), (tested on an M2 chip)

data size bytes	diff bytes	compute diff time (ms)
2e8 (today's size)	1e3	584.03
2e8	1e4	635.24
2e8	1e5	778.79
2e8	1e6	1507.0
1e9 (future size)	1e3	4895.6
1e9	1e4	3885.0
1e9	1e5	6301.1
1e9	1e6	15858.0

For 1GB input data those are pretty terrifying numbers. I'm hitting Request changes to:

Make this branch future proof to allow us to improve the diff algorithm in the future. For example, version the diffs and use more advance strategies latter
Come up with a better diff algorithm now

michaelsproul · 2024-10-31T21:46:59Z

Versioning the diff algorithm sounds like a great idea. I also think we could probably include #6556

* Split hdiff computation * Dedicated logic for historical roots and summaries * Benchmark against real states * Mutated source? * Version the hdiff * Add lighthouse DB config for hierarchy exponents * Tidy up hierarchy exponents flag * Apply suggestions from code review Co-authored-by: Michael Sproul <[email protected]> * Address PR review * Remove hardcoded paths in benchmarks * Delete unused function in benches * lint --------- Co-authored-by: Michael Sproul <[email protected]>

michaelsproul · 2024-11-14T06:20:12Z

I'll merge this PR first and then resolve merge conflicts:

Improving blob propagation post-PeerDAS with Decentralized Blob Building #6268

already addressed

michaelsproul

Approved ✔️

michaelsproul · 2024-11-18T01:17:26Z

@mergify queue

mergify · 2024-11-18T01:17:46Z

queue

✅ The pull request has been merged automatically

The pull request has been merged automatically at 9fdd53d

* Start extracting freezer changes for tree-states * Remove unused config args * Add comments * Remove unwraps * Subjective more clear implementation * Clean up hdiff * Update xdelta3 * Tree states archive metrics (sigp#6040) * Add store cache size metrics * Add compress timer metrics * Add diff apply compute timer metrics * Add diff buffer cache hit metrics * Add hdiff buffer load times * Add blocks replayed metric * Move metrics to store * Future proof some metrics --------- Co-authored-by: Michael Sproul <[email protected]> * Port and clean up forwards iterator changes * Add and polish hierarchy-config flag * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Cleaner errors * Fix beacon_chain test compilation * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Patch a few more freezer block roots * Fix genesis block root bug * Fix test failing due to pending updates * Beacon chain tests passing * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Fix doc lint * Implement DB schema upgrade for hierarchical state diffs (sigp#6193) * DB upgrade * Add flag * Delete RestorePointHash * Update docs * Update docs * Implement hierarchical state diffs config migration (sigp#6245) * Implement hierarchical state diffs config migration * Review PR * Remove TODO * Set CURRENT_SCHEMA_VERSION correctly * Fix genesis state loading * Re-delete some PartialBeaconState stuff --------- Co-authored-by: Michael Sproul <[email protected]> * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Fix test compilation * Update schema downgrade test * Fix tests * Fix null anchor migration * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Fix tree states upgrade migration (sigp#6328) * Towards crash safety * Fix compilation * Move cold summaries and state roots to new columns * Rename StateRoots chunked field * Update prune states * Clean hdiff CLI flag and metrics * Fix "staged reconstruction" * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Fix alloy issues * Fix staged reconstruction logic * Prevent weird slot drift * Remove "allow" flag * Update CLI help * Remove FIXME about downgrade * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Remove some unnecessary error variants * Fix new test * Tree states archive - review comments and metrics (sigp#6386) * Review PR comments and metrics * Comments * Add anchor metrics * drop prev comment * Update metadata.rs * Apply suggestions from code review --------- Co-authored-by: Michael Sproul <[email protected]> * Update beacon_node/store/src/hot_cold_store.rs Co-authored-by: Lion - dapplion <[email protected]> * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Clarify comment and remove anchor_slot garbage * Simplify database anchor (sigp#6397) * Simplify database anchor * Update beacon_node/store/src/reconstruct.rs * Add migration for anchor * Fix and simplify light_client store tests * Fix incompatible config test * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * More metrics * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * New historic state cache (sigp#6475) * New historic state cache * Add more metrics * State cache hit rate metrics * Fix store metrics * More logs and metrics * Fix logger * Ensure cached states have built caches :O * Replay blocks in preference to diffing * Two separate caches * Distribute cache build time to next slot * Re-plumb historic-state-cache flag * Clean up metrics * Update book * Update beacon_node/store/src/hdiff.rs Co-authored-by: Lion - dapplion <[email protected]> * Update beacon_node/store/src/historic_state_cache.rs Co-authored-by: Lion - dapplion <[email protected]> --------- Co-authored-by: Lion - dapplion <[email protected]> * Update database docs * Update diagram * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Update lockbud to work with bindgen/etc * Correct pkg name for Debian * Remove vestigial epochs_per_state_diff * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Markdown lint * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Address Jimmy's review comments * Simplify ReplayFrom case * Fix and document genesis_state_root * Typo Co-authored-by: Jimmy Chen <[email protected]> * Merge branch 'unstable' into tree-states-archive * Compute diff of validators list manually (sigp#6556) * Split hdiff computation * Dedicated logic for historical roots and summaries * Benchmark against real states * Mutated source? * Version the hdiff * Add lighthouse DB config for hierarchy exponents * Tidy up hierarchy exponents flag * Apply suggestions from code review Co-authored-by: Michael Sproul <[email protected]> * Address PR review * Remove hardcoded paths in benchmarks * Delete unused function in benches * lint --------- Co-authored-by: Michael Sproul <[email protected]> * Test hdiff binary format stability (sigp#6585) * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Add deprecation warning for SPRP * Update xdelta to get rid of duplicate deps * Document test

Start extracting freezer changes for tree-states

ec974b8

michaelsproul added work-in-progress PR is a work-in-progress optimization Something to make Lighthouse run more efficiently. database tree-states Upcoming state and database overhaul labels Jun 21, 2024

michaelsproul mentioned this pull request Jun 21, 2024

Database schema changes 2024 #5912

Open

dapplion added 4 commits June 21, 2024 11:57

Remove unused config args

df5e716

Add comments

17ce7d0

Remove unwraps

3c5d722

Subjective more clear implementation

31bcd84

dapplion reviewed Jun 21, 2024

View reviewed changes

beacon_node/store/src/hot_cold_store.rs Show resolved Hide resolved

Clean up hdiff

394abba

This was referenced Jul 3, 2024

Tree states archive metrics #6040

Merged

Track store metrics by DB column #6041

Merged

michaelsproul and others added 12 commits July 4, 2024 12:24

Update xdelta3

cac7672

Port and clean up forwards iterator changes

e578f5d

Add and polish hierarchy-config flag

bdcc818

Merge remote-tracking branch 'origin/unstable' into tree-states-archive

1501ba5

Cleaner errors

aba6b8b

Fix beacon_chain test compilation

420e524

Merge remote-tracking branch 'origin/unstable' into tree-states-archive

3bec78b

Patch a few more freezer block roots

0500e64

Fix genesis block root bug

2715f60

Fix test failing due to pending updates

fa1a941

Beacon chain tests passing

ee032df

jimmygchen reviewed Oct 31, 2024

View reviewed changes

beacon_node/store/src/hdiff.rs Show resolved Hide resolved

jimmygchen approved these changes Oct 31, 2024

View reviewed changes

jimmygchen added ready-for-merge This PR is ready to merge. and removed ready-for-review The code is ready for review labels Oct 31, 2024

Merge branch 'unstable' into tree-states-archive

f5b42e2

dapplion previously requested changes Oct 31, 2024

View reviewed changes

dapplion mentioned this pull request Oct 31, 2024

Compute diff of validators list manually #6556

Merged

michaelsproul added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-merge This PR is ready to merge. labels Oct 31, 2024

michaelsproul mentioned this pull request Nov 12, 2024

Database growth during non-finality #6580

Open

michaelsproul added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels Nov 14, 2024

michaelsproul added 3 commits November 17, 2024 08:03

Test hdiff binary format stability (#6585)

dc2b665

Merge remote-tracking branch 'origin/unstable' into tree-states-archive

fa79803

Add deprecation warning for SPRP

6b6f796

michaelsproul added 2 commits November 18, 2024 11:39

Update xdelta to get rid of duplicate deps

9b4b069

Document test

b0a5bbe

michaelsproul commented Nov 18, 2024

View reviewed changes

michaelsproul added ready-for-merge This PR is ready to merge. and removed ready-for-review The code is ready for review labels Nov 18, 2024

mergify bot merged commit 9fdd53d into unstable Nov 18, 2024
30 checks passed

mergify bot deleted the tree-states-archive branch November 18, 2024 01:51

dapplion mentioned this pull request Jan 2, 2025

Hierarchical state diffs in hot DB #6750

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hierarchical state diffs #5978

Hierarchical state diffs #5978

michaelsproul commented Jun 21, 2024 •

edited

Loading

dapplion commented Jun 21, 2024 •

edited

Loading

michaelsproul commented Jun 21, 2024

dapplion commented Jun 21, 2024 •

edited

Loading

dapplion commented Jun 21, 2024 •

edited

Loading

jimmygchen left a comment

dapplion left a comment •

edited

Loading

michaelsproul commented Oct 31, 2024 •

edited

Loading

michaelsproul commented Nov 14, 2024 •

edited

Loading

michaelsproul left a comment

michaelsproul commented Nov 18, 2024

mergify bot commented Nov 18, 2024 •

edited

Loading

Hierarchical state diffs #5978

Hierarchical state diffs #5978

Conversation

michaelsproul commented Jun 21, 2024 • edited Loading

Proposed Changes

Additional Info

dapplion commented Jun 21, 2024 • edited Loading

michaelsproul commented Jun 21, 2024

dapplion commented Jun 21, 2024 • edited Loading

TODOs

dapplion commented Jun 21, 2024 • edited Loading

jimmygchen left a comment

Choose a reason for hiding this comment

dapplion left a comment • edited Loading

Choose a reason for hiding this comment

michaelsproul commented Oct 31, 2024 • edited Loading

michaelsproul commented Nov 14, 2024 • edited Loading

michaelsproul left a comment

Choose a reason for hiding this comment

michaelsproul commented Nov 18, 2024

mergify bot commented Nov 18, 2024 • edited Loading

✅ The pull request has been merged automatically

michaelsproul commented Jun 21, 2024 •

edited

Loading

dapplion commented Jun 21, 2024 •

edited

Loading

dapplion commented Jun 21, 2024 •

edited

Loading

dapplion commented Jun 21, 2024 •

edited

Loading

dapplion left a comment •

edited

Loading

michaelsproul commented Oct 31, 2024 •

edited

Loading

michaelsproul commented Nov 14, 2024 •

edited

Loading

mergify bot commented Nov 18, 2024 •

edited

Loading