Feature/Era1 import/export #6547

ak88 · 2024-01-16T10:53:20Z

Introduces a new archive format called Era1, which is based on the Era spec made by nimbus, and this proposal from arnetheduck.

The purpose of Era1 is to provide the same functionality for EL, that Era provides for the CL. An easily verifiable and shareable format for pre-merge historic EL data, which is considered a prerequisite for eip-4444 . Era1 only concerns pre-merge data, since post-merge EL payload is contained in the beacon chain.

In the long term the idea is to distribute historic data through the portal network.

Underneath Era1 is based on e2store which is a simple type-length-value scheme.

Specification

An Era1 archive can be expressed like so

   era1 := Version | block-tuple* | other-entries* | Accumulator | BlockIndex
   block-tuple :=  CompressedHeader | CompressedBody | CompressedReceipts | TotalDifficulty
   block-index := starting-number | index | index | index ... | count

Headers, bodies and receipts are compressed using the snappy framing format.
Additionally each file contains a block index for fast lookup, and an Epoch accumulator for verification.

The epoch accumulator can be used to verify the entire archive with accumulators from a trusted source. Additionally it allows a node to download a header with a merkle-proof, that proves it belongs to a certain epoch.

The Epoch accumulator is defined in the portal network spec here.

Behaviour

Two operation is added, import and export can be invoked via rpc or cli.
- --Era.ImportDirectory "something" --Era.From 0 --Era.To 0 --Era.TrustedAccumulatorFile "somefile"
  - From, To and TrustedAccumulatorFile is optional. Will import everything when range is set to 0. Will trust the era directory if TrustedAccumulatorFile is not specified.
  - RPC is admin_importHistory.
- --Era.ExportDirectory "something" --Era.From 0 --Era.To 0
  - From, To is optional. Will export everything when set to 0.
  - accumulators.txt and checksums.txt is always created.
  - RPC is admin_exportHistory.
For simpliciy, CLI will block the application from starting until it is completed. RPC does not block but only one import/export can run at the same time.
During import, block range before head will be inserted in parallel like old bodies, and after head will be "suggested" like forward sync. So it will process new imported block.
- It will not complete until the blocks are processed, and will pause adding if the processing queue reaches 2000 block.
- An exception to this is if fastsync is enabled and the head is 0, in which case, it will only insert blocks like old bodies.
- Accumulator and content is always verified/checksumed, along with IBlockValidator.
During export, it export the range in parallel, each era file having its own task.

Types of changes

What types of changes does your code introduce?

New feature (a non-breaking change that adds functionality)

Testing

Requires testing

Yes

If yes, did you write tests?

Yes

Documentation

Requires documentation update

Yes

Requires explanation in Release Notes

Yes

R#eceipt decoder

into feature/era-history

Test import export unit tests

commit 68c722cad69d082da7e6283c396b4f5706a3540f Author: Anders Holmbjerg Kristiansen <[email protected]> Date: Sun Nov 5 23:38:52 2023 +0100 Verify accumulator commit c48ca9de6ce767173b9674b665ac7ecd0f2e7172 Author: Anders Holmbjerg Kristiansen <[email protected]> Date: Sun Nov 5 15:14:37 2023 +0100 refactor EraReader

…a-import

LukaszRozmej

Overall magnificent work! Just pointing out there is potential for a lot more optimizations, some micro, some maybe more substantial. I don't see anything particularly wrong, not approving yet only because I want to have those discussions and potential improvements here and not loose the conversations.

Missing if (_logger.IsXXX) in many places.
Use is null/is not null.

LukaszRozmej · 2024-11-05T12:22:11Z

src/Nethermind/Nethermind.Serialization.Rlp/ByteBufferExtensions.cs

+        /// <param name="startIndex">Optional start index of the underlying buffer.</param>
+        /// <returns></returns>
+        /// <exception cref="InvalidOperationException"></exception>
+        public static Memory<byte> AsMemory(this IByteBuffer buffer, int? startIndex = null)


maybe have optional length for completeness?

LukaszRozmej · 2024-11-05T12:23:57Z

src/Nethermind/Nethermind.JsonRpc/Modules/Admin/IAdminRpcModule.cs

+    [JsonRpcMethod(Description = "Exports a range of historic block in era1 format.",
+    EdgeCaseHint = "",
+    ExampleResponse = "\"Export task started.\"",
+    IsImplemented = true)]
+    Task<ResultWrapper<string>> admin_exportHistory(string destination, int start, int end);
+
+    [JsonRpcMethod(Description = "Import a range of historic block from era1 directory.",
+    EdgeCaseHint = "",
+    ExampleResponse = "\"Export task started.\"",
+    IsImplemented = true)]
+    Task<ResultWrapper<string>> admin_importHistory(string source, int start, int end, string? accumulatorFile);


Add JsonRpcParameter for parameter docs.

LukaszRozmej · 2024-11-05T12:24:28Z

src/Nethermind/Nethermind.Init/Steps/RegisterRpcModules.cs

+        IContainer eraContainer = _api.ConfigureContainerBuilderFromApiWithBlockchain(new ContainerBuilder())
+            .AddModule(new EraModule())
+            .Build();
+        _api.DisposeStack.Push((IAsyncDisposable)eraContainer);


Why separate container?

I don't wanna touch NethermindApi.

LukaszRozmej · 2024-11-05T12:24:53Z

src/Nethermind/Nethermind.Init/Steps/EraStep.cs

+        await using IContainer container = _api.ConfigureContainerBuilderFromApiWithBlockchain(new ContainerBuilder())
+            .AddModule(new EraModule())
+            .Build();
+        await container.Resolve<EraCliRunner>().Run(cancellationToken);


Why separate container?

LukaszRozmej · 2024-11-05T12:26:53Z

src/Nethermind/Nethermind.Era1/IEraConfig.cs

+    [ConfigItem(Description = "Network name used for era directory naming. When null, it will imply from network.", DefaultValue = "null", HiddenFromDocs = true)]
+    string? NetworkName { get; set; }


When null, it will imply from network is not understandable what you mean.

LukaszRozmej · 2024-11-06T09:42:05Z

src/Nethermind/Nethermind.Era1/EraImporter.cs

+        if (accumulatorFile != null)
+        {
+            string[] lines = _fileSystem.File.ReadAllLines(accumulatorFile);
+            trustedAccumulators = lines.Select(s => new ValueHash256(s)).ToHashSet();


Shouldn't the accumulators be in the same order as the data?
If yes then:

Shouldn't be a Set but more of a list/enumerable

Don't need to allocate a collection, but could be lazy loaded from the file when needed as IEnumerable<> or better IAsyncEnumerable.

Not sure. Imagine you have an accumulator file from EF website for example, and another era archive for only a portion of the history not necessarily starting from genesis.

LukaszRozmej · 2024-11-06T09:43:27Z

src/Nethermind/Nethermind.Era1/EraImporter.cs

+                _receiptsDb.Tune(ITunableDb.TuneType.Default);
+                _blocksDb.Tune(ITunableDb.TuneType.Default);
+
+                if (t.IsFaulted)
+                    throw t.Exception?.InnerException!;


do await and put Tune(ITunableDb.TuneType.Default) in finally

LukaszRozmej · 2024-11-06T09:44:56Z

src/Nethermind/Nethermind.Era1/EraImporter.cs

+        // I wish I could say that EraStore can be run used in parallel in any way you like but I could not make it so.
+        // This make the `blockNumber` aligned to era file boundary so that when running parallel, each thread does not
+        // work on the same era file as other thread.


What was the problem with parallelization?

The EraReader sharing in EraStore is limited. If a thread try to access an era reader that is currently being used, it will create a new EraReader. Need a more sophisticated reference counting mechanism to make work.

LukaszRozmej · 2024-11-06T09:47:22Z

src/Nethermind/Nethermind.Era1/EraImporter.cs

+                await SuggestAndProcessBlock(b);
+            }
+            else
+                InsertBlockAndReceipts(b, r);


Do we need to decode blocks and receipts from RLP just to insert it here again and code it again (maybe we do)

LukaszRozmej · 2024-11-06T09:49:55Z

src/Nethermind/Nethermind.Era1/EraStore.cs

+        if (!(_verifiedEpochs.TryGetValue(epoch, out bool verified) && verified))
+        {
+            ValueHash256 checksum = reader.CalculateChecksum();
+            ValueHash256 expectedChecksum = _checksums[epoch - FirstEpoch];


checksums could be loaded lazily?

Co-authored-by: Lukasz Rozmej <[email protected]>

ak88 added 30 commits October 9, 2023 16:34

ERA history import/export start

301180c

R#eceipt decoder

Rename in unit test

5a0b8c4

Fix block index not set

04209fb

Start export

4b10a61

Era builder

bfcbfb5

Accumulator calculation

e9d91b6

ERA history import/export

7b88e17

Rename in unit test

1543a39

Fix block index not set

e07cca5

Start export

42131e6

Era builder

551d974

Accumulator calculation

aba6cfb

Builder finalize

65a9225

Merge branch 'feature/era-history' of https://github.com/ak88/nethermind

d0741e5

into feature/era-history

Unit tests

508f1ae

Test import export unit tests

Unit tests

c7636c2

import geth files

ee53da7

Reader with netty

8503f08

1mb read buffer

66eacd1

refactor test and use of netty

8d48264

Big blocks test

904ecab

Change to IByteBuffer

74d14ad

buffer allocater injected

e1e86ae

Merge branch 'NettyBuffers' into feature/era-history

d8f4ad8

module

539423b

Merge from master

27fc5cd

assert files exist

756b170

reverted

e3a6647

rollback gitbook

f18e742

asdacap added 19 commits October 29, 2024 14:15

Era store factory

c3c5802

Network name from config

5bb8cde

Add comments

617d0ff

Checksum support

fddcc6d

Cli runner test

203159a

Minor comment

36b39d8

Move admin era service to era directory

df8a91b

Merge remote-tracking branch 'origin/master' into feature/era-import

d66eb61

Whitespace

f89e644

Fix test

67e2a8c

Add option to not recover sender

31b469f

Merge branch 'feature/allow-skipping-sender-recovery' into feature/er…

a654d9a

…a-import

Fix tests, disable sender ccalculatioon

4d5944e

Add option to specify write flags

7028b52

Merge branch 'feature/allow-skipping-sender-recovery' into feature/er…

3833ff5

…a-import

More robust validation

d417ac0

Whitespace

4df3719

Add tests for admin era service

097fd7a

More consistent name

d9e5f0e

LukaszRozmej requested review from kamilchodola and rubo October 30, 2024 12:29

asdacap added 4 commits November 1, 2024 08:30

Merge remote-tracking branch 'origin/master' into feature/era-import

69fbf66

Whitespace

1683e87

Cleanup

a6a4fb9

Fix misleading cli

a8e8154

LukaszRozmej reviewed Nov 6, 2024

View reviewed changes

asdacap and others added 4 commits November 9, 2024 09:29

Merge remote-tracking branch 'origin/master' into feature/era-import

2476a51

Update src/Nethermind/Nethermind.Era1/E2StoreWriter.cs

240f5c8

Co-authored-by: Lukasz Rozmej <[email protected]>

Update src/Nethermind/Nethermind.Era1/E2StoreWriter.cs

4f0021e

Co-authored-by: Lukasz Rozmej <[email protected]>

Update src/Nethermind/Nethermind.Era1/AccumulatorCalculator.cs

08cf69c

Co-authored-by: Lukasz Rozmej <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/Era1 import/export #6547

Feature/Era1 import/export #6547

ak88 commented Jan 16, 2024 •

edited by asdacap

Loading

LukaszRozmej left a comment

LukaszRozmej Nov 5, 2024

LukaszRozmej Nov 5, 2024

LukaszRozmej Nov 5, 2024

asdacap Nov 9, 2024

LukaszRozmej Nov 5, 2024

LukaszRozmej Nov 5, 2024

LukaszRozmej Nov 6, 2024

asdacap Nov 9, 2024

LukaszRozmej Nov 6, 2024

LukaszRozmej Nov 6, 2024

asdacap Nov 9, 2024

LukaszRozmej Nov 6, 2024

asdacap Nov 9, 2024

LukaszRozmej Nov 6, 2024

		[ConfigItem(Description = "Network name used for era directory naming. When null, it will imply from network.", DefaultValue = "null", HiddenFromDocs = true)]
		string? NetworkName { get; set; }

Feature/Era1 import/export #6547

Are you sure you want to change the base?

Feature/Era1 import/export #6547

Conversation

ak88 commented Jan 16, 2024 • edited by asdacap Loading

Specification

Behaviour

Types of changes

What types of changes does your code introduce?

Testing

Requires testing

If yes, did you write tests?

Documentation

Requires documentation update

Requires explanation in Release Notes

LukaszRozmej left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ak88 commented Jan 16, 2024 •

edited by asdacap

Loading