Sequencing and syncing

[arnaubennassar]

Note: This discussion is part of the broader Ethereum vanilla clients topic.

Context

For a while the CDK team has been willing to re-architect the relation between the execution and the CDK client in a way that it uses the Engine API. This is a no brainer when the goal is to be as vanilla as possible, since all the execution clients use this interface for Ethereum. Not only that, but the separation of execution / consensus client of Ethereum, matches very well with our approach of execution / cdk clients. Additionally, this path seems to future proof interesting forward looking features such as sequencer decentralisation (there are already tools for that such as beacon kit), PBS style goodies like rollup boost, based rollups, ...

Challenges

tl;dr:

• build blocks using forced transactions
• sync from DA

Unfortunately, using the Engine API to driver the execution client of a rollup is not as straight forward as it may seem. This is the case mostly because in order to build a block, the consensus client (CC, in our case the cdk), requests this to the execution client (EC) to do so. Upon request the EC will initiate the block building by adding transactions from the pool in the block. Later on, the CC will propagate this block to other CCs which will validate the block against the EC. The problem is that:

• There is no way for the CC to indicate the EC to add a block to the chain, unless it already knows the block of the hash. That conflicts with:
  • Syncing from DA: it's not natural to include block hashes in the DA, although this could be done (at a cost of more data being posted)
  • Forced inclusion: this feature is not enabled today, but once it's added to the stack, the CC will need to indicate the EC to execute a specific block / tx without knowing it's outcome before hand
• It's not possible to pass L1 messages specific to the protocol, such as GERs. We already know how to work around this issue, by using AggOracle

Interesting resources

• Engine API spec
• Wire Protocol spec
• This doc is super helpful to understand how the Engine API works
• OP spec for the engine API, where they've modified the original by adding, among other things the ability to pass a list of txs without the block hash when syncing/building.

[arnaubennassar]

Proposal A

The problem boils down to: how to build blocks without knowing the execution outcome (block hash) before hand? It turns out that there are solutions that need this feature: block builders such as rbuilder from flashobts. And besides potentially overcoming the challenges, they are a good thing to have in itself as they bring some nice sequencing features on their own. Here's how they work on the system

Happy path

sequenceDiagram

sequencer (CDK) ->> execution client (sequencer): build block `engine_forkchoiceUpdatedV2(ForkchoiceState, PayloadAttributes)`
execution client (sequencer) -->> sequencer (CDK): {payloadStatus: {status: VALID, ...}, payloadId: buildProcessId}
sequencer (CDK) ->> execution client (sequencer): get status `engine_getPayloadV2(PayloadId)`
execution client (sequencer) -->> sequencer (CDK): {executionPayload, blockValue}
sequencer (CDK) ->> sync node (CDK): distribute new block
sync node (CDK) ->> execution client (sync): go to new tip of the chain: engine_forkchoiceUpdatedV2(ForkchoiceState, null)

Loading

Note that:

• this step doesn't involve the builder at all, although it could be used to incorporate advanced sequencing features
• block distribution is left out of the scope of this doc

Sync from DA

sequenceDiagram

CDK ->> L1: get verified block hash
L1 -->> CDK: block hash
CDK ->> execution client: check block hash
execution client -->> CDK: block hash does not exist
CDK ->> L1: download DA
L1 -->> CDK: DA
CDK ->> builder: build block using DA
builder -->> CDK: new block
CDK ->> execution client: new tip of the chain
execution client ->> CDK: get block body
CDK -->> execution client: block body

Loading

The key parts of this flow:

• The builder is able to craft a new block by taking the data from the DA
• CDK and execution client only communicate through standard APIs:
  • Engine API: for CDK to communicate the new block header using engine_forkchoiceUpdatedV2
  • Wire protocol: for the execution client to request the new block body

Build blocks with forced transactions

In order to build block with forced transactions / add forced blocks, the builder is also used: CDK would download the froced data, and request the builder to compute it. Once it's done it will share the new block with the execution client

Considerations

The complex part here is implementing the builder, ideally we should be able to reuse an existing software for that. Unfortunately, builders themselves depend on execution clients as it can be seen from flashbots moving from geth based to reth based builder implementation. This could lead the project to a situation where we need to do specific implementations for the different execution clients (rETH, erigon, ...), which is something to avoid, one of the goals of going vanilla

A potential remediation would be to not support all the features on all the clients, for instance, only reth would be supported for sequencing. On the other hand, when it comes to syncing, as long as there is a single honest node that has the blocks as per the DA, the network could use that node to sync. This opens the room for having a dedicated service (that anyone could run) that is in charge of building the state from the DA

Alternatively, we could force the DA to include block hashes, and pretend that forced transactions will never exist. Doing so would enable a vanilla usage of the Engine API without any trick or add on. But it would increase the cost of DA, and eventually, forced transactions should make it into the protocol

Builder implementation

Reth looks the way to go here, the modularity of the projects makes it easier to build blocks, it looks like we would be able to reuse certian part of the codebase to build blocks without forking

[vcastellm]

In Proposal A we won't know what's the content of the block.

What is check block hash in EC?
Why Download DA from L1?
Why is the Execution client separated from the builder? aren't they the same?
If they use the wire protocol are you proposing that we implement part of devp2p in the CDK?

We can definitely use Reth for building a builder, even as an Execution Extension but how we're going to provide that block to the execution client?

[arnaubennassar]

What is check block hash in EC?

engine_newPayloadV2(ExecutionPayload) used to validate

Why Download DA from L1?

from L1 or elsewhere, but the point is that we need to be able to fetch the DA in a scenario where the sequencer miss-behaves (doesn't share the DA or the state roots that are shared don't match the execution of the DA)

Why is the Execution client separated from the builder? aren't they the same?

builder software is not necessary the same as execution client, it overlaps or interacts with it depending on the builder implementation. Looking at flashbots builders, the one based on geth is a fork and could be considered as an execution client, but the one based on rETH is a separated softwarem(binary/process) that uses some crates from reth and connects to the DB. However, they're working on making the builder run in the same process a reth

If they use the wire protocol are you proposing that we implement part of devp2p in the CDK?

yes

We can definitely use Reth for building a builder, even as an Execution Extension but how we're going to provide that block to the execution client?

once we have a full block built, we can just share the block hash using the fork choice endpoint, and the execution client will request the block body through devp2p. It's not optimal as this requires double execution, but keep in mind that this process should only be triggered when:

• the sequencer is building a forced block
• a node is syncing from the DA, this means that there is no single node on the P2P mesh that can be reached and has already synced (one honest node is enough). So if the sequencer is not sharing the trusted state as expected, some nodes will need to do this double processing, but once they have done that, other nodes can sync from them. Since we should get the block hashes from L1, we can use that to drive the fork of choice, and then share the intermediary block headers and all the block bodies through devp2p