Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(l2): multiple provers at the same time #1466

Merged
merged 54 commits into from
Jan 6, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
fe49344
add sp1 program, and makefile target
fborello-lambda Dec 3, 2024
1901ee6
rename Prover to Risc0Prover
fborello-lambda Dec 3, 2024
f88c308
add Sp1Prover structure -> test [wip]
fborello-lambda Dec 3, 2024
d42a65c
standalone test for Sp1Prover
fborello-lambda Dec 3, 2024
5b6dc3c
wip: integrate prover_client <-> prover_server
fborello-lambda Dec 3, 2024
72e8a08
feat: test target
fborello-lambda Dec 4, 2024
425ea97
Merge branch 'l2/test_without_docker' into l2/implement_sp1
fborello-lambda Dec 4, 2024
eaa18ad
feat: add errors.rs module
fborello-lambda Dec 4, 2024
f4decda
wip: basic interaction prover_server <-> prover_client
fborello-lambda Dec 4, 2024
d9c6873
fix: expect in sp1 program
fborello-lambda Dec 4, 2024
d560fef
feat: run prover_client based on the argument
fborello-lambda Dec 4, 2024
bc3a50f
Merge branch 'main' into l2/implement_sp1
fborello-lambda Dec 4, 2024
ebd15e9
chore: lint and fmt
fborello-lambda Dec 5, 2024
a87dbb5
fix feature
fborello-lambda Dec 5, 2024
34f3be0
add CI for prover
fborello-lambda Dec 5, 2024
757cce1
test: prover CI
fborello-lambda Dec 5, 2024
d0ceec7
feat(l2): deploy `Verifier` contracts conditionally (#1409)
fborello-lambda Dec 5, 2024
5a34cc4
fix: prover_server dev_mode
fborello-lambda Dec 5, 2024
d64a1a7
refactor: prover_server handle_proof_submission
fborello-lambda Dec 5, 2024
926fecf
fix: proof_submission
fborello-lambda Dec 5, 2024
c6229b9
chore: lint
fborello-lambda Dec 5, 2024
663ec1f
Merge branch 'main' into l2/implement_sp1
fborello-lambda Dec 5, 2024
cdf1d93
Merge branch 'main' into l2/implement_sp1
fborello-lambda Dec 5, 2024
2ae588c
fix: proof verification on_chain
fborello-lambda Dec 6, 2024
9262414
fix: transaction encoding
fborello-lambda Dec 9, 2024
d1c42f2
fix: add ?
fborello-lambda Dec 9, 2024
fddd8bd
Merge branch 'main' into l2/implement_sp1
fborello-lambda Dec 9, 2024
b1ccfa9
chore(prover): comments
fborello-lambda Dec 9, 2024
3a49baf
refactor: unify building process and crates
fborello-lambda Dec 9, 2024
6ef5133
refactor: improve Makefile
fborello-lambda Dec 10, 2024
a59b55c
docs: improve docs
fborello-lambda Dec 10, 2024
44d0177
chore: docs
fborello-lambda Dec 10, 2024
e73da08
chore: add sp1 NOTICE
fborello-lambda Dec 10, 2024
2bf0c04
Merge branch 'main' into l2/implement_sp1
fborello-lambda Dec 10, 2024
6082f4e
fix: ci
fborello-lambda Dec 10, 2024
0708065
Update Cargo.toml
fborello-lambda Dec 10, 2024
ab5d42a
refactor: remove unneeded struct
fborello-lambda Dec 10, 2024
903a6e9
fix: cli
fborello-lambda Dec 10, 2024
c07b2a7
wip
fborello-lambda Dec 10, 2024
d840b58
fix: panic
fborello-lambda Dec 11, 2024
dcf737d
fix: timing issues
fborello-lambda Dec 11, 2024
30770ec
refactor: reorganize prover_server
fborello-lambda Dec 11, 2024
b779396
fix: avoid storing old proofs
fborello-lambda Dec 11, 2024
88b97fc
chore: remove dbg message
fborello-lambda Dec 11, 2024
c7aa3b7
pr_comments: implement suggestions
fborello-lambda Dec 30, 2024
cb992f6
Merge branch 'main' into l2/multiple_provers
fborello-lambda Dec 30, 2024
41ed0fb
Merge branch 'main' into l2/multiple_provers
fborello-lambda Jan 2, 2025
5eeeab5
fix: as_u64 conversions
fborello-lambda Jan 2, 2025
f5bbd87
Merge branch 'main' into l2/multiple_provers
ilitteri Jan 2, 2025
d6e8825
Merge branch 'main' into l2/multiple_provers
fborello-lambda Jan 2, 2025
ad45506
feat(l2): multiple provers in file (#1477)
fborello-lambda Jan 3, 2025
4e88db8
Merge branch 'main' into l2/multiple_provers
fborello-lambda Jan 3, 2025
98cc25a
fix(l2_prover_ci): set the crates to optional
fborello-lambda Jan 6, 2025
44aca79
Merge branch 'l2/fix_prover_ci' into l2/multiple_provers
fborello-lambda Jan 6, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 71 additions & 10 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 5 additions & 2 deletions cmd/ethrex_l2/src/commands/prove.rs
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
use clap::Args;
use ethrex_l2::utils::test_data_io::{generate_program_input, read_chain_file, read_genesis_file};
use ethrex_l2::utils::{
prover::proving_systems::ProverType,
test_data_io::{generate_program_input, read_chain_file, read_genesis_file},
};
use ethrex_prover_lib::prover::create_prover;

#[derive(Args)]
Expand Down Expand Up @@ -30,7 +33,7 @@ impl Command {
let chain = read_chain_file(&self.chain);
let program_input = generate_program_input(genesis, chain, self.block_number)?;

let mut prover = create_prover(ethrex_l2::proposer::prover_server::ProverType::RISC0);
let mut prover = create_prover(ProverType::RISC0);
prover.prove(program_input).expect("proving failed");
println!(
"Total gas consumption: {}",
Expand Down
12 changes: 5 additions & 7 deletions crates/blockchain/metrics/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -6,23 +6,21 @@ edition = "2021"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
tokio.workspace = true
tracing.workspace = true
tokio = { workspace = true, optional = true }
tracing = { workspace = true, optional = true }
thiserror.workspace = true
serde_json.workspace = true
serde.workspace = true

ethrex-core = { path = "../../common", default-features = false }

prometheus = "0.13.4"

# TODO: remove?
axum = "0.7.9"
prometheus = { version = "0.13.4", optional = true }
axum = { version = "0.8.1", optional = true }


[lib]
path = "./mod.rs"

[features]
default = ["api"]
api = []
api = ["dep:axum", "dep:prometheus", "dep:tokio", "dep:tracing"]
3 changes: 3 additions & 0 deletions crates/l2/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,10 @@ secp256k1.workspace = true
keccak-hash.workspace = true
envy = "0.4.2"
thiserror.workspace = true
directories = "5.0.1"

zkvm_interface = { path = "./prover/zkvm/interface/", default-features = false }

# risc0
risc0-zkvm = { version = "1.2.0" }
# sp1
Expand Down
36 changes: 24 additions & 12 deletions crates/l2/Makefile
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
.PHONY: help init down clean restart cli update-cli-contracts init-local-l1 init-l1 down-local-l1 restart-local-l1 rm-db-l1 clean-contract-deps restart-contract-deps deploy-l1 init-l2 down-l2 restart-l2 init-prover rm-db-l2 purge_prover_state ci_test test
.DEFAULT_GOAL := help

L2_GENESIS_FILE_PATH=../../test_data/genesis-l2.json
L1_GENESIS_FILE_PATH=../../test_data/genesis-l1.json

# Basic
.PHONY: help init down clean restart

help: ## 📚 Show help for each of the Makefile recipes
@grep -E '^[a-zA-Z0-9_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'

Expand All @@ -18,8 +17,6 @@ clean: clean-contract-deps ## 🧹 Cleans the localnet
restart: restart-local-l1 deploy-l1 restart-l2 ## 🔄 Restarts the localnet

# CLI
.PHONY: cli update-cli-contracts

cli: ## 🛠️ Installs the L2 Lambda ethrex CLI
cargo install --path ${ethrex_PATH}/cmd/ethrex_l2/ --force

Expand Down Expand Up @@ -60,8 +57,6 @@ L2_AUTH_PORT=8552
L2_PROMETHEUS_METRICS_PORT = 3702

# Local L1
.PHONY: init-local-l1 init-l1 down-local-l1 restart-local-l1 rm-db-l1

init-local-l1: ## 🚀 Initializes an L1 Lambda ethrex Client with Docker (Used with make init)
docker compose -f ${ethrex_DEV_DOCKER_COMPOSE_PATH} -f ${ethrex_METRICS_OVERRIDES_L1_DOCKER_COMPOSE_PATH} up -d

Expand Down Expand Up @@ -94,8 +89,6 @@ rm-db-l1: ## 🛑 Removes the DB used by the L1
cargo run --release --manifest-path ../../Cargo.toml --bin ethrex -- removedb --datadir ${ethrex_L1_DEV_LIBMDBX}

# Contracts
PHONY: clean-contract-deps restart-contract-deps deploy-l1

clean-contract-deps: ## 🧹 Cleans the dependencies for the L1 contracts.
rm -rf contracts/solc_out
rm -rf contracts/lib
Expand All @@ -106,8 +99,6 @@ deploy-l1: ## 📜 Deploys the L1 contracts
DEPLOYER_CONTRACTS_PATH=contracts cargo run --release --bin ethrex_l2_l1_deployer --manifest-path ${ethrex_L2_CONTRACTS_PATH}/Cargo.toml

# L2
PHONY: init-l2 down-l2 restart-l2 init-prover rm-db-l2

init-l2: init-metrics ## 🚀 Initializes an L2 Lambda ethrex Client
cargo run --release --manifest-path ../../Cargo.toml --bin ethrex --features "l2,metrics" -- \
--network ${L2_GENESIS_FILE_PATH} \
Expand Down Expand Up @@ -148,8 +139,6 @@ rm-db-l2: ## 🛑 Removes the DB used by the L2
cargo run --release --manifest-path ../../Cargo.toml --bin ethrex -- removedb --datadir ${ethrex_L2_DEV_LIBMDBX}

# Testing
PHONY: ci_test test

ci_test: ## 🚧 Runs the L2's integration test, used by the github's CI
docker compose -f ${ethrex_L2_DOCKER_COMPOSE_PATH} down
docker compose -f ${ethrex_L2_DOCKER_COMPOSE_PATH} up -d --build
Expand All @@ -158,3 +147,26 @@ ci_test: ## 🚧 Runs the L2's integration test, used by the github's CI

test: ## 🚧 Runs the L2's integration test, run `make init` and in a new terminal make test
BRIDGE_ADDRESS=$$(grep 'L1_WATCHER_BRIDGE_ADDRESS' .env | cut -d= -f2) ON_CHAIN_PROPOSER_ADDRESS=$$(grep 'COMMITTER_ON_CHAIN_PROPOSER_ADDRESS' .env | cut -d= -f2) cargo test --release testito -- --nocapture


# Purge L2's state
UNAME_S:=$(shell uname -s)
# This directory is set by crates/l2/utils/prover/save_state.rs -> const DEFAULT_DATADIR
PROJECT_NAME:=ethrex_l2_state

ifeq ($(UNAME_S),Linux)
PROJECT_PATH := $(HOME)/.local/share/${PROJECT_NAME}
else ifeq ($(UNAME_S),Darwin)
PROJECT_PATH := $(HOME)/Library/Application\ Support/${PROJECT_NAME}
else
$(error Unsupported platform: $(UNAME_S))
endif

purge_prover_state: ## 🧹 Removes the L2 state, only use to start fresh.
@echo "Are you sure you want to delete the directory: $(PROJECT_PATH) ? [y/n]"
@read answer; \
if [ "$$answer" != "y" ]; then \
echo "Operation canceled."; \
fi; \
rm -rf $(PROJECT_PATH); \
echo "Directory deleted."
6 changes: 2 additions & 4 deletions crates/l2/contracts/deployer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -81,10 +81,8 @@ async fn main() -> Result<(), DeployError> {
)
.await?;

let sp1_contract_verifier_address = match sp1_verifier_address {
Some(address) => address,
None => setup_result.sp1_contract_verifier_address,
};
let sp1_contract_verifier_address =
sp1_verifier_address.unwrap_or(setup_result.sp1_contract_verifier_address);

initialize_contracts(
setup_result.deployer_address,
Expand Down
15 changes: 12 additions & 3 deletions crates/l2/proposer/errors.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
use std::sync::mpsc::SendError;

use crate::utils::config::errors::ConfigError;
use crate::utils::prover::errors::SaveStateError;
use ethereum_types::FromStrRadixErr;
use ethrex_core::types::{BlobsBundleError, FakeExponentialError};
use ethrex_dev::utils::engine_client::errors::EngineClientError;
Expand All @@ -22,6 +23,8 @@ pub enum L1WatcherError {
FailedToRetrieveChainConfig(String),
#[error("L1Watcher failed to get config: {0}")]
FailedToGetConfig(#[from] ConfigError),
#[error("{0}")]
Custom(String),
}

#[derive(Debug, thiserror::Error)]
Expand All @@ -32,9 +35,9 @@ pub enum ProverServerError {
EthClientError(#[from] EthClientError),
#[error("ProverServer failed to send transaction: {0}")]
FailedToVerifyProofOnChain(String),
#[error("ProverServer failed retrieve block from storage: {0}")]
FailedToRetrieveBlockFromStorage(#[from] StoreError),
#[error("ProverServer failed retrieve block from storaga, data is None.")]
#[error("ProverServer failed to access Store: {0}")]
FailedAccessingStore(#[from] StoreError),
#[error("ProverServer failed to retrieve block from storaga, data is None.")]
StorageDataIsNone,
#[error("ProverServer failed to create ProverInputs: {0}")]
FailedToCreateProverInputs(#[from] EvmError),
Expand All @@ -44,6 +47,12 @@ pub enum ProverServerError {
JoinError(#[from] JoinError),
#[error("ProverServer failed: {0}")]
Custom(String),
#[error("ProverServer failed to write to TcpStream: {0}")]
WriteError(String),
#[error("ProverServer failed to get data from Store: {0}")]
ItemNotFoundInStore(String),
#[error("ProverServer encountered a SaveStateError: {0}")]
SaveStateError(#[from] SaveStateError),
#[error("Failed to encode calldata: {0}")]
CalldataEncodeError(#[from] CalldataEncodeError),
}
Expand Down
8 changes: 3 additions & 5 deletions crates/l2/proposer/l1_watcher.rs
Original file line number Diff line number Diff line change
Expand Up @@ -227,11 +227,9 @@ impl L1Watcher {

let gas_price = self.l2_client.get_gas_price().await?;
// Avoid panicking when using as_u64()
let gas_price = if gas_price > u64::MAX.into() {
u64::MAX
} else {
gas_price.as_u64()
};
let gas_price: u64 = gas_price
.try_into()
.map_err(|_| L1WatcherError::Custom("Failed at gas_price.try_into()".to_owned()))?;
fborello-lambda marked this conversation as resolved.
Show resolved Hide resolved

let mut mint_transaction = self
.eth_client
Expand Down
Loading
Loading