Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Custom Base Token Gas Price Oracle #996

Merged
merged 71 commits into from
Apr 9, 2024
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
a15cd2c
[WIP] Gas oracle to convert native token price
jrchatruc Feb 1, 2024
dcd2721
zk fmt
jrchatruc Feb 1, 2024
c8626a7
Connect with fetcher with config data to get the host and poll interval
jrchatruc Feb 1, 2024
06ec84d
Remove unused fields
jrchatruc Feb 1, 2024
5c6b647
add erc20 fetcher to gas adjuster
juan518munoz Feb 8, 2024
9a055f2
fix env vars names, remove unused variable
Feb 16, 2024
a16252b
remove cast to u64
Feb 16, 2024
4fe5b7a
remove unused field from ZkSyncStateKeeper
Feb 16, 2024
b7cbd70
Merge remote-tracking branch 'upstream/kl-factory' into gas-oracle
jrchatruc Feb 20, 2024
c3a8d80
Merge remote-tracking branch 'upstream/kl-factory' into gas-oracle
jrchatruc Feb 20, 2024
5de1233
Use = &mut instead of ref mut
jrchatruc Feb 26, 2024
773ff7a
Fetch the conversion rate on Fetcher initialization to avoid incorrec…
jrchatruc Feb 26, 2024
3cb3e5c
Merge remote-tracking branch 'upstream/kl-factory' into gas-oracle
jrchatruc Feb 26, 2024
fd3e7d2
Add http client as a field to the native erc20 fetcher
jrchatruc Feb 27, 2024
6838588
Rename NativeErc20 stuff to be NativeToken instead
jrchatruc Feb 27, 2024
2ebe86b
Merge remote-tracking branch 'upstream/lambdaclass_gas_oracle' into c…
jrchatruc Feb 27, 2024
67286bc
initial commit
juan518munoz Feb 28, 2024
08e8b5a
remove comment
juan518munoz Feb 28, 2024
7ca95b8
remove needless variable
juan518munoz Feb 28, 2024
0315e46
initial commit
juan518munoz Feb 28, 2024
c79dccb
initial commit
juan518munoz Feb 28, 2024
b6367ad
rmv unwrap
juan518munoz Feb 28, 2024
252312a
improve err handling
juan518munoz Feb 28, 2024
83dd481
Merge pull request #1263 from lambdaclass/cache-http-client
jrchatruc Feb 29, 2024
e91c27c
Merge pull request #1272 from lambdaclass/gas-oracle-noop-conversion-…
jrchatruc Feb 29, 2024
7214d5e
Merge branch 'lambdaclass_gas_oracle' into gas-oracle-change-err-type
juan518munoz Feb 29, 2024
7b33934
store native token address in fetcher config, use it in the conversio…
Feb 29, 2024
e4fb1c4
add mission EOF
Feb 29, 2024
6d114f3
remove print
Feb 29, 2024
3d4addd
Merge pull request #1281 from lambdaclass/gas-oracle-change-err-type
jrchatruc Feb 29, 2024
97f9eec
Merge branch 'lambdaclass_gas_oracle' into gas-oracle-imprv-fetcher-err
juan518munoz Feb 29, 2024
5eb92bc
impl ErrorReporter
juan518munoz Feb 29, 2024
13b84d0
use Address struct for token address
Mar 1, 2024
7f61a38
move struct & add context to err
juan518munoz Mar 1, 2024
a98a0b7
add context to init
juan518munoz Mar 1, 2024
e182381
add conversion rate component
Mar 1, 2024
6bb64ef
apply zk fmt
Mar 1, 2024
e7241fc
add token address to config
Mar 1, 2024
bfae1c6
Add missing trait
Mar 1, 2024
7077b40
add token address to config
Mar 1, 2024
3ab731b
Merge pull request #1301 from lambdaclass/add_token_address_to_api_su…
jrchatruc Mar 4, 2024
1154e54
merge lambdaclass_gas_oracle
juan518munoz Mar 4, 2024
79a8eb2
Merge pull request #1282 from lambdaclass/gas-oracle-imprv-fetcher-err
jrchatruc Mar 4, 2024
984e052
Merge branch 'lambdaclass_gas_oracle' into conversion_rate_component
juan518munoz Mar 4, 2024
dd11e4a
fix up code
juan518munoz Mar 4, 2024
1a20f2a
[WIP] Remove native token fetcher singleton
jrchatruc Mar 5, 2024
3103dca
Fix compilation (add missing await)
jrchatruc Mar 5, 2024
0e2f572
Merge remote-tracking branch 'upstream/lambdaclass_gas_oracle' into r…
jrchatruc Mar 5, 2024
cf5f2ff
fix stop handle for converstion rate api
juan518munoz Mar 6, 2024
46b6ef4
add logging for conversion rate
juan518munoz Mar 6, 2024
6e0c812
Remove unnecessary alert_spawned boolean
jrchatruc Mar 7, 2024
13cf343
Revert "Remove unnecessary alert_spawned boolean"
jrchatruc Mar 8, 2024
fbf66b1
Use a mutex for the errorreporter
jrchatruc Mar 8, 2024
a19be60
Revert changes to .init.env
jrchatruc Mar 8, 2024
d99e9bf
rmv flask app, rename component & refactor func
juan518munoz Mar 11, 2024
8d37924
rename module
juan518munoz Mar 11, 2024
feaf132
Merge pull request #1319 from lambdaclass/conversion_rate_component
jrchatruc Mar 11, 2024
3fff7d9
Merge remote-tracking branch 'upstream/lambdaclass_gas_oracle' into r…
jrchatruc Mar 11, 2024
ba7762c
Merge pull request #1360 from lambdaclass/remove-token-fetcher-singleton
jrchatruc Mar 11, 2024
d81aef0
Merge branch 'kl-factory' into lambdaclass_gas_oracle
jrchatruc Mar 12, 2024
5680df8
zk fmt
juan518munoz Mar 19, 2024
47c444f
Merge remote-tracking branch 'matter-labs/kl-factory' into gas-oracle
juan518munoz Mar 20, 2024
b520744
rename native token to base token
juan518munoz Mar 25, 2024
29289c9
fix env load
juan518munoz Mar 25, 2024
7c8ec45
add gas oracle documentation
Mar 27, 2024
2a7761d
fmt
Mar 27, 2024
55ae89f
Merge branch 'kl-factory' into gas-oracle
fkrause98 Mar 27, 2024
bac7a4c
Merge branch 'kl-factory' into gas-oracle
fkrause98 Apr 8, 2024
cd2e52b
chore(fmt): zk fmt
fkrause98 Apr 8, 2024
e25b33e
feat: add resources/conversion_rate_fetcher.rs
fborello-lambda Apr 8, 2024
1d85502
Merge branch 'kl-factory' into gas-oracle
fkrause98 Apr 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion contracts
2 changes: 2 additions & 0 deletions core/bin/zksync_server/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ use zksync_config::{
},
fri_prover_group::FriProverGroupConfig,
house_keeper::HouseKeeperConfig,
native_erc20_fetcher::NativeErc20FetcherConfig,
FriProofCompressorConfig, FriProverConfig, FriWitnessGeneratorConfig, PrometheusConfig,
ProofDataHandlerConfig, WitnessGeneratorConfig,
},
Expand Down Expand Up @@ -119,6 +120,7 @@ async fn main() -> anyhow::Result<()> {
eth_watch_config: ETHWatchConfig::from_env().ok(),
gas_adjuster_config: GasAdjusterConfig::from_env().ok(),
object_store_config: ObjectStoreConfig::from_env().ok(),
native_erc20_fetcher_config: NativeErc20FetcherConfig::from_env().ok(),
};

let postgres_config = configs.postgres_config.clone().context("PostgresConfig")?;
Expand Down
1 change: 1 addition & 0 deletions core/lib/config/src/configs/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ pub mod fri_prover_group;
pub mod fri_witness_generator;
pub mod fri_witness_vector_generator;
pub mod house_keeper;
pub mod native_erc20_fetcher;
pub mod object_store;
pub mod proof_data_handler;
pub mod snapshots_creator;
Expand Down
7 changes: 7 additions & 0 deletions core/lib/config/src/configs/native_erc20_fetcher.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
use serde::Deserialize;

#[derive(Debug, Clone, Deserialize, PartialEq)]
pub struct NativeErc20FetcherConfig {
pub poll_interval: u64,
pub host: String,
}
1 change: 1 addition & 0 deletions core/lib/env_config/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ mod fri_prover_group;
mod fri_witness_generator;
mod fri_witness_vector_generator;
mod house_keeper;
pub mod native_erc20_fetcher;
pub mod object_store;
mod proof_data_handler;
mod snapshots_creator;
Expand Down
9 changes: 9 additions & 0 deletions core/lib/env_config/src/native_erc20_fetcher.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
use zksync_config::configs::native_erc20_fetcher::NativeErc20FetcherConfig;

use crate::{envy_load, FromEnv};

impl FromEnv for NativeErc20FetcherConfig {
fn from_env() -> anyhow::Result<Self> {
envy_load("native_erc20_fetcher", "NATIVE_ERC20_FETCHER_")
}
}
1 change: 1 addition & 0 deletions core/lib/zksync_core/src/consensus/testonly.rs
Original file line number Diff line number Diff line change
Expand Up @@ -375,6 +375,7 @@ impl StateKeeperRunner {
Box::new(io),
Box::new(MockBatchExecutorBuilder),
Box::new(NoopSealer),
Box::new(None),
)
.run(),
);
Expand Down
17 changes: 14 additions & 3 deletions core/lib/zksync_core/src/l1_gas_price/gas_adjuster/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ use zksync_system_constants::L1_GAS_PER_PUBDATA_BYTE;

use self::metrics::METRICS;
use super::{L1GasPriceProvider, L1TxParamsProvider};
use crate::state_keeper::metrics::KEEPER_METRICS;
use crate::{native_erc20_fetcher::Erc20Fetcher, state_keeper::metrics::KEEPER_METRICS};

mod metrics;
#[cfg(test)]
Expand All @@ -25,10 +25,15 @@ pub struct GasAdjuster<E> {
pub(super) statistics: GasStatistics,
pub(super) config: GasAdjusterConfig,
eth_client: E,
erc20_fetcher_dyn: Option<Arc<dyn Erc20Fetcher>>,
}

impl<E: EthInterface> GasAdjuster<E> {
pub async fn new(eth_client: E, config: GasAdjusterConfig) -> Result<Self, Error> {
pub async fn new(
eth_client: E,
config: GasAdjusterConfig,
erc20_fetcher_dyn: Option<Arc<dyn Erc20Fetcher>>,
) -> Result<Self, Error> {
// Subtracting 1 from the "latest" block number to prevent errors in case
// the info about the latest block is not yet present on the node.
// This sometimes happens on Infura.
Expand All @@ -44,6 +49,7 @@ impl<E: EthInterface> GasAdjuster<E> {
statistics: GasStatistics::new(config.max_base_fee_samples, current_block, &history),
eth_client,
config,
erc20_fetcher_dyn,
})
}

Expand Down Expand Up @@ -125,7 +131,12 @@ impl<E: EthInterface> L1GasPriceProvider for GasAdjuster<E> {
(self.config.internal_l1_pricing_multiplier * effective_gas_price as f64) as u64;

// Bound the price if it's too high.
self.bound_gas_price(calculated_price)
let conversion_rate = match self.erc20_fetcher_dyn.as_ref() {
Some(fetcher) => fetcher.conversion_rate().unwrap_or(1),
mationorato marked this conversation as resolved.
Show resolved Hide resolved
None => 1,
};

self.bound_gas_price(calculated_price) * conversion_rate
mationorato marked this conversation as resolved.
Show resolved Hide resolved
}

fn estimate_effective_pubdata_price(&self) -> u64 {
Expand Down
20 changes: 15 additions & 5 deletions core/lib/zksync_core/src/l1_gas_price/singleton.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ use tokio::{
use zksync_config::GasAdjusterConfig;
use zksync_eth_client::clients::QueryClient;

use crate::l1_gas_price::GasAdjuster;
use crate::{l1_gas_price::GasAdjuster, native_erc20_fetcher::Erc20Fetcher};

/// Special struct for creating a singleton of `GasAdjuster`.
/// This is needed only for running the server.
Expand All @@ -17,6 +17,7 @@ pub struct GasAdjusterSingleton {
web3_url: String,
gas_adjuster_config: GasAdjusterConfig,
singleton: OnceCell<Result<Arc<GasAdjuster<QueryClient>>, Error>>,
erc20_fetcher_dyn: Option<Arc<dyn Erc20Fetcher>>,
}

#[derive(thiserror::Error, Debug, Clone)]
Expand All @@ -30,11 +31,16 @@ impl From<anyhow::Error> for Error {
}

impl GasAdjusterSingleton {
pub fn new(web3_url: String, gas_adjuster_config: GasAdjusterConfig) -> Self {
pub fn new(
web3_url: String,
gas_adjuster_config: GasAdjusterConfig,
erc20_fetcher_dyn: Option<Arc<dyn Erc20Fetcher>>,
mationorato marked this conversation as resolved.
Show resolved Hide resolved
) -> Self {
Self {
web3_url,
gas_adjuster_config,
singleton: OnceCell::new(),
erc20_fetcher_dyn,
}
}

Expand All @@ -44,9 +50,13 @@ impl GasAdjusterSingleton {
.get_or_init(|| async {
let query_client =
QueryClient::new(&self.web3_url).context("QueryClient::new()")?;
let adjuster = GasAdjuster::new(query_client.clone(), self.gas_adjuster_config)
.await
.context("GasAdjuster::new()")?;
let adjuster = GasAdjuster::new(
query_client.clone(),
self.gas_adjuster_config,
self.erc20_fetcher_dyn.clone(),
)
.await
.context("GasAdjuster::new()")?;
Ok(Arc::new(adjuster))
})
.await;
Expand Down
54 changes: 50 additions & 4 deletions core/lib/zksync_core/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ use std::{net::Ipv4Addr, str::FromStr, sync::Arc, time::Instant};
use anyhow::Context as _;
use fee_model::MainNodeFeeInputProvider;
use futures::channel::oneshot;
use native_erc20_fetcher::Erc20Fetcher;
use prometheus_exporter::PrometheusExporterConfig;
use temp_config_store::TempConfigStore;
use tokio::{sync::watch, task::JoinHandle};
Expand Down Expand Up @@ -69,6 +70,7 @@ use crate::{
l1_gas_price::{GasAdjusterSingleton, L1GasPriceProvider},
metadata_calculator::{MetadataCalculator, MetadataCalculatorConfig},
metrics::{InitStage, APP_METRICS},
native_erc20_fetcher::NativeErc20FetcherSingleton,
state_keeper::{
create_state_keeper, MempoolFetcher, MempoolGuard, MiniblockSealer, SequencerSealer,
},
Expand All @@ -88,6 +90,7 @@ pub mod house_keeper;
pub mod l1_gas_price;
pub mod metadata_calculator;
mod metrics;
pub mod native_erc20_fetcher;
pub mod proof_data_handler;
pub mod reorg_detector;
pub mod state_keeper;
Expand Down Expand Up @@ -230,6 +233,8 @@ pub enum Component {
Housekeeper,
/// Component for exposing APIs to prover for providing proof generation data and accepting proofs.
ProofDataHandler,
/// Native ERC20 fetcher
NativeERC20Fetcher,
mationorato marked this conversation as resolved.
Show resolved Hide resolved
}

#[derive(Debug)]
Expand Down Expand Up @@ -264,6 +269,7 @@ impl FromStr for Components {
"eth_tx_aggregator" => Ok(Components(vec![Component::EthTxAggregator])),
"eth_tx_manager" => Ok(Components(vec![Component::EthTxManager])),
"proof_data_handler" => Ok(Components(vec![Component::ProofDataHandler])),
"native_erc20_fetcher" => Ok(Components(vec![Component::NativeERC20Fetcher])),
other => Err(format!("{} is not a valid component name", other)),
}
}
Expand Down Expand Up @@ -320,14 +326,44 @@ pub async fn initialize_components(
panic!("Circuit breaker triggered: {}", err);
});

let query_client = QueryClient::new(&eth_client_config.web3_url).unwrap();
let gas_adjuster_config = configs.gas_adjuster_config.context("gas_adjuster_config")?;
let mut gas_adjuster =
GasAdjusterSingleton::new(eth_client_config.web3_url.clone(), gas_adjuster_config);
// spawn the native ERC20 fetcher if it is enabled
let mut fetcher_component = if components.contains(&Component::NativeERC20Fetcher) {
let fetcher = NativeErc20FetcherSingleton::new(
configs
.native_erc20_fetcher_config
.clone()
.context("native_erc20_fetcher_config")?,
);

Some(fetcher)
} else {
None
};
let (stop_sender, stop_receiver) = watch::channel(false);
let (cb_sender, cb_receiver) = oneshot::channel();

let native_erc20_fetcher = if let Some(ref mut fetcher_singleton) = fetcher_component {
popzxc marked this conversation as resolved.
Show resolved Hide resolved
let fetcher = fetcher_singleton
.get_or_init()
.await
.context("fetcher.get_or_init()")?;
Some(fetcher)
} else {
None
};

let erc20_fetcher_dyn: Option<Arc<dyn Erc20Fetcher>> = native_erc20_fetcher
.as_ref()
.map(|fetcher| fetcher.clone() as Arc<dyn Erc20Fetcher>);

let query_client = QueryClient::new(&eth_client_config.web3_url).unwrap();
let gas_adjuster_config = configs.gas_adjuster_config.context("gas_adjuster_config")?;
let mut gas_adjuster = GasAdjusterSingleton::new(
eth_client_config.web3_url.clone(),
gas_adjuster_config,
erc20_fetcher_dyn,
);

// Prometheus exporter and circuit breaker checker should run for every component configuration.
let prom_config = configs
.prometheus_config
Expand Down Expand Up @@ -482,6 +518,7 @@ pub async fn initialize_components(
.get_or_init()
.await
.context("gas_adjuster.get_or_init()")?;

add_state_keeper_to_task_futures(
&mut task_futures,
&postgres_config,
Expand Down Expand Up @@ -671,6 +708,15 @@ pub async fn initialize_components(
if let Some(task) = gas_adjuster.run_if_initialized(stop_receiver.clone()) {
task_futures.push(task);
}

// check if the native ERC20 fetcher is enabled and run it if it is
fetcher_component
.and_then(|c| c.run_if_initialized(stop_receiver.clone()))
.into_iter()
.for_each(|handle| {
task_futures.push(handle);
});

Ok((task_futures, stop_sender, cb_receiver, health_check_handle))
}

Expand Down
107 changes: 107 additions & 0 deletions core/lib/zksync_core/src/native_erc20_fetcher/mod.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
use std::{sync::Arc, time::Duration};

use async_trait::async_trait;
use metrics::atomics::AtomicU64;
use tokio::{
sync::{watch, OnceCell},
task::JoinHandle,
};
use zksync_config::configs::native_erc20_fetcher::NativeErc20FetcherConfig;

// TODO: this error type is also defined by the gasAdjuster module,
// we should consider moving it to a common place
#[derive(thiserror::Error, Debug, Clone)]
#[error(transparent)]
pub struct Error(Arc<anyhow::Error>);
mationorato marked this conversation as resolved.
Show resolved Hide resolved

impl From<anyhow::Error> for Error {
fn from(err: anyhow::Error) -> Self {
Self(Arc::new(err))
}
}
mationorato marked this conversation as resolved.
Show resolved Hide resolved

pub trait Erc20Fetcher: 'static + std::fmt::Debug + Send + Sync {
popzxc marked this conversation as resolved.
Show resolved Hide resolved
fn conversion_rate(&self) -> anyhow::Result<u64>;
mationorato marked this conversation as resolved.
Show resolved Hide resolved
}

pub(crate) struct NativeErc20FetcherSingleton {
mationorato marked this conversation as resolved.
Show resolved Hide resolved
native_erc20_fetcher_config: NativeErc20FetcherConfig,
singleton: OnceCell<Result<Arc<NativeErc20Fetcher>, Error>>,
}

impl NativeErc20FetcherSingleton {
pub fn new(native_erc20_fetcher_config: NativeErc20FetcherConfig) -> Self {
Self {
native_erc20_fetcher_config,
singleton: OnceCell::new(),
}
}

pub async fn get_or_init(&mut self) -> Result<Arc<NativeErc20Fetcher>, Error> {
let adjuster = self
.singleton
.get_or_init(|| async {
let fetcher = NativeErc20Fetcher::new(self.native_erc20_fetcher_config.clone());
Ok(Arc::new(fetcher))
})
.await;
adjuster.clone()
}

pub fn run_if_initialized(
self,
stop_signal: watch::Receiver<bool>,
) -> Option<JoinHandle<anyhow::Result<()>>> {
let fetcher = self.singleton.get()?.clone();
Some(tokio::spawn(async move { fetcher?.run(stop_signal).await }))
}
}

#[derive(Debug)]
pub(crate) struct NativeErc20Fetcher {
// TODO: we probably need to add a http client here
// to avoid creating a new one for each request
pub config: NativeErc20FetcherConfig,
mationorato marked this conversation as resolved.
Show resolved Hide resolved
pub latest_to_eth_conversion_rate: AtomicU64,
}

impl NativeErc20Fetcher {
pub(crate) fn new(config: NativeErc20FetcherConfig) -> Self {
Self {
config,
latest_to_eth_conversion_rate: AtomicU64::new(0),
mationorato marked this conversation as resolved.
Show resolved Hide resolved
}
}

pub(crate) async fn run(&self, stop_receiver: watch::Receiver<bool>) -> anyhow::Result<()> {
loop {
if *stop_receiver.borrow() {
tracing::info!("Stop signal received, eth_tx_manager is shutting down");
break;
}

let conversion_rate = reqwest::get(format!("{}/conversion_rate", &self.config.host))
mationorato marked this conversation as resolved.
Show resolved Hide resolved
.await?
mationorato marked this conversation as resolved.
Show resolved Hide resolved
.json::<u64>()
.await
.unwrap();
mationorato marked this conversation as resolved.
Show resolved Hide resolved

self.latest_to_eth_conversion_rate
.store(conversion_rate, std::sync::atomic::Ordering::Relaxed);
mationorato marked this conversation as resolved.
Show resolved Hide resolved

tokio::time::sleep(Duration::from_secs(self.config.poll_interval)).await;
}

Ok(())
}
}

#[async_trait]
impl Erc20Fetcher for NativeErc20Fetcher {
fn conversion_rate(&self) -> anyhow::Result<u64> {
anyhow::Ok(
self.latest_to_eth_conversion_rate
.load(std::sync::atomic::Ordering::Relaxed),
)
}
}
1 change: 1 addition & 0 deletions core/lib/zksync_core/src/state_keeper/tests/tester.rs
Original file line number Diff line number Diff line change
Expand Up @@ -201,6 +201,7 @@ impl TestScenario {
Box::new(io),
Box::new(batch_executor_base),
Box::new(sealer),
Box::new(None),
);
let sk_thread = tokio::spawn(sk.run());

Expand Down
1 change: 1 addition & 0 deletions core/lib/zksync_core/src/sync_layer/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ impl StateKeeperHandles {
Box::new(io),
Box::new(batch_executor_base),
Box::new(NoopSealer),
Box::new(None),
);
Self {
stop_sender,
Expand Down
Loading
Loading