Skip to content

feat(l1): ethrex_replay add tooling for periodic replay runs #4108

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 117 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
117 commits
Select commit Hold shift + click to select a range
c8c3e24
Fix ExecutionWitness serde
ilitteri Aug 7, 2025
92ccf11
Rewrite rebuild trie function
ilitteri Aug 7, 2025
f4ea7de
Fix chain config used in `ethrex-replay execute block`
ilitteri Aug 7, 2025
dd26674
Fix chain config in every command
ilitteri Aug 7, 2025
4c0fcbe
Retrieve only canonical blocks
ilitteri Aug 7, 2025
b3fb896
Improve logging
ilitteri Aug 7, 2025
6bb5b56
Remove leftover `dbg!`s
ilitteri Aug 7, 2025
df14985
Add network to cache file name
ilitteri Aug 7, 2025
af82b43
Update fetch_rangedata_from_client
ilitteri Aug 7, 2025
8741e6f
Add error log when debug_executionWitness request fails
ilitteri Aug 7, 2025
048ec5c
Remove needless field
ilitteri Aug 7, 2025
859e5fc
Implement `Deserialize` for `ExecutionWitnessResult`
ilitteri Aug 7, 2025
02e1362
Implement `Serialize` for `ExecutionWitnessResult`
ilitteri Aug 7, 2025
601c312
Cleanup
ilitteri Aug 7, 2025
e890ab3
Inserts must come before deletes, otherwise deletes might require ext…
ilitteri Aug 7, 2025
de51069
Update cmd/ethrex_replay/src/fetcher.rs
JereSalo Aug 8, 2025
098f901
Update cmd/ethrex_replay/src/cli.rs
JereSalo Aug 8, 2025
b8847f7
Update crates/common/types/block_execution_witness.rs
JereSalo Aug 8, 2025
f2d3c48
Rethink serde
ilitteri Aug 8, 2025
31fd4c5
Fix
ilitteri Aug 8, 2025
806fcdf
Fix trie rebuild in deserialization
ilitteri Aug 8, 2025
3d35ca6
Merge branch 'main' of github.com:lambdaclass/ethrex into fix_executi…
ilitteri Aug 8, 2025
646adf5
fill state and keys
JereSalo Aug 8, 2025
8cc7c69
Merge branch 'fix_execution_witness_serde' of github.com:lambdaclass/…
JereSalo Aug 8, 2025
fcdb1ed
Fix `ExecutionWitnessResult` serialization
ilitteri Aug 8, 2025
0421704
Add warn log when cache fails to be loaded
ilitteri Aug 8, 2025
d9dcabb
Add chain config to ProgramInput for convenience
ilitteri Aug 8, 2025
3bfe0f2
add chain_config in missing places
JereSalo Aug 8, 2025
f27d06d
Use correct chain id
ilitteri Aug 8, 2025
7cf2648
Merge branch 'fix_execution_witness_serde' of github.com:lambdaclass/…
ilitteri Aug 8, 2025
e721b08
Avoid rebuilding trie twice in L1
ilitteri Aug 8, 2025
f3bf3df
Merge branch 'main' into fix_execution_witness_serde
ilitteri Aug 8, 2025
24157c7
fix chain config problem
JereSalo Aug 8, 2025
fb7ac8a
Decouple `RpcExecutionWitness` from `ExecutionWitnessResult`
ilitteri Aug 9, 2025
15c10f9
Initial version of the forever run binary
ilitteri Aug 9, 2025
c950039
Wait at most 12 seconds for next execution
ilitteri Aug 9, 2025
5c305e7
Add comment
ilitteri Aug 9, 2025
022e9d9
Add useful log
ilitteri Aug 9, 2025
ade4e83
Fix log
ilitteri Aug 9, 2025
a5fa2d6
fix blockchain tests
JereSalo Aug 11, 2025
88566e5
Merge branch 'fix_execution_witness_serde' of github.com:lambdaclass/…
ilitteri Aug 11, 2025
6ce281a
Name script
ilitteri Aug 11, 2025
d8585f2
run cargo fmt
JereSalo Aug 11, 2025
e895dd8
make rebuild trie private
JereSalo Aug 11, 2025
94ebdda
remove temp log
ilitteri Aug 11, 2025
3da1c46
Rename replayer bin
ilitteri Aug 11, 2025
4c60354
Reorg replayer bin files
ilitteri Aug 11, 2025
b95f02c
remove chain config from ProverInput
JereSalo Aug 11, 2025
bed4fc1
initialize witness with ChainConfig and remove it from ProgramInput
JereSalo Aug 11, 2025
b57b5e2
remove duplicate function
JereSalo Aug 11, 2025
8e2546b
Add support for sending reports to slack
ilitteri Aug 11, 2025
6a15716
remove chainconfig from load_cache
JereSalo Aug 11, 2025
ab75cfe
cargo fmt
JereSalo Aug 11, 2025
9b3eff6
leave a todo
JereSalo Aug 11, 2025
d07c4dc
Add sepolia rpc url
ilitteri Aug 11, 2025
2926250
rename witness to rpc_witness in function
JereSalo Aug 11, 2025
33447ed
remove storage_trie_nodes because we have keys
JereSalo Aug 11, 2025
35ef067
simplify network clap
JereSalo Aug 11, 2025
6548ed7
refactor(l1): create cmd common module and move network to it (#4016)
JereSalo Aug 11, 2025
bf490fb
Add args to `replayer` bin
ilitteri Aug 11, 2025
1a8804b
create config crate
JereSalo Aug 11, 2025
e290ccb
run cargo fmt
JereSalo Aug 11, 2025
727f745
remove fixme comment
JereSalo Aug 11, 2025
0b5df09
Remove old error log
ilitteri Aug 11, 2025
bd585a7
Error log when execution fails and info log when it succeeds
ilitteri Aug 11, 2025
984140b
remove argument from rebuild_tries
JereSalo Aug 11, 2025
09cb2ee
change error type for parent block header not found
JereSalo Aug 11, 2025
7c40845
Cleanup
ilitteri Aug 11, 2025
3fc118a
Enable the possibility to prove and execute
ilitteri Aug 11, 2025
052355b
apply suggested nit when loading cache
JereSalo Aug 11, 2025
5575cdb
Fix display
ilitteri Aug 11, 2025
65f4562
remove unnecessary code and change comment
JereSalo Aug 12, 2025
4a1059e
Merge branch 'main' into fix_execution_witness_serde
JereSalo Aug 12, 2025
b3f5c96
nit in test_runner witness
JereSalo Aug 12, 2025
d8b701c
remove p2p from replay workspace
JereSalo Aug 12, 2025
e387035
add issue number
JereSalo Aug 12, 2025
04bd47d
Fix proving part of the script
ilitteri Aug 12, 2025
a49c67e
Merge branch 'main' of github.com:lambdaclass/ethrex into fix_executi…
ilitteri Aug 12, 2025
e119ef5
Merge branch 'fix_execution_witness_serde' of github.com:lambdaclass/…
ilitteri Aug 12, 2025
eb7e08d
Fix block run report generation
ilitteri Aug 13, 2025
6bb27dc
merge main
JereSalo Aug 13, 2025
5e66561
Merge branch 'fix_execution_witness_serde' of github.com:lambdaclass/…
ilitteri Aug 13, 2025
19e6d74
Add log to display which block is being replayed
ilitteri Aug 14, 2025
5cdb8c6
reafctor: make only one rpc url required
tomip01 Aug 14, 2025
a04a175
Merge branch 'fix_execution_witness_serde_bin' of github.com:lambdacl…
tomip01 Aug 14, 2025
366373b
Improve recently added log to display etherscan's block URLs
ilitteri Aug 14, 2025
53af66d
Notify to Slack all proven block results
ilitteri Aug 14, 2025
5d5544d
Fix format duration (show hours now)
ilitteri Aug 14, 2025
3e1f814
Merge branch 'main' into fix_execution_witness_serde
JereSalo Aug 14, 2025
efdd49b
Merge branch 'main' into fix_execution_witness_serde
ilitteri Aug 14, 2025
87b42b4
Merge branch 'fix_execution_witness_serde' of github.com:lambdaclass/…
ilitteri Aug 14, 2025
17385e5
Print time taken in slack message
ilitteri Aug 15, 2025
33521d9
Make slack message header consistent
ilitteri Aug 15, 2025
7427a23
Revert "Make slack message header consistent"
ilitteri Aug 15, 2025
0b22a4f
Merge branch 'main' into fix_execution_witness_serde
ilitteri Aug 18, 2025
9ce30b4
Merge branch 'main' of github.com:lambdaclass/ethrex into fix_executi…
ilitteri Aug 18, 2025
c20a5ed
Merge branch 'main' of github.com:lambdaclass/ethrex into fix_executi…
ilitteri Aug 19, 2025
0f59dcb
Fix bad merge
ilitteri Aug 19, 2025
98b41b3
Update cmd/ethrex_replay/src/fetcher.rs
JereSalo Aug 19, 2025
9c01482
remove duplicate code for rebuilding tries
JereSalo Aug 19, 2025
230259d
Merge branch 'fix_execution_witness_serde' of github.com:lambdaclass/…
ilitteri Aug 19, 2025
e9d8ac0
mention issue
JereSalo Aug 19, 2025
59a61ea
Merge branch 'fix_execution_witness_serde' of github.com:lambdaclass/…
ilitteri Aug 19, 2025
0966582
get parent block header correctly
JereSalo Aug 20, 2025
0190b3e
move config to common and remove src folder
JereSalo Aug 20, 2025
59cbbf2
replace panic for error
JereSalo Aug 20, 2025
627d67d
improve error parent block header
JereSalo Aug 20, 2025
437f399
merge main
JereSalo Aug 20, 2025
720fd0a
Merge branch 'main' into fix_execution_witness_serde
ilitteri Aug 20, 2025
c787cac
Merge branch 'fix_execution_witness_serde' of github.com:lambdaclass/…
ilitteri Aug 20, 2025
592011e
Merge main
ilitteri Aug 20, 2025
61fe7f1
Move replayer to tooling
ilitteri Aug 20, 2025
a87cece
Leftovers
ilitteri Aug 20, 2025
8bd0e41
Merge branch 'main' of github.com:lambdaclass/ethrex into fix_executi…
ilitteri Aug 20, 2025
9b9415f
Merge branch 'main' of github.com:lambdaclass/ethrex into fix_executi…
ilitteri Aug 21, 2025
6bc4db3
Merge branch 'main' of github.com:lambdaclass/ethrex into fix_executi…
ilitteri Aug 21, 2025
7392be9
Try send slack message in every prove result and only in failed execu…
ilitteri Aug 21, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ members = [
"tooling/load_test",
"tooling/loc",
"tooling/archive_sync",
"tooling/replayer",
"crates/common/config",
]
resolver = "2"
Expand Down
1 change: 1 addition & 0 deletions cmd/ethrex_replay/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ eyre.workspace = true
tokio = { version = "1.21", default-features = false, features = ["full"] }
clap.workspace = true
charming = { version = "0.4.0", features = ["ssr"] }
futures.workspace = true

jemallocator = { version = "0.5.4", optional = true }

Expand Down
13 changes: 6 additions & 7 deletions cmd/ethrex_replay/src/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ use crate::{bench::run_and_measure, fetcher::get_batchdata};
use ethrex_config::networks::Network;

pub const VERSION_STRING: &str = env!("CARGO_PKG_VERSION");
pub const BINARY_NAME: &str = env!("CARGO_BIN_NAME");

#[cfg(feature = "sp1")]
const BACKEND: Backend = Backend::SP1;
Expand All @@ -25,14 +24,14 @@ const BACKEND: Backend = Backend::RISC0;
const BACKEND: Backend = Backend::Exec;

#[derive(Parser)]
#[command(name=BINARY_NAME, author, version=VERSION_STRING, about, long_about = None)]
#[command(name="ethrex-replay", author, version=VERSION_STRING, about, long_about = None)]
pub struct EthrexReplayCLI {
#[command(subcommand)]
command: EthrexReplayCommand,
}

#[derive(Subcommand)]
enum SubcommandExecute {
pub enum SubcommandExecute {
#[command(about = "Execute a single block.")]
Block {
#[arg(help = "Block to use. Uses the latest if not specified.")]
Expand Down Expand Up @@ -198,13 +197,13 @@ impl SubcommandExecute {
}

#[derive(Subcommand)]
enum SubcommandProve {
pub enum SubcommandProve {
#[command(about = "Proves a single block.")]
Block {
#[arg(help = "Block to use. Uses the latest if not specified.")]
block: Option<usize>,
#[arg(long, env = "RPC_URL", required = true)]
rpc_url: String,
rpc_url: Url,
#[arg(
long,
help = "Name of the network or genesis file. Supported: mainnet, holesky, sepolia, hoodi. Default: mainnet",
Expand Down Expand Up @@ -260,7 +259,7 @@ impl SubcommandProve {
network,
bench,
} => {
let eth_client = EthClient::new(&rpc_url)?;
let eth_client = EthClient::new(rpc_url.as_str())?;
let block = or_latest(block)?;
let cache = get_blockdata(eth_client, network.clone(), block).await?;
let future = async {
Expand Down Expand Up @@ -313,7 +312,7 @@ impl SubcommandProve {
}

#[derive(Subcommand)]
enum EthrexReplayCommand {
pub enum EthrexReplayCommand {
#[command(
subcommand,
about = "Execute blocks, ranges of blocks, or individual transactions."
Expand Down
6 changes: 6 additions & 0 deletions cmd/ethrex_replay/src/lib.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
mod bench;
mod cache;
pub mod cli;
mod fetcher;
mod plot_composition;
mod run;
9 changes: 1 addition & 8 deletions cmd/ethrex_replay/src/main.rs
Original file line number Diff line number Diff line change
@@ -1,13 +1,6 @@
use std::str::FromStr;
use tracing_subscriber::filter::Directive;

mod bench;
mod cache;
mod cli;
mod fetcher;
mod plot_composition;
mod run;

#[cfg(feature = "jemalloc")]
use jemallocator::Jemalloc;

Expand All @@ -30,7 +23,7 @@ async fn main() {
.finish(),
)
.expect("setting default subscriber failed");
if let Err(e) = cli::start().await {
if let Err(e) = ethrex_replay::cli::start().await {
tracing::error!("{e:?}");
std::process::exit(1);
}
Expand Down
30 changes: 30 additions & 0 deletions tooling/replayer/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
[package]
name = "ethrex-replayer"
version = "0.1.0"
edition = "2024"

[dependencies]
ethrex-config.workspace = true
ethrex-common.workspace = true
ethrex-rpc.workspace = true
ethrex-replay = { path = "../../cmd/ethrex_replay" }

serde.workspace = true
hex.workspace = true
serde_json.workspace = true
serde_with.workspace = true
reqwest.workspace = true
tracing-subscriber = { workspace = true, features = ["env-filter"] }
tracing.workspace = true
eyre.workspace = true
tokio = { version = "1.21", default-features = false, features = ["full"] }
clap.workspace = true
futures.workspace = true

[features]
risc0 = ["ethrex-replay/risc0"]
sp1 = ["ethrex-replay/sp1"]
gpu = ["ethrex-replay/gpu"]
l2 = ["ethrex-replay/l2"]
profiling = ["ethrex-replay/profiling"]
ci = ["ethrex-replay/ci"]
Loading
Loading