feat(l1): `ethrex_replay` add tooling for periodic replay runs #4108

tomip01 · 2025-08-20T21:22:31Z

Motivation

We urge the need for a tool for internal usage that replays L1 blocks from public networks like Hoodi, Sepolia, and Mainnet periodically (either execution with or without SP1, or proving with SP1) to ensure ethrex-replay works correctly and also to find potential bugs in ethrex.

Description

Replayer script that executes or proves the latest public network blocks in a loop and notifies to a Slack channel through a slack web hook if the flag --slack-web-hook is passed.

Usage

Usage: ethrex-replayer [OPTIONS] <--hoodi-rpc-url <URL>|--sepolia-rpc-url <URL>|--mainnet-rpc-url <URL>>

Options:
  -h, --help  Print help

Replayer options:
      --slack-webhook-url <URL>  [env: SLACK_WEBHOOK_URL=]
      --hoodi-rpc-url <URL>      [env: HOODI_RPC_URL=]
      --sepolia-rpc-url <URL>    [env: SEPOLIA_RPC_URL=]
      --mainnet-rpc-url <URL>    [env: MAINNET_RPC_URL=]
      --execute                  Replayer will execute blocks
      --prove                    Replayer will prove blocks

How to test

To periodically execute the latest Hoodi blocks, run:

cargo run --release -p ethrex-replayer --bin ethrex-replayer -- --hoodi-rpc-url http://65.108.69.58:8545 --execute

…ra nodes

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…on_witness_serde

…ethrex into fix_execution_witness_serde

…ethrex into fix_execution_witness_serde_bin

Copilot

Pull Request Overview

This PR removes the redb database backend from the codebase and adds a new replayer tool for automated block execution/proving. The changes focus on streamlining the database architecture while introducing tooling for periodic replay runs across different networks.

Remove redb database support entirely, keeping only libmdbx as the database backend
Refactor execution witness data structures to use a simpler, more consistent format
Add a new replayer tool that can execute or prove latest blocks from public networks with Slack notifications

Reviewed Changes

Copilot reviewed 49 out of 51 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tooling/replayer/src/*	New replayer tool for periodic block execution/proving with Slack integration
crates/storage/*	Remove all `redb` backend code and dependencies
crates/common/types/block_execution_witness.rs	Major refactoring of witness data structures
crates/common/config/*	New config crate for network configurations
crates/networking/rpc/debug/*	Move execution witness RPC code to separate module

Comments suppressed due to low confidence (1)

crates/common/types/block_execution_witness.rs:174

[nitpick] Using expect with a panic message in production code should be avoided. Consider using proper error handling and returning a Result type instead.

                            Trie::from_nodes(None, &[]).expect("failed to create empty trie")

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-08-20T21:23:59Z

tooling/replayer/src/main.rs

+            let (result, _index, _remaining_handles) = select_all(revalidation_handles).await;
+            result
+        } => {
+            handle_rpc_revalidation_handle_result(res, opts.hoodi_rpc_url.unwrap(), opts.slack_webhook_url.clone()).await; // TODO: change hoodi rpc to generic


This line will panic if hoodi_rpc_url is None. The code assumes it exists but this is not guaranteed. Consider using the actual failed RPC URL from the revalidation handle result instead.

Suggested change

handle_rpc_revalidation_handle_result(res, opts.hoodi_rpc_url.unwrap(), opts.slack_webhook_url.clone()).await; // TODO: change hoodi rpc to generic

let ((result, url), _index, _remaining_handles) = select_all(revalidation_handles).await;

(result, url)

} => {

let (res, url) = res;

handle_rpc_revalidation_handle_result(res, url, opts.slack_webhook_url.clone()).await;

Copilot · 2025-08-20T21:23:59Z

tooling/replayer/src/main.rs

+        )
+        .await
+        .unwrap_or_else(|e| {
+            tracing::error!("Failed to notify Slack about invalid Sepolia RPC: {e}");


The error message hardcodes 'Sepolia RPC' but this function is called generically for any RPC URL. The error message should reflect the actual RPC URL that failed.

Suggested change

tracing::error!("Failed to notify Slack about invalid Sepolia RPC: {e}");

network: Network,

slack_webhook_url: Option<Url>,

) {

if let Err(e) = res {

tracing::error!("RPC URL `{}` for network `{}` failed: {}", rpc_url, network, e);

try_notify_no_longer_valid_rpc_to_slack(

rpc_url,

network,

slack_webhook_url,

)

.await

.unwrap_or_else(|e| {

tracing::error!("Failed to notify Slack about invalid RPC URL `{}` for network `{}`: {}", rpc_url, network, e);

Copilot · 2025-08-20T21:23:59Z

tooling/replayer/src/block_execution_report.rs

+    let hours = total_seconds / 3600;
+    let minutes = (total_seconds % 3600) / 60;
+    let seconds = total_seconds % 60;
+    let milliseconds = total_seconds / 1000;


This calculation is incorrect. It should be duration.subsec_millis() or (duration.as_millis() % 1000) to get the millisecond component. Currently it's dividing total seconds by 1000, which will always be 0 for durations less than 1000 seconds.

Suggested change

let milliseconds = total_seconds / 1000;

let milliseconds = duration.subsec_millis();

Copilot · 2025-08-20T21:24:00Z

cmd/ethrex_replay/src/fetcher.rs

+        }
+        Err(e) => {
+            error!("{e}");
+            todo!("Retry with eth_getProofs")


The todo! macro will panic at runtime. This should be replaced with proper error handling or a placeholder implementation that doesn't cause the program to crash.

Suggested change

todo!("Retry with eth_getProofs")

return Err(eyre::eyre!(

"Failed to retrieve witness for block {requested_block_number}: {e}. Retry with eth_getProofs not yet implemented."

));

Copilot · 2025-08-20T21:24:00Z

tooling/replayer/src/main.rs

+            .await?;
+        }
+        let elapsed = start.elapsed().unwrap_or_else(|e| {
+            panic!("SystemTime::elapsed failed: {e}");


[nitpick] Using panic! inside unwrap_or_else defeats the purpose of graceful error handling. Consider logging the error and using a default duration instead of panicking.

Suggested change

panic!("SystemTime::elapsed failed: {e}");

tracing::error!("SystemTime::elapsed failed: {e}. Using default duration of 12 seconds.");

Duration::from_secs(12)

…on_witness_serde_bin

github-actions · 2025-08-20T21:59:33Z

Lines of code report

Total lines added: 714
Total lines removed: 7
Total lines changed: 721

Detailed view

+-------------------------------------------------------+-------+------+
| File                                                  | Lines | Diff |
+-------------------------------------------------------+-------+------+
| ethrex/cmd/ethrex_replay/src/cli.rs                   | 401   | -1   |
+-------------------------------------------------------+-------+------+
| ethrex/cmd/ethrex_replay/src/lib.rs                   | 6     | +6   |
+-------------------------------------------------------+-------+------+
| ethrex/cmd/ethrex_replay/src/main.rs                  | 26    | -6   |
+-------------------------------------------------------+-------+------+
| ethrex/tooling/replayer/src/block_execution_report.rs | 295   | +295 |
+-------------------------------------------------------+-------+------+
| ethrex/tooling/replayer/src/main.rs                   | 379   | +379 |
+-------------------------------------------------------+-------+------+
| ethrex/tooling/replayer/src/slack.rs                  | 34    | +34  |
+-------------------------------------------------------+-------+------+

…on_witness_serde_bin

…tions

ilitteri and others added 30 commits August 7, 2025 15:05

Fix ExecutionWitness serde

c8c3e24

Rewrite rebuild trie function

92ccf11

Fix chain config used in ethrex-replay execute block

f4ea7de

Fix chain config in every command

dd26674

Retrieve only canonical blocks

4c0fcbe

Improve logging

b3fb896

Remove leftover dbg!s

6bb5b56

Add network to cache file name

df14985

Update fetch_rangedata_from_client

af82b43

Add error log when debug_executionWitness request fails

8741e6f

Remove needless field

048ec5c

Implement Deserialize for ExecutionWitnessResult

859e5fc

Implement Serialize for ExecutionWitnessResult

02e1362

Cleanup

601c312

Inserts must come before deletes, otherwise deletes might require ext…

e890ab3

…ra nodes

Update cmd/ethrex_replay/src/fetcher.rs

de51069

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update cmd/ethrex_replay/src/cli.rs

098f901

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update crates/common/types/block_execution_witness.rs

b8847f7

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Rethink serde

f2d3c48

Fix

31fd4c5

Fix trie rebuild in deserialization

806fcdf

Merge branch 'main' of github.com:lambdaclass/ethrex into fix_executi…

3d35ca6

…on_witness_serde

fill state and keys

646adf5

Merge branch 'fix_execution_witness_serde' of github.com:lambdaclass/…

8cc7c69

…ethrex into fix_execution_witness_serde

Fix ExecutionWitnessResult serialization

fcdb1ed

Add warn log when cache fails to be loaded

0421704

Add chain config to ProgramInput for convenience

d9dcabb

add chain_config in missing places

3bfe0f2

Use correct chain id

f27d06d

Merge branch 'fix_execution_witness_serde' of github.com:lambdaclass/…

7cf2648

…ethrex into fix_execution_witness_serde

JereSalo and others added 13 commits August 19, 2025 11:06

remove duplicate code for rebuilding tries

9c01482

Merge branch 'fix_execution_witness_serde' of github.com:lambdaclass/…

230259d

…ethrex into fix_execution_witness_serde_bin

mention issue

e9d8ac0

Merge branch 'fix_execution_witness_serde' of github.com:lambdaclass/…

59a61ea

…ethrex into fix_execution_witness_serde_bin

get parent block header correctly

0966582

move config to common and remove src folder

0190b3e

replace panic for error

59cbbf2

improve error parent block header

627d67d

merge main

437f399

Merge branch 'main' into fix_execution_witness_serde

720fd0a

Merge branch 'fix_execution_witness_serde' of github.com:lambdaclass/…

c787cac

…ethrex into fix_execution_witness_serde_bin

Merge main

592011e

Move replayer to tooling

61fe7f1

Copilot AI review requested due to automatic review settings August 20, 2025 21:22

tomip01 requested a review from a team as a code owner August 20, 2025 21:22

tomip01 assigned tomip01 and ilitteri and unassigned tomip01 Aug 20, 2025

tomip01 added L2 Rollup client L1 Ethereum client labels Aug 20, 2025

github-project-automation bot added this to ethrex_l2 and ethrex_l1 Aug 20, 2025

Copilot AI reviewed Aug 20, 2025

View reviewed changes

ilitteri added 2 commits August 20, 2025 18:54

Leftovers

a87cece

Merge branch 'main' of github.com:lambdaclass/ethrex into fix_executi…

8bd0e41

…on_witness_serde_bin

ilitteri added 2 commits August 20, 2025 21:55

Merge branch 'main' of github.com:lambdaclass/ethrex into fix_executi…

9b9415f

…on_witness_serde_bin

Merge branch 'main' of github.com:lambdaclass/ethrex into fix_executi…

6bc4db3

…on_witness_serde_bin

xqft moved this to In Review in ethrex_l2 Aug 21, 2025

Try send slack message in every prove result and only in failed execu…

7392be9

…tions

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(l1): `ethrex_replay` add tooling for periodic replay runs #4108

feat(l1): `ethrex_replay` add tooling for periodic replay runs #4108

Uh oh!

tomip01 commented Aug 20, 2025 •

edited by ilitteri

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Aug 20, 2025

Uh oh!

Copilot AI Aug 20, 2025

Uh oh!

Copilot AI Aug 20, 2025

Uh oh!

Copilot AI Aug 20, 2025

Uh oh!

Copilot AI Aug 20, 2025

Uh oh!

github-actions bot commented Aug 20, 2025 •

edited

Loading

Uh oh!

Uh oh!

-            handle_rpc_revalidation_handle_result(res, opts.hoodi_rpc_url.unwrap(), opts.slack_webhook_url.clone()).await; // TODO: change hoodi rpc to generic
+            let ((result, url), _index, _remaining_handles) = select_all(revalidation_handles).await;
+            (result, url)
+        } => {
+            let (res, url) = res;
+            handle_rpc_revalidation_handle_result(res, url, opts.slack_webhook_url.clone()).await;

-            tracing::error!("Failed to notify Slack about invalid Sepolia RPC: {e}");
+    network: Network,
+    slack_webhook_url: Option<Url>,
+) {
+    if let Err(e) = res {
+        tracing::error!("RPC URL `{}` for network `{}` failed: {}", rpc_url, network, e);
+        try_notify_no_longer_valid_rpc_to_slack(
+            rpc_url,
+            network,
+            slack_webhook_url,
+        )
+        .await
+        .unwrap_or_else(|e| {
+            tracing::error!("Failed to notify Slack about invalid RPC URL `{}` for network `{}`: {}", rpc_url, network, e);

	let milliseconds = total_seconds / 1000;
	let milliseconds = duration.subsec_millis();

-            todo!("Retry with eth_getProofs")
+            return Err(eyre::eyre!(
+                "Failed to retrieve witness for block {requested_block_number}: {e}. Retry with eth_getProofs not yet implemented."
+            ));

	panic!("SystemTime::elapsed failed: {e}");
	tracing::error!("SystemTime::elapsed failed: {e}. Using default duration of 12 seconds.");
	Duration::from_secs(12)

feat(l1): ethrex_replay add tooling for periodic replay runs #4108

Are you sure you want to change the base?

feat(l1): ethrex_replay add tooling for periodic replay runs #4108

Uh oh!

Conversation

tomip01 commented Aug 20, 2025 • edited by ilitteri Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Lines of code report

Uh oh!

Uh oh!

feat(l1): `ethrex_replay` add tooling for periodic replay runs #4108

feat(l1): `ethrex_replay` add tooling for periodic replay runs #4108

tomip01 commented Aug 20, 2025 •

edited by ilitteri

Loading

github-actions bot commented Aug 20, 2025 •

edited

Loading