feat(levm): improve state EF Tests (#2490)

**Motivation**

- Make running and modifying EF Tests a better experience

**Description**

<!-- A clear and concise general description of the changes this PR
introduces -->
- Remove spinner and just use prints
- We now can filter the tests we want to run by fork name(we do the
filtering in the parsing). Default is all forks.
- Upgrade tests to more recent version
- Run some Legacy Tests that we weren't running before. This adds a lot
of tests more, it is the folder Cancun under LegacyTests. There will be
repeated tests with the folder GeneralStateTests, we may want to find a
solution for that so that it takes less time to execute.
- Create docs in `README.md`
- Implement some nits in the runner, making code easier to understand.
- Ignore a few tests that take too long to run so that we can check for
breaking changes fast.
- Fix comparison report against LEVM, they weren't working correctly
mostly because we were mishandling our Cache
- Tidy the report, now it is much more clear and easier for debugging.
Also the code is easier to follow and more concise too!
- Fix some tests with REVM, basically now using constructor of
`BlobExcessGasAndPrice` and setting chain id to 1 (as we do in LEVM).
- Changed `get_state_transitions`
[here](#2518) so that REVM and
LEVM account updates are mostly the same and the comparison is more
accurate for the person who is debugging any test.

<!-- Link to issues: Resolves #111, Resolves #222 -->

Closes #issue_number

Author: JereSalo

cmd/ef_tests/state/README.md

@@ -1,9 +1,36 @@
 # State Tests
 
-https://ethereum.github.io/execution-spec-tests/main/consuming_tests/state_test/
+The state tests are individual transactions not related one to each other that test particular behavior of the EVM. Tests are usually run for multiple forks and the result of execution may vary between forks.
+Some [docs](https://ethereum.github.io/execution-spec-tests/main/consuming_tests/state_test/).
+
 
 ## Running the tests
 
 ```bash
-make run-evm-ef-tests
+make run-evm-ef-tests flags=<flags>
+```
+or
+```bash
+cargo test --package ef_tests-state --test all --release -- <flags>
 ```
+
+**Flags**
+- forks: Forks for which we want to run the tests for.
+- tests: Tests (.json files) we want to run
+- specific_tests: For running tests with a specific name. (Sometimes a .json file has multiple tests)
+- summary: For not doing a re-run with REVM of failed tests after LEVM's run.
+- skip: For skipping tests
+- verbose: For more info while running, like tests names being run.
+- revm: For running EFTests ONLY with REVM.
+
+
+Example usage: 
+```bash
+cargo test --package ef_tests-state --test all --release -- --forks Prague,Cancun --summary --tests push0.json,invalidAddr.json
+```
+This runs 2 specific tests with LEVM just for Prague and Cancun. If they fail they are not re-run with REVM.
+
+Most of the tests that we run are from [this repository](https://github.com/ethereum/tests). We run the `GeneralStateTests` from that repo and also from `LegacyTests`, which is another repository that has snapshots of tests from previous forks. 
+
+
+Beware: Sometimes there is a test overlap between the tests folders we have downloaded and we may run the same test for a recent fork (Cancun ATTOW) twice. The impact of this in performance is minimal because we are doing runs for other forks anyway so one more run won't harm, but we should be aware that may lead to an inaccurate test count. We chose not to handle this because it wasn't a huge problem, but be conscious about this.

cmd/ef_tests/state/deserialize.rs

@@ -68,7 +68,7 @@ where
                 "TransactionException.INSUFFICIENT_MAX_FEE_PER_BLOB_GAS" => {
                     TransactionExpectedException::InsufficientMaxFeePerBlobGas
                 }
-                other => panic!("Unexpected error type: {}", other), // Should not fail, TODO is to return an error
+                _other => TransactionExpectedException::Other, //TODO: Support exceptions that enter here.
             })
             .collect();
 

cmd/ef_tests/state/parser.rs

@@ -2,10 +2,8 @@ use crate::{
     report::format_duration_as_mm_ss,
     runner::EFTestRunnerOptions,
     types::{EFTest, EFTests},
-    utils::{spinner_success_or_print, spinner_update_text_or_print},
 };
 use colored::Colorize;
-use spinoff::{spinners::Dots, Color, Spinner};
 use std::fs::DirEntry;
 
 #[derive(Debug, thiserror::Error)]
@@ -20,9 +18,12 @@ pub enum EFTestParseError {
     FailedToParseTestFile(String),
 }
 
-const IGNORED_TESTS: [&str; 8] = [
-    "ValueOverflowParis.json",                 // Skip because of errors
-    "loopMul.json",                            // Skip because it takes too long to run
+const IGNORED_TESTS: [&str; 11] = [
+    "static_Call50000_sha256.json", // Skip because it takes longer to run than some tests, but not a huge deal.
+    "CALLBlake2f_MaxRounds.json",   // Skip because it takes extremely long to run, but passes.
+    "ValueOverflow.json",           // Skip because it tries to deserialize number > U256::MAX
+    "ValueOverflowParis.json",      // Skip because it tries to deserialize number > U256::MAX
+    "loopMul.json",                 // Skip because it takes too long to run
     "dynamicAccountOverwriteEmpty_Paris.json", // Skip because it fails on REVM
     "RevertInCreateInInitCreate2Paris.json", // Skip because it fails on REVM. See https://github.com/lambdaclass/ethrex/issues/1555
     "RevertInCreateInInit_Paris.json", // Skip because it fails on REVM. See https://github.com/lambdaclass/ethrex/issues/1555
@@ -35,10 +36,8 @@ pub fn parse_ef_tests(opts: &EFTestRunnerOptions) -> Result<Vec<EFTest>, EFTestP
     let parsing_time = std::time::Instant::now();
     let cargo_manifest_dir = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"));
     let ef_general_state_tests_path = cargo_manifest_dir.join("vectors");
-    let mut spinner = Spinner::new(Dots, "Parsing EF Tests".bold().to_string(), Color::Cyan);
-    if !opts.spinner {
-        spinner.stop();
-    }
+    println!("{}", "Parsing EF Tests".bold().cyan());
+
     let mut tests = Vec::new();
     for test_dir in std::fs::read_dir(ef_general_state_tests_path.clone())
         .map_err(|err| {
@@ -49,17 +48,13 @@ pub fn parse_ef_tests(opts: &EFTestRunnerOptions) -> Result<Vec<EFTest>, EFTestP
         })?
         .flatten()
     {
-        let directory_tests = parse_ef_test_dir(test_dir, opts, &mut spinner)?;
+        let directory_tests = parse_ef_test_dir(test_dir, opts)?;
         tests.extend(directory_tests);
     }
 
-    spinner_success_or_print(
-        &mut spinner,
-        format!(
-            "Parsed EF Tests in {}",
-            format_duration_as_mm_ss(parsing_time.elapsed())
-        ),
-        opts.spinner,
+    println!(
+        "Parsed EF Tests in {}",
+        format_duration_as_mm_ss(parsing_time.elapsed())
     );
 
     Ok(tests)
@@ -68,13 +63,8 @@ pub fn parse_ef_tests(opts: &EFTestRunnerOptions) -> Result<Vec<EFTest>, EFTestP
 pub fn parse_ef_test_dir(
     test_dir: DirEntry,
     opts: &EFTestRunnerOptions,
-    directory_parsing_spinner: &mut Spinner,
 ) -> Result<Vec<EFTest>, EFTestParseError> {
-    spinner_update_text_or_print(
-        directory_parsing_spinner,
-        format!("Parsing directory {:?}", test_dir.file_name()),
-        opts.spinner,
-    );
+    println!("Parsing directory {:?}", test_dir.file_name());
 
     let mut directory_tests = Vec::new();
     for test in std::fs::read_dir(test_dir.path())
@@ -90,7 +80,7 @@ pub fn parse_ef_test_dir(
             })?
             .is_dir()
         {
-            let sub_directory_tests = parse_ef_test_dir(test, opts, directory_parsing_spinner)?;
+            let sub_directory_tests = parse_ef_test_dir(test, opts)?;
             directory_tests.extend(sub_directory_tests);
             continue;
         }
@@ -132,13 +122,9 @@ pub fn parse_ef_test_dir(
             .skip
             .contains(&test_dir.file_name().to_str().unwrap().to_owned())
         {
-            spinner_update_text_or_print(
-                directory_parsing_spinner,
-                format!(
-                    "Skipping test {:?} as it is in the folder of tests to skip",
-                    test.path().file_name().unwrap()
-                ),
-                opts.spinner,
+            println!(
+                "Skipping test {:?} as it is in the folder of tests to skip",
+                test.path().file_name().unwrap()
             );
             continue;
         }
@@ -153,13 +139,9 @@ pub fn parse_ef_test_dir(
                 .unwrap()
                 .to_owned(),
         ) {
-            spinner_update_text_or_print(
-                directory_parsing_spinner,
-                format!(
-                    "Skipping test {:?} as it is in the list of tests to skip",
-                    test.path().file_name().unwrap()
-                ),
-                opts.spinner,
+            println!(
+                "Skipping test file {:?} as it is in the list of tests to skip",
+                test.path().file_name().unwrap()
             );
             continue;
         }
@@ -173,6 +155,24 @@ pub fn parse_ef_test_dir(
         for test in tests.0.iter_mut() {
             test.dir = test_dir.file_name().into_string().unwrap();
         }
+
+        // We only want to include tests that have post states from the specified forks in EFTestsRunnerOptions.
+        if let Some(forks) = &opts.forks {
+            for test in tests.0.iter_mut() {
+                let test_forks_numbers: Vec<u8> = forks.iter().map(|fork| *fork as u8).collect();
+
+                test.post.forks = test
+                    .post
+                    .forks
+                    .iter()
+                    .filter(|a| test_forks_numbers.contains(&(*a.0 as u8)))
+                    .map(|(k, v)| (*k, v.clone()))
+                    .collect();
+            }
+
+            tests.0.retain(|test| !test.post.forks.is_empty());
+        }
+
         directory_tests.extend(tests.0);
     }
     Ok(directory_tests)