Eigenda implementation merge main (#271)

* fix(tee-prover): passthrough env vars to the SGX enclave (matter-labs#2824)

## What ❔

Passthrough env vars to the SGX enclave.

Relevant logs showcasing the issue:
https://grafana.matterlabs.dev/goto/1iFHMIeIg?orgId=1

## Why ❔

To fix the bug.

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.

* feat(zk_toolbox): Add setup keys step to prover init (matter-labs#2811)

## What ❔
Add setup keys step to prover init

* feat(zk_toolbox): Allow running docker images for provers (matter-labs#2800)

## What ❔

Add `--docker` flag for `zk_inception prover run` which will allow
prover components to run from docker images.

## Why ❔

To decrease setup time/improve UX

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

* chore: Add README for verified sources fetcher (matter-labs#2829)

## What ❔

Adds README for verified sources fetcher

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

* chore(ci): Limit tokio/rayon pools for zk_toolbox CI (matter-labs#2828)

## What ❔

Limits rayon threadpool size to 2 and tokio threadpool size to 4 in
zk_toolbox CI.
I have checked locally, and with this configuration time to run
integration tests is pretty close to the default configuration.

## Why ❔

By default, both tokio and rayon will try to use all the CPUs. When we
run multiple Rust binaries at the same time (3 servers and 3 ENs in our
case), it causes a lot of conflict for resources, regardless of the
number of CPUs.

* feat: attester committees data extractor (BFT-434) (matter-labs#2684)

Extraction of the attester committee from consensus registry state. If
consensus registry address is not specified, we fall back to attester
committee from genesis. This pr does NOT enable the dynamic attestation,
as the registry address needs to be added to the main node config first.

---------

Co-authored-by: Moshe Shababo <17073733+moshababo@users.noreply.github.com>
Co-authored-by: Igor Aleksanov <popzxc@yandex.ru>

* test: Improve revert integration test (matter-labs#2822)

## What ❔

- Removes a data race from the revert integration test (a single node
one); a batch gets executed after it is verified that not all batches
are executed.
- Removes copy-paste between single-node and EN tests.
- Structures revert tests into smaller steps.

## Why ❔

Makes the test less flaky and easier to maintain.

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.

* test: Fix "missing revert data" error; fix / debug integration tests (matter-labs#2804)

## What ❔

- Fixes the "missing revert data" error by updating the used reth Docker
image. The error is probably caused by [this
issue](paradigmxyz/reth#7381) fixed in the new
reth versions.
- Removes "web3 API compatibility tests › Should check API returns error
when there are too many logs in eth_getLogs" test as fundamentally flaky
and able to poison other tests.
- Adds logging for upgrade test to investigate L1 "nonce too low"
errors.

## Why ❔

Flaky CI bad.

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.

* refactor(api): Extract oneshot VM executor to executor crate (matter-labs#2806)

## What ❔

Extracts oneshot VM executor to the executor crate.

## Why ❔

To make executor logic more reusable and maintainable.

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.

* fix(en): Fix connection starvation during snapshot recovery (matter-labs#2836)

## What ❔

- Fixes DB connection starvation during snapshot recovery. Caused by the
insufficient number of connections in the DB pool provided to recovery
logic.
- Additionally, fixes max concurrency of recovery not being set.

## Why ❔

Connection starvation errors degrade UX.

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.

* fix: handling of HTTP 403 thrown by proxyd (matter-labs#2835)

When a method is missing:
proxyd returns HTTP 403:  methodnotfound
while api server returns HTTP 200: methodnotfound

we need to handle both.

* chore: log the whole error message when the task fails (matter-labs#2834)

"{err}" is not useful, because it truncates the stack trace of anyhow
errors.

* feat: allow specifying what tests to run with zks (matter-labs#2841)

`zks t i` now accepts a `-t` flag that can can be used to specify a
pattern. Only matching tests are run.

* chore: clean up dependencies (matter-labs#2839)

## What ❔

Resolves matter-labs#2783

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

* feat(utils): Rework locate_workspace, introduce Workspace type (matter-labs#2830)

## What

- Removes `locate_workspace` and `workspace_dir_or_current_dir` methods.
- Instead, introduces `Workspace` type that is aware of different Cargo
workspaces in the codebase.

## Why

The approach with a single `locate_workspace` doesn't work well for our
codebase, since we have multiple workspaces. It resulted in some very
implicit and convoluted code (see the removed `get_base_dir` in prover
workspace).

New approach handles all 3 workspaces _plus_ the lack of a workspace.

* feat(zk_toolbox):  `zk_supervisor prover` subcommand (matter-labs#2820)

## What ❔

Add prover subcommand for `zk_supervisor`. Add the following subcommand:
* `zk_supervisor prover info` - Prints information about current prover
setup.
* `zk_supervisor prover insert-version` - Insert new protocol version in
prover database(integration with `prover_cli`).
* `zk_supervisor prover insert-batch` - Insert new batch in prover
database(integration with `prover_cli`).

Add automatic creation of `prover/artifacts/witness_inputs` dirs if the
storage is file backed on init.

## Why ❔

To improve UX of working with provers.

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

* fix(zk-toolbox): Make token multiplier optional (matter-labs#2843)

## What ❔

<!-- What are the changes this PR brings about? -->
<!-- Example: This PR adds a PR template to the repo. -->
<!-- (For bigger PRs adding more context is appreciated) -->

## Why ❔

<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

Signed-off-by: Danil <deniallugo@gmail.com>

* feat: Smaller zk_environment image (matter-labs#1920)

## What ❔

* Reduced uncompressed size from 7.5GB to 4.81 GB.
* Changes to make zk_environment image smaller
* compile solc with release mode
* remove unnecessary components from google cli
* build things first, and then only copy the artifacts.

## Why ❔

* This image is used on ALL of our CI builds - so reducing its size will
speed up the CIs.

* fix(zk_toolbox): secrets path, artifacts path (matter-labs#2850)

## What ❔

Fix getting chain config in zk_supervisor prover
Fix artifacts path when initializing.
Setup data path for provers

## Why ❔

<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

* ci: fix using cargo nextest (matter-labs#2855)

## What ❔

`cargo install --list` doesn't work properly in CI because we copy only
binaries to zk-environment
also increase reth block-time and fixes upgrade-test which improves
tests stability

## Why ❔

we rely on cargo-nextest in tests

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

* feat: add da clients (matter-labs#2743)

## What ❔

This PR changes the approach to managing 3rd party DA clients. It was
assumed before that they will be stored in a separate repository
(hyperchain-da), but to simplify the processes and solve a recursive
dependency problem, we decided to manage those within `zksync-era`.

The config now defines which DA client will be used, for proto-based
configuration it requires adding these lines to general.yaml:
```
da_client:
  avail:
    api_node_url: wss://turing-rpc.avail.so/ws
    bridge_api_url: undefined
    seed: SEED_PHRASE
    app_id: 82
    timeout: 3
    max_retries: 5
``` 
for env-based:
```
    DA_CLIENT="Avail"
    DA_API_NODE_URL="localhost:12345"
    DA_BRIDGE_API_URL="localhost:54321"
    DA_SEED="SEED_PHRASE"
    DA_APP_ID=1
    DA_TIMEOUT=2
    DA_MAX_RETRIES=3
```
If no config is provided - the default behavior is to use NoDA client
(same as now).

The `da_client` config might be merged with `da_dispatcher` at some
point as the second depends on the first one, so their separation does
not make much sense (apart from simplification of the configs). But I'd
prefer to do it as a separate PR in case we decide to merge them.

The client was reimplemented using only lightweight libraries from
crates.io, so it doesn't have any visible impact on build time.

## Why ❔

To enable seamless integration with 3rd party DA clients in
`zksync-era`.

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

* feat(zk_toolbox): deploy legacy bridge (matter-labs#2837)

## What ❔

Support legacy bridges in zk toolbox 

## Why ❔

We still have some legacy bridges in production, so we have to have an
opportunity to test it

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

---------

Signed-off-by: Danil <deniallugo@gmail.com>

* feat(zk_toolbox): Add external_node consensus support (matter-labs#2821)

## What ❔
Add external_node consensus support

---------

Signed-off-by: Danil <deniallugo@gmail.com>
Co-authored-by: Danil <deniallugo@gmail.com>

* feat: Selector generator tool (matter-labs#2844)

## What ❔

* A small tool to generate the selector hashes based on the ABI from
json files

## Why ❔

* The output json can be useful for humans to better understand some of
the errors (and calldata)
* It can also be read by our tools, to make the debugging easier.

In the future, we could call this tool regularly on each contracts
version change, but for now it can stay as manual.

* feat(prover): Optimize setup keys loading (matter-labs#2847)

## What ❔

- Loads setup keys to memory in parallel (for GPU prover only).
- Refactors a bunch of related code for simplicity.

## Why ❔

- Locally I've observed load time going from ~30s to ~12s, so ~60%
improvement for prover start time.
- Readability & maintainability.

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

* feat: Bump crypto and protocol deps (matter-labs#2825)

## What ❔

- Use latest versions of crypto, protocol, gpu, and consensus crates.
- Remove solved cargo deny advisories from the allowlist.

## Why ❔

- A bunch of fixes/improvements were done.
- Optimization of dependency graph.
- Solving cargo deny advisories.

* feat(prover): Refactor WitnessGenerator (matter-labs#2845)

## What ❔

Introduce new structure for witness generators.
Introduce `ArtifactsManager` trait responsible for operations with
object store and artifacts.

## Why ❔

<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

* fix(zk-toolbox): use chain admin for bridgehub  (matter-labs#2857)

## What ❔

<!-- What are the changes this PR brings about? -->
<!-- Example: This PR adds a PR template to the repo. -->
<!-- (For bigger PRs adding more context is appreciated) -->

## Why ❔

<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

---------

Signed-off-by: Danil <deniallugo@gmail.com>

* fix: count SECP256 precompile to account validation gas limit as well (matter-labs#2859)

Account validation counts all the precompile's extra gas costs. This PR
adds a missing precompile.

* chore(vm): Bump `zksync_vm2` revision (matter-labs#2838)

## What ❔

Updates the fast VM revision to incorporate latest changes.

## Why ❔

Mainly to check that these changes work.

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.

* feat: Add 80 arch to cuda archs (matter-labs#2871)

## What ❔

Add CUDA ARCH 80 to list of CUDA archs.

## Why ❔

To be able to run provers on A100.
## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

* fix: Docker build for compressor (matter-labs#2874)

## What ❔

Pass CUDA_ARCH to compressor image build.

## Why ❔

<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

* fix(prover): fix setup_metadata_to_setup_data_key (matter-labs#2875)

* fix(zksync_tee_prover): strip binary for nix package (matter-labs#2876)

## What ❔

Strip `zksync_tee_prover` binary for nix package.

## Why ❔

Somehow llvm decided to produce different symbol names each build

```diff
- 12916: 000000000194d628    48 OBJECT  LOCAL  HIDDEN    23 anon.c880a573512c356e53163839ee7cd669.742.llvm.17979232152659092167
+ 12916: 000000000194d628    48 OBJECT  LOCAL  HIDDEN    23 anon.c880a573512c356e53163839ee7cd669.742.llvm.14235143304249741099
[...]
```

Remove those to make the builds reproducible again.

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

Signed-off-by: Harald Hoyer <harald@matterlabs.dev>

* fix(eth_watch): fix `get_events_inner` (matter-labs#2882)

## What ❔

- `get_events_inner` should recursively call itself
- `get_events_inner` should allow passing `None` as topics and/or
addresses

## Why ❔

bug fix

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

* refactor(vm-runner): Improve VM runner / VM playground (matter-labs#2840)

## What ❔

Various minor improvements to VM runner / VM playground:

- Get batch storage asynchronously, so that it works efficiently with
snapshot storage.
- Add metrics / logs to ensure that snapshot storage works as expected.

## Why ❔

Improves usability.

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.

* fix(zk_toolbox): increase confirmations in testing (matter-labs#2878)

## What ❔

<!-- What are the changes this PR brings about? -->
<!-- Example: This PR adds a PR template to the repo. -->
<!-- (For bigger PRs adding more context is appreciated) -->
Increase confirmations in testing.

## Why ❔

<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->
Should make CI workflows less flaky.

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.

* feat(zk_toolbox): Verbose version message (matter-labs#2884)

Modifies the version message to include stuff like git commit, branch,
build timestamp, and submodules versions (non-recursive).
Sample output:

```
$ zki --version
zk_inception v0.1.0-c03229138
Branch: popzxc-version-with-git-commit
Submodules:
  - contracts: 3a1b5d4
Build timestamp: 2024-09-16 12:07:09
```

* feat(en): Resume incomplete snapshot in snapshot creator in more cases (matter-labs#2886)

## What ❔

Resumes an incomplete snapshot in the snapshot creator if the creator
config doesn't specify an L1 batch.

## Why ❔

This effectively reverts the relevant changes from
matter-labs#2256. It makes the
snapshot creator resilient by default without additional setup, at the
cost of parallel creator jobs working incorrectly (unless they all
specify L1 batches).

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.

* feat(build-base): Add sccache to build-base Dockerfile (matter-labs#2889)

## What ❔

Adding installation of sccache to build-base Dockerfile

## Why ❔

To speed-up docker builds 

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

* refactor(selector-generator): Polish the code (matter-labs#2888)

## What ❔

- Changes procedural approach to more idiomatic for rust (introduces
`Selectors` + `App` structures). `Selectors` are not FS aware, `App` is.
- Introduces more strong typing -- when both selectors and function
names are `String`, it's easy to misuse them (I did several times, while
working on this PR).
- Removes manual selector calculation and uses `ethabi` for that.
- Makes the code async.
- Adds docs for CLI (e.g. to see with `--help` argument).
- Adds a test for `Selectors`.
- Reduces number of panics and uses `anyhow` to propagate errors.
- Adds new selectors (I guess I have more contracts compiled locally).

## Why ❔

Maintainability and testability, mostly. With the refactoring, it should
be easier to add new functionality and tests for it.
Also, the runtime has improved.

Before

```
$ time cargo run --bin selector_generator -- contracts etc/selector-generator-data/selectors.json
   Compiling selector_generator v0.1.0 (/home/popzxc/workspace/current/zksync-era/core/bin/selector_generator)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.54s
     Running `target/debug/selector_generator contracts etc/selector-generator-data/selectors.json`
Analyzed 401 files. Added 452 selectors (before: 516 after: 968)
cargo run --bin selector_generator -- contracts   4.48s user 7.02s system 100% cpu 11.486 total
```

After

```
$ time cargo run --bin selector_generator -- contracts etc/selector-generator-data/selectors.json
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.20s
     Running `target/debug/selector_generator contracts etc/selector-generator-data/selectors.json`
Analyzed 1087 files. Added 0 selectors (before: 1023 after: 1023)
cargo run --bin selector_generator -- contracts   1.24s user 0.12s system 100% cpu 1.360 total
```

## Next steps

I would love to see `--check` mode introduced (which checks that 0 new
entries were added) and this tool is added to CI, so that we continously
run it and prevent from rotting. @mm-zk leaving it to you.

* feat(zk_toolbox): add `zki ecosystem build` subcommand (matter-labs#2787)

## What ❔

<!-- What are the changes this PR brings about? -->
<!-- Example: This PR adds a PR template to the repo. -->
<!-- (For bigger PRs adding more context is appreciated) -->
Add `zki ecosystem build` subcommand, which builds L1 transactions
without signing and broadcasting them.
This allows a security team to sign them using keys in cold storage and
multisig.

```bash
Create transactions to build ecosystem contracts

Usage: zki ecosystem build-transactions [OPTIONS]

Options:
      --sender <SENDER>
          Address of the transaction sender

      --l1-rpc-url <L1_RPC_URL>
          L1 RPC URL

  -o, --out <OUT>
          Output directory for the generated files

  -h, --help
          Print help (see a summary with '-h')
```

```bash
Create unsigned transactions for chain deployment

Usage: zki chain build-transactions [OPTIONS]

Options:
  -o, --out <OUT>
          Output directory for the generated files

  -h, --help
          Print help (see a summary with '-h')
```

### Output
```json
# /transactions/deploy.json
{
  "transactions": [
    {
      "hash": null,
      "transactionType": "CREATE",
      "contractName": null,
      "contractAddress": "0xddca24376aa96d8f56667d78306a3bbbdc65b1ff",
      "function": null,
      "arguments": null,
      "transaction": {
        "from": "0xaf9d732e8a5607caccb72df525851849c33edf9e",
        "gas": "0x15a02",
        "value": "0x0",
        "input": "0x604580600e600039806000f350fe7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffe03601600081602082378035828234f58015156039578182fd5b8082525050506014600cf3",
        "nonce": "0x0",
        "chainId": "0x9"
      },
      "additionalContracts": [],
      "isFixedGasLimit": false
    },
    {
      "hash": null,
      "transactionType": "CALL",
      "contractName": null,
      "contractAddress": "0xddca24376aa96d8f56667d78306a3bbbdc65b1ff",
      "function": null,
      "arguments": null,
      "transaction": {
        "from": "0xaf9d732e8a5607caccb72df525851849c33edf9e",
        "to": "0xddca24376aa96d8f56667d78306a3bbbdc65b1ff",
        "gas": "0xfb603",
        "value": "0x0",
        "input": ...
```



## Why ❔

<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.

* feat: deployment of multicall3 on L2 by zk_toolbox (matter-labs#2872)

For complex atomic operations on consensus registry.
Should be merged after
matter-labs/era-contracts#805
Tested manually by running `cargo -p zk_inception -- e init --dev`,
which populated the multicall3 address in contract.yaml to non-zero
value.

* fix(zk_toolbox): Clone era observability (matter-labs#2892)

## What ❔

<!-- What are the changes this PR brings about? -->
<!-- Example: This PR adds a PR template to the repo. -->
<!-- (For bigger PRs adding more context is appreciated) -->

## Why ❔

<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

Signed-off-by: Danil <deniallugo@gmail.com>

* feat: updated the docker examples to generate consensus keys and use gossipnet (matter-labs#2476)

Tested on a hetzner instance.

---------

Co-authored-by: Bruno França <bruno@franca.xyz>

* fix: token multiplier setter is taken from chain config (matter-labs#2893)

Fix `zki chain update-toen-multiplier-setter` to take the address from
the chain config instead of the ecosystem.

* feat: emit errors in prover API metrics (matter-labs#2890)

## What ❔

Report reasons of failed calls in prover API metrics.

## Why ❔

To have more observability

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

* ci: Deprecate zk in ci tests (matter-labs#2860)

## What ❔

Do not use old zk tool inside our ci. Only zk docker is still in place
will be replaced soon by DevOps team

## Why ❔

We are deprecating old zk tool and now it's not needed anymore 

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

---------

Signed-off-by: Danil <deniallugo@gmail.com>
Co-authored-by: perekopskiy <mikeson.dp@gmail.com>

* chore: increase max_nonce_ahead (matter-labs#2896)

## What ❔

increases max_nonce_ahead in default config

## Why ❔

fix flaky test

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

* feat(zk_toolbox): Add zk_supervisor config-writer command (matter-labs#2866)

## What ❔
Add zk_supervisor override configs command

## Why ❔
In some cases we want to override some default configs. Right now it is
done manually and we need proper automation for regular use/ci.

---------

Signed-off-by: Danil <deniallugo@gmail.com>
Signed-off-by: Harald Hoyer <harald@matterlabs.dev>
Co-authored-by: Patrick <pab@matterlabs.dev>
Co-authored-by: Matías Ignacio González <maigonzalez@fi.uba.ar>
Co-authored-by: Artem Fomiuk <88630083+Artemka374@users.noreply.github.com>
Co-authored-by: perekopskiy <53865202+perekopskiy@users.noreply.github.com>
Co-authored-by: Igor Aleksanov <popzxc@yandex.ru>
Co-authored-by: Grzegorz Prusak <pompon.pompon@gmail.com>
Co-authored-by: Moshe Shababo <17073733+moshababo@users.noreply.github.com>
Co-authored-by: Alex Ostrovski <slowli@users.noreply.github.com>
Co-authored-by: Joonatan Saarhelo <joon.saar@gmail.com>
Co-authored-by: Danil <deniallugo@gmail.com>
Co-authored-by: Marcin M <128217157+mm-zk@users.noreply.github.com>
Co-authored-by: Dima Zhornyk <55756184+dimazhornyk@users.noreply.github.com>
Co-authored-by: Harald Hoyer <harald@matterlabs.dev>
Co-authored-by: Manuel Mauro <manuel.mauro@protonmail.com>
Co-authored-by: Artem Makhortov <13339874+artmakh@users.noreply.github.com>
Co-authored-by: Bruno França <bruno@franca.xyz>
Co-authored-by: Ivan Schasny <31857042+ischasny@users.noreply.github.com>
Co-authored-by: perekopskiy <mikeson.dp@gmail.com>

Author: 18 people

.github/workflows/build-contract-verifier-template.yml

@@ -113,18 +113,15 @@ jobs:
           ci_run git config --global --add safe.directory /usr/src/zksync/sdk/binaryen
           ci_run git config --global --add safe.directory /usr/src/zksync/contracts/system-contracts
           ci_run git config --global --add safe.directory /usr/src/zksync/contracts
-          ci_run zk || true
-          ci_run yarn zk build
+          ci_run ./bin/zkt || true
+          ci_run ./bin/zk || true
           ci_run run_retried curl -LO https://storage.googleapis.com/matterlabs-setup-keys-us/setup-keys/setup_2\^26.key
 
       - name: build contracts
         if: env.BUILD_CONTRACTS == 'true'
         run: |
-          ci_run zk run yarn
           ci_run cp etc/tokens/{test,localhost}.json
-          ci_run zk compiler all
-          ci_run zk contract build
-          ci_run zk f yarn run l2-contracts build
+          ci_run zk_supervisor contracts
 
       - name: Login to Docker registries
         if: ${{ inputs.action == 'push' }}

.github/workflows/build-core-template.yml

@@ -71,11 +71,15 @@ jobs:
             if [ $(jq length <<<"$tags") -eq 0 ]; then
               echo "No tag found on all pages."
               echo "BUILD_CONTRACTS=true" >> "$GITHUB_ENV"
+              # TODO Remove it when we migrate to foundry inside contracts repository
+              mkdir -p contracts/l1-contracts/artifacts/ 
               exit 0
             fi
             filtered_tag=$(jq -r --arg commit_sha "$commit_sha" 'map(select(.commit.sha == $commit_sha)) | .[].name' <<<"$tags")
             if [[ ! -z "$filtered_tag" ]]; then
               echo "BUILD_CONTRACTS=false" >> "$GITHUB_ENV"
+              # TODO Remove it when we migrate to foundry inside contracts repository
+              mkdir -p contracts/l1-contracts/out
               break
             fi
             ((page++))
@@ -122,18 +126,15 @@ jobs:
           ci_run git config --global --add safe.directory /usr/src/zksync/sdk/binaryen
           ci_run git config --global --add safe.directory /usr/src/zksync/contracts/system-contracts
           ci_run git config --global --add safe.directory /usr/src/zksync/contracts
-          ci_run zk || true
-          ci_run yarn zk build
+          ci_run ./bin/zk || true
+          ci_run ./bin/zkt || true
           ci_run run_retried curl -LO https://storage.googleapis.com/matterlabs-setup-keys-us/setup-keys/setup_2\^26.key
 
       - name: build contracts
         if: env.BUILD_CONTRACTS == 'true'
         run: |
-          ci_run zk run yarn
           ci_run cp etc/tokens/{test,localhost}.json
-          ci_run zk compiler all
-          ci_run zk contract build
-          ci_run zk f yarn run l2-contracts build
+          ci_run zk_supervisor contracts
 
       - name: Login to Docker registries
         if: ${{ inputs.action == 'push' }}

.github/workflows/build-docker-from-tag.yml

@@ -102,7 +102,7 @@ jobs:
     with:
       image_tag_suffix: ${{ needs.setup.outputs.image_tag_suffix }}-avx512
       ERA_BELLMAN_CUDA_RELEASE: ${{ vars.ERA_BELLMAN_CUDA_RELEASE }}
-      CUDA_ARCH: "60;70;75;89"
+      CUDA_ARCH: "60;70;75;80;89"
       WITNESS_GENERATOR_RUST_FLAGS: "-Ctarget_feature=+avx512bw,+avx512cd,+avx512dq,+avx512f,+avx512vl"
     secrets:
       DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}

.github/workflows/build-local-node-docker.yml

@@ -16,7 +16,7 @@ on:
 jobs:
   build-images:
     name: Local Node - Build and Push Docker Image
-    runs-on: [matterlabs-ci-runner-high-performance]
+    runs-on: [ matterlabs-ci-runner-high-performance ]
     steps:
       - uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4
         with:
@@ -61,9 +61,9 @@ jobs:
           ci_run git config --global --add safe.directory /usr/src/zksync/contracts
 
           ci_run zk
+          ci_run zkt
           ci_run cp etc/tokens/{test,localhost}.json
-          ci_run zk compiler all
-          ci_run zk contract build
+          ci_run zk_supervisor contracts 
 
       - name: update-image
         run: |

.github/workflows/ci-common-reusable.yml

@@ -28,11 +28,10 @@ jobs:
 
       - name: Init
         run: |
-          ci_run zk
-          ci_run run_retried rustup show
-          ci_run zk db setup
+          ci_run zkt
 
       # This does both linting and "building". We're using `zk lint prover` as it's common practice within our repo
       # `zk lint prover` = cargo clippy, which does cargo check behind the scenes, which is a lightweight version of cargo build
       - name: Lints
-        run: ci_run zk lint prover
+        run: ci_run zk_supervisor lint -t rs --check
+