feat: Use JSON-RPC for core <> prover interaction #3626

Artemka374 · 2025-02-19T13:54:39Z

What ❔

Implement communication between proof data handler and gateway through web sockets.
The workflow:

Long-living connection is established
Data handler subscribes for proofs from gateway
Data handler sends proof generation data to gateway
On receiving proofs through subscription, data handler saves it's data and sends a notification to gateway, that proof was successfully received(to not lose the data in between)

Why ❔

For the sake of prover cluster, where we will have a single prover subsystem with multiple sequencers.

Is this a breaking change?

Yes
No

Operational changes

Config values:
Prover gateway: ws_port
Proof data handler: api_url, batch_readiness_check_interval_in_secs, retry_connection_interval_in_secs

Checklist

PR title corresponds to the body of PR (we generate changelog entries from PRs).
Tests for the changes have been added / updated.
Documentation comments have been added / updated.
Code has been formatted via zkstack dev fmt and zkstack dev lint.

core/lib/config/src/configs/proof_data_handler.rs

core/lib/prover_interface/src/rpc.rs

core/node/proof_data_handler/src/rpc_client/mod.rs

core/lib/dal/src/proof_generation_dal.rs

etc/env/base/proof_data_handler.toml

core/node/proof_data_handler/src/tee_proof_api/processor.rs

EmilLuta

Checked changes since last review -- LGTM.

🤖 I have created a release *beep* *boop* --- ## [27.0.0](core-v26.7.0...core-v27.0.0) (2025-03-21) ### ⚠ BREAKING CHANGES * Remove old prover stack ([#3729](#3729)) * V27 update ([#3580](#3580)) ### Features * **eigenda:** EigenDA M0 ([#3650](#3650)) ([2a3cae9](2a3cae9)) * **eth-watch:** split heavy get logs requests if 503 ([#3706](#3706)) ([406a3ff](406a3ff)) * **gateway:** Requirement to stop L1->L2 transactions before v26 upgrade ([#3707](#3707)) ([0a095b7](0a095b7)) * Remove old prover stack ([#3729](#3729)) ([fbbdc76](fbbdc76)) * Starting from v26 version, always fetch the address of the validator timelock from CTM ([#3721](#3721)) ([d3db521](d3db521)) * **storage:** rocksdb storage extension for zkos ([#3698](#3698)) ([2779245](2779245)) * Use JSON-RPC for core <> prover interaction ([#3626](#3626)) ([4e74730](4e74730)) * V27 update ([#3580](#3580)) ([9e18550](9e18550)) * **zksync_tee_prover:** add support for TDX and None TEE types ([#3711](#3711)) ([11d166b](11d166b)) * **zksync_tee_prover:** read config in TDX from google metadata ([#3702](#3702)) ([e50201c](e50201c)) ### Bug Fixes * **contract-verifier:** Allow reverification of system contracts in Yul ([#3735](#3735)) ([e99b548](e99b548)) * **contract-verifier:** contract verifier to return fully matched verification info when available ([#3734](#3734)) ([1a0f27f](1a0f27f)) * **contract-verifier:** Correctly process partial verification for EVM contracts ([#3688](#3688)) ([8292234](8292234)) * **contract-verifier:** Ignore suppressable errors during compilation ([#3747](#3747)) ([a5955c4](a5955c4)) * **data-availability-fetcher:** prevent EN database from being populated with unnecessary inclusion data ([#3742](#3742)) ([037bac3](037bac3)) * make eigenda_eth_rpc in Eigen config optional in file-based configs ([#3732](#3732)) ([69d0efc](69d0efc)) * make secret optionnal when da client object store is selected ([#3715](#3715)) ([f05fffd](f05fffd)) * Serialization issues ([#3589](#3589)) ([606d5af](606d5af)) * **vm:** Fix another VM divergence in validation ([#3675](#3675)) ([85dfc13](85dfc13)) ### Performance Improvements * **zkos:** Choose optimal amortization radix for ZK OS Merkle tree ([#3685](#3685)) ([fc1e230](fc1e230)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: zksync-era-bot <zksync-era-bot@users.noreply.github.com>

This reverts commit 4e74730

🤖 I have created a release *beep* *boop* --- ## [20.0.0](prover-v19.0.0...prover-v20.0.0) (2025-04-01) ### ⚠ BREAKING CHANGES * Remove old prover stack ([#3729](#3729)) * V27 update ([#3580](#3580)) ### Features * Add S3 implementation for object_store ([#3664](#3664)) ([a848927](a848927)) * **consensus:** Add consensus protocol versioning ([#3720](#3720)) ([d1b4308](d1b4308)) * **gateway:** Migration to Gateway ([#3654](#3654)) ([2858ba0](2858ba0)) * Remove old prover stack ([#3729](#3729)) ([fbbdc76](fbbdc76)) * Use JSON-RPC for core <> prover interaction ([#3626](#3626)) ([4e74730](4e74730)) * V27 update ([#3580](#3580)) ([9e18550](9e18550)) ### Bug Fixes * API URL for prover gateway ([#3716](#3716)) ([ca2c4a4](ca2c4a4)) * make proof data handler backwards compatible ([#3767](#3767)) ([bdbbaaa](bdbbaaa)) * Prover job ordering ([#3738](#3738)) ([8f7f831](8f7f831)) * **prover:** Fixing shutdown time for circuit-prover ([#3768](#3768)) ([6c9de26](6c9de26)) * **prover:** Reevaluation of 'heavy' jobs for WVG ([#3754](#3754)) ([2a8d33b](2a8d33b)) * **prover:** Remove deleted pods from autoscaler-agent cluster cache ([#3739](#3739)) ([e94985f](e94985f)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: zksync-era-bot <147085853+zksync-era-bot@users.noreply.github.com> Co-authored-by: zksync-era-bot <zksync-era-bot@users.noreply.github.com>

## What ❔ Implement communication between proof data handler and gateway through web sockets. The workflow: * Long-living connection is established * Data handler subscribes for proofs from gateway * Data handler sends proof generation data to gateway * On receiving proofs through subscription, data handler saves it's data and sends a notification to gateway, that proof was successfully received(to not lose the data in between) ## Why ❔ For the sake of prover cluster, where we will have a single prover subsystem with multiple sequencers. ## Is this a breaking change? - [x] Yes - [ ] No ## Operational changes Config values: Prover gateway: ws_port Proof data handler: api_url, batch_readiness_check_interval_in_secs, retry_connection_interval_in_secs ## Checklist   - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zkstack dev fmt` and `zkstack dev lint`.

🤖 I have created a release *beep* *boop* --- ## [27.0.0](core-v26.7.0...core-v27.0.0) (2025-03-21) ### ⚠ BREAKING CHANGES * Remove old prover stack ([#3729](#3729)) * V27 update ([#3580](#3580)) ### Features * **eigenda:** EigenDA M0 ([#3650](#3650)) ([2a3cae9](2a3cae9)) * **eth-watch:** split heavy get logs requests if 503 ([#3706](#3706)) ([406a3ff](406a3ff)) * **gateway:** Requirement to stop L1->L2 transactions before v26 upgrade ([#3707](#3707)) ([867cfa5](867cfa5)) * Remove old prover stack ([#3729](#3729)) ([4a001a7](4a001a7)) * Starting from v26 version, always fetch the address of the validator timelock from CTM ([#3721](#3721)) ([d3db521](d3db521)) * **storage:** rocksdb storage extension for zkos ([#3698](#3698)) ([2779245](2779245)) * Use JSON-RPC for core <> prover interaction ([#3626](#3626)) ([1971fe4](1971fe4)) * V27 update ([#3580](#3580)) ([e7e930e](e7e930e)) * **zksync_tee_prover:** add support for TDX and None TEE types ([#3711](#3711)) ([11d166b](11d166b)) * **zksync_tee_prover:** read config in TDX from google metadata ([#3702](#3702)) ([e50201c](e50201c)) ### Bug Fixes * **contract-verifier:** Allow reverification of system contracts in Yul ([#3735](#3735)) ([e99b548](e99b548)) * **contract-verifier:** contract verifier to return fully matched verification info when available ([#3734](#3734)) ([1a0f27f](1a0f27f)) * **contract-verifier:** Correctly process partial verification for EVM contracts ([#3688](#3688)) ([8292234](8292234)) * **contract-verifier:** Ignore suppressable errors during compilation ([#3747](#3747)) ([a5955c4](a5955c4)) * **data-availability-fetcher:** prevent EN database from being populated with unnecessary inclusion data ([#3742](#3742)) ([037bac3](037bac3)) * make eigenda_eth_rpc in Eigen config optional in file-based configs ([#3732](#3732)) ([69d0efc](69d0efc)) * make secret optionnal when da client object store is selected ([#3715](#3715)) ([f05fffd](f05fffd)) * Serialization issues ([#3589](#3589)) ([606d5af](606d5af)) * **vm:** Fix another VM divergence in validation ([#3675](#3675)) ([85dfc13](85dfc13)) ### Performance Improvements * **zkos:** Choose optimal amortization radix for ZK OS Merkle tree ([#3685](#3685)) ([fc1e230](fc1e230)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: zksync-era-bot <zksync-era-bot@users.noreply.github.com>

🤖 I have created a release *beep* *boop* --- ## [27.0.0](matter-labs/zksync-era@core-v26.7.0...core-v27.0.0) (2025-03-21) ### ⚠ BREAKING CHANGES * Remove old prover stack ([#3729](matter-labs/zksync-era#3729)) * V27 update ([#3580](matter-labs/zksync-era#3580)) ### Features * **eigenda:** EigenDA M0 ([#3650](matter-labs/zksync-era#3650)) ([2a3cae9](matter-labs/zksync-era@2a3cae9)) * **eth-watch:** split heavy get logs requests if 503 ([#3706](matter-labs/zksync-era#3706)) ([406a3ff](matter-labs/zksync-era@406a3ff)) * **gateway:** Requirement to stop L1->L2 transactions before v26 upgrade ([#3707](matter-labs/zksync-era#3707)) ([1fea419](matter-labs/zksync-era@1fea419)) * Remove old prover stack ([#3729](matter-labs/zksync-era#3729)) ([ba05a88](matter-labs/zksync-era@ba05a88)) * Starting from v26 version, always fetch the address of the validator timelock from CTM ([#3721](matter-labs/zksync-era#3721)) ([d3db521](matter-labs/zksync-era@d3db521)) * **storage:** rocksdb storage extension for zkos ([#3698](matter-labs/zksync-era#3698)) ([2779245](matter-labs/zksync-era@2779245)) * Use JSON-RPC for core <> prover interaction ([#3626](matter-labs/zksync-era#3626)) ([6a6cead](matter-labs/zksync-era@6a6cead)) * V27 update ([#3580](matter-labs/zksync-era#3580)) ([6548ace](matter-labs/zksync-era@6548ace)) * **zksync_tee_prover:** add support for TDX and None TEE types ([#3711](matter-labs/zksync-era#3711)) ([11d166b](matter-labs/zksync-era@11d166b)) * **zksync_tee_prover:** read config in TDX from google metadata ([#3702](matter-labs/zksync-era#3702)) ([e50201c](matter-labs/zksync-era@e50201c)) ### Bug Fixes * **contract-verifier:** Allow reverification of system contracts in Yul ([#3735](matter-labs/zksync-era#3735)) ([e99b548](matter-labs/zksync-era@e99b548)) * **contract-verifier:** contract verifier to return fully matched verification info when available ([#3734](matter-labs/zksync-era#3734)) ([1a0f27f](matter-labs/zksync-era@1a0f27f)) * **contract-verifier:** Correctly process partial verification for EVM contracts ([#3688](matter-labs/zksync-era#3688)) ([8292234](matter-labs/zksync-era@8292234)) * **contract-verifier:** Ignore suppressable errors during compilation ([#3747](matter-labs/zksync-era#3747)) ([a5955c4](matter-labs/zksync-era@a5955c4)) * **data-availability-fetcher:** prevent EN database from being populated with unnecessary inclusion data ([#3742](matter-labs/zksync-era#3742)) ([037bac3](matter-labs/zksync-era@037bac3)) * make eigenda_eth_rpc in Eigen config optional in file-based configs ([#3732](matter-labs/zksync-era#3732)) ([69d0efc](matter-labs/zksync-era@69d0efc)) * make secret optionnal when da client object store is selected ([#3715](matter-labs/zksync-era#3715)) ([f05fffd](matter-labs/zksync-era@f05fffd)) * Serialization issues ([#3589](matter-labs/zksync-era#3589)) ([606d5af](matter-labs/zksync-era@606d5af)) * **vm:** Fix another VM divergence in validation ([#3675](matter-labs/zksync-era#3675)) ([85dfc13](matter-labs/zksync-era@85dfc13)) ### Performance Improvements * **zkos:** Choose optimal amortization radix for ZK OS Merkle tree ([#3685](matter-labs/zksync-era#3685)) ([fc1e230](matter-labs/zksync-era@fc1e230)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: zksync-era-bot <zksync-era-bot@users.noreply.github.com>

Artemka374 added 11 commits February 19, 2025 15:53

initial implementation

7472166

NL

03a1859

fix build, update logging

b9c444e

prepare db

6558563

add logging for receive_final_proof

757af8e

fmt

b33136d

fix test

610649e

fix test

26a6682

use execute instead of fetch

df0d47a

fix test

0920963

fix test

84d3a68

EmilLuta reviewed Feb 24, 2025

View reviewed changes

Artemka374 added 3 commits March 11, 2025 13:16

Merge branch 'main' into afo/web-socket-gateway

43f0d3e

fix cargo.lock

ca0d222

address comments

565edc8

EmilLuta approved these changes Mar 12, 2025

View reviewed changes

Artemka374 added this pull request to the merge queue Mar 14, 2025

Merged via the queue into main with commit 4e74730 Mar 14, 2025
46 checks passed

Artemka374 deleted the afo/web-socket-gateway branch March 14, 2025 13:48

This was referenced Mar 14, 2025

chore(main): release core 27.0.0 #3686

Merged

chore(main): release zkstack_cli 0.2.0 #3086

Open

chore(main): release prover 20.0.0 #3672

Merged

Artemka374 added a commit that referenced this pull request Mar 26, 2025

Revert "feat: Use JSON-RPC for core <> prover interaction (#3626)"

0c09587

This reverts commit 4e74730

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Use JSON-RPC for core <> prover interaction #3626

feat: Use JSON-RPC for core <> prover interaction #3626

Uh oh!

Artemka374 commented Feb 19, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

EmilLuta left a comment

Uh oh!

Uh oh!

Uh oh!

feat: Use JSON-RPC for core <> prover interaction #3626

feat: Use JSON-RPC for core <> prover interaction #3626

Uh oh!

Conversation

Artemka374 commented Feb 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What ❔

Why ❔

Is this a breaking change?

Operational changes

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

EmilLuta left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Artemka374 commented Feb 19, 2025 •

edited

Loading