feat(consensus): Add consensus protocol versioning #3720

brunoffranca · 2025-03-17T14:46:10Z

What ❔

It makes some necessary changes to allow supporting multiple consensus protocol versions. Most of the changes are on the era-consensus side (which this PR updates to v0.9). On the zksync-era side, I changed the DB to store versioned block certificates.
This PR also removes the genesis column on the consensus_replica_state table since that was deprecated a long time ago.
Also updates the Rust toolchain a bit (needed for a stable feature used in era-consensus). This caused a bunch of lints to need to also be fixed in this PR (mostly too_long_first_doc_paragraph).

Part of BFT-483

Why ❔

We need to be able to support at least 2 consensus protocol versions concurrently in order to be able to do consensus protocol upgrades.

Is this a breaking change?

Yes
No

Operational changes

None.

Checklist

PR title corresponds to the body of PR (we generate changelog entries from PRs).
Tests for the changes have been added / updated.
Documentation comments have been added / updated.
Code has been formatted via zkstack dev fmt and zkstack dev lint.

core/lib/basic_types/src/macros.rs

core/lib/dal/migrations/20250314155250_consensus_versioning.up.sql

core/lib/dal/src/consensus_dal/mod.rs

github-actions · 2025-03-25T15:09:35Z

Detected VM performance changes

Benchmark name	Est. cycles	Change in est. cycles
init/fast	14923416	+4.8%
init/fast_no_sigs	14957579	+4.8%

hatemosphere

approving only changes in CI

🤖 I have created a release *beep* *boop* --- ## [27.1.0](core-v27.0.0...core-v27.1.0) (2025-03-27) ### Features * **consensus:** Add consensus protocol versioning ([#3720](#3720)) ([d1b4308](d1b4308)) * **zkos:** Implement ZK OS tree manager ([#3730](#3730)) ([efc0007](efc0007)) ### Bug Fixes * **api:** Fix panic applying nonce override ([#3748](#3748)) ([944059b](944059b)) * **contract_verifier:** order deploy events in `get_contract_info_for_verification` ([#3766](#3766)) ([6e3c031](6e3c031)) * make proof data handler backwards compatible ([#3767](#3767)) ([bdbbaaa](bdbbaaa)) * **proof_data_handler:** update save_proof_artifacts_metadata UPDATE ([#3758](#3758)) ([ed4926f](ed4926f)) * **vm:** Fix VM divergence in revert data ([#3570](#3570)) ([b82e2e4](b82e2e4)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: zksync-era-bot <zksync-era-bot@users.noreply.github.com>

🤖 I have created a release *beep* *boop* --- ## [20.0.0](prover-v19.0.0...prover-v20.0.0) (2025-04-01) ### ⚠ BREAKING CHANGES * Remove old prover stack ([#3729](#3729)) * V27 update ([#3580](#3580)) ### Features * Add S3 implementation for object_store ([#3664](#3664)) ([a848927](a848927)) * **consensus:** Add consensus protocol versioning ([#3720](#3720)) ([d1b4308](d1b4308)) * **gateway:** Migration to Gateway ([#3654](#3654)) ([2858ba0](2858ba0)) * Remove old prover stack ([#3729](#3729)) ([fbbdc76](fbbdc76)) * Use JSON-RPC for core <> prover interaction ([#3626](#3626)) ([4e74730](4e74730)) * V27 update ([#3580](#3580)) ([9e18550](9e18550)) ### Bug Fixes * API URL for prover gateway ([#3716](#3716)) ([ca2c4a4](ca2c4a4)) * make proof data handler backwards compatible ([#3767](#3767)) ([bdbbaaa](bdbbaaa)) * Prover job ordering ([#3738](#3738)) ([8f7f831](8f7f831)) * **prover:** Fixing shutdown time for circuit-prover ([#3768](#3768)) ([6c9de26](6c9de26)) * **prover:** Reevaluation of 'heavy' jobs for WVG ([#3754](#3754)) ([2a8d33b](2a8d33b)) * **prover:** Remove deleted pods from autoscaler-agent cluster cache ([#3739](#3739)) ([e94985f](e94985f)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: zksync-era-bot <147085853+zksync-era-bot@users.noreply.github.com> Co-authored-by: zksync-era-bot <zksync-era-bot@users.noreply.github.com>

…mmittee rotation (#3775) ## What ❔ It introduces some code necessary for validator committee rotation. This doesn't enable that functionality yet as we still need the new consensus registry contract to be merged (matter-labs/era-contracts#1365) and also change to matter-labs/era-consensus. A future PR will integrate those changes and enable it. It deprecates almost all the code related to the attester role. Since ZK OS won't have L1 batches, this role won't be needed in the future. Plus, if we decide to do L1 consensus verification before ZK OS, it's simpler to just do an operator multisig (the functionality is reduced but it's a temporary solution anyway). It also deprecates the genesis column in the consensus_replica_state Postgres table. Storing genesis has been deprecated in favor of storing global config a long time ago. #3720 already removed all code related to it, this completes the deprecation by removing the column. My next PR will remove the attester method from ZK Stack CLI. I'll do it in a separate PR since this one is already somewhat large and ZK Stack CLI changes are best reviewed by someone familiar with it. Part of PLA-1112 and PLA-1113 ## Why ❔ It prepares the repo for validator committee rotation, which is the ability to change the validator set by interacting with a L2 smart contract (the consensus registry). This then enables staking, all being required after it is a contract that handles the staking UI and updates the validator set on the consensus registry. ## Is this a breaking change? - [X] Yes - [ ] No Technically, yes. Any chain that has a working attester set will stop attesting batches, but nothing else will break. ## Operational changes On the consensus config of the main node, the attesters field should be removed. Not removing it won't break anything, the field will just be ignored by protobuf. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zkstack dev fmt` and `zkstack dev lint`.

## What ❔ It makes some necessary changes to allow supporting multiple consensus protocol versions. Most of the changes are on the era-consensus side (which this PR updates to v0.9). On the zksync-era side, I changed the DB to store versioned block certificates. This PR also removes the genesis column on the consensus_replica_state table since that was deprecated a long time ago. Also updates the Rust toolchain a bit (needed for a stable feature used in era-consensus). This caused a bunch of lints to need to also be fixed in this PR (mostly `too_long_first_doc_paragraph`). Part of BFT-483 ## Why ❔ We need to be able to support at least 2 consensus protocol versions concurrently in order to be able to do consensus protocol upgrades. ## Is this a breaking change? - [ ] Yes - [x] No ## Operational changes None. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zkstack dev fmt` and `zkstack dev lint`.

🤖 I have created a release *beep* *boop* --- ## [27.1.0](core-v27.0.0...core-v27.1.0) (2025-03-27) ### Features * **consensus:** Add consensus protocol versioning ([#3720](#3720)) ([1e3d17e](1e3d17e)) * **zkos:** Implement ZK OS tree manager ([#3730](#3730)) ([efc0007](efc0007)) ### Bug Fixes * **api:** Fix panic applying nonce override ([#3748](#3748)) ([944059b](944059b)) * **contract_verifier:** order deploy events in `get_contract_info_for_verification` ([#3766](#3766)) ([6e3c031](6e3c031)) * make proof data handler backwards compatible ([#3767](#3767)) ([d7b0122](d7b0122)) * **proof_data_handler:** update save_proof_artifacts_metadata UPDATE ([#3758](#3758)) ([ed4926f](ed4926f)) * **vm:** Fix VM divergence in revert data ([#3570](#3570)) ([b82e2e4](b82e2e4)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: zksync-era-bot <zksync-era-bot@users.noreply.github.com>

🤖 I have created a release *beep* *boop* --- ## [27.1.0](matter-labs/zksync-era@core-v27.0.0...core-v27.1.0) (2025-03-27) ### Features * **consensus:** Add consensus protocol versioning ([#3720](matter-labs/zksync-era#3720)) ([15771c5](matter-labs/zksync-era@15771c5)) * **zkos:** Implement ZK OS tree manager ([#3730](matter-labs/zksync-era#3730)) ([efc0007](matter-labs/zksync-era@efc0007)) ### Bug Fixes * **api:** Fix panic applying nonce override ([#3748](matter-labs/zksync-era#3748)) ([944059b](matter-labs/zksync-era@944059b)) * **contract_verifier:** order deploy events in `get_contract_info_for_verification` ([#3766](matter-labs/zksync-era#3766)) ([6e3c031](matter-labs/zksync-era@6e3c031)) * make proof data handler backwards compatible ([#3767](matter-labs/zksync-era#3767)) ([c1788df](matter-labs/zksync-era@c1788df)) * **proof_data_handler:** update save_proof_artifacts_metadata UPDATE ([#3758](matter-labs/zksync-era#3758)) ([ed4926f](matter-labs/zksync-era@ed4926f)) * **vm:** Fix VM divergence in revert data ([#3570](matter-labs/zksync-era#3570)) ([b82e2e4](matter-labs/zksync-era@b82e2e4)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: zksync-era-bot <zksync-era-bot@users.noreply.github.com>

brunoffranca added 13 commits March 4, 2025 15:20

update toml

253e949

lock update

a87d244

Merge branch 'main' into bf-consensus-0.9

5108e64

Added BlockCertificate proto.

24424fd

Created migrations

5c266e8

Merge branch 'main' into bf-consensus-0.9

75da2d6

Fixed consensus_dal

995f635

update toolchain

3b1e344

started work on consensus crate

7dc9f3a

more work on consensus crate. missing tests.

9fd0de7

Unit tests pass.

3ab9e99

fmt

993be26

Merge branch 'main' into bf-consensus-0.9

6305387

brunoffranca force-pushed the bf-consensus-0.9 branch from 67acdd6 to 6305387 Compare March 19, 2025 18:43

brunoffranca marked this pull request as ready for review March 19, 2025 18:49

brunoffranca added 11 commits March 19, 2025 18:49

Merge branch 'main' into bf-consensus-0.9

3614ee5

nullable

5ab015d

dal patch

ca433e6

lints

fa22f54

more lints

0ac5436

more lints

13adf1e

Merge branch 'main' into bf-consensus-0.9

3d3aad0

ci fix

0bf956e

toolchain zkstack

89d6196

Merge branch 'main' into bf-consensus-0.9

c80bcb5

more toolchain

e16b3e8

brunoffranca requested a review from a team as a code owner March 21, 2025 14:59

brunoffranca requested review from yorik, alexandrst88 and artmakh March 21, 2025 14:59

RomanBrodetski reviewed Mar 24, 2025

View reviewed changes

brunoffranca added 7 commits March 24, 2025 17:01

Merge branch 'main' into bf-consensus-0.9

3a4e237

small fixes from PR review

37d2e9d

block_certificate

047c43f

migration

b3c7714

fix lint

6c9e96a

encapsulate BlockCertificate logic

27de2e7

sqlx prepare

d2161bb

brunoffranca requested review from RomanBrodetski and perekopskiy March 24, 2025 19:42

perekopskiy approved these changes Mar 25, 2025

View reviewed changes

Merge branch 'main' into bf-consensus-0.9

d55bc14

brunoffranca enabled auto-merge March 25, 2025 16:42

perekopskiy approved these changes Mar 25, 2025

View reviewed changes

hatemosphere approved these changes Mar 25, 2025

View reviewed changes

brunoffranca added this pull request to the merge queue Mar 25, 2025

Merged via the queue into main with commit d1b4308 Mar 25, 2025
54 checks passed

brunoffranca deleted the bf-consensus-0.9 branch March 25, 2025 21:16

This was referenced Mar 25, 2025

chore(main): release core 27.1.0 #3755

Merged

chore(main): release prover 20.0.0 #3672

Merged

chore(main): release zkstack_cli 0.2.0 #3086

Open

brunoffranca mentioned this pull request Apr 9, 2025

refactor(consensus): Deprecate attesters and prepare for validator committee rotation #3775

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(consensus): Add consensus protocol versioning #3720

feat(consensus): Add consensus protocol versioning #3720

Uh oh!

brunoffranca commented Mar 17, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Mar 25, 2025

Uh oh!

hatemosphere left a comment

Uh oh!

Uh oh!

Uh oh!

feat(consensus): Add consensus protocol versioning #3720

feat(consensus): Add consensus protocol versioning #3720

Uh oh!

Conversation

brunoffranca commented Mar 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What ❔

Why ❔

Is this a breaking change?

Operational changes

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Mar 25, 2025

Detected VM performance changes

Uh oh!

hatemosphere left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

brunoffranca commented Mar 17, 2025 •

edited

Loading