feat(l1): improve rebuilding speed during snap sync #2447
Conversation
fmoletta
changed the title
…_ethereum_rust into improve-rebuild-speed
Lines of code reportTotal lines added: Detailed view |
| /// Max storages to rebuild in parallel | ||
| const MAX_PARALLEL_REBUILDS: usize = 15; | ||
|
|
||
| const MAX_SNAPSHOT_READS_WITHOUT_COMMIT: usize = 5; |
There was a problem hiding this comment.
what's the criteria for this number?
There was a problem hiding this comment.
Trial and error mostly. The gist is that less commits make rebuild go faster but large commits can slow down other processes. I sought the highest value that would not slow down state sync.
There was a problem hiding this comment.
Wonder if these values that might be ideal for Holesky, are also appropriate for the other networks
There was a problem hiding this comment.
These values should be the least affected by a change of network as rebuild happens in the background and only interacts with the DB, it is not really affected by peer availability/speed at which we receive data from peers, and we cannot really split it into different processes to scale to a bigger trie size.
They could possibly change along with changes/improvements to trie caches and the DB.
These are also strong candidates to become customizable via config file (such as in #2331 )
**Motivation** After recent changes in main, rebuilding now takes a lot longer than state sync. This PR aims to mitigate this hit by introducing other performance upgrades <!-- Why does this pull request exist? What are its goals? --> **Description** * Increase parallelism when rebuilding storages * Reduce intermediate hashing when rebuilding state tries <!-- A clear and concise general description of the changes this PR introduces --> These changes have increased storage rebuild speed to around the same as before the changes to store, and has reduced time estimates for state rebuild, but doesn't manage to make the state rebuild keep up with the state sync. These changes have not affected state sync speed <!-- Link to issues: Resolves lambdaclass#111, Resolves lambdaclass#222 --> Closes #issue_number
Motivation
After recent changes in main, rebuilding now takes a lot longer than state sync. This PR aims to mitigate this hit by introducing other performance upgrades
Description
These changes have increased storage rebuild speed to around the same as before the changes to store, and has reduced time estimates for state rebuild, but doesn't manage to make the state rebuild keep up with the state sync. These changes have not affected state sync speed
Closes #issue_number