Add compilation stats #1339
Open
Add compilation stats #1339
+253
−31
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Benchmark results Main vs HEAD.Base
Head
Base
Head
Base
Head
Base
Head
Base
Head
Base
Head
|
Benchmarking resultsBenchmark for program
|
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
Cairo-vm (Rust, Cairo 1) |
11.608 ± 0.080 | 11.505 | 11.754 | 2.18 ± 0.03 |
cairo-native (embedded AOT) |
5.317 ± 0.071 | 5.234 | 5.462 | 1.00 |
cairo-native (embedded JIT using LLVM's ORC Engine) |
5.416 ± 0.047 | 5.315 | 5.480 | 1.02 ± 0.02 |
Benchmark for program dict_snapshot
Open benchmarks
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
Cairo-vm (Rust, Cairo 1) |
879.5 ± 13.2 | 860.0 | 899.6 | 1.00 |
cairo-native (embedded AOT) |
5152.8 ± 70.8 | 5043.4 | 5259.7 | 5.86 ± 0.12 |
cairo-native (embedded JIT using LLVM's ORC Engine) |
5170.2 ± 78.2 | 5038.6 | 5287.7 | 5.88 ± 0.13 |
Benchmark for program factorial_2M
Open benchmarks
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
Cairo-vm (Rust, Cairo 1) |
5.387 ± 0.046 | 5.315 | 5.466 | 1.00 |
cairo-native (embedded AOT) |
5.479 ± 0.081 | 5.390 | 5.635 | 1.02 ± 0.02 |
cairo-native (embedded JIT using LLVM's ORC Engine) |
5.511 ± 0.043 | 5.436 | 5.577 | 1.02 ± 0.01 |
Benchmark for program fib_2M
Open benchmarks
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
Cairo-vm (Rust, Cairo 1) |
5.183 ± 0.061 | 5.130 | 5.347 | 1.05 ± 0.02 |
cairo-native (embedded AOT) |
4.918 ± 0.047 | 4.829 | 4.979 | 1.00 |
cairo-native (embedded JIT using LLVM's ORC Engine) |
4.966 ± 0.070 | 4.876 | 5.071 | 1.01 ± 0.02 |
Benchmark for program linear_search
Open benchmarks
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
Cairo-vm (Rust, Cairo 1) |
858.4 ± 14.8 | 838.1 | 875.4 | 1.00 |
cairo-native (embedded AOT) |
5017.4 ± 78.7 | 4913.2 | 5120.7 | 5.84 ± 0.14 |
cairo-native (embedded JIT using LLVM's ORC Engine) |
5235.3 ± 93.9 | 5073.7 | 5360.2 | 6.10 ± 0.15 |
Benchmark for program logistic_map
Open benchmarks
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
Cairo-vm (Rust, Cairo 1) |
558.8 ± 6.4 | 552.0 | 570.5 | 1.00 |
cairo-native (embedded AOT) |
5097.5 ± 56.2 | 5001.5 | 5189.4 | 9.12 ± 0.15 |
cairo-native (embedded JIT using LLVM's ORC Engine) |
5313.8 ± 51.5 | 5241.8 | 5384.3 | 9.51 ± 0.14 |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1339 +/- ##
==========================================
- Coverage 79.16% 78.81% -0.36%
==========================================
Files 111 111
Lines 30440 30578 +138
==========================================
Hits 24099 24099
- Misses 6341 6479 +138 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
DiegoCivi
requested review from
edg-l,
igaray,
azteca1998 and
jrchatruc
as code owners
DiegoCivi
requested review from
Oppen,
pefontana and
gabrielbosio
as code owners
|
I don't like the idea of saving max values as statistics. Maybe a contract can have one really big function and a lot of small functions, while other contract have a lot of medium functions. The max value wouldn't be too significant. I would try to save the full data in every case, and not some aggregated value. If this ends up being too much data, we could have at least a more accurate representation of the distribution (like some percentiles/quartiles). Is there a way to link declaration with usage? For example:
Is there a way to record this information? We need to find some value/set of values that predicts with acceptable accuracy the compilation effort, and I feel like that involves "combining" different metrics together. What do you think? |
JulianGCalderon
requested changes
Aug 6, 2025
| let type_id = type_declaration.id.to_string(); | ||
| let type_size = type_concrete.layout(®istry).unwrap().size(); | ||
| if !type_concrete.is_builtin() { | ||
| // We dont want to add the builtins to the stats |
There was a problem hiding this comment.
Why are builtins skipped?
There was a problem hiding this comment.
I figured they are always the same size. So I though it didn't make much sense to have them.
There was a problem hiding this comment.
Its not worth treating them any differently. There are many types that are always the same size (i.e. u8). I would keep them, it would also make the code a bit simpler.
Co-authored-by: Julian Gonzalez Calderon <gonzalezcalderonjulian@gmail.com>
Co-authored-by: Julian Gonzalez Calderon <gonzalezcalderonjulian@gmail.com>
5 tasks
| @@ -416,6 +423,20 @@ pub fn layout_repeat(layout: &Layout, n: usize) -> Result<(Layout, usize), Layou | |||
| Ok((layout, padded_size)) | |||
| } | |||
|
|
|||
| /// Gets the size of the full set of params of a Sierra function | |||
There was a problem hiding this comment.
nit: we can use any type, not just the params of a function, so we could update the documentation to be more generic.
Suggested change
| /// Gets the size of the full set of params of a Sierra function | |
| /// Returns the total layout size for the given types. |
Comment on lines
+56
to
+57
| /// - params_total_size: Total size of all the params | ||
| /// - return_types_total_size: Total size of all the params |
There was a problem hiding this comment.
Please move each bullet point to the appropiate field doc comment. Idem for the other struct.
Something like:
/// Contains the following stats about a Sierra function:
pub struct SierraFuncStats {
/// Total size of all the params
pub params_total_size: usize,
/// Total size of all the params
pub return_types_total_size: usize,
pub times_used: usize,
}
Comment on lines
+25
to
+26
| /// Number of times each circuit gate is used. | ||
| sierra_circuit_gates_count: CircuitGatesStats, |
There was a problem hiding this comment.
Shouldn't this field be public?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We have contracts that take quite a bit of time to compile. To understand why this is happening we want to add more compilation stats.
Added stats:
sierra_declared_types_stats: Contains the stats for each Sierra declared typesierra_func_stats: Stats about params and return types of each Sierra functionsierra_circuit_gates_count: Number of times each circuit gate is usedChecklist