Skip to content

feat(makefile,ci): add cairo_test_suite_programs target and CI integration#2380

Draft
naor-starkware wants to merge 2 commits intonaor/feat/add_test_helpersfrom
naor/refactor/Makefile-include-test_cairo_suite_target
Draft

feat(makefile,ci): add cairo_test_suite_programs target and CI integration#2380
naor-starkware wants to merge 2 commits intonaor/feat/add_test_helpersfrom
naor/refactor/Makefile-include-test_cairo_suite_target

Conversation

@naor-starkware
Copy link
Copy Markdown
Collaborator

@naor-starkware naor-starkware commented Mar 29, 2026

TITLE

Description

Description of the pull request changes and motivation.

Checklist

  • Linked to Github Issue
  • Unit tests added
  • Integration tests added.
  • This change requires new documentation.
    • Documentation has been added/updated.
    • CHANGELOG has been updated.

This change is Reviewable

Copy link
Copy Markdown
Collaborator Author

naor-starkware commented Mar 29, 2026

@github-actions
Copy link
Copy Markdown

github-actions bot commented Mar 29, 2026

Benchmark Results for unmodified programs 🚀

Command Mean [s] Min [s] Max [s] Relative
base big_factorial 2.121 ± 0.022 2.100 2.167 1.00 ± 0.01
head big_factorial 2.116 ± 0.008 2.103 2.132 1.00
Command Mean [s] Min [s] Max [s] Relative
base big_fibonacci 2.050 ± 0.018 2.036 2.093 1.00
head big_fibonacci 2.059 ± 0.049 2.032 2.192 1.00 ± 0.03
Command Mean [s] Min [s] Max [s] Relative
base blake2s_integration_benchmark 7.454 ± 0.127 7.358 7.732 1.00 ± 0.03
head blake2s_integration_benchmark 7.451 ± 0.152 7.359 7.816 1.00
Command Mean [s] Min [s] Max [s] Relative
base compare_arrays_200000 2.170 ± 0.005 2.164 2.178 1.00
head compare_arrays_200000 2.173 ± 0.007 2.162 2.182 1.00 ± 0.00
Command Mean [s] Min [s] Max [s] Relative
base dict_integration_benchmark 1.429 ± 0.007 1.422 1.447 1.00 ± 0.01
head dict_integration_benchmark 1.422 ± 0.017 1.410 1.467 1.00
Command Mean [s] Min [s] Max [s] Relative
base field_arithmetic_get_square_benchmark 1.227 ± 0.004 1.223 1.236 1.00
head field_arithmetic_get_square_benchmark 1.231 ± 0.013 1.221 1.266 1.00 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
base integration_builtins 7.495 ± 0.013 7.476 7.519 1.00
head integration_builtins 7.503 ± 0.055 7.449 7.654 1.00 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
base keccak_integration_benchmark 7.596 ± 0.028 7.550 7.637 1.00
head keccak_integration_benchmark 7.622 ± 0.028 7.567 7.662 1.00 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
base linear_search 2.174 ± 0.008 2.162 2.187 1.00
head linear_search 2.186 ± 0.009 2.171 2.196 1.01 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
base math_cmp_and_pow_integration_benchmark 1.519 ± 0.003 1.515 1.526 1.00 ± 0.01
head math_cmp_and_pow_integration_benchmark 1.514 ± 0.010 1.500 1.535 1.00
Command Mean [s] Min [s] Max [s] Relative
base math_integration_benchmark 1.467 ± 0.004 1.460 1.474 1.00
head math_integration_benchmark 1.478 ± 0.014 1.466 1.514 1.01 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
base memory_integration_benchmark 1.228 ± 0.006 1.220 1.235 1.00 ± 0.01
head memory_integration_benchmark 1.223 ± 0.005 1.219 1.233 1.00
Command Mean [s] Min [s] Max [s] Relative
base operations_with_data_structures_benchmarks 1.542 ± 0.004 1.538 1.552 1.00
head operations_with_data_structures_benchmarks 1.557 ± 0.006 1.549 1.570 1.01 ± 0.00
Command Mean [ms] Min [ms] Max [ms] Relative
base pedersen 535.0 ± 1.8 532.9 538.2 1.00 ± 0.00
head pedersen 534.8 ± 1.7 532.6 537.8 1.00
Command Mean [ms] Min [ms] Max [ms] Relative
base poseidon_integration_benchmark 626.7 ± 1.2 625.2 628.6 1.00 ± 0.00
head poseidon_integration_benchmark 624.7 ± 1.7 622.9 627.7 1.00
Command Mean [s] Min [s] Max [s] Relative
base secp_integration_benchmark 1.819 ± 0.022 1.802 1.869 1.00
head secp_integration_benchmark 1.828 ± 0.022 1.804 1.880 1.00 ± 0.02
Command Mean [ms] Min [ms] Max [ms] Relative
base set_integration_benchmark 659.6 ± 1.6 658.0 663.3 1.00
head set_integration_benchmark 664.2 ± 10.1 658.6 691.5 1.01 ± 0.02
Command Mean [s] Min [s] Max [s] Relative
base uint256_integration_benchmark 4.228 ± 0.013 4.211 4.254 1.00
head uint256_integration_benchmark 4.260 ± 0.067 4.217 4.441 1.01 ± 0.02

@naor-starkware naor-starkware changed the base branch from naor/feat/add_math_cairo_test to graphite-base/2380 March 29, 2026 13:00
@naor-starkware naor-starkware force-pushed the naor/refactor/Makefile-include-test_cairo_suite_target branch from cf572eb to 646dd9e Compare March 29, 2026 13:01
@naor-starkware naor-starkware changed the base branch from graphite-base/2380 to naor/feat/add_test_helpers March 29, 2026 13:01
@naor-starkware naor-starkware force-pushed the naor/refactor/Makefile-include-test_cairo_suite_target branch from 646dd9e to 2bc600b Compare March 29, 2026 13:13
@codecov
Copy link
Copy Markdown

codecov bot commented Mar 29, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.06%. Comparing base (6ec5328) to head (2bc600b).

Additional details and impacted files
@@                     Coverage Diff                     @@
##           naor/feat/add_test_helpers    #2380   +/-   ##
===========================================================
  Coverage                       96.06%   96.06%           
===========================================================
  Files                             107      107           
  Lines                           37964    37964           
===========================================================
  Hits                            36471    36471           
  Misses                           1493     1493           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@naor-starkware naor-starkware force-pushed the naor/refactor/Makefile-include-test_cairo_suite_target branch from 2bc600b to 6fd8964 Compare March 29, 2026 14:19
@naor-starkware naor-starkware force-pushed the naor/feat/add_test_helpers branch from 6ec5328 to ce3aeb1 Compare March 29, 2026 14:19
naor-starkware and others added 2 commits March 29, 2026 19:09
…ation

Makefile:
- Add CAIRO_TEST_SUITE_ROOT, CAIRO_TEST_SUITE_FILES, COMPILED_CAIRO_TEST_SUITE
  variables to track cairo test suite sources
- Add pattern rule to compile each .cairo to .json via cairo-compile
- Add cairo_test_suite_programs target
- Hook cairo_test_suite_programs into the test target
- Add cairo_test_suite_programs to .PHONY

CI (rust.yml):
- Add cairo_test_suite_programs to the build-programs matrix
- Add vm/src/tests/cairo_test_suite/**/*.json to CAIRO_PROGRAMS_PATH cache
- Update all program cache keys to hash test suite source files
- Add function_runner feature flag to the tests job

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@naor-starkware naor-starkware force-pushed the naor/refactor/Makefile-include-test_cairo_suite_target branch from 6fd8964 to c106a57 Compare March 29, 2026 16:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant