Skip to content
This repository was archived by the owner on Oct 4, 2026. It is now read-only.
This repository was archived by the owner on Oct 4, 2026. It is now read-only.

Check what EvoTrees is doing #8

Open

Activity

  1. mmikhasenko commented on Sep 29, 2026

    @mmikhasenko
    MemberAuthor

    EvoTrees comparison: results and reproducible scripts

    Compared JLBoost at 06d9924 with the requested EvoTrees commit 0d83f261 (version 0.18.8), on 2026-09-29.

    EvoTrees uses gradient/Newton boosting, not classical AdaBoost. The controlled logistic calculations match, including positive observation weights. The packages are not interchangeable under their defaults, and the comparison exposed correctness issues in this JLBoost fork.

    Scripts, report, and raw results

    Downloadable reproduction bundle on GitHub Gist — use Download ZIP on the Gist page. It contains:

    • compare.jl: controlled numerical checks and warmed training/prediction benchmarks.
    • diagnostics.jl: independent weighted-stump oracle and minimal regression reproduction.
    • summarize.py: validates agreement/invariants and generates the measurement tables.
    • README.md: setup and reproduction commands, with pinned source revisions.
    • COMPARISON.md: full functionality, weights, API, documentation, and correctness review.
    • measurements.md, raw result TOMLs, and dependency-manifest snapshots.

    Do the results match?

    Check Result
    Binary logistic, 1/10 rounds, uniform/nonuniform positive weights, L2 penalties 0/2 Maximum probability difference below 3.4e-8 on controlled discrete-feature data
    Additional stump and matched gamma cases Differences below 3.9e-8
    Positive integer weights vs duplicated observations Agree within numerical tolerance in both packages
    All positive weights scaled by 10, with penalties/threshold effects disabled Predictions unchanged within numerical tolerance
    Native L2DistLoss() regression Does not match: reversed gradient arguments in JLBoost
    Diagnostic custom MSE with correctly ordered arguments Eight controlled configurations agree within 1.8e-7
    Nonzero warm start One round agrees; after two rounds the maximum residual-probability difference is 0.03527

    On continuous features, predictions are not pointwise identical: EvoTrees uses histogram boundaries, while JLBoost searches exact distinct-value splits. Held-out logistic quality was very similar in the tested datasets.

    Correctness findings

    1. Regression derivative argument order. g_h.jl calls deriv(loss, target, prediction), while LossFunctions 1.1.0 expects deriv(loss, output, target). With a constant feature, target 3, one round, eta 1, and no penalty, L2DistLoss() predicts −3; EvoTrees predicts 3. JLBoost's own target-first LogitLogLoss is unaffected. The custom MSE in the bundle is a diagnostic, not a library fix.
    2. Warm-start baseline is dropped after the first round. jlboost-fit.jl evaluates later rounds using only prior-tree predictions, omitting the original baseline.
    3. Child threshold only checks the left child. find_best_split.jl checks the selected split's left Hessian, without checking the right child or choosing the best eligible alternative.
    4. Weight validation is incomplete. JLBoost accepts negative weights; all-zero weights produced nonfinite predictions. EvoTrees rejects any nonpositive weight at this commit. JLBoost accepts some zero weights, but this does not universally guarantee equivalence to deleting those rows.

    All 63 existing JLBoost test assertions pass, so these cases reveal coverage gaps. Library source was left unchanged.

    Timing and quality

    Apple M2, Julia 1.11.5, one Julia compute thread, CPU only. Eight continuous features, positive nonuniform weights, 20 rounds, path depth three, no sampling. Median of three warmed repetitions, packages run sequentially. Fit includes initialization/binning; input-table construction is excluded. EvoTrees uses 64 bins and JLBoost exact splits.

    Logistic training rows JLBoost fit EvoTrees fit EvoTrees speedup JLBoost held-out log loss EvoTrees held-out log loss
    1,000 31.7 ms 5.22 ms 6.1× 0.569373 0.569025
    10,000 238 ms 14.5 ms 16.4× 0.557754 0.557357
    50,000 1.276 s 60.3 ms 21.2× 0.555512 0.555266

    For the 50k-trained model, prediction on 2,000 held-out rows took 0.358 ms for JLBoost vs 0.740 ms for EvoTrees. This small-batch result should not be extrapolated to large-batch inference. Logistic allocation volume per fit was about 2.33 GB vs 11.4 MB, respectively; these are cumulative allocations, not peak memory.

    Native JLBoost regression RMSE was 8.4–8.7 because of the derivative bug. The diagnostic custom MSE gave RMSE 0.747–0.759, compared with EvoTrees' 0.751–0.764. Native regression timings therefore do not represent equal-quality work. Full timings and raw samples are in the bundle. These are synthetic, single-seed CPU measurements, not GPU/thread-scaling or production-data benchmarks.

    Functionality, API, and documentation

    Area JLBoost fork EvoTrees at the pinned commit
    Objectives Binary logistic; generic LossFunctions adapter currently affected by the bug above Regression objectives, binary/multiclass classification, multiple targets and distributional objectives
    Features Numeric/Boolean; missing and categorical features unsupported per README Numeric/Boolean and categorical features; missing features require preprocessing
    Tree growth Depth-wise or best-first max_leaves Binary depth-wise or oblivious/symmetric trees
    Training controls Row/column sampling; explicit tree scaling Sampling, early stopping, monotonic constraints, bagging, explicit seed
    Integration/compute Tables API; serial implementation; old MLJ wrapper archived Tables/matrix APIs, in-package MLJ, threaded CPU and GPU extensions
    Observation weights weights=w; multiplies gradients and Hessians weight_name or w_train; same gradient/Hessian scaling
    Inspection Node objects, gain/coverage/frequency, per-tree eta wrapper Array-based trees, gain importance, plotting, TreeSHAP
    Documentation Concise README and Markdown tutorials; hosted Quarto site planned Broader stable/dev documentation and tutorials, but some examples have stale keywords

    Capabilities outside the shared logistic/MSE paths were source-reviewed, not all runtime-tested. EvoTrees documentation issues include README fnames instead of feature_names, overview examples using rng instead of seed, and assigning dtrain.y .= ... before creating the column.

    Required parameter translation: JLBoost max_depth=d corresponds to EvoTrees max_depth=d+1; JLBoost lambda=λ corresponds to EvoTrees L2=λ, lambda=0; JLBoost gamma=γ corresponds to EvoTrees gamma=γ/2 for the shared objectives. JLBoost returns raw logistic margins; EvoTrees returns probabilities. Neutral initialization must be requested in EvoTrees, whose default bias is computed from the unweighted targets. Child thresholds have different units: Hessian mass vs observation-weight mass.

    Dependency compatibility: these versions do not resolve together in one clean environment: SortingLab 0.3 requires CategoricalArrays 0.10, while EvoTrees requires CategoricalArrays 1. The bundle runs them in separate processes and environments without overriding compatibility.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions