linq_fold: Phase 2A loop planner (where|select array + counter lanes)

[borisbat]

Summary

Phase 2A of the linq_fold splice-mode rewrite (plan: ~/.claude/plans/keen-hopping-balloon.md; foundation landed in #2687). _fold now emits an explicit for-loop inside invoke($block, $src) for two narrow shape families. Anything outside scope returns the raw chain unfolded — no dispatch to _old_fold or to fold_linq_default's nested-pass emitter.

In scope:

• Array lane — [where_*][select*] (implicit to_array). Chained _where|_where|... fuses via &&; chained workhorse _select|_select|... fuses via intermediate var v_N = projection_N let-bindings (each next lambda's _ is renamed straight to the prior binding's name, no expression substitution).
• Counter lane — same intermediates terminated by _count. Emits var n = 0; for... if... n++; return n with no array materialization.

Out of scope (falls through to raw chain): _select|_where, sum, min, max, average, first, any, all, long_count, _order, _distinct, _take, _skip, _zip, _reverse, and non-workhorse chained selects. These compile to plain linq (m3f ≈ m3) — accepted regressions vs the historical _old_fold baseline. Phase 2B picks them up; the regressions are the forcing function so we feel the gap immediately.

_old_fold is untouched and continues to own the comprehension-emission contract; 10 existing AST tests were retargeted to _old_fold so the frozen baseline still has explicit coverage, and 8 new AST tests + 6 behavioral parity tests cover the new loop emission.

Phase 2A benchmark deltas (100K rows, ns/op per element, INTERP)

Benchmark	Shape	m3f_old	m3f	Delta
count_aggregate	where → count	5	5	parity
chained_where	where → where → count	17	8	2.1× faster
select_count	select → count	15	2	7.5× faster
to_array_filter	where → select → to_array	11	11	parity

Implementation notes

Four small tricks closed the to_array_filter gap from a first-cut 13 → 11 ns/op (parity with the comprehension baseline):

1. Workhorse decision at macro time. The projection's _type is resolved by the time LinqFold.visit() fires, so the planner reads projection._type.isWorkhorseType directly and emits exactly one branch — no static_if (typeinfo is_workhorse(...)) at runtime.
2. Peel each(<array>). each(arr) reports as iterator<T>, so the array-only reserve path got skipped on benchmark sources like each(arr)._where(...)._select(...).to_array(). The planner detects each(<source-with-length>) and unwraps it. Iteration semantics are unchanged — for (it in arr) and for (it in each(arr)) yield the same element refs.
3. Pre-reserve. Once the source has a known length (post-peel), emit acc |> reserve(length(src)) before the loop — matches what ExprArrayComprehension lowering does internally.
4. No emplace in emission. emplace moves out of its argument and can corrupt the source when the projection returns a ref into it (e.g. _._field). The planner emits push for workhorse and push_clone for non-workhorse — no intermediate var v <- proj; emplace(v) dance.

Chained _select|_select was the original Phase 2A gap. Plain Template.replaceVariable("it", proj_prev) + apply_template substitution fails because the typer wraps it reads in ExprRef2Value; the Template visitor only replaces the inner ExprVar, leaving ExprRef2Value(<non-ref value>) and a "can only dereference a reference" error. The fix here is to not substitute at all — bind the prior projection to a fresh local in the loop body and rename the next lambda's _ to that name via the existing fold_linq_cond. Non-workhorse chained selects still fall through (would need := clone for the intermediate binding since <- can corrupt source for lvalue projections; deferred to Phase 2B).

Files changed

• daslib/linq_fold.das — new plan_loop_or_count planner; LinqFold.visit() rewired to plan-then-fall-through. _old_fold, fold_linq_default, g_foldSeq, linqCalls, flatten_linq, fold_linq_cond, and every existing fold_* helper are untouched.
• tests/linq/test_linq_fold_ast.das — 10 AST tests retargeted to _old_fold (parallel target_*_old_fold functions added); 8 new AST tests covering the new loop-emission shapes; 6 new behavioral parity tests for chained-where, chained-select, where-count, select-count, count, bare-count.
• benchmarks/sql/select_count.das — new 4-way bench. chained_where.das and to_array_filter.das already existed; their m3f columns now show the speedup / parity.
• benchmarks/sql/LINQ.md — Phase 2A row in the phase-status table; new "Phase 2A — Loop planner" section with the delta table and implementation notes.

Test plan

• mcp__daslang__lint on all changed .das files — 0 issues
• mcp__daslang__format_file — all already formatted
• mcp__daslang__compile_check — all 4 files clean
• Full tests/linq/ suite (15 files, 592 tests) — pass
• tests/dasSQLITE/test_05_sql_macro.das (19 tests) — pass (no transitive regression)
• AOT: test_aot -use-aot dastest/dastest.das -- --use-aot --test tests/linq — 614 tests, all pass
• Phase 2A 4 benchmarks at 100K — m3f at parity-or-better on every shape

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR advances the linq_fold splice-mode rewrite by introducing a Phase 2A planner that emits explicit invoke(...) + for loops for a narrow set of LINQ shapes, while making _fold fall through to the raw (unfolded) LINQ chain for everything out of scope. It also preserves the historical comprehension-emission contract via _old_fold and updates tests/benchmarks/docs accordingly.

Changes:

• Add plan_loop_or_count in daslib/linq_fold.das and rewire _fold to “plan-then-fall-through” (no dispatch to _old_fold / fold_linq_default when unmatched).
• Retarget existing AST-shape tests to _old_fold and add new AST + behavioral tests for the new loop-emission shapes.
• Add a new select → count benchmark and document Phase 2A status/deltas in benchmarks/sql/LINQ.md.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
daslib/linq_fold.das	Adds the Phase 2A loop planner and changes _fold to emit loops for supported shapes or fall through unchanged.
tests/linq/test_linq_fold_ast.das	Splits AST-contract coverage: _old_fold retains comprehension baseline tests; _fold gains loop-emission and parity tests.
benchmarks/sql/select_count.das	Introduces a benchmark for select → count to measure Phase 2A counter-lane behavior.
benchmarks/sql/LINQ.md	Updates the phase status table and documents Phase 2A scope and benchmark deltas.

Comments suppressed due to low confidence (1)

tests/linq/test_linq_fold_ast.das:474

• Same issue as the where-case: qm_resolve_comprehension is run on the invoke’s block argument (ExprMakeBlock), which will always return null and doesn’t validate the absence of comprehensions inside the generated loop body. The assertion should inspect the block’s body/return expression instead of the block node itself.

        let inv = body_expr as ExprInvoke
        var arg0 = clone_expression(inv.arguments[0])
        var maybe_comp <- qm_resolve_comprehension(arg0)
        t |> success(maybe_comp == null, "loop planner must NOT emit a comprehension")
    }

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.