feat: Lambda function support from DataFusion, illustrated with array_filter

docs/source/contributor-guide/roadmap.md

@@ -41,14 +41,16 @@ supported functions, frames, and fallback cases.
 
 ## Native Lambda Evaluation
 
-Spark supports higher-order functions on arrays and maps that take a lambda, including `transform`, `exists`,
-`forall`, `aggregate`, `zip_with`, `map_filter`, and `map_zip_with`. Comet evaluates these today through a JVM
+Spark supports higher-order functions on arrays and maps that take a lambda, including `filter`, `transform`, `exists`,
+`forall`, `aggregate`, `zip_with`, `map_filter`, and `map_zip_with`. Comet historically evaluated these through a JVM
 codegen-dispatch bridge (`CometScalaUDF`, `CometBatchKernelCodegen`) instead of falling back to Spark, but the
-lambda body is still interpreted row-at-a-time on the JVM rather than natively in DataFusion. DataFusion added
-native higher-order function support (`array_transform`, `array_filter`, `array_any_match`, etc.) that Comet
-does not yet use; it's not yet known whether their semantics are Spark-compatible. We'll explore whether
-DataFusion's implementations can replace the JVM codegen-dispatch bridge to remove that round-trip and let
-these expressions benefit from vectorized native execution.
+lambda body was still interpreted row-at-a-time on the JVM rather than natively in DataFusion.
+
+Comet now supports native evaluation for unary `filter` lambdas leveraging DataFusion's `array_filter`, with strict
+per-element short-circuiting and empty-batch protection, while retaining the JVM codegen-dispatch bridge as a fallback
+for indexed or unsupported shapes. Future work will explore whether DataFusion's native implementations for other
+higher-order functions (`array_transform`, `array_any_match`, etc.) can similarly replace the JVM codegen-dispatch bridge
+to remove that round-trip and allow more expressions to benefit from vectorized native execution.
 
 ## Native Coverage for Codegen-Dispatched Expressions
 

docs/source/user-guide/latest/expressions.md

@@ -383,11 +383,11 @@ to Spark ([#2837](https://github.com/apache/datafusion-comet/issues/2837)). Enab
 ## lambda_funcs
 
 | Function | Status | Implementation | Notes |
-| --- | --- | --- | --- |
+| --- | --- |--------| --- |
 | `aggregate` | ✅ | Codegen dispatch |  |
 | `array_sort` | ✅ | Codegen dispatch |  |
 | `exists` | ✅ | Codegen dispatch |  |
-| `filter` | ✅ | Native | General lambda routed through the JVM codegen dispatcher; the `array_compact` form runs natively |
+| `filter` | ✅ | Hybrid | Single-argument lambdas and array_compact run natively with strict per-element short-circuiting; two-argument lambdas (with index) and expressions requiring JVM codegen degrade to Codegen dispatch. |
 | `forall` | ✅ | Codegen dispatch |  |
 | `map_filter` | ✅ | Codegen dispatch |  |
 | `map_zip_with` | ✅ | Codegen dispatch |  |