[SPARK-54776][SQL] Improved the logs message regarding lambda function with SQL UDF #53542

Shubhambhusate · 2025-12-19T11:53:02Z

What changes were proposed in this pull request?

Changes made:

Added new error condition in error-conditions.json:
UNSUPPORTED_FEATURE.LAMBDA_FUNCTION_WITH_SQL_UDF - A clear error message for SQL UDFs used in lambda functions.

Why are the changes needed?

Currently, when a SQL UDF is used inside a higher-order function like transform, the error message is confusing:


CREATE FUNCTION lower_udf(s STRING) RETURNS STRING RETURN lower(s);
SELECT transform(array('A', 'B'), x -> lower_udf(x));

Before (confusing error):

[MISSING_ATTRIBUTES.RESOLVED_ATTRIBUTE_MISSING_FROM_INPUT]
Resolved attribute(s) "x" missing from in operator !Project [cast(lambda x#20395 as string) AS s#20397].
SQLSTATE: XX000

This error doesn't explain why the attribute is missing or what the user should do.

After (clear error):

[UNSUPPORTED_FEATURE.LAMBDA_FUNCTION_WITH_SQL_UDF] The feature is not supported: Lambda function with SQL UDF "spark_catalog.default.lower_udf(lambda x)" in a higher order function. SQLSTATE: 0A000

This is consistent with the existing error message for Python UDFs in the same scenario (UNSUPPORTED_FEATURE.LAMBDA_FUNCTION_WITH_PYTHON_UDF).

Does this PR introduce any user-facing change?

Yes. Users will now see a clearer, more actionable error message when attempting to use a SQL UDF inside a higher-order function's lambda expression.

How was this patch tested?

Test 1:

Added a new test case "SQL UDF in higher-order function should fail with clear error message" in SQLFunctionSuite.scala that:
Creates a SQL UDF
Attempts to use it in a transform higher-order function
Verifies the error condition is UNSUPPORTED_FEATURE.LAMBDA_FUNCTION_WITH_SQL_UDF
Verifies the error message contains the function name and lambda x

Test 2:
Manual testing
spark.sql("CREATE OR REPLACE FUNCTION test_lower_udf(s STRING) RETURNS STRING RETURN lower(s)") spark.sql("SELECT transform(array('A', 'B'), x -> test_lower_udf(x))").show()

Was this patch authored or co-authored using generative AI tooling?

No

allisonwang-db

Thanks for improving the error message! cc @cloud-fan

allisonwang-db · 2025-12-22T18:47:36Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala

+        throw new AnalysisException(
+          errorClass = "UNSUPPORTED_FEATURE.LAMBDA_FUNCTION_WITH_SQL_UDF",
+          messageParameters = Map(
+            "funcName" -> ("\"" + function.name.unquotedString + "(" + formattedInputs + ")\"")))
+      }


Can we add this in UserDefinedFunctionErrors

As @cloud-fan mentioned that we don't include the function inputs in the error message. So i have removed it.

cloud-fan · 2025-12-23T08:03:22Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala

+          case v: NamedLambdaVariable => "lambda " + v.name
+          case v: UnresolvedNamedLambdaVariable => "lambda " + v.name
+          case e => e.sql
+        }.mkString(", ")


We usually don't include the function inputs in the error message, just function name is sufficient.

Thanks for the response i have change made required change in the code to remove function input in the error. Please review it

cloud-fan · 2026-01-06T09:50:43Z

sql/core/src/test/scala/org/apache/spark/sql/execution/SQLFunctionSuite.scala

+          |RETURNS STRING
+          |RETURN lower(s)
+          |""".stripMargin)
+      val exception = intercept[AnalysisException] {


let's use checkError to test errors.

I have made the changes used checkError to test errors

…functions

…functions removed the input parameter

cloud-fan · 2026-01-08T08:50:34Z

thanks, merging to master!

…n with SQL UDF ### What changes were proposed in this pull request? **Changes made:** Added new error condition in error-conditions.json: UNSUPPORTED_FEATURE.LAMBDA_FUNCTION_WITH_SQL_UDF - A clear error message for SQL UDFs used in lambda functions. ### Why are the changes needed? Currently, when a SQL UDF is used inside a higher-order function like transform, the error message is confusing: ``` CREATE FUNCTION lower_udf(s STRING) RETURNS STRING RETURN lower(s); SELECT transform(array('A', 'B'), x -> lower_udf(x)); ``` **Before (confusing error):** [MISSING_ATTRIBUTES.RESOLVED_ATTRIBUTE_MISSING_FROM_INPUT] Resolved attribute(s) "x" missing from in operator !Project [cast(lambda x#20395 as string) AS s#20397]. SQLSTATE: XX000 <img width="1728" height="427" alt="Screenshot 2025-12-18 at 6 13 29 PM" src="https://github.com/user-attachments/assets/8d7e79dd-bd86-4199-8b16-fae0b9313d46" /> This error doesn't explain why the attribute is missing or what the user should do. **After (clear error):** [UNSUPPORTED_FEATURE.LAMBDA_FUNCTION_WITH_SQL_UDF] The feature is not supported: Lambda function with SQL UDF "spark_catalog.default.lower_udf(lambda x)" in a higher order function. SQLSTATE: 0A000 <img width="1728" height="314" alt="Screenshot 2025-12-18 at 6 14 11 PM" src="https://github.com/user-attachments/assets/76b30d2d-1c3a-4a8d-8feb-65a5295d6d35" /> This is consistent with the existing error message for Python UDFs in the same scenario (UNSUPPORTED_FEATURE.LAMBDA_FUNCTION_WITH_PYTHON_UDF). ### Does this PR introduce _any_ user-facing change? Yes. Users will now see a clearer, more actionable error message when attempting to use a SQL UDF inside a higher-order function's lambda expression. ### How was this patch tested? **Test 1:** Added a new test case "SQL UDF in higher-order function should fail with clear error message" in SQLFunctionSuite.scala that: Creates a SQL UDF Attempts to use it in a transform higher-order function Verifies the error condition is UNSUPPORTED_FEATURE.LAMBDA_FUNCTION_WITH_SQL_UDF Verifies the error message contains the function name and lambda x **Test 2:** Manual testing spark.sql("CREATE OR REPLACE FUNCTION test_lower_udf(s STRING) RETURNS STRING RETURN lower(s)") spark.sql("SELECT transform(array('A', 'B'), x -> test_lower_udf(x))").show() ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#53542 from Shubhambhusate/fix/LAMBDA_FUNCTION_WITH_SQL_UDF. Authored-by: Shubhambhusate <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

github-actions bot added the SQL label Dec 19, 2025

Shubhambhusate changed the title ~~Improved the logs message regarding lambda function with SQL UDF~~ [SPARK-54776][SQL] Improved the logs message regarding lambda function with SQL UDFImproved the logs message regarding lambda function with SQL UDF Dec 19, 2025

Shubhambhusate force-pushed the fix/LAMBDA_FUNCTION_WITH_SQL_UDF branch from e87c41c to 6f50c79 Compare December 19, 2025 18:31

Shubhambhusate changed the title ~~[SPARK-54776][SQL] Improved the logs message regarding lambda function with SQL UDFImproved the logs message regarding lambda function with SQL UDF~~ [SPARK-54776][SQL] Improved the logs message regarding lambda function with SQL UDF Dec 20, 2025

Shubhambhusate force-pushed the fix/LAMBDA_FUNCTION_WITH_SQL_UDF branch from 6f50c79 to 5d61f52 Compare December 21, 2025 10:19

allisonwang-db reviewed Dec 22, 2025

View reviewed changes

cloud-fan reviewed Dec 23, 2025

View reviewed changes

Shubhambhusate force-pushed the fix/LAMBDA_FUNCTION_WITH_SQL_UDF branch from 6c2e653 to 1251461 Compare December 23, 2025 10:18

cloud-fan reviewed Jan 6, 2026

View reviewed changes

cloud-fan approved these changes Jan 6, 2026

View reviewed changes

Shubhambhusate force-pushed the fix/LAMBDA_FUNCTION_WITH_SQL_UDF branch from f66e849 to 42677f5 Compare January 7, 2026 10:18

Shubhambhusate added 4 commits January 7, 2026 16:02

[SPARK-54776][SQL] Improve error message for SQL UDF in higher-order …

1b8f550

…functions

[SPARK-54776][SQL] Improve error message for SQL UDF in higher-order …

8aeedc3

…functions removed the input parameter

Retrigger CI workflow

c97e05b

Use checkError to test SQL UDF in higher-order function error

32dd332

Shubhambhusate force-pushed the fix/LAMBDA_FUNCTION_WITH_SQL_UDF branch 3 times, most recently from 7792601 to 123f817 Compare January 7, 2026 15:19

Add missing AnalysisException import to SessionCatalog

daba475

Shubhambhusate force-pushed the fix/LAMBDA_FUNCTION_WITH_SQL_UDF branch from 123f817 to daba475 Compare January 7, 2026 17:53

Shubhambhusate added 2 commits January 8, 2026 01:36

Retrigger CI

e0dad78

Retrigger CI - flaky Docker integration tests

6557dea

cloud-fan closed this in 2fff424 Jan 8, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-54776][SQL] Improved the logs message regarding lambda function with SQL UDF #53542

[SPARK-54776][SQL] Improved the logs message regarding lambda function with SQL UDF #53542

Uh oh!

Shubhambhusate commented Dec 19, 2025 •

edited

Loading

Uh oh!

allisonwang-db left a comment

Uh oh!

allisonwang-db Dec 22, 2025

Uh oh!

Shubhambhusate Dec 23, 2025

Uh oh!

cloud-fan Dec 23, 2025

Uh oh!

Shubhambhusate Dec 23, 2025

Uh oh!

cloud-fan Jan 6, 2026

Uh oh!

Shubhambhusate Jan 7, 2026 •

edited

Loading

Uh oh!

cloud-fan commented Jan 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[SPARK-54776][SQL] Improved the logs message regarding lambda function with SQL UDF #53542

[SPARK-54776][SQL] Improved the logs message regarding lambda function with SQL UDF #53542

Uh oh!

Conversation

Shubhambhusate commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

allisonwang-db left a comment

Choose a reason for hiding this comment

Uh oh!

allisonwang-db Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

Shubhambhusate Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

cloud-fan Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

Shubhambhusate Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

cloud-fan Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

Shubhambhusate Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented Jan 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Shubhambhusate commented Dec 19, 2025 •

edited

Loading

Shubhambhusate Jan 7, 2026 •

edited

Loading