[SPARK-53573][SQL] Allow coalescing string literals everywhere #52638

srielau · 2025-10-16T18:30:50Z

What changes were proposed in this pull request?

As part of generalizing the usage of parameter markers, we want to also generalize the ability to chain string literals in more places than only expressions.
This allows for fuctionality such as: spark.sql("CREATE TABLE ... LOCATION :root '/subdir'", Map("root" -> "/data")
to turn into:
CREATE TABLE ... LOCATION '/data/subdir'

String coalescing works now nearly everywhere with two cave-ats:

typed-literals
TIMESTAMP :param collides with json path jsonstr:path.
I think tat's acceptable as most typed literals take strings and allow :: or CAST casting. And of course one can always pass a typed parameter marker
property lists allow equal signs to be optional. That means we don't know where the split key and value. OPTIONS however already allowed constants which included coalescing. So precedence clarifies that, in teh absesne of an equal sign only the first string literal is the key. All subsequent literals compose the value.

Why are the changes needed?

Greatly increases the power of parameter markers without having to build heavy expressions and refactoring code

Does this PR introduce any user-facing change?

Yes, it is an additive feature

How was this patch tested?

Written a new set of tests StringLiteralCoalescingSuite

Was this patch authored or co-authored using generative AI tooling?

Claude Sonnet 4.5

dtenedor

Thanks for working on this, it will make the SQL syntax better!

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala

sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala

sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4

dtenedor

Thanks for working on this and making our SQL language better!

dtenedor · 2025-10-20T22:30:18Z

sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala

+
+    // Check if any of the tokens are R-strings.
+    val hasRString = tokens.exists { token =>
+      val text = token.getText


This looks duplicated with L187-190 and L207-210 below, let's dedup into a helper? Bonus point for adding a bunch of unit tests specifically for the helper with many different input strings (AI can maybe help generate a bunch of test cases).

dtenedor · 2025-10-20T22:31:06Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala

+        case p: ExpressionPropertyWithKeyNoEqualsContext =>
+          val k = visitPropertyKeyOrStringLitNoCoalesce(p.key)
+          val v = Option(p.value).map(expression).getOrElse {
+            operationNotAllowed(s"A value must be specified for the key: $k.", ctx)


this looks duplicated with the above, please dedup?

dtenedor · 2025-10-20T22:33:04Z

sql/core/src/test/scala/org/apache/spark/sql/StringLiteralCoalescingSuite.scala

+  // ========================================================================
+
+  test("string coalescing in LIKE pattern") {
+    checkAnswer(


There are many tests of this format, with

test("category") { checkAnswer( sql("some query"), Row(true) ) }

It seems like we could simplify this a bunch by just putting a big Seq of strings and then doing a .foreach on it and calling checkAnswer(sql(query), Row(true)) on it. We can still port the existing test case names as comments and put them in comments in the Seq instead.

dtenedor · 2025-10-20T22:33:21Z

sql/core/src/test/scala/org/apache/spark/sql/StringLiteralCoalescingSuite.scala

+    withTable("test_table_123", "test_table_456", "other_table") {
+      sql("CREATE TABLE test_table_123 (id INT)")
+      sql("CREATE TABLE test_table_456 (id INT)")
+      sql("CREATE TABLE other_table (id INT)")


do a Seq of the table names and .foreach on it?

dtenedor · 2025-10-20T22:33:35Z

sql/core/src/test/scala/org/apache/spark/sql/StringLiteralCoalescingSuite.scala

+
+      // The pattern is coalesced into 'test_table_*' (regex pattern where * matches any chars)
+      val result = sql("SHOW TABLES LIKE 'test' '_table_' '*'").collect()
+      assert(result.length == 2,


should we checkAnswer on it?

cloud-fan · 2025-10-21T05:56:06Z

sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4

 expressionProperty
-    : key=propertyKey (EQ? value=expression)?
+    : key=propertyKeyOrStringLit EQ value=expression                  #expressionPropertyWithKeyAndEquals
+    | key=propertyKeyOrStringLitNoCoalesce value=expression           #expressionPropertyWithKeyNoEquals


Suggested change

| key=propertyKeyOrStringLitNoCoalesce value=expression #expressionPropertyWithKeyNoEquals

| key=propertyKeyOrStringLitNoCoalesce value=expression? #expressionPropertyWithKeyNoEquals

cloud-fan · 2025-10-21T06:03:03Z

sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala

+      tokens.head
+    } else {
+      // Multiple tokens: create coalesced token
+      createCoalescedStringToken(tokens)


hmm shall we put everything in AstBuilder? Looks weird to combine tokens here.

github-actions bot added the SQL label Oct 16, 2025

dtenedor reviewed Oct 16, 2025

View reviewed changes

srielau added 3 commits October 16, 2025 16:59

Initial solution

b82a19d

Fix escaping

a9e38eb

Add parm support

e7a726b

srielau force-pushed the stringlitplusplus branch from e2aa35b to e7a726b Compare October 17, 2025 01:59

github-actions bot added the DOCS label Oct 17, 2025

srielau added 10 commits October 16, 2025 19:48

r-literals

609b876

Fix PROPERTIES and OPTIONS

de697ad

Fixes around property keys

0475705

Fix SHWO PROPERTIES

1717fed

fixes

1e8b1b2

more fixes

29093ed

Disable for type literals

7b749e8

fixes

c49387f

more fixes

fc4dc5c

Retore md files, restore literals.sql.out

db85a7f

github-actions bot removed the DOCS label Oct 18, 2025

srielau added 2 commits October 18, 2025 14:03

Cleanup

5a34988

address commment

3d8fc41

cloud-fan reviewed Oct 20, 2025

View reviewed changes