[SPARK-54971] Add WITH SCHEMA EVOLUTION syntax for SQL INSERT

[longvu-db]

What changes were proposed in this pull request?

Similar to the MERGE WITH SCHEMA EVOLUTION PR, this PR introduces a syntax WITH SCHEMA EVOLUTION to the SQL INSERT command. Since this syntax is not fully implemented for any table formats yet, users will receive an exception if they try to use it.

When WITH SCHEMA EVOLUTION is specified, schema evolution-related features must be turned on for this single statement and only in this statement.

In this PR, Spark is only responsible for recognizing the existence or absence of the syntax WITH SCHEMA EVOLUTION, and the recognition info is passed down from the Analyzer. When WITH SCHEMA EVOLUTION is detected, Spark sets the mergeSchema write option to true in the respective V2 Insert Command nodes.

Data sources must respect the syntax and give appropriate reactions: Turn on features that are categorised as "schema evolution" when the WITH SCHEMA EVOLUTION Syntax exists.

Why are the changes needed?

This intuitive SQL Syntax allows the user to specify Automatic Schema Evolution for a specific INSERT operation.

Some users would like Schema Evolution for DML commands like MERGE, INSERT,... where the schema between the table and query relations can mismatch.

Does this PR introduce any user-facing change?

Yes, Introducing the SQL Syntax WITH SCHEMA EVOLUTION to SQL INSERT.

How was this patch tested?

Added UTs.

Was this patch authored or co-authored using generative AI tooling?

No.

[github-actions]

JIRA Issue Information

=== Improvement SPARK-54971 ===
Summary: Recognizing the existence of the SQL Syntax WITH SCHEMA EVOLUTION for SQL INSERT statements in the Parser
Assignee: None
Status: Open
Affected: ["4.2.0"]

---

This comment was automatically generated by GitHub Actions

[szehon-ho]

I was thinking it can be interesting to have Spark optionally call alterTable , if the V2 data source has TableCapability.AUTOMATIC_SCHEMA_EVOLUTION (which we introduced when doing MERGE INTO schema evolution implementation in DSV2). That will ease the burden on the data sources. But it can be a future enhancement.

[longvu-db]

To cover the first

case InsertIntoStatement(l @ LogicalRelationWithTable(_: InsertableRelation, _),

    parts, _, query, overwrite, false, _) if parts.isEmpty =>
    parts, _, query, overwrite, false, _, withSchemaEvolution)
    if parts.isEmpty && !withSchemaEvolution =>

[longvu-db]

To cover the 2nd case

case i @ InsertIntoStatement(l @ LogicalRelationWithTable(t: HadoopFsRelation, table),

    parts, _, query, overwrite, _, _, withSchemaEvolution)
    if query.resolved && !withSchemaEvolution =>