Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] When using insert overwrite, a StackOverflowError exception is thrown #5331

Closed
1 of 2 tasks
hugw777 opened this issue Mar 24, 2025 · 3 comments
Closed
1 of 2 tasks

[Bug] When using insert overwrite, a StackOverflowError exception is thrown #5331

hugw777 opened this issue Mar 24, 2025 · 3 comments
Labels
bug Something isn't working

Comments

@hugw777
Copy link

hugw777 commented Mar 24, 2025

Search before asking

  • I searched in the issues and found nothing similar.

Paimon version

paimon0.9.0

Compute Engine

spark3.3.0
hive3.1.2

Minimal reproduce step

for example
a double partition table
CREATE TABLE IF NOT EXISTS paimon.${dbname}.${tablename}(
column1 string COMMENT 'column1',
column2 string COMMENT 'column2',
column3 string COMMENT 'column3',
column4 string COMMENT 'column4',
p_col_1 string COMMENT 'p_col_1',
p_col_2 string COMMENT 'p_col_2'
) PARTITIONED BY (p_col_1,p_col_2)
COMMENT ''
TBLPROPERTIES (
'primary-key' = 'p_col_1,p_col_2,column1,column2,column3',
'bucket-key' = 'column1,column2,column3',
'bucket' = '1',
'file.format' = 'parquet',
'deletion-vectors.enabled' = 'true',
'metastore.partitioned-table' = 'true',
'scan.mode' = 'compacted-full',
'compaction.optimization-interval' = '3600000',
'target-file-size' = '256mb',
'sink.parallelism' = '20',
'num-sorted-run.stop-trigger' = '2147483647',
'sort-spill-threshold' = '10',
'merge-engine' = 'deduplicate'
);
insert overwrite table paimon.${dbname}.${tablename} partition(cdate,p_col_2)
select
t1.column1
,t1.column2
,t1.column3
,t2.column4
,t2.column5
,t1.cdate
,t1.p_col_2
from (
select cdate,p_col_2,column1,column2,column3
from hive_db.hive_table1
where cdate between '${start_day}' and '${end_day}'
)t1 join(
select cdate,p_col_2,column1,column2,column3,column4,column5
from hive_db.hive_table2
where cdate between '${start_day}' and '${end_day}'
)t2 on
t1.cdate = t2.cdate and t1.p_col_2 = t2.p_col_2 and t1.column1 = t2.column1 and t1.column2 = t2.column2
and t1.column3 = t2.column3;
hive_db.hive_table1:unique key is cdate,p_col_2,column1,column2,column3
hive_db.hive_table2:unique key is cdate,p_col_2,column1,column2,column3,column4
The estimated number of partitions written is around 4500.

What doesn't meet your expectations?

When the last stage is written, an error is reported
java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.StackOverflowError
at org.apache.paimon.spark.commands.PaimonSparkWriter.commit(PaimonSparkWriter.scala:285)
at org.apache.paimon.spark.commands.WriteIntoPaimonTable.run(WriteIntoPaimonTable.scala:64)
at org.apache.paimon.spark.commands.PaimonDynamicPartitionOverwriteCommand.run(PaimonDynamicPartitionOverwriteCommand.scala:69)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:560)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79)
at org.apache.spark.sql.Dataset.(Dataset.scala:220)
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:100)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:622)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:617)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:651)
at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:67)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:384)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1(SparkSQLCLIDriver.scala:504)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1$adapted(SparkSQLCLIDriver.scala:498)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processLine(SparkSQLCLIDriver.scala:498)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:207)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Anything else?

No response

Are you willing to submit a PR?

  • I'm willing to submit a PR!
@hugw777 hugw777 added the bug Something isn't working label Mar 24, 2025
@hugw777 hugw777 closed this as completed Mar 24, 2025
@hugw777 hugw777 reopened this Mar 24, 2025
@hugw777
Copy link
Author

hugw777 commented Mar 24, 2025

spark sql conf

set spark.sql.optimizer.dynamicPartitionPruning.enabled=false;

If I don't set the value of the spark.sql.optimizer.dynamicPartitionPruning.enabled parameter to false, I get Error in query: unresolved operator 'Filter dynamicpruning950'

If I set the value of the spark.sql.optimizer.dynamicPartitionPruning.enabled parameter to false, throw java.lang.StackOverflowError.

java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.StackOverflowError
		at org.apache.paimon.spark.commands.PaimonSparkWriter.commit(PaimonSparkWriter.scala:285)
		at org.apache.paimon.spark.commands.WriteIntoPaimonTable.run(WriteIntoPaimonTable.scala:64)
		at org.apache.paimon.spark.commands.PaimonDynamicPartitionOverwriteCommand.run(PaimonDynamicPartitionOverwriteCommand.scala:69)
		at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
		at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
		at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
		at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
		at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
		at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
		at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
		at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
		at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
		at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
		at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
		at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)
		at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
		at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584)
		at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
		at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
		at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
		at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
		at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
		at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:560)
		at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94)
		at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81)
		at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79)
		at org.apache.spark.sql.Dataset.<init>(Dataset.scala:220)
		at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:100)
		at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
		at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
		at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:622)
		at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
		at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:617)
		at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:651)
		at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:67)
		at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:384)
		at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1(SparkSQLCLIDriver.scala:504)
		at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1$adapted(SparkSQLCLIDriver.scala:498)
		at scala.collection.Iterator.foreach(Iterator.scala:943)
		at scala.collection.Iterator.foreach$(Iterator.scala:943)
		at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
		at scala.collection.IterableLike.foreach(IterableLike.scala:74)
		at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
		at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
		at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processLine(SparkSQLCLIDriver.scala:498)
		at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
		at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:207)
		at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:498)
		at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
		at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
		at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
		at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
		at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
		at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
		at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
		at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
	Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.StackOverflowError
		at org.apache.paimon.utils.ScanParallelExecutor$1.advanceIfNeeded(ScanParallelExecutor.java:85)
		at org.apache.paimon.utils.ScanParallelExecutor$1.hasNext(ScanParallelExecutor.java:60)
		at org.apache.paimon.manifest.FileEntry.mergeEntries(FileEntry.java:126)
		at org.apache.paimon.manifest.FileEntry.mergeEntries(FileEntry.java:112)
		at org.apache.paimon.operation.AbstractFileStoreScan.readAndMergeFileEntries(AbstractFileStoreScan.java:395)
		at org.apache.paimon.operation.AbstractFileStoreScan.doPlan(AbstractFileStoreScan.java:299)
		at org.apache.paimon.operation.AbstractFileStoreScan.plan(AbstractFileStoreScan.java:223)
		at org.apache.paimon.operation.FileStoreCommitImpl.tryOverwrite(FileStoreCommitImpl.java:729)
		at org.apache.paimon.operation.FileStoreCommitImpl.overwrite(FileStoreCommitImpl.java:451)
		at org.apache.paimon.table.sink.TableCommitImpl.commitMultiple(TableCommitImpl.java:225)
		at org.apache.paimon.table.sink.TableCommitImpl.commit(TableCommitImpl.java:200)
		at org.apache.paimon.table.sink.TableCommitImpl.commit(TableCommitImpl.java:179)
		at org.apache.paimon.table.sink.TableCommitImpl.commit(TableCommitImpl.java:163)
		at org.apache.paimon.spark.commands.PaimonSparkWriter.commit(PaimonSparkWriter.scala:283)
		... 59 more
	Caused by: java.util.concurrent.ExecutionException: java.lang.StackOverflowError
		at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
		at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
		at org.apache.paimon.utils.ScanParallelExecutor$1.advanceIfNeeded(ScanParallelExecutor.java:83)
		... 72 more
	Caused by: java.lang.StackOverflowError
		at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
		at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
		at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
		at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
		at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
		at java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:677)
		at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:735)
		at java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:714)
		at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
		at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
		at org.apache.paimon.operation.AbstractFileStoreScan.lambda$readAndMergeFileEntries$6(AbstractFileStoreScan.java:386)
		at org.apache.paimon.utils.ScanParallelExecutor$1.lambda$advanceIfNeeded$0(ScanParallelExecutor.java:81)
		at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
		at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1596)
		at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
		at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067)
		at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703)
		at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)
	Caused by: java.lang.StackOverflowError
		at org.apache.paimon.predicate.Or.test(Or.java:54)
		at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
		at org.apache.paimon.predicate.Or.test(Or.java:55)
		at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
		at org.apache.paimon.predicate.Or.test(Or.java:55)
		at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
		at org.apache.paimon.predicate.Or.test(Or.java:55)
		at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
		at org.apache.paimon.predicate.Or.test(Or.java:55)
		at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
		at org.apache.paimon.predicate.Or.test(Or.java:55)
		at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
		at org.apache.paimon.predicate.Or.test(Or.java:55)
		at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
		at org.apache.paimon.predicate.Or.test(Or.java:55)
		at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
		at org.apache.paimon.predicate.Or.test(Or.java:55)
		at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
		at org.apache.paimon.predicate.Or.test(Or.java:55)
		at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
		at org.apache.paimon.predicate.Or.test(Or.java:55)
		at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
		at org.apache.paimon.predicate.Or.test(Or.java:55)
		at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
		at org.apache.paimon.predicate.Or.test(Or.java:55)
		at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
		at org.apache.paimon.predicate.Or.test(Or.java:55)
		at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
		at org.apache.paimon.predicate.Or.test(Or.java:55)
		at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
		at org.apache.paimon.predicate.Or.test(Or.java:55)
		at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
		at org.apache.paimon.predicate.Or.test(Or.java:55)
		at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
		at org.apache.paimon.predicate.Or.test(Or.java:55)
		at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
		at org.apache.paimon.predicate.Or.test(Or.java:55)
		at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
		at org.apache.paimon.predicate.Or.test(Or.java:55)
		at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
		at org.apache.paimon.predicate.Or.test(Or.java:55)

@Zouxxyy
Copy link
Contributor

Zouxxyy commented Mar 28, 2025

Can you try the latest 1.0.1

@hugw777
Copy link
Author

hugw777 commented Mar 28, 2025

Can you try the latest 1.0.1
I've used 1.0.1
This problem seems to be brought about by Spark 3
When I set the following three parameters, I didn't encounter any problems
set spark.sql.optimizer.dynamicPartitionPruning.enabled=false;
set spark.sql.optimizer.nestedSchemaPruning.enabled=false;
set spark.sql.hive.convertMetastoreParquet=false;

@hugw777 hugw777 closed this as completed Mar 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Zouxxyy @hugw777