Skip to content

Commit 4f10262

Browse files
yeshengmcloud-fan
authored andcommitted
[SPARK-53623][SQL] improve reading large table properties performance
### What changes were proposed in this pull request? The CatalogColumnStat.readLargeTableProp is an O(N) operation. Considering a table can have a lot of table properties, this effectively becomes an O(N^2) operation, which can be very slow for tables with a lot of table properties. This PR improves the algorithmic complexity to O(N) by only constructing the large table properties if numParts exists. ### Why are the changes needed? For fixing a performance issue unintentionally introduced before. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existing unit tests. A previous patch already tested the side effect of this change #52355 ### Was this patch authored or co-authored using generative AI tooling? No Closes #52374 from yeshengm/improve-read-large-prop. Authored-by: Yesheng Ma <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
1 parent db13a38 commit 4f10262

File tree

1 file changed

+8
-13
lines changed
  • sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog

1 file changed

+8
-13
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala

Lines changed: 8 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -765,20 +765,15 @@ object CatalogTable {
765765

766766
def readLargeTableProp(props: Map[String, String], key: String): Option[String] = {
767767
props.get(key).orElse {
768-
if (props.exists { case (mapKey, _) => mapKey.startsWith(key) }) {
769-
props.get(s"$key.numParts") match {
770-
case None => None
771-
case Some(numParts) =>
772-
val parts = (0 until numParts.toInt).map { index =>
773-
val keyPart = s"$key.part.$index"
774-
props.getOrElse(keyPart, {
775-
throw QueryCompilationErrors.insufficientTablePropertyPartError(keyPart, numParts)
776-
})
777-
}
778-
Some(parts.mkString)
768+
// only construct the property from parts if numParts exist,
769+
props.get(s"$key.numParts").map { numParts =>
770+
val parts = (0 until numParts.toInt).map { index =>
771+
val keyPart = s"$key.part.$index"
772+
props.getOrElse(keyPart, {
773+
throw QueryCompilationErrors.insufficientTablePropertyPartError(keyPart, numParts)
774+
})
779775
}
780-
} else {
781-
None
776+
parts.mkString
782777
}
783778
}
784779
}

0 commit comments

Comments
 (0)