Skip to content

Commit 4eb56bb

Browse files
Pratham Manjadongjoon-hyun
authored andcommitted
[MINOR][DOCS] Document minimum executor memory requirement
### What changes were proposed in this pull request? This PR adds documentation for the minimum allowable executor memory value (450M) to prevent configuration errors especially when using offHeap memory configurations. ### Why are the changes needed? Currently, Spark enforces a minimum executor memory of 450M as reserved memory to prevent OOMs, through validation logic in the below snippets https://github.com/apache/spark/blob/1a802e36ed7698b7108ea7c0000efda1e6d5f949/core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala#L264 https://github.com/apache/spark/blob/1a802e36ed7698b7108ea7c0000efda1e6d5f949/core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala#L466 However, this constraint is not documented in the configuration guide or in the code-level configuration definition. The lack of documentation makes it hard for users to configure off-heap memory (spark.memory.offHeap.size) along with executor memory, leading to validation failures. ### Does this PR introduce _any_ user-facing change? Only Doc changes ### How was this patch tested? CI ### Was this patch authored or co-authored using generative AI tooling? No Closes #53017 from pratham76/fix-docs. Authored-by: Pratham Manja <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent ecaec3d commit 4eb56bb

File tree

2 files changed

+4
-2
lines changed

2 files changed

+4
-2
lines changed

core/src/main/scala/org/apache/spark/internal/config/package.scala

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -407,7 +407,8 @@ package object config {
407407
.createWithDefault(1)
408408

409409
private[spark] val EXECUTOR_MEMORY = ConfigBuilder(SparkLauncher.EXECUTOR_MEMORY)
410-
.doc("Amount of memory to use per executor process, in MiB unless otherwise specified.")
410+
.doc("Amount of memory to use per executor process, in MiB unless otherwise specified, " +
411+
"and with minimum value 450m")
411412
.version("0.7.0")
412413
.bytesConf(ByteUnit.MiB)
413414
.createWithDefaultString("1g")

docs/configuration.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -282,7 +282,8 @@ of the most common options to set are:
282282
<td>1g</td>
283283
<td>
284284
Amount of memory to use per executor process, in the same format as JVM memory strings with
285-
a size unit suffix ("k", "m", "g" or "t") (e.g. <code>512m</code>, <code>2g</code>).
285+
a size unit suffix ("k", "m", "g" or "t") (e.g. <code>512m</code>, <code>2g</code>),
286+
and with minimum value <code>450m</code>.
286287
</td>
287288
<td>0.7.0</td>
288289
</tr>

0 commit comments

Comments
 (0)