Commit 4eb56bb
[MINOR][DOCS] Document minimum executor memory requirement
### What changes were proposed in this pull request?
This PR adds documentation for the minimum allowable executor memory value (450M) to prevent configuration errors especially when using offHeap memory configurations.
### Why are the changes needed?
Currently, Spark enforces a minimum executor memory of 450M as reserved memory to prevent OOMs, through validation logic in the below snippets
https://github.com/apache/spark/blob/1a802e36ed7698b7108ea7c0000efda1e6d5f949/core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala#L264
https://github.com/apache/spark/blob/1a802e36ed7698b7108ea7c0000efda1e6d5f949/core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala#L466
However, this constraint is not documented in the configuration guide or in the code-level configuration definition. The lack of documentation makes it hard for users to configure off-heap memory (spark.memory.offHeap.size) along with executor memory, leading to validation failures.
### Does this PR introduce _any_ user-facing change?
Only Doc changes
### How was this patch tested?
CI
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #53017 from pratham76/fix-docs.
Authored-by: Pratham Manja <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>1 parent ecaec3d commit 4eb56bb
File tree
2 files changed
+4
-2
lines changed- core/src/main/scala/org/apache/spark/internal/config
- docs
2 files changed
+4
-2
lines changedLines changed: 2 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
407 | 407 | | |
408 | 408 | | |
409 | 409 | | |
410 | | - | |
| 410 | + | |
| 411 | + | |
411 | 412 | | |
412 | 413 | | |
413 | 414 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
282 | 282 | | |
283 | 283 | | |
284 | 284 | | |
285 | | - | |
| 285 | + | |
| 286 | + | |
286 | 287 | | |
287 | 288 | | |
288 | 289 | | |
| |||
0 commit comments