What would you like to happen?
PGBKCVOperation (sdks/python/apache_beam/runners/worker/operations.py) caps its precombine table by key count (max_keys, default 100k / 1M for Count/Mean/min/max/sum), flushing ~10% of keys once the count is hit. The code already flags this as a stopgap:
# TODO(b/36567833): Bound by in-memory size rather than key count.
A fixed key-count cap is a poor memory proxy when accumulators vary in size, so workers can OOM well below max_keys (or flush needlessly when accumulators are tiny). Request: bound the table by estimated in-memory size instead of (or in addition to) key count.
Notes:
b/36567833 is a Google-internal Buganizer ref with no public page; this issue tracks it publicly.- Sibling
PGBKOperation has the same shape (max_size = 10 * 1000 elements).
- Sizing accurately is the hard part —
sys.getsizeof undercounts wrapped/native (e.g. C-extension) accumulators; a coder-based or sampled-RSS estimate is likely needed.
Issue Priority
Priority: 3 (nice-to-have improvement)
Issue Components
What would you like to happen?
PGBKCVOperation(sdks/python/apache_beam/runners/worker/operations.py) caps its precombine table by key count (max_keys, default 100k / 1M for Count/Mean/min/max/sum), flushing ~10% of keys once the count is hit. The code already flags this as a stopgap:# TODO(b/36567833): Bound by in-memory size rather than key count.A fixed key-count cap is a poor memory proxy when accumulators vary in size, so workers can OOM well below
max_keys(or flush needlessly when accumulators are tiny). Request: bound the table by estimated in-memory size instead of (or in addition to) key count.Notes:
b/36567833is a Google-internal Buganizer ref with no public page; this issue tracks it publicly.PGBKOperationhas the same shape (max_size = 10 * 1000elements).sys.getsizeofundercounts wrapped/native (e.g. C-extension) accumulators; a coder-based or sampled-RSS estimate is likely needed.Issue Priority
Priority: 3 (nice-to-have improvement)
Issue Components