Add BlockIdManagerSelector #3560

matthewc2003 · 2024-07-31T17:14:11Z

Description

Adds the BlockIdManagerSelector to the list of available manager selectors. This selector returns a sorted list of managers by their block id, from greatest (newest) to least (oldest). Will be used to more effectively distribute tasks. Eventually, this selector will be used in conjunction with some method of sorting the task queue based on runtime to a) ensure tasks won't get assigned to managers that have a shorter remaining wall time than that of the task's execution time and b) put tasks with similar execution times on the same block, thereby resulting in a higher likelihood that the block completely empties of tasks, allowing the scale down logic to kick in.

Changed Behavior

Users can now choose the BlockIdManagerSelector() in the HTEX config.

Fixes

Work as part of #3323

Type of change

New feature

benclifford · 2024-07-31T17:15:37Z

you might elaborate on "Will be used to more effectively distribute tasks."

benclifford · 2024-08-06T10:10:50Z

switching this to draft status because @matthewc2003 is working on gathering evidence on when this does and does not help.

matthewc2003 · 2024-08-21T18:44:54Z

As a result of some experimentation I've found the following:

BlockID manager selector helps with workloads that see a varying amount of tasks over time. Example: Cholesky factorization workload run through the TaPS benchmarking suite
Random Manager Selector:

Block Id Manager Selector:

We see new blocks are prioritized with the blockID manager selector, when used with 'htex_auto_scaling', results in compute cost savings.
Doesn't really work with bag-of-tasks workloads. When all the tasks are put into the queue upfront, all blocks operate at near full utilization for the majority of the workload, which task goes where doesn't really matter.

benclifford · 2024-08-22T06:34:57Z

some documentation ideas:

the same summary you posted above, put into a docstring for the block id manager selector, and then add to docs/reference.rst (perhaps in a new section)

add a description of manager_selector to the HighThroughputExecutor selector, with a link to the docstring on block id manager selector.

benclifford · 2024-09-06T12:46:31Z

parsl/executors/high_throughput/manager_selector.py

@@ -19,6 +19,14 @@ def sort_managers(self, ready_managers: Dict[bytes, ManagerRecord], manager_list

 class RandomManagerSelector(ManagerSelector):

+    """Returns a shuffled list of interesting_managers
+
+    Maintains the behavior of the original interchange. Works well


the docs should primarily be about the version now, rather than about changes since some other version the user isn't using, so you could get rid of this first sentence.

a related thing to express would be that this is (I think) the default - see how some other docstrings on HighThroughputExecutor talk about defaults

... but the commit message that this gets merged under (which hopefully will come from the description text of this PR) is the right place to describe how the new code is different/similar to the old code - so describing how the default behaviour now is the same or different to the previous behaviour before this is merged is relevant there: the audience for that is people who want to know what has changed in parsl, vs audience for the user guide is what is stuff like now

Reworded to identify the random manager selector as the default

benclifford · 2024-09-06T12:58:25Z

docs/reference.rst

@@ -78,6 +78,16 @@ Executors
    parsl.executors.FluxExecutor
    parsl.executors.radical.RadicalPilotExecutor

+Executors Miscellaneous


you could call this Manager selectors

benclifford · 2024-09-06T14:45:09Z

It would be good to get some testing in here for the newly introduced code.

There are a few things you could do:

make a test that invokces BlockIdManagerSelector.sort_managers with some example inputs and check that it sorts them the right way

launch a DFK with the BlockIdManagerSelector configured and check that a simple no-op task runs.

level 2 of that: run a configuration with 2 blocks, no scaling, run a few tasks and check that they all end up in the 2nd block and never in the first block: you should see that in the PARSL_WORKER_BLOCK_ID environment variable inside a task

@khk-globus might have some other ideas too around this space

benclifford · 2024-10-14T09:35:31Z

parsl/tests/test_htex/test_manager_selector_by_block.py

+@pytest.mark.local
+def test_block_id_selection(try_assert):
+    blockids = []
+    try_assert(lambda: (time.sleep(1) or get_worker_pid().result() == "1"), timeout_ms=20000)


whats the sleep(1) for? in test cases that's usually something that should be waited for explicitly, or is something broken that the author hopes goes away after a while.

I think maybe you should be checking that both blocks are available when this test runs - eg lambda: len(htex.connected_managers()) == 2 - so that there is the opportunity for buggy behaviour to place blocks onto block 0, and then the opportunity for the test to detect that.

benclifford · 2024-10-14T09:38:04Z

parsl/executors/high_throughput/executor.py

+    manager_selector: ManagerSelector
+        Determines what strategy the interchange uses to select managers during task distribution.
+        See API reference under "Manager Selectors" regarding the various manager selectors.
+        Default: RandomManagerSelector()


Do this and you'll get a hyperlink to the documentation for RandomManagerSelector:

$ git diff diff --git a/parsl/executors/high_throughput/executor.py b/parsl/executors/high_throughput/executor.py index 09e634f41d..92a256ae54 100644 --- a/parsl/executors/high_throughput/executor.py +++ b/parsl/executors/high_throughput/executor.py @@ -149,7 +149,7 @@ GENERAL_HTEX_PARAM_DOCS = """provider : :class:`~parsl.providers.base.ExecutionP manager_selector: ManagerSelector Determines what strategy the interchange uses to select managers during task distribution. See API reference under "Manager Selectors" regarding the various manager selectors. - Default: RandomManagerSelector() + Default: `RandomManagerSelector` """ # Documentation for params used by both HTEx and MPIEx

benclifford · 2024-10-14T09:39:20Z

parsl/executors/high_throughput/manager_selector.py

+    in distributing workloads equally across all availble compute
+    resources. Is not effective in conjunction with elastic scaling
+    behavior as it leads to wasted resource consumption.
+    """


maybe put subjects in these sentences such as it - this isn't charged by the word.

benclifford · 2024-10-14T09:40:41Z

parsl/executors/high_throughput/manager_selector.py

+
+    Observations:
+    1. BlockID manager selector helps with workloads that see a varying
+    amount of tasks over time. We see new blocks are prioritized with the


avoid academic-we in user facing documentation: just write "New blocks are prioritizied"

benclifford · 2024-10-14T09:41:20Z

parsl/tests/test_htex/test_manager_selector_by_block.py

+    for i in range(10):
+        future = get_worker_pid()
+        blockids.append(future.result())
+        print(future.result())


don't print in tests - the output is either passing or failing.

benclifford · 2024-10-14T09:41:47Z

parsl/executors/high_throughput/manager_selector.py

+
+class BlockIdManagerSelector(ManagerSelector):
+
+    """Returns a interesting_managers list sorted by block ID


benclifford · 2024-10-15T14:17:13Z

I removed most of your PR description from the commit message, rather than harrassing @matthewc2003 to go rewrite it - it was mostly about stuff the isn't implemented, rather than stuff that this PR implements.

matthewc2003 added 2 commits July 31, 2024 17:00

add BlockIdManagerSelector to ManagerSelector

d2a1988

clean up extra comments

f4e04d8

benclifford marked this pull request as draft August 6, 2024 10:10

Merge branch 'master' into add_manager_selector_by_block

1f16dcb

matthewc2003 marked this pull request as ready for review August 21, 2024 20:07

matthewc2003 added 2 commits September 5, 2024 19:13

Merge branch 'master' into add_manager_selector_by_block

e914493

add docs

d745670

benclifford reviewed Sep 6, 2024

View reviewed changes

fix docs

705cf99

benclifford reviewed Sep 6, 2024

View reviewed changes

matthewc2003 added 3 commits September 6, 2024 13:06

re:fix docs

85555fa

re:re:fix docs

e5456bf

Merge branch 'master' into add_manager_selector_by_block

85ce33a

matthewc2003 added 7 commits October 10, 2024 23:25

Merge branch 'master' into add_manager_selector_by_block

f8dde30

add block_id unit test

5a2a4fe

fix unit test

e5af4c8

add e2e test

6ebcd5b

fix e2e test

e44262f

fix format error

2cda101

fix unit test

0b03b46

matthewc2003 requested a review from benclifford October 13, 2024 00:11

benclifford reviewed Oct 14, 2024

View reviewed changes

matthewc2003 and others added 4 commits October 14, 2024 17:19

minor fixes

734c788

reworked e2e test

f876939

Merge branch 'master' into add_manager_selector_by_block

d096457

Merge branch 'master' into add_manager_selector_by_block

2bdbd98

benclifford approved these changes Oct 15, 2024

View reviewed changes

benclifford merged commit ac62e04 into Parsl:master Oct 15, 2024
7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add BlockIdManagerSelector #3560

Add BlockIdManagerSelector #3560

matthewc2003 commented Jul 31, 2024 •

edited

Loading

benclifford commented Jul 31, 2024

benclifford commented Aug 6, 2024

matthewc2003 commented Aug 21, 2024

benclifford commented Aug 22, 2024

benclifford Sep 6, 2024

benclifford Sep 6, 2024

matthewc2003 Sep 6, 2024

benclifford Sep 6, 2024

matthewc2003 Sep 6, 2024

benclifford commented Sep 6, 2024

benclifford Oct 14, 2024

benclifford Oct 14, 2024

benclifford Oct 14, 2024

benclifford Oct 14, 2024

benclifford Oct 14, 2024

benclifford Oct 14, 2024

benclifford Oct 14, 2024

benclifford commented Oct 15, 2024


		class BlockIdManagerSelector(ManagerSelector):

		"""Returns a interesting_managers list sorted by block ID

Add BlockIdManagerSelector #3560

Add BlockIdManagerSelector #3560

Conversation

matthewc2003 commented Jul 31, 2024 • edited Loading

Description

Changed Behavior

Fixes

Type of change

benclifford commented Jul 31, 2024

benclifford commented Aug 6, 2024

matthewc2003 commented Aug 21, 2024

benclifford commented Aug 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benclifford commented Sep 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benclifford commented Oct 15, 2024

matthewc2003 commented Jul 31, 2024 •

edited

Loading