DynaMPI

CI Status

---

Library for efficient manager-worker dynamic load distribution using MPI

This project is licensed under the terms of the MIT license.

Usage

For dynamic load distribution of a static number of tasks, we provide the simple API

template <typename ResultT>
std::optional<std::vector<ResultT>> mpi_manager_worker_distribution(
    size_t n_tasks, std::function<ResultT(size_t)> worker_function, MPI_Comm comm = MPI_COMM_WORLD,
    int manager_rank = 0);

Allowing

#include <dynampi/dynampi.hpp>

auto worker_task = [](size_t task) -> size_t { return task * task; };
auto result = dynampi::mpi_manager_worker_distribution<size_t>(4, worker_task);
if (result.has_value())
    assert(result == std::vector<size_t>({0, 1, 4, 9}));

The order of the result is guaranteed to be in order of the task indexes.

It is common for the number of tasks to not be static. It can also be inefficient to form all tasks prior to

template <typename TaskT, typename ResultT>
class MPIDynampicWorkDistributor {

};

Allowing

  using Task = int;
  using Result = std::vector<int>;
  auto worker_task = [](Task task) -> Result {
    return Result{task, task * task, task * task * task};
  };
  {
    dynampi::MPIDynamicWorkDistributor<Task, Result> work_distributer(worker_task);
    if (work_distributer.is_manager()) {
      work_distributer.insert_tasks({1, 2, 3, 4, 5});
      auto results = work_distributer.distribute_tasks();
      EXPECT_EQ(results.size(), 5);
      work_distributer.insert_tasks({6, 7, 8});
      results = work_distributer.distribute_tasks();
      EXPECT_EQ(results.size(), 8);
    }
  }

This allows the manager to begin distributing tasks before all of the tasks have been formed. The manager can also alternate between inserting tasks and receiving results for when task formation is dependent on the results of previous tasks. There are many configuration additional options including work prioritization, custom datatypes, and error handling.

Installation

DynaMPI is a header-only library with dependence only on MPI, so you can simply copy the include folder into your project. Alternatively, DynaMPI can be installed using CMake:

include(FetchContent)
FetchContent_Declare(
    dynampi
    GIT_REPOSITORY https://github.com/ryanstocks00/DynaMPI.git
    GIT_TAG        main
)
FetchContent_MakeAvailable(dynampi)
add_subdirectory(${dynampi_SOURCE_DIR} ${dynampi_BINARY_DIR})