Skip to content

Request MPI_THREAD_MULTIPLE when slate is present #4166

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
May 8, 2025
Merged

Request MPI_THREAD_MULTIPLE when slate is present #4166

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
487f562
If user hasn't specified mpi thread type provide default with slate
lindsayad May 6, 2025
4a012a7
Add comment
lindsayad May 7, 2025
6eac173
Add more comment
lindsayad May 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 16 additions & 8 deletions src/base/libmesh.C
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -442,30 +442,38 @@ LibMeshInit::LibMeshInit (int argc, const char * const * argv,
int mpi_thread_request = using_threads;
const auto mpi_thread_type = libMesh::command_line_value("--mpi-thread-type", std::string(""));
if (mpi_thread_type.empty())
check_empty_command_line_value(*command_line, "--mpi-thread-type");
{
check_empty_command_line_value(*command_line, "--mpi-thread-type");
#if defined(PETSC_HAVE_STRUMPACK) && defined(PETSC_HAVE_SLATE)
// For GPU computations, the solver strumpack uses slate which always requests
// MPI_THREAD_MULTIPLE. The solution here is not perfect because the run may never be
// using strumpack, but we believe it's better to force the MPI library to use locks
// whenever it accesses the message queue, that is, when processing any sends and receive,
// than it is to require users to pre-announce/signal what solvers they are using through
// --mpi-thread-type
mpi_thread_request = 3;
#endif
}
else
{
int cli_mpi_thread_request;
if (mpi_thread_type == "single")
{
if (using_threads)
libmesh_error_msg("We are using threads, so we require more mpi thread support "
"than '--mpi-thread-type=single'");
cli_mpi_thread_request = 0;
mpi_thread_request = 0;
}
else if (mpi_thread_type == "funneled")
cli_mpi_thread_request = 1;
mpi_thread_request = 1;
else if (mpi_thread_type == "serialized")
cli_mpi_thread_request = 2;
mpi_thread_request = 2;
else if (mpi_thread_type == "multiple")
cli_mpi_thread_request = 3;
mpi_thread_request = 3;
else
libmesh_error_msg(
"Unsupported mpi thread type '"
<< mpi_thread_type
<< "'. Allowed options are 'single', 'funneled', 'serialized', and 'multiple'");

mpi_thread_request = cli_mpi_thread_request;
}

this->_timpi_init =
Expand Down