Skip to content

Margo (Mercury) Data Transfer Layer Implementation #138

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
JaeseungYeom merged 62 commits into from
Jun 11, 2025
Merged

Margo (Mercury) Data Transfer Layer Implementation #138

JaeseungYeom merged 62 commits into from
Jun 11, 2025

Conversation

@wangvsa
Copy link
Collaborator

@wangvsa wangvsa commented May 29, 2025 •
edited
Loading

Will add more details tomorrow.

This is a big PR with many changes.

  1. Implemented the Margo (mercury) DTL. Margo supports CXI driver, libfabric and UCX. We can use "ofi+verbs" for Infiniband, "ofi+cxi" for Slingshot, and "ofi+tcp" for everything else.
  2. Tested on Corona, Tioga, and Tuolumne.
  3. Fixed many minor bugs.
  4. Changed many debug outputs fromDYAD_LOG_INFO level to DYAD_LOG_DEBUG.
  5. Changed some CMAKE options to make it more intuitive and user-friendly.
  6. Included dyad executable. So users can simply do dyad start and dyad stop to start/stop the DYAD service.

wangvsa added 22 commits March 3, 2025 19:45
@wangvsa
Create a service folder and Move flux module code there
4353315 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Change lib to lib64
ad99b9a 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Implement margo dtl
b95671c
@wangvsa
Fix conflicts
2430d03
@wangvsa
Remove debug output
e74ee24
@wangvsa
Add developer comment
73b0f68
@wangvsa
Bug fix
65f2c92 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Remove flux_open call during finalization
ed6402a 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Set ENABLE_UCX_DATA_RMA to OFF by default
bdcc64e 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Delete the old test code
a685325 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Fix typo
a1743f8 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Use int64_t instead of int32_t for margo buffer size
595347f 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Clean code
f0e96f6 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Add MARGO in python binding
3ad3b3a 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Add cmake comments
378f28a 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Fix cmake debug settings
c204367 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Fix excessive debug outputs
1b230d1 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Fix format issues in dyad util
21f8c9e 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Bug fix
27f9e58 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Fix debug formats
14b4d24 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Remove the example margo service
bdcb8e0 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Rename margo rpc parameters
88564aa 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Copy link
Collaborator Author

wangvsa commented May 30, 2025 •
edited
Loading

TODO:

  1. PUSH vs PULL?
  2. Runtime network layer (cxi, tcp, ucx, verbs) selection.
  3. Debug information should include client/service information (e.g., ranks, node id)
  4. Margo-based service: KVS and RPC (do PRC first)

@JaeseungYeom
Copy link
Contributor

JaeseungYeom commented Jun 2, 2025

Can you take a look at the test errors?

wangvsa added 29 commits June 11, 2025 10:42
@wangvsa
Create a service folder and Move flux module code there
624a3af 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Implement margo dtl
b524487
@wangvsa
Remove debug output
12ee406
@wangvsa
Add developer comment
50538b1
@wangvsa
Bug fix
98d9c55 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Remove flux_open call during finalization
d3fceff 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Set ENABLE_UCX_DATA_RMA to OFF by default
2cd3112 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Delete the old test code
40aa04a 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Fix typo
edc9108 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Use int64_t instead of int32_t for margo buffer size
2923e47 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Clean code
34d72b5 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Add MARGO in python binding
5bd81d4 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Add cmake comments
72cd26a 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Fix cmake debug settings
63287fc 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Fix excessive debug outputs
f3b0590 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Fix format issues in dyad util
3ba78d9 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Bug fix
a5913fa 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Fix debug formats
96fcbc0 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Remove the example margo service
4b285f0 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Rename margo rpc parameters
d0611ea 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Perform clang-formating
19f0cab 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Update format-check.yml to use clang-format-18
d4ca9ef
@wangvsa
Update format-check.yml
7777096
@wangvsa
Update clang-format version
4b68785 
Signed-off-by: Chen Wang <[email protected]>
@wangvsa
Merge branch 'margo_dtl' of https://github.com/flux-framework/dyad in…
ca1cc28 
…to margo_dtl
@wangvsa
Include mochi-margo as dependency
f348351
@wangvsa
Fix typo
7113064
@wangvsa
Add enable margo to compile flags
f7df573
@wangvsa
Update compile_test.yaml
20b76a0
@JaeseungYeom JaeseungYeom merged commit 1e3b48d into main Jun 11, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JaeseungYeom JaeseungYeom Awaiting requested review from JaeseungYeom

Assignees

@wangvsa wangvsa

Labels

enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@wangvsa @JaeseungYeom