Feature/split gauge reuse #1543

aroadant · 2025-03-19T16:47:35Z

These modifications will allow the reuse of the redistributed gauge buffers.

QudaGaugeParam.use_split_gauge_bkup is added to determine whether to delete the buffer after use or not. Its default value is 1. The internal flag update_split_gauge is used to determine whether an update of buffer is needed or not.

The initial test with staggered_invert_test.cpp gives the same results with gauge_param.use_split_gauge_bkup = 0 or 1. Also inversions can be done correctly with inv_param.split_grid[i] = 1 after split gauge inversions. Results from various default tests are good with default value use_split_gauge_bkup=0/1.

The test staggered_invert_test is updated with the flag --use_split_gauge_bkup. Results from the following input looks good

staggered_invert_test --dim 6 6 6 6 --gridsize 2 2 2 2 --grid-partition 2 2 2 2
-dslash-type staggered --compute-fat-long false --unit-gauge 0 --tadpole-coeff 1.0 --verbosity-precondition debug --recon 13
--mass 0.02 --solve-type direct-pc --solution-type mat --inv-type cg --matpc even-even
--prec double --prec-sloppy single
--nsrc 64 --nsrc-tile 16 --niter 10000 --tol 1e-100
--inv-multigrid false --mass-normalization mass --fermion-t-boundary periodic --verify 1 --use_split_gauge_bkup 1

…e-reuse

maddyscientist · 2025-04-10T23:03:52Z

@bjoo and @hummingtree can you review this PR? @bjoo is this helpful for Chroma at all?

include/quda.h

…e-reuse

maddyscientist · 2025-04-11T18:19:18Z

Just a note regarding the the CI failing on CSCS at the moment: the Alps machine is being used for Gordon Bell runs at the moment. We expect the CI to be restored again next week.

hummingtree

Thanks @aroadant for this - I have left some comments, and I feel we could use some better documentation for the newly added functions.

hummingtree · 2025-04-23T20:52:29Z

lib/interface_quda.cpp

+  callMultiSrcQuda related gauge split
+  update_split_gauge : -1 delete buffer after usage, 0 split gauge in buf, 1 split gauge not in buf
+  update_split_gauge set to -1 if QudaGaugeParam.use_split_gauge_bkup == QUDA_BOOLEAN_FALSE, 1 if QudaGaugeParam.use_split_gauge_bkup == QUDA_BOOLEAN_TRUE
+  split_grid_bkup will be used to check whether split layout is changed or not
+  update_split_gauge > 0 and split gauge in buf, the original links will swap with *_bkup 
+  */
+int update_split_gauge = -1;
+int split_grid_bkup[QUDA_MAX_DIM];
+quda::GaugeField   *collected_gauge = nullptr;
+quda::GaugeField   *collected_milc_fatlink_field = nullptr;
+quda::GaugeField   *collected_milc_longlink_field = nullptr;
+quda::CloverField *collected_clover = nullptr;
+GaugeBundleBackup* thin_links_bkup = nullptr;
+GaugeBundleBackup* fat_links_bkup  = nullptr;
+GaugeBundleBackup* long_links_bkup = nullptr;
+CloverBundleBackup* clov_bkup = nullptr;
+lat_dim_t X_bkup;
+int is_clover_bkup = -1;
+int is_asqtad_bkup = -1;
+


Better to use Enum to replace the -1/0/1 so it's more readable.

Same for is_clover_bkup/is_asqtad_bkup.

I have removed the unnecessary parameters and changed to Enum.

hummingtree · 2025-04-23T20:53:35Z

lib/interface_quda.cpp

@@ -138,6 +159,7 @@ static TimeProfile profileInvert("invertQuda");

 //!< Profiler for invertMultiSrcQuda
 static TimeProfile profileInvertMultiSrc("invertMultiSrcQuda");
+static TimeProfile profileUpdate_split_gauge("Update_split_gauge");


Let's use CamelCase consistently.

hummingtree · 2025-04-23T20:54:40Z

lib/interface_quda.cpp

@@ -568,6 +590,15 @@ void freeUniqueSloppyGaugeUtility(GaugeField *&precise, GaugeField *&sloppy, Gau
 void freeUniqueGaugeUtility(GaugeField *&precise, GaugeField *&sloppy, GaugeField *&precondition, GaugeField *&refinement,
                            GaugeField *&eigensolver, GaugeField *&extended, bool preserve_precise);

+// Generate or swap to re-distributed gauge links 
+void Update_split_gauge(QudaInvertParam *param, const int is_asqtad, const bool is_clover, CommKey& split_key);


Let's use CamelCase consistently.

hummingtree · 2025-04-23T20:56:04Z

lib/interface_quda.cpp

+void Update_split_gauge(QudaInvertParam *param, const int is_asqtad, const bool is_clover, CommKey& split_key);
+
+// free current gauges if keep_buffer == 0, restore previous gauge links, could be used to swap back to split gauge
+void swapGaugeSplit(const int keep_buffer );


keep_buffer can be a bool? Also I am not too sure about what this function does. Could probably use some more documentations.

hummingtree · 2025-04-23T20:58:04Z

lib/interface_quda.cpp

@@ -3112,6 +3157,237 @@ void loadFatLongGaugeQuda(QudaInvertParam *inv_param, QudaGaugeParam *gauge_para
  }
 }

+void swapGaugeSplit(const int keep_buffer = 0)


Could use some documentation on what this does.

hummingtree · 2025-04-23T21:01:04Z

lib/interface_quda.cpp

+  swapGaugeSplit(0);
+}
+
+void Update_split_gauge(QudaInvertParam *param, const int is_asqtad, const bool is_clover, CommKey& split_key)


Could use some documentation here.

hummingtree · 2025-04-23T21:03:54Z

tests/utils/command_line_params.cpp

@@ -570,6 +571,7 @@ std::shared_ptr<QUDAApp> make_app(std::string app_description, std::string app_n
    "--laplace3D", laplace3D,
    "Restrict laplace operator to omit the t dimension (n=3), or include all dims (n=4) (default 4)");
  quda_app->add_option("--load-gauge", latfile, "Load gauge field \" file \" for the test (requires QIO)");
+  quda_app->add_option("--use_split_gauge_bkup", use_split_gauge_bkup, "Use gauge split buff or not");


Let's do --use-split-gauge-bkup

…e-reuse

aroadant · 2025-05-08T17:10:32Z

@hummingtree Thanks for the suggestions. I have updated the function documentation and made some changes. There is one issue about clover which may need to be resolved. The check of changes of clover is not good enough and will cause trouble if one needs to switch different clover parameters.

quda/lib/interface_quda.cpp

Line 3262 in d1c3eb0

int is_clover_bkup = 0;

hummingtree

@bjoo When you get a chance, can you check and see if you are comfortable with the changes as it changes many of the codes you added. It would be even better if you could see if there is anything that you can use in Chroma.

hummingtree · 2025-05-08T20:09:19Z

include/enum_quda.h

@@ -648,6 +648,13 @@ typedef enum QudaWFlowStepType_s {
  WFLOW_FOURTH_ORDER_STEP_6,
 } QudaWFlowStepType;

+//Used by update_split_gauge
+typedef enum QudaUpdateSpliteGauge_s {


Suggested change

typedef enum QudaUpdateSpliteGauge_s {

typedef enum QudaUpdateSplitGauge_s {

hummingtree · 2025-05-08T20:10:17Z

include/enum_quda.h

@@ -648,6 +648,13 @@ typedef enum QudaWFlowStepType_s {
  WFLOW_FOURTH_ORDER_STEP_6,
 } QudaWFlowStepType;

+//Used by update_split_gauge


Could use a more descriptive comment by explaining what these 3 options mean.

hummingtree · 2025-05-08T20:10:48Z

include/enum_quda.h

+  QUDA_UPDATE_SPLITE_GAUGE_OFF,
+  QUDA_UPDATE_SPLITE_GAUGE_TRUE  = 1,
+  QUDA_UPDATE_SPLITE_GAUGE_FALSE = 0,
+} QudaUpdateSpliteGauge;


Suggested change

QUDA_UPDATE_SPLITE_GAUGE_OFF,

QUDA_UPDATE_SPLITE_GAUGE_TRUE = 1,

QUDA_UPDATE_SPLITE_GAUGE_FALSE = 0,

} QudaUpdateSpliteGauge;

QUDA_UPDATE_SPLIT_GAUGE_OFF,

QUDA_UPDATE_SPLIT_GAUGE_TRUE = 1,

QUDA_UPDATE_SPLIT_GAUGE_FALSE = 0,

} QudaUpdateSplitGauge;

hummingtree · 2025-05-08T20:11:47Z

lib/interface_quda.cpp

+/** 
+ * callMultiSrcQuda related gauge split
+ * update_split_gauge : 
+ *   QUDA_UPDATE_SPLITE_GAUGE_OFF delete buffer after usage


Let's use split instead of splite consistently.

hummingtree · 2025-05-08T20:13:34Z

lib/interface_quda.cpp

+ * callMultiSrcQuda related gauge split
+ * update_split_gauge : 
+ *   QUDA_UPDATE_SPLITE_GAUGE_OFF delete buffer after usage
+ *   QUDA_UPDATE_SPLITE_GAUGE_FALSE split gauge in buf


"in buf" probably means reusing the split gauge that has been backed up previously?

hummingtree · 2025-05-08T20:22:37Z

lib/interface_quda.cpp

+
+void freeGaugeSplit()
+{
+  // swap current gauge with the splitted ones if splits are not null


Suggested change

// swap current gauge with the splitted ones if splits are not null

// swap current gauges with the split ones if split ones are not null

hummingtree · 2025-05-08T20:24:14Z

lib/interface_quda.cpp

+  // swap current gauge with the splitted ones if splits are not null
+  swapGaugeSplit(true);
+
+  // free the splitted gauge and swap to loaded gauge, if splits are not null


Suggested change

// free the splitted gauge and swap to loaded gauge, if splits are not null

// free the split gauges and swap the split gauges with the buffered gauges, if split gauges are not null

hummingtree · 2025-05-08T20:26:24Z

lib/interface_quda.cpp

-      gaugeLongRefinement = long_links_bkup.refinement;
-      gaugeLongEigensolver = long_links_bkup.eigensolver;
-      gaugeLongExtended = long_links_bkup.extended;
+    // swap back to original links, detete split gauge if update_split_gauge == QUDA_UPDATE_SPLITE_GAUGE_OFF


Suggested change

// swap back to original links, detete split gauge if update_split_gauge == QUDA_UPDATE_SPLITE_GAUGE_OFF

// switch back to the original links, detete split gauge if update_split_gauge == QUDA_UPDATE_SPLITE_GAUGE_OFF

hummingtree · 2025-05-08T20:27:32Z

include/quda.h

@@ -70,6 +70,7 @@ extern "C" {
    int staple_pad;   /**< Used by link fattening */
    int llfat_ga_pad; /**< Used by link fattening */
    int mom_ga_pad;   /**< Used by the gauge and fermion forces */
+    QudaBoolean use_split_gauge_bkup; /**< Used by gauge split buffers (default=true keep split gauge after usage)*/


We can use bool directly - C99 supports boolean types now.

hummingtree · 2025-05-08T20:34:06Z

lib/interface_quda.cpp

+ *   QUDA_UPDATE_SPLITE_GAUGE_TRUE  split gauge not in buf, need to call split
+ * update_split_gauge set to QUDA_UPDATE_SPLITE_GAUGE_OFF if QudaGaugeParam.use_split_gauge_bkup == QUDA_BOOLEAN_FALSE, QUDA_UPDATE_SPLITE_GAUGE_TRUE if QudaGaugeParam.use_split_gauge_bkup == QUDA_BOOLEAN_TRUE
+ * update_split_gauge != QUDA_UPDATE_SPLITE_GAUGE_OFF and split gauge in buf, the original links will swap with *_bkup 
+ * split_grid_bkup will be used to check whether split layout is changed or not
+ * change in gauge precsions will need loadGaugeQuda which results in re-distribute
+ * assumed no change in clover parameters


I modified the comment here to make it clearer - but please correct me if I say anything wrong.

Suggested change

* QUDA_UPDATE_SPLITE_GAUGE_TRUE split gauge not in buf, need to call split

* update_split_gauge set to QUDA_UPDATE_SPLITE_GAUGE_OFF if QudaGaugeParam.use_split_gauge_bkup == QUDA_BOOLEAN_FALSE, QUDA_UPDATE_SPLITE_GAUGE_TRUE if QudaGaugeParam.use_split_gauge_bkup == QUDA_BOOLEAN_TRUE

* update_split_gauge != QUDA_UPDATE_SPLITE_GAUGE_OFF and split gauge in buf, the original links will swap with *_bkup

* split_grid_bkup will be used to check whether split layout is changed or not

* change in gauge precsions will need loadGaugeQuda which results in re-distribute

* assumed no change in clover parameters

* - QUDA_UPDATE_SPLITE_GAUGE_TRUE: the input gauge fields will be split and the buffered (split) gauges will be updated accordingly;

* - QUDA_UPDATE_SPLITE_GAUGE_FALSE: the input gauge fields will not be split and the buffered (split) gauges will be used for split grid solves;

* - QUDA_UPDATE_SPLITE_GAUGE_OFF: nothing will be done.

* split_grid_bkup will be used to check whether split layout is changed or not

* change in gauge precsions will need loadGaugeQuda which results in re-distribute

* assumed no change in clover parameters

aroadant · 2025-05-08T21:00:48Z

@hummingtree Thanks! It's very helpful to make the comments to be clear!

aroadant added 4 commits March 17, 2025 15:24

Update interface_quda.cpp to have split gauge reuse

c31de79

Update quda.h

cbcb26b

Update check_params.h

ddee0e3

Update split gauge buffer reuse

c1b2583

aroadant requested a review from a team as a code owner March 19, 2025 16:47

aroadant added 11 commits March 19, 2025 11:03

Merge remote-tracking branch 'origin/develop' into feature/split-gaug…

4d7145c

…e-reuse

update inversion parameter

382c465

remove length log

309b44b

fix free buf bugs

3c965ee

test default : reuse buf

75f9fdb

reset default to not using buf

86c4fa1

try again default use buf

80d005a

re-try server

d42d9b7

free buf when not needed and default of using buf

de9f4a0

change test default to use buf

6516b7b

Merge remote-tracking branch 'origin/develop' into feature/split-gaug…

4a68f68

…e-reuse

maddyscientist reviewed Apr 10, 2025

View reviewed changes

include/quda.h Outdated Show resolved Hide resolved

aroadant added 3 commits April 11, 2025 01:40

Merge remote-tracking branch 'origin/develop' into feature/split-gaug…

d847d9e

…e-reuse

change to Quda Bool

354debb

retry tests

24b0e14

aroadant added 2 commits April 15, 2025 06:25

rerun tests, default test reuse 0

fca0a11

rerun tests, default test reuse 1

e94d273

hummingtree requested changes Apr 23, 2025

View reviewed changes

aroadant added 4 commits May 8, 2025 06:59

Merge remote-tracking branch 'origin/develop' into feature/split-gaug…

01f3ac5

…e-reuse

update split gauge parameters

938eb05

change keep_buffer into bool

471626c

test default use buff

d1c3eb0

fix loadCloverQuda with split gauge reuse

c9dc15e

hummingtree reviewed May 8, 2025

View reviewed changes

update type and more clear comments

0017931

aroadant requested a review from hummingtree May 9, 2025 13:49

	typedef enum QudaUpdateSpliteGauge_s {
	typedef enum QudaUpdateSplitGauge_s {

	// swap current gauge with the splitted ones if splits are not null
	// swap current gauges with the split ones if split ones are not null

	// free the splitted gauge and swap to loaded gauge, if splits are not null
	// free the split gauges and swap the split gauges with the buffered gauges, if split gauges are not null

	// swap back to original links, detete split gauge if update_split_gauge == QUDA_UPDATE_SPLITE_GAUGE_OFF
	// switch back to the original links, detete split gauge if update_split_gauge == QUDA_UPDATE_SPLITE_GAUGE_OFF

Feature/split gauge reuse #1543

Are you sure you want to change the base?

Feature/split gauge reuse #1543

Uh oh!

Conversation

aroadant commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maddyscientist commented Apr 10, 2025

Uh oh!

Uh oh!

maddyscientist commented Apr 11, 2025

Uh oh!

hummingtree left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aroadant commented May 8, 2025

Uh oh!

hummingtree left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aroadant commented May 8, 2025

Uh oh!

Uh oh!

aroadant commented Mar 19, 2025 •

edited

Loading