Feature/adjoint flow

[rkarur]

Implemented the adjoint fermion flow algorithm (both the serial and hierarchical versions detailed in Appendix E of https://arxiv.org/pdf/1302.5246) and wrote a test that tests both the adjoint and regular (forward) gradient flow, in which the equivalence of the serial and hierarchical methods is demonstrated, as well as a numerical verification that the forward and adjoint fermionic flows are indeed adjoint methods of one another (Identity E.4 in https://arxiv.org/pdf/1302.5246)

[rkarur]

refined errors tests

[cpviolator]

@maddyscientist This looks pretty goof from my perspective, if you give it the once over for @rkarur we can merge and I can refactor into classes as a separate PR.

[weinbe2]

The test executable ./su3_fermion_test doesn't run out of the box with no arguments passed in:

> mpirun -np 1 ./su3_fermion_test
[...]
WARNING: Using device memory pool allocator
WARNING: Using pinned memory pool allocator
cublasCreated successfully
ERROR: Smear type 1 not supported - only Wilson or Symanzik supported (rank 0, host ipp1-2318.nvidia.com, su3_fermion_test.cpp:147 in int main(int, char**)())
       last kernel called was (name=cudaMemcpyDeviceToHost,volume=bytes=47775744,aux=copy,gauge_field.cpp,1024)
[...]

Smear type "1" is stout smearing, which is the default specified in the command_line_params files. Should we change the default there, or manually override it in ./su3_fermion_test?

[weinbe2]

This also fails an error check after manually specifying wilson-type smearing:

> mpirun -np 1 ./su3_fermion_test --su3-smear-type wilson --verbosity verbose
[...]
Fractional error of (<check,adj_check> - <check,fwd_check>.conj()) = 2.82863e-14
adjoint safe/hier match precision passed!
ERROR: fractional error precision failed
 (rank 0, host ipp1-2318.nvidia.com, su3_fermion_test.cpp:279 in int main(int, char**)())
       last kernel called was (name=cudaMemcpyDefault,volume=bytes=63700992,aux=copy,color_spinor_field.cpp,498)

Symanzik has the same issue.

[weinbe2]

Further, it looks like setting the precision with --prec double vs --prec single doesn't do anything---it seems like this is intentional?

Lines 75 to 80 of the test executable:

  QudaGaugeParam gauge_param = newQudaGaugeParam();
  if (prec_sloppy == QUDA_INVALID_PRECISION) {
    prec = QUDA_DOUBLE_PRECISION;
    prec_sloppy = prec;
  }
  if (link_recon_sloppy == QUDA_RECONSTRUCT_INVALID) link_recon_sloppy = link_recon;

If there's a reason for this that's fine, but there should be a warningQuda noting so, and maybe some other warning or error deeper down if there's a fundamental concern about running this in single precision.

[maddyscientist]

Further, it looks like setting the precision with --prec double vs --prec single doesn't do anything---it seems like this is intentional?

Lines 75 to 80 of the test executable:

  QudaGaugeParam gauge_param = newQudaGaugeParam();
  if (prec_sloppy == QUDA_INVALID_PRECISION) {
    prec = QUDA_DOUBLE_PRECISION;
    prec_sloppy = prec;
  }
  if (link_recon_sloppy == QUDA_RECONSTRUCT_INVALID) link_recon_sloppy = link_recon;

If there's a reason for this that's fine, but there should be a warningQuda noting so, and maybe some other warning or error deeper down if there's a fundamental concern about running this in single precision.

This was a bug: fixed in my cleanup.

[maddyscientist]

This also fails an error check after manually specifying wilson-type smearing:

> mpirun -np 1 ./su3_fermion_test --su3-smear-type wilson --verbosity verbose
[...]
Fractional error of (<check,adj_check> - <check,fwd_check>.conj()) = 2.82863e-14
adjoint safe/hier match precision passed!
ERROR: fractional error precision failed
 (rank 0, host ipp1-2318.nvidia.com, su3_fermion_test.cpp:279 in int main(int, char**)())
       last kernel called was (name=cudaMemcpyDefault,volume=bytes=63700992,aux=copy,color_spinor_field.cpp,498)

Symanzik has the same issue.

This was due to requesting too tight a tolerance in double precision. Fixed.

[maddyscientist]

The test executable ./su3_fermion_test doesn't run out of the box with no arguments passed in:

> mpirun -np 1 ./su3_fermion_test
[...]
WARNING: Using device memory pool allocator
WARNING: Using pinned memory pool allocator
cublasCreated successfully
ERROR: Smear type 1 not supported - only Wilson or Symanzik supported (rank 0, host ipp1-2318.nvidia.com, su3_fermion_test.cpp:147 in int main(int, char**)())
       last kernel called was (name=cudaMemcpyDeviceToHost,volume=bytes=47775744,aux=copy,gauge_field.cpp,1024)
[...]

Smear type "1" is stout smearing, which is the default specified in the command_line_params files. Should we change the default there, or manually override it in ./su3_fermion_test?

Fixed in my cleanup: the default smear type is Stout (which is inherited from pre-existing state). If the user doesn't specify, then Wilson-flow is now set (and a message printed to this effect).

[maddyscientist]

This is great addition @rkarur, thank you.

I have cleaned up the test a bit, and it now uses google test when the --enable-testing true flag is passed to the exe. Also, I've added the test to the ctest list, so it will be run in our CI. Can you check what I've done to make sure I've not done anything untoward?

I'm approving this PR now, but before we merge we should wait for @weinbe2's sign off.

[weinbe2]

This all looks good to me. Thanks @rkarur and @maddyscientist !