test_linear2d_layer fails: incorrect updated weights

[barracuda156]

This is on macOS on powerpc with gfortran 14.2.0.

--->  Testing neural-fortran
Warning: neural-fortran is configured to build for the architecture(s) 'ppc', but did not install any Mach-O files.
Executing:  cd "/opt/local/var/macports/build/_opt_local_ppcports_fortran_neural-fortran/neural-fortran/work/build" && /usr/bin/make test 
Running tests...
/opt/local/bin/ctest --force-new-ctest-process 
Test project /opt/local/var/macports/build/_opt_local_ppcports_fortran_neural-fortran/neural-fortran/work/build
      Start  1: test_input1d_layer
 1/26 Test  #1: test_input1d_layer ...............   Passed    0.02 sec
      Start  2: test_input2d_layer
 2/26 Test  #2: test_input2d_layer ...............   Passed    0.01 sec
      Start  3: test_input3d_layer
 3/26 Test  #3: test_input3d_layer ...............   Passed    0.01 sec
      Start  4: test_dropout_layer
 4/26 Test  #4: test_dropout_layer ...............   Passed    1.16 sec
      Start  5: test_linear2d_layer
 5/26 Test  #5: test_linear2d_layer ..............***Failed    0.01 sec
      Start  6: test_parametric_activation
 6/26 Test  #6: test_parametric_activation .......   Passed    0.01 sec
      Start  7: test_dense_layer
 7/26 Test  #7: test_dense_layer .................   Passed    0.01 sec
      Start  8: test_conv1d_layer
 8/26 Test  #8: test_conv1d_layer ................   Passed    0.01 sec
      Start  9: test_conv2d_layer
 9/26 Test  #9: test_conv2d_layer ................   Passed    0.01 sec
      Start 10: test_locally_connected1d_layer
10/26 Test #10: test_locally_connected1d_layer ...   Passed    0.01 sec
      Start 11: test_maxpool1d_layer
11/26 Test #11: test_maxpool1d_layer .............   Passed    0.01 sec
      Start 12: test_maxpool2d_layer
12/26 Test #12: test_maxpool2d_layer .............   Passed    0.01 sec
      Start 13: test_flatten_layer
13/26 Test #13: test_flatten_layer ...............   Passed    0.01 sec
      Start 14: test_insert_flatten
14/26 Test #14: test_insert_flatten ..............   Passed    0.02 sec
      Start 15: test_reshape_layer
15/26 Test #15: test_reshape_layer ...............   Passed    0.01 sec
      Start 16: test_reshape2d_layer
16/26 Test #16: test_reshape2d_layer .............   Passed    0.01 sec
      Start 17: test_multihead_attention_layer
17/26 Test #17: test_multihead_attention_layer ...   Passed    0.40 sec
      Start 18: test_embedding_layer
18/26 Test #18: test_embedding_layer .............   Passed    0.01 sec
      Start 19: test_layernorm
19/26 Test #19: test_layernorm ...................   Passed    0.04 sec
      Start 20: test_dense_network
20/26 Test #20: test_dense_network ...............   Passed    0.01 sec
      Start 21: test_get_set_network_params
21/26 Test #21: test_get_set_network_params ......   Passed    0.01 sec
      Start 22: test_conv1d_network
22/26 Test #22: test_conv1d_network ..............   Passed    0.05 sec
      Start 23: test_conv2d_network
23/26 Test #23: test_conv2d_network ..............   Passed    0.17 sec
      Start 24: test_optimizers
24/26 Test #24: test_optimizers ..................   Passed    0.02 sec
      Start 25: test_loss
25/26 Test #25: test_loss ........................   Passed    0.01 sec
      Start 26: test_metrics
26/26 Test #26: test_metrics .....................   Passed    0.01 sec

96% tests passed, 1 tests failed out of 26

Total Test time (real) =   2.07 sec

The following tests FAILED:
      5 - test_linear2d_layer (Failed)
Errors while running CTest

Test log is not greatly helpful:

5/26 Testing: test_linear2d_layer
5/26 Test: test_linear2d_layer
Command: "/opt/local/var/macports/build/_opt_local_ppcports_fortran_neural-fortran/neural-fortran/work/build/bin/test_linear2d_layer"
Directory: /opt/local/var/macports/build/_opt_local_ppcports_fortran_neural-fortran/neural-fortran/work/build/test
"test_linear2d_layer" start time: Mar 28 11:05 CST
Output:
----------------------------------------------------------
incorrect updated weights.. failed
test_linear2d_layer: One or more tests failed.
STOP 1
<end of output>
Test time =   0.01 sec
----------------------------------------------------------
Test Failed.
"test_linear2d_layer" end time: Mar 28 11:05 CST
"test_linear2d_layer" time elapsed: 00:00:00

[OneAdder]

I have an idea what could have gone wrong but I don't own any PPC hardware to test it. The issue most likely is that PowerPC might yield slightly different results for floating point operations from AMD64. I use allclose function in some of my tests to avoid this problems but not here. @milancurcic I think we should create smth like nf_utils that will contain implementation from my tests here and then use it everywhere:

neural-fortran/test/test_multihead_attention_layer.f90

Line 42 in 2ed7b6a

function allclose(x, y) result(res)

What do you think?