Correctness/Robustness Checking

[MrBurmark]

When running multiple tunings or adding new tunings the answers can be inconsistent if there is a platform, software, or logic bug.

Dying on inconsistent checksums would help quickly identify these. We would need to separate out cases (like atomics) where checksums are not reproducible.

It would be helpful to be able to distinguish cases where the checksum is always the same, differs due to operation order between tunings, or always differs because atomics are used.

[rhornung67]

We do something like this in the test we have https://github.com/llnl/RAJAPerf/blob/develop/test/test-raja-perf-suite.cpp#L183 However, this can be improved. I just set one tolerance for all kernels. We could expand this to define a different tolerance for each kernel, each subset of kernels, etc. Is this what you were thinking?

[MrBurmark]

I was thinking about setting the tolerance to 0 for kernels that should always get the same answer. But other than that I wasn't thinking about changing the tolerance.