CTest Discussion Redux

[nickelpro]

I would like to talk about enabling testing in Beman repos

History

This discussion now has a bit of a history:

• The introduction of unconditional include(CTest): Define the test build target more often #91
• To enable_testing() + CTestUseLaunchers: Support the use of CTest launchers #109
• Back to include(CTest): Unconditional test target #153
• (River added:) Do only include(CTest) if dashboard is needed #60

And finally I stumbled into the issue updating beman.scope: Update CMakeLists, prepare for packaging #31

I think the above fully hash out the issue, and this now comes down to a direct discussion between the handful of us who care way too much about the intersection of CMake and CTest.

Discussion

For the reasons discussed in @purpleKarrot's PR and my comment on beman.scope, unconditional test targets via include(CTest) are not workable in the general case. They do not exist on all generators, and they can be misleading even in cases where they do exist.

The generalize-able and fully understood behavior is that if you try to run tests and the tests have not been built, the command should fail by one mechanism or another. The intent of the user is to run the tests provided by the project, and if they didn't build the tests they shouldn't be told a test run passed. Moreover, the mechanism for running tests is ctest, the target driven approach is entirely superfluous.

The only remaining question in my mind is if the cornucopia of other targets dragged in by include(CTest) is desirable for Beman. My answer to this is No. At best they annoyingly overflow target lists, at worst they borderline violate [CMAKE.PASSIVE_PROJECTS].

Proposal

I want to switch Exemplar and the other Beman projects back to the approach used in #109. I'm uncertain if this decision is worthy of the Beman standard. I would want it as something even less than a recommendation, just, "this is how we do it in Beman because a decision needed to be made".

I do not think the other approaches are invalid always, or even in bad taste, but I don't think they meet Beman's particular use case.

[nickelpro]

cc @wusatosi, @bretbrownjr

I don't think anyone else has a dog in this race other than us four. If there's not unanimous consensus I recommend a foot race across Central Park at 72nd St, first one to the information kiosk decides how we enable testing.

[wusatosi]

I recommend a foot race across Central Park at 72nd St

You gotta buy my ticket to fly to NYC.

Note that I don't have that much opinion on this, I just wanted to point out this stream of discussion...

[steve-downey]

The docs are super unclear as to what CTestUseLaunchers does, but if this is the minimum plumbing to get tests to work, I'm fine.

https://cmake.org/cmake/help/latest/module/CTestUseLaunchers.html

Does it allow for an actual test target that I could make depend on all, or will I need to continue to wire up a ctest in the Makefile I drive the build statemachine through?

https://github.com/bemanproject/optional/blob/main/Makefile#L74-L77

( I remember tar commands, I cannot be expected to learn any more tool options. Obligatory XKCD )

[nickelpro]

CTestUseLaunchers is specifically for doing advanced CTest scripting, it has no bearing on the rest of the discussion. The core of the question is whether to use top-level include(CTest) or the much narrower, conditional enable_testing(). Your makefile continues to exist exactly the way it is.

I'm quoting @bretbrownjr from the beman.scope PR:

It's very helpful for the higher level workflow orchestrator (me as a contributor, a common conan workflow, or a shared github workflow) to know exactly how and when to run tests, especially from a plain vanilla checkout of the code in a plain vanilla environment. Having an always-available test target that, yes, sometimes runs zero tests is a positive validation that all available tests are passing.

And I guess I just don't understand the use case that the test target is fulfilling here vs ctest.

A couple of points in my mind:

• The canonical way to run tests is ctest. If you want to run tests and get a positive answer to whether all available tests pass, ctest is the way you do that.
• ctest is configurable with regards to whether it considers no-test a failure or not, with the --no-test= flag, which meets both Bret's "tell me if everything available passes" criteria, and my "if I didn't actually build the tests, please tell me by failing" criteria, and the user invoking the tests can communicate their intent one way or the other
• I do not see a difference between cmake --build --target test, make test and ctest, except that the final one is both the shortest to type actually does what you want in all circumstances. It has the added benefit of being canonical for CMake.

The biggest thing I see as a blocker to enable_testing() + "just use ctest" is that in mixed workflows, where many build systems are in use including plain makefiles, the "just use ctest" answer is incompatible with pure-makefile repos.

If you guarantee that everyone writing makefiles is adding a test target, and you use include(CTest), and you always the Unix Makefile generator for CMake, then make test works everywhere. However, Beman is universally CMake projects, and anyway if you want a test target just add one to your CML, same way you would with a Makefile (ok not exact same don't @ me).

[purpleKarrot]

+1 to everything @nickelpro wrote. Except:

If ..., then 'make test' works everywhere.

Not even then. Note that 'test' is not one of the predefined targets for autotools: https://www.gnu.org/software/automake/manual/html_node/Standard-Targets.html

[dietmarkuehl]

If you guarantee that everyone writing makefiles is adding a test target, and you use include(CTest), and you always the Unix Makefile generator for CMake, then make test works everywhere. However, Beman is universally CMake projects, and anyway if you want a test target just add one to your CML, same way you would with a Makefile (ok not exact same don't @ me).

I believe we eventually want to use modules. I was made to understand that the Makefile generator does not (and will not?) understand modules. Thus, my assumption is that no Makefile can be assumed. Also, POSIX/make isn’t universal. The implication is that the reliable invocation is to use either cmake … or ctest ….

For human consumption which can adapt, that is acceptable. If tooling wants to have something different or even humans appreciate a hand, an optional(?) driver external to cmake/ctest is entirely viable: we don’t built in the root directory
of the repository. Thus, the root directory can reasonably contain a Makefile which invokes cmake/ctest appropriately. I would argue there can also be a configure (and configure.bat?) with a default invocation of cmake, possibly even adapting some of the common arguments (like —prefix). The resulting built (on POSIX systems) would be the familiar (of course, other common approach could/should be supported):

configure
make
make install

Of course, suitable uses of cmake/ctest should/would work as well. I think we are just learning what UI developers [the good ones] learned decades ago: separate presentation and implementation! In my repositories I certainly have a Makefile driving things because I can remember typing make but off-hand I don't know what the [currently] preferred method of invoking cmake is: I'm using exact same make, make test, make install, and make clean since 30+ years (actually, I can't be bothered to type make and have a script mk which also colours the output and tees it into a file I can later less). I don't recall if I also added a configure script.

tl;dr: make the interface common/familiar; implement as appropriate!

[ClausKlein]

Today and in near future we should use cmake and ninja, too on windows to install cxx modules:

cmake --workflow Release
should all do including package step for project uses.

And

cmake --workflow Debug
all checks and tests including temporary check_install for developer!

All other things are for CI or ninja console professionals.

[dietmarkuehl]

Today and in near future we should use cmake and ninja, too on windows to install cxx modules:

cmake --workflow Release should all do including package step for project uses.

The way interpret this request is to have the workflow use a default compiler rather than having a gcc, llvm, appleclang, etc. prefix! I think that’s a more sane direction than the prefixed version. I’m not against having tbe prefixes, too.

[wusatosi]

this request is to have the workflow use a default compiler rather than having a gcc, llvm, appleclang, etc. prefix!

Then we must define sanitizer settings in the main CMake file after project.

[nickelpro]

We're off-topic for the discussion of which testing strategy to use.

Bret's out so the main proponent of the current scheme isn't here to defend it. I will take blatant advantage of this and PR a change to the conditional enable testing approach, since this has now had many eyes on it and no objections.

[wusatosi]

Bret's out so the main proponent of the current scheme isn't here to defend it. I will take blatant advantage of this and PR a change to the conditional enable testing approach, since this has now had many eyes on it and no objections.

He is aware of this thread.