Make test strategy uniform

[anshumanmohan]

I have learned a bunch about testing by stumbling through slow-odgi, and I'd love to pipe that around to other parts of the testing setup. See also #31 (comment)

Just making it an issue for whenever @susan-garry has a moment to chat about it!

[anshumanmohan]

Susan and I talked about this on a Zoom call today. Not clear whether any immediate steps are necessary, but at least we know each others' strategies and thought processes.

[susan-garry]

To follow up on this, my general philosophy for testing is that we should try to keep the testing somewhat separated so that it's easy to figure out what a test is doing for a given command and we don't have one devilishly long turnt.toml file with dozens of different environments corresponding to various different commands. In a perfect world, I could just look at a list of directories, pick the directory corresponding to the command that I want to use, and then run the test file in that directory, rather than combing through a turnt.toml file looking for the appropriate test to use and worrying about whether there are any preprocessing, or setup steps, necessary to run the actual test. This doesn't feel like a waste of space to me since most odgi commands have various options and subcommands that we will eventually want to add some degree of support for (e.g., depth might look at only a subset of paths and/or compute the unique depth of a node), and these subdirectories are a logical place to store files that relate only to specific odgi tests (e.g., considering depth again, we may want a file which lists the subset of paths that we want to consider).

Practically speaking, however, turnt's bottom-up strategy of looking for turnt.toml in the parent directories for a given test file means that running separate test files on the same test requires storing duplicate copies of the test, hence the symlinks, which have proven to be something of a menace for people trying to clone the pollen repo. I think that the only way to get rid of these symlinks is to continue adding all of our tests to one turnt.toml file, and then invoking them from the Makefile. A large, cluttered, messy turnt.toml file is exactly what I would like to avoid, but may be the lesser of two evils in this situation.

[sampsyo]

Yeah, this is a little tricky. I think there are two issues going on here:

1. which tests get which treatment?

• Turnt environments are great when you want to take N different files and treat them in M different ways, for a total of N×M tests. For example, if you're writing an interpreter, you might have 100 benchmarks and 3 different optimization levels, and you want to run all of them, so that's 300 commands to execute. This argues for the "big long turnt.toml" file.
• They are less great when you have some files that are only relevant to some treatments. Like, we currently have some flip-specific GFAs. Then you have to be careful about which tests run in which environments. I discussed this problem in Collapse make targets for testing #113 (comment); it could imply a new Turnt feature we want.

2. how do I know which environments to run to test a given piece of functionality?

Quoting @susan-garry:

In a perfect world, I could just look at a list of directories, pick the directory corresponding to the command that I want to use, and then run the test file in that directory, rather than combing through a turnt.toml file looking for the appropriate test to use and worrying about whether there are any preprocessing, or setup steps, necessary to run the actual test.

Here, Susan is imaging a world where—for example—you might be working solely on flip functionality. Then you'd do cd tests/flip ; turnt *.gfa, for example, and be done with it. Currently, you also have to remember to do some setup steps before you can do that.

However, I would refine this and argue that it would be just as nice to be able to do turnt -e flip tests/*.gfa (or, in a fantasy world, just turnt -e flip and it figures out which tests you want). The point is not about the directories; it's about having a single command to run that does multiple steps.

Therefore, I think what we're missing is some way to know which of several steps to run if you want to run a given kind of test. It's actually not so bad currently; you just have to know that you need to do this stuff once:

make slow-odgi-setup
make slow-odgi-oracles

Then you have all the setup you need; you can then just do turnt -e flip tests/whatever.gfa as many times as you like, while you're iterating on the flip implementation, and life is good. But this still leaves some things to be desired, including the fact that we have to record stuff in a Makefile at all instead of it just being part of invoking Turnt itself.

We will need to extend Turnt to make this work. cucapra/turnt#26 is part of this, although it needs some careful thinking about how exactly it would work. This functionality would at least let us group together the environments that are currently called flip_oracle and flip_test so they could both be part of a single group called flip, and Turnt would somehow be aware of the need to rerun one to test against the other. We could also imagine teaching these groups about "setup" commands that need to run first, which would also obviate the need for the slow-odgi-setup stuff in the Makefile.

[anshumanmohan]

Today we decided to make this change. That is, we'll move away from symlinks and towards a long turnt file.