[SR-1691] Run longer tests first

[gribozavr]

Previous ID	SR-1691
Radar	None
Original Reporter	@gribozavr
Type	New Feature

Additional Detail from JIRA

Votes	0
Component/s	Project Infrastructure
Labels	New Feature, Python, StarterBug
Assignee	None
Priority	Medium

md5: d2501feeca870b45e5e159e000fd4619

Issue Description:

In the Swift testsuite there many short tests and a few long tests. Sometimes the longer tests happen to be scheduled last, creating a "long tail" and keeping just a few cores busy. It would be better to start the longer tests first, and then run the short tests. This can shave off 1..2 minutes from testing time.

The suggested solution is to add a new feature to lit, that would allow to attach arbitrary labels to tests. lit could embed definition of special label type, or test suite could have extra code to take specific label names and then attach more semantic attributes to test.

// RUN: some-long-test
// LABEL: run_first

We could also migrate the limit_to_features feature to use labels, it would be a much better fit.

[ddunbar]

This is a good starter bug for learning about how lit and the Swift testing infrastructure works.

[Rajveer100]

An idea for this is very similar to C++ pair-sort using std::sort in which the we can store the tests in this fashion std::vector<std::pair<int, {id_type}>>, and then use std::sort(v.rbegin(), v.rend()), so that could do the work, once we can assign the appropriate id type for each test...and maybe later using lambda comparators.

[Rajveer100]

So how would it start, I mean what could be the potential first step...

[Rajveer100]

@AnthonyLatsis

[AnthonyLatsis]

I am not familiar with this area; @ddunbar might have some advice.

[Rajveer100]

Cool! @ddunbar please reply...