Improve GitHub Actions test execution time

[michaelsauter]

Ideas:

• save time during image build
• reduce reliance on external services
• Run Tekton tests in parallel #14

[michaelsauter]

Looking forward to what you come up with! Please do not hesitate to run ideas past me - we might have tried some things in the past already and I might be able to provide some more pointers in that case.

[kuebler]

Was thinking about running the tests concurrently as well or at least make this configurable. Will definitely give that a try and see how it affects overall execution time. I do fear, however, that running the tests concurrently will make the console outputs less readable since they might end up being interleaved. @henrjk @buegelbeatz @henninggross @michaelsauter Do you have any ideas how to deal with that? Maybe there is a way to prefix the outputs with an identifier to be able to trace them back to an individual test case?

[henninggross]

I am currently playing around with running tests in parallel by making use of t.Parallel()and the -parallel n option for go test, but I noticed that I need to remove/comment out the following line for this to even work:

ods-pipeline/test/tasks/common_test.go

Line 135 in 14d5129

defer tasktesting.TearDown(t, c, ns)

This is because test will be stopped and continued when running in parallel. If they are stopped the teardown function will be called resulting in the namespace and the pvc being deleted and all tests failing.

[henninggross]

I do fear, however, that running the tests concurrently will make the console outputs less readable since they might end up being interleaved.

This definitely is the case and I also thought about adding the name of the test case to each output, but wasn't sure if this is going to be a cumbersome process to implement on each output and also if this will be polluting the console in the end.

[kuebler]

@henninggross think the teardown issue can be solved by wrapping the loop over the parallel test runs in another test run like so (example code from https://pkg.go.dev/testing#hdr-Subtests_and_Sub_benchmarks):

func TestTeardownParallel(t *testing.T) {
    // This Run will not return until the parallel tests finish.
    t.Run("group", func(t *testing.T) {
        t.Run("Test1", parallelTest1)
        t.Run("Test2", parallelTest2)
        t.Run("Test3", parallelTest3)
    })
    // <tear-down code>
}

[michaelsauter]

Cool! One thing to keep in mind for parallel tests is resource consumption on GitHub Actions. I do not know how much room we have, the main limitations is memory I think with is max 8GB afaik.

Re:output - I do not know of anything besides Go standard behaviour to deal with this. Thinking out loud: maybe one could redirect output of one test into one file in addition to printing to STDOUT, which is then easier to debug if things go wrong?

[henninggross]

I just tried running the TypeScript tests in parallel because I reckoned that duration wise they take the longest and might show the highest potential of time savings.

On my machine with all 6 tests running in parallel it took 576s in total compared to 903s when running them consecutively cutting roughly a third of test execution time.

[henninggross]

Thinking out loud: maybe one could redirect output of one test into one file in addition to printing to STDOUT, which is then easier to debug if things go wrong?

I might not know the capabilities of GitHub actions, but how would one be able to access said files if tests fail during the CI test execution?
When running tests locally this really could be an option.

[michaelsauter]

I might not know the capabilities of GitHub actions, but how would one be able to access said files if tests fail during the CI test execution?

You can attach artifacts to the build. The Jenkins shared library repo is doing this for example.

[henninggross]

I was playing around with writing the outputs to files inside test/testdata/test-results/<task_name>/<test-case>.log but hit dead ends when it came to getting this to work in parallel execution of the tests.
When running the tests consecutively the outputs are written to the correct file, but when running them in parallel evry test output of one specific task test is put together in one file (named after the first test to run).

After doing some debugging I found out that inside of t.Run(name, func(t *testing.T) name is always defined as the first test case that is being run (only when also adding t.Parallel()) and I wasn't able to get down to the bottom of it by now.

For example, this:

t.Run(taskName, func(t *testing.T) {
  for name, tc := range testCases {
    t.Run(name, func(t *testing.T) {
      t.Parallel()
      fmt.Printf("Name: %s\n", name) // -> debug output
      ...
    }
    ...
  }
  ...
}

prints this for the typescript task tests:

=== PAUSE TestTaskODSBuildTypescript/ods-build-typescript/build_typescript_app_in_subdirectory
=== CONT  TestTaskODSBuildTypescript/ods-build-typescript/fail_linting_typescript_app_and_generate_lint_report
Name: build typescript app in subdirectory
=== CONT  TestTaskODSBuildTypescript/ods-build-typescript/build_typescript_app_with_custom_build_directory
Name: build typescript app in subdirectory
=== CONT  TestTaskODSBuildTypescript/ods-build-typescript/build_typescript_app_in_subdirectory
Name: build typescript app in subdirectory
=== CONT  TestTaskODSBuildTypescript/ods-build-typescript/build_typescript_app
Name: build typescript app in subdirectory
=== CONT  TestTaskODSBuildTypescript/ods-build-typescript/build_backend_typescript_app
Name: build typescript app in subdirectory
=== CONT  TestTaskODSBuildTypescript/ods-build-typescript/fail_pulling_image_if_unsupported_node_version_is_specified
Name: build typescript app in subdirectory

and I have no idea why...

[kuebler]

@henninggross I think the name loop variable is captured in the anonymous function passed to r.Run. This can be problematic (e.g., refer to https://www.tddfellow.com/blog/2016/01/08/why-do-you-need-to-be-careful-with-loop-variable-in-go/). Maybe something like this might work:

t.Run(taskName, func(t *testing.T) {
  for name, tc := range testCases {
    t.Run(name, func(name string) {
      return func(t *testing.T) {
        t.Parallel()
        fmt.Printf("Name: %s\n", name) // -> debug output
        ...
      }
    }(name))
    ...
  }
  ...
}