Run tests in parallel

CHANGELOG.md

@@ -2,6 +2,10 @@
 
 ## Added
 
+- You can now parallelize tests by enabling the `:parallel` key or
+    the `--parallel` flag. This is still a beta feature, but works on a variety
+    of code bases.
+
 ## Fixed
 
 ## Changed

doc/03_configuration.md

@@ -29,7 +29,8 @@ Here's an example test configuration with a single test suite:
  :kaocha.plugin.randomize/seed       950716166
  :kaocha.plugin.randomize/randomize? true
  :kaocha.plugin.profiling/count      3
- :kaocha.plugin.profiling/profiling? true}
+ :kaocha.plugin.profiling/profiling? true
+ :kaocha/parallize?                  false}
 ```
 
 Writing a full test configuration by hand is tedious, which is why in

doc/04_running_kaocha_cli.md

@@ -95,6 +95,32 @@ unhelpful output in a particular scenario, you can turn it off using the
 
 ![Terminal screenshot showing an expected value of "{:expected-key 1}" and an actual value. ":unexpected-key 1" is in green because it is an extra key not expected and "expected-key 1" is in red because it was expected but not present.](./deep-diff.png)
 
+## Parallelization
+
+Kaocha allows you to run tests in parallel using the `:parallel` key or
+`--parallel` flag. This is primarily useful for I/O heavy tests, but could also
+be useful for CPU-bound tests.
+
+Before enabling parallelization, strongly consider timing it to ensure it
+actually makes a difference. Consider using a tool like
+`bench` or `hyperfine`. While Kaocha's built-in profiling tools are great for
+identifying specific tests that take a disproportionate amount of time, they don't repeatedly measure your entire test suite
+to account for variation and noise. If you want to use parallelization to
+speed up continuous integration, try to use the CI service itself or similar hardware. CI runners are often lower powered than even a middle-of-the-road laptop.
+
+`test.check` tests consist of repeatedly testing a property against random data.
+In principle, these tests would be an excellent use case for parallelization.
+However, because this repeated testing happens within `test.check`, Kaocha sees each `defspec` as a
+single test. If you have many property-based tests that take a significant amount of
+time, parallelization is a great fit. However, if you have one or two
+property-based tests that take up the bulk of the runtme time, parallelization may not
+make a significant difference because the work cannot be split up.
+
+If you want to disable parallelization that's enabled in your configuration, you can
+pass `--no-parallel`. If you find yourself frequently reaching for this flag,
+it's probably worth reconsidering your configuration—having to frequently
+disable parallelization might be negating any time saved by parallelization.
+
 ## Debug information
 
 `--version` prints version information, whereas `--test-help` will print the

doc/11_parallelization.md

@@ -0,0 +1,85 @@
+# 11. Parallelization
+
+Parallelization is an optional Kaocha feature, where it distributes your test
+workload across multiple threads, to make better use of multiple CPU cores.
+
+This is still a relatively new feature, one that has a chance of interfering in
+various ways with plugins, custom hooks, or the particulars of test setups that
+people have "in the wild". We very much welcome feedback and improvements.
+Please be mindful and respectful of the maintainers though and report issues
+clearly, preferably with a link to a git repository containing a minimal
+reproduction of the issue.
+
+## Configuration and high-level behavior
+
+You can enable parallelization either via the `--parallelize` command line flag,
+or by setting `:parallelize? true` in your `tests.edn`. This is assuming that
+you are using the `#kaocha/v1` reader literal to provide normalization. The
+canonical configuration key is `:kaocha/parallelize?`.
+
+Kaocha looks at your tests as a hierarchy, at the top level there are your test
+suites (e.g. unit vs intergration, or clj vs cljs). These contain groups of
+tests (their children), e.g. one for each namespace, and these in turn contain
+multiple tests, e.g. one for each test var.
+
+Setting `:parallelize true?` at the top-level configuration, or using the
+command line flag, will run any suites you have in parallel, as well making
+parallelization the default for any type of testable that has children. So say
+for instance you have a suite of type `clojure.test`, then multiple test
+namespaces will be run in parallel, and individual test vars within those
+namespaces will also be started in parallel.
+
+Test type implementations need to opt-in to parallelization. For instance,
+Clojure is multi-threaded, but ClojureScript (running on a JavaScript runtime)
+is not, so thre is little benefit in trying to parallelize ClojureScript tests.
+So even with parallelization on, `kaocha.cljs` or `kaocha.cljs2` tests will
+still run in series.
+
+## Fine-grained opting in or out
+
+Using the command line flag or setting `:parallelize? true` at the top-level of
+tests.edn will cause any testable that is parallelizable to be run in parallel.
+If you want more fine-grained control you can configure specific test suites to
+be parallelized, or set metadata on namespaces to opt in or out of
+parallelization.
+
+```clj
+#kaocha/v1
+{:tests [{:id :unit, :parallelize? true}])
+```
+
+This will cause all namespaces in the unit test suite to be run in parallel, but
+since the default (top-level config) is not set, vars within those namespaces
+will not run in parallel. But you can again opt-in at that specific level,
+through metadata.
+
+```clj
+(ns ^{:kaocha/parallelize? true} my-test
+ (:require [clojure.test :as t]))
+
+...
+```
+
+Conversely you can opt-out of parallelization on the test suite or test
+namespace level by setting `:kaocha/parallelize? false`.
+
+## Caveats
+
+When you start running your tests in parallel you will likely notice one or two
+things. The first is that your output looks all messed up. Before you might see
+something like `[(....)(.......)][(.......)]`, whereas now it looks more like
+`[[(..(..).....(..)...]....)]`. This will be even more pronounced if you are for
+instance using the documentation reporter. Output from multiple tests gets
+interleaved, causing a garbled mess.
+
+The default dots reporter is probably the most usable reporter right now.
+
+The second thing you might notice is that you are getting failures where before
+you got none. This likely indicates that your tests themselves are not thread
+safe. They may for instance be dealing with shared mutable state.
+
+You will have to examine your code carefully. Starting out with a more piecemeal
+opting in might be helpful to narrow things down.
+
+It is also possible that you encounters failures caused by Kaocha itself. In
+that case please report them on our issue tracker.

repl_sessions/benchmark.clj

@@ -0,0 +1,97 @@
+
+(ns benchmark
+  (:require [criterium.core :as c])
+  (:import [java.util.concurrent Executors ])
+  )
+
+(def thread-pool (Executors/newFixedThreadPool 10))
+
+(defn math-direct []
+  (+ 1 1))
+
+(defn math-future []
+  (deref 
+    (future (+ 1 1))))
+
+(defn math-thread []
+  (let [result (atom nil)]
+    (doto (Thread. (fn [] (reset! result (+ 1 1))))
+      (.start)
+      (.join))
+    @result))
+
+(defn math-threadpool []
+  (let [result (atom nil)]
+    (.get (.submit thread-pool (fn [] (reset! result (+ 1 1))) ))
+    @result))
+
+(defn math-threadpool-no-atom []
+  (.get (.submit thread-pool (fn [] (+ 1 1)) )))
+
+
+(c/bench (math-direct) )
+; (out) Evaluation count : 6215391600 in 60 samples of 103589860 calls.
+; (out)              Execution time mean : 2,015262 ns
+; (out)     Execution time std-deviation : 0,497743 ns
+; (out)    Execution time lower quantile : 1,442374 ns ( 2,5%)
+; (out)    Execution time upper quantile : 3,392990 ns (97,5%)
+; (out)                    Overhead used : 7,915626 ns
+; (out) 
+; (out) Found 5 outliers in 60 samples (8,3333 %)
+; (out) 	low-severe	 3 (5,0000 %)
+; (out) 	low-mild	 2 (3,3333 %)
+; (out)  Variance from outliers : 94,6147 % Variance is severely inflated by outliers
+
+(c/bench (math-future) )
+; (out) Evaluation count : 3735420 in 60 samples of 62257 calls.
+; (out)              Execution time mean : 16,635809 µs
+; (out)     Execution time std-deviation : 1,104338 µs
+; (out)    Execution time lower quantile : 15,397518 µs ( 2,5%)
+; (out)    Execution time upper quantile : 19,751883 µs (97,5%)
+; (out)                    Overhead used : 7,915626 ns
+; (out) 
+; (out) Found 6 outliers in 60 samples (10,0000 %)
+; (out) 	low-severe	 3 (5,0000 %)
+; (out) 	low-mild	 3 (5,0000 %)
+; (out)  Variance from outliers : 50,0892 % Variance is severely inflated by outliers
+
+(c/bench (math-thread))
+
+; (out) Evaluation count : 774420 in 60 samples of 12907 calls.
+; (out)              Execution time mean : 82,513236 µs
+; (out)     Execution time std-deviation : 5,706987 µs
+; (out)    Execution time lower quantile : 75,772237 µs ( 2,5%)
+; (out)    Execution time upper quantile : 91,971212 µs (97,5%)
+; (out)                    Overhead used : 7,915626 ns
+; (out) 
+; (out) Found 1 outliers in 60 samples (1,6667 %)
+; (out) 	low-severe	 1 (1,6667 %)
+; (out)  Variance from outliers : 51,7849 % Variance is severely inflated by outliers
+
+(c/bench (math-threadpool))
+; (out) Evaluation count : 3815100 in 60 samples of 63585 calls.
+; (out)              Execution time mean : 16,910124 µs
+; (out)     Execution time std-deviation : 2,443261 µs
+; (out)    Execution time lower quantile : 14,670118 µs ( 2,5%)
+; (out)    Execution time upper quantile : 23,743868 µs (97,5%)
+; (out)                    Overhead used : 7,915626 ns
+; (out) 
+; (out) Found 3 outliers in 60 samples (5,0000 %)
+; (out) 	low-severe	 2 (3,3333 %)
+; (out) 	low-mild	 1 (1,6667 %)
+; (out)  Variance from outliers : 82,4670 % Variance is severely inflated by outliers
+
+
+(c/bench (math-threadpool-no-atom))
+
+; (out) Evaluation count : 3794940 in 60 samples of 63249 calls.
+; (out)              Execution time mean : 16,182655 µs
+; (out)     Execution time std-deviation : 1,215451 µs
+; (out)    Execution time lower quantile : 14,729393 µs ( 2,5%)
+; (out)    Execution time upper quantile : 18,549902 µs (97,5%)
+; (out)                    Overhead used : 7,915626 ns
+; (out) 
+; (out) Found 3 outliers in 60 samples (5,0000 %)
+; (out) 	low-severe	 2 (3,3333 %)
+; (out) 	low-mild	 1 (1,6667 %)
+; (out)  Variance from outliers : 56,7625 % Variance is severely inflated by outliers

src/kaocha/api.clj

@@ -137,8 +137,7 @@
                                       ;; don't know where in the process we've
                                       ;; been interrupted, output capturing may
                                       ;; still be in effect.
-                                      (System/setOut
-                                       orig-out)
+                                      (System/setOut orig-out)
                                       (System/setErr
                                        orig-err)
                                       (binding [history/*history* history]
@@ -155,13 +154,13 @@
                                             on-exit)
                         (let [test-plan (plugin/run-hook :kaocha.hooks/pre-run test-plan)]
                           (binding [testable/*test-plan* test-plan]
-                            (let [test-plan-tests (:kaocha.test-plan/tests test-plan)
-                                  result-tests    (testable/run-testables test-plan-tests test-plan)
-                                  result          (plugin/run-hook :kaocha.hooks/post-run
-                                                                   (-> test-plan
-                                                                       (dissoc :kaocha.test-plan/tests)
-                                                                       (assoc :kaocha.result/tests result-tests)))]
-                              (assert (= (count test-plan-tests) (count (:kaocha.result/tests result))))
+                            (let [result-tests (testable/run-testables-parent test-plan test-plan)
+                                  result       (plugin/run-hook :kaocha.hooks/post-run
+                                                          (-> test-plan
+                                                              (dissoc :kaocha.test-plan/tests)
+                                                              (assoc :kaocha.result/tests result-tests)))]
+                              (assert (= (count (:kaocha.test-plan/tests test-plan))
+                                         (count (:kaocha.result/tests result))))
                               (-> result
                                   result/testable-totals
                                   result/totals->clojure-test-summary

src/kaocha/config.clj

@@ -31,7 +31,7 @@
         (update config k vary-meta assoc :replace true)
         (do
           (output/error "Test suite configuration value with key " k " should be a collection or symbol, but got '" v "' of type " (type v))
-          (throw+ {:kaocha/early-exit 250}))))
+          (throw+ {:kaocha/early-exit 252}))))
     config))
 
 (defn merge-config
@@ -55,7 +55,8 @@
               (rename-key :skip :kaocha.filter/skip)
               (rename-key :focus :kaocha.filter/focus)
               (rename-key :skip-meta :kaocha.filter/skip-meta)
-              (rename-key :focus-meta :kaocha.filter/focus-meta))]
+              (rename-key :focus-meta :kaocha.filter/focus-meta)
+              (rename-key :parallelize? :kaocha/parallelize?))]
     (as-> m $
       (merge-config (first (:kaocha/tests (default-config))) $)
       (merge {:kaocha.testable/desc (str (name (:kaocha.testable/id $))
@@ -82,7 +83,8 @@
                 randomize?
                 capture-output?
                 watch?
-                bindings]} config
+                bindings
+                parallelize?]} config
         tests (some->> tests (mapv normalize-test-suite))]
     (cond-> {}
       tests                   (assoc :kaocha/tests (vary-meta tests assoc :replace true))
@@ -95,6 +97,7 @@
       (some? watch?)          (assoc :kaocha/watch? watch?)
       (some? randomize?)      (assoc :kaocha.plugin.randomize/randomize? randomize?)
       (some? capture-output?) (assoc :kaocha.plugin.capture-output/capture-output? capture-output?)
+      (some? parallelize?)    (assoc :kaocha/parallelize? parallelize?)
       :->                     (merge (dissoc config :tests :plugins :reporter :color? :fail-fast? :watch? :randomize?)))))
 
 (defmethod aero/reader 'kaocha [_opts _tag value]
@@ -195,16 +198,16 @@
      config
      (read-config nil opts))))
 
-
 (defn apply-cli-opts [config options]
   (cond-> config
-    (some? (:fail-fast options))  (assoc :kaocha/fail-fast? (:fail-fast options))
-    (:reporter options)           (assoc :kaocha/reporter (:reporter options))
-    (:watch options)              (assoc :kaocha/watch? (:watch options))
-    (some? (:color options))      (assoc :kaocha/color? (:color options))
-    (some? (:diff-style options)) (assoc :kaocha/diff-style (:diff-style options))
-    (:plugin options)             (update :kaocha/plugins #(distinct (concat % (:plugin options))))
-    true                          (assoc :kaocha/cli-options options)))
+    (some? (:fail-fast options))   (assoc :kaocha/fail-fast? (:fail-fast options))
+    (:reporter options)            (assoc :kaocha/reporter (:reporter options))
+    (:watch options)               (assoc :kaocha/watch? (:watch options))
+    (some? (:color options))       (assoc :kaocha/color? (:color options))
+    (some? (:diff-style options))  (assoc :kaocha/diff-style (:diff-style options))
+    (:plugin options)              (update :kaocha/plugins #(distinct (concat % (:plugin options))))
+    (some? (:parallelize options)) (assoc :kaocha/parallelize? (:parallelize options))
+    true                           (assoc :kaocha/cli-options options)))
 
 (defn apply-cli-args [config args]
   (if (seq args)
@@ -227,7 +230,7 @@
       cli-opts (apply-cli-opts cli-opts)
       cli-args (apply-cli-args cli-args)))
 
-(defn find-config-and-warn 
+(defn find-config-and-warn
   [config-file]
   (let [final-config-file (or config-file "tests.edn")]
     (when (not (.exists (io/file (or config-file "tests.edn"))))