Repository navigation
Workaround func-test's flakiness by forcing serial testing - #5152
Conversation
7baa181 to
ad98eb5
Compare
|
Very interesting! If it works 🤷 But it does show us where the issue lies, if these groups running in parallel are the reason for the failure. |
|
The failures are due to In the same log, I read that I suppose the issue is that stackage-lts23 and stackage-lts24 ship with I wonder what happens if I enforce a lower bound of Incidentally, why that ugly spacing??? |
That's just the default formatting with stylish-haskell, our formatter for HLS. Is this the first time you are working outside the case-split plugin? :D The purpose is to align it with
Stack would ignore it (or throw the error earlier), you need to add that version as an |
No, probably it's the first time I run stylish-haskell on a file where there's no
Yeah, I know that, I just don't understand the preemptive spacing in the event that somebody will add/make an I would say that suffering the diff of spaces in that event at review time is little cost. (That's the only reason I can see for the current behavior.) |
The point of this is to keep Also, you get used to it. Best not to think about it too much. |
Avoid `#if`-branching on the presence/absence of the plugins, and use `idePlugins` to retrieve the currently available plugins (as determined by the passed `Config`). --- I think there's definitely some flakiness around, as I've very sparingly observed the failure below. It's not unlikely that we just need to serialize tests a bit more, which I only partly did via haskell#5152. ``` haskell-language-server part 1 plugin config generic plugin config custom defaults: FAIL Exception: ghc-internal:GHC.Internal.IO.Exception.IOException: /home/enrico/haskell-language-server/test/testdata/diagnostics/test/testdata/diagnostics: changeWorkingDirectory: does not exist (No such file or directory) HasCallStack backtrace: ioError, called at libraries/unix/System/Posix/PosixPath/FilePath.hsc:106:5 in unix-2.8.8.0-fd61:System.Posix.PosixPath.FilePath Use -p '$0=="haskell-language-server.part 1.plugin config.generic plugin config.custom defaults"' to rerun this test only. custom defaults and user config on some other plugin: OK (0.43s) custom defaults and non overlapping user config: FAIL Exception: ghc-internal:GHC.Internal.IO.Exception.IOException: /home/enrico/haskell-language-server/test/testdata/diagnostics/test/testdata/diagnostics: changeWorkingDirectory: does not exist (No such file or directory) HasCallStack backtrace: ioError, called at libraries/unix/System/Posix/PosixPath/FilePath.hsc:106:5 in unix-2.8.8.0-fd61:System.Posix.PosixPath.FilePath Use -p '/custom defaults and non overlapping user config/' to rerun this test only. custom defaults and overlapping user plugin config: FAIL Exception: ghc-internal:GHC.Internal.IO.Exception.IOException: /home/enrico/haskell-language-server/test/testdata/diagnostics/test/testdata/diagnostics: changeWorkingDirectory: does not exist (No such file or directory) HasCallStack backtrace: ioError, called at libraries/unix/System/Posix/PosixPath/FilePath.hsc:106:5 in unix-2.8.8.0-fd61:System.Posix.PosixPath.FilePath Use -p '/custom defaults and overlapping user plugin config/' to rerun this test only. custom defaults and non plugin user config: OK (0.24s) generate schema vscode-extension-schema: OK (0.47s) generate-default-config: OK (0.48s) plugins-custom-config-markdown-reference: OK (0.47s) part 2 format document lsp formatting provider respects none: OK (0.03s) can change on the fly: OK (0.44s) behaviour on malformed projects Missing module diagnostic: OK (0.86s) Missing module diagnostic - no matching prefix: OK (0.54s) hie-bios loads main-is module: OK (13.91s) window/workDoneProgress sends indefinite progress notifications: OK (0.69s) eval plugin sends progress reports: OK (19.10s) ormolu plugin sends progress notifications: OK (0.98s) fourmolu plugin sends progress notifications: OK (0.75s) 3 out of 17 tests failed (19.58s) ```
Avoid `#if`-branching on the presence/absence of the plugins, and use `idePlugins` to retrieve the currently available plugins (as determined by the passed `Config`). --- I think there's definitely some flakiness around, as I've very sparingly observed the failure below. It's not unlikely that we just need to serialize tests a bit more, which I only partly did via haskell#5152. ``` haskell-language-server part 1 plugin config generic plugin config custom defaults: FAIL Exception: ghc-internal:GHC.Internal.IO.Exception.IOException: /home/enrico/haskell-language-server/test/testdata/diagnostics/test/testdata/diagnostics: changeWorkingDirectory: does not exist (No such file or directory) HasCallStack backtrace: ioError, called at libraries/unix/System/Posix/PosixPath/FilePath.hsc:106:5 in unix-2.8.8.0-fd61:System.Posix.PosixPath.FilePath Use -p '$0=="haskell-language-server.part 1.plugin config.generic plugin config.custom defaults"' to rerun this test only. custom defaults and user config on some other plugin: OK (0.43s) custom defaults and non overlapping user config: FAIL Exception: ghc-internal:GHC.Internal.IO.Exception.IOException: /home/enrico/haskell-language-server/test/testdata/diagnostics/test/testdata/diagnostics: changeWorkingDirectory: does not exist (No such file or directory) HasCallStack backtrace: ioError, called at libraries/unix/System/Posix/PosixPath/FilePath.hsc:106:5 in unix-2.8.8.0-fd61:System.Posix.PosixPath.FilePath Use -p '/custom defaults and non overlapping user config/' to rerun this test only. custom defaults and overlapping user plugin config: FAIL Exception: ghc-internal:GHC.Internal.IO.Exception.IOException: /home/enrico/haskell-language-server/test/testdata/diagnostics/test/testdata/diagnostics: changeWorkingDirectory: does not exist (No such file or directory) HasCallStack backtrace: ioError, called at libraries/unix/System/Posix/PosixPath/FilePath.hsc:106:5 in unix-2.8.8.0-fd61:System.Posix.PosixPath.FilePath Use -p '/custom defaults and overlapping user plugin config/' to rerun this test only. custom defaults and non plugin user config: OK (0.24s) generate schema vscode-extension-schema: OK (0.47s) generate-default-config: OK (0.48s) plugins-custom-config-markdown-reference: OK (0.47s) part 2 format document lsp formatting provider respects none: OK (0.03s) can change on the fly: OK (0.44s) behaviour on malformed projects Missing module diagnostic: OK (0.86s) Missing module diagnostic - no matching prefix: OK (0.54s) hie-bios loads main-is module: OK (13.91s) window/workDoneProgress sends indefinite progress notifications: OK (0.69s) eval plugin sends progress reports: OK (19.10s) ormolu plugin sends progress notifications: OK (0.98s) fourmolu plugin sends progress notifications: OK (0.75s) 3 out of 17 tests failed (19.58s) ```
I looked up Tasty's documentation and specifically the Dependencies section, which enlightened me about the existence of
dependentTestGroupand similar functions.With this PR I'm using
dependentTestGroupto force seriality between two portions of the tests. The choice of not running them all serially is empirical.I've manually verified that
passes, despite the
-j32flag.Fixes #5149.