Skip to content

Workaround func-test's flakiness by forcing serial testing - #5152

Merged
Aster89 merged 7 commits into
haskell:masterfrom
Aster89:fix-5149-with-serial-tests
Oct 22, 2026
Merged

Aster89 merged 7 commits into
haskell:masterfrom
Aster89:fix-5149-with-serial-tests

Conversation

@Aster89

@Aster89 Aster89 commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

I looked up Tasty's documentation and specifically the Dependencies section, which enlightened me about the existence of dependentTestGroup and similar functions.

With this PR I'm using dependentTestGroup to force seriality between two portions of the tests. The choice of not running them all serially is empirical.

I've manually verified that

ionice nice ghcup run --ghc 9.10.3 -- cabal run -w ghc-9.10.3 func-test -- -j32

passes, despite the -j32 flag.


Fixes #5149.


@Aster89 Aster89 self-assigned this Oct 22, 2026
@Aster89
Aster89 requested a review from fendor as a code owner October 22, 2026 05:58
@Aster89
Aster89 force-pushed the fix-5149-with-serial-tests branch from 7baa181 to ad98eb5 Compare October 22, 2026 06:02
@Aster89 Aster89 added flaky test type: bug Something isn't right: doesn't work as intended, documentation is missing/outdated, etc.. labels Oct 22, 2026
@fendor

fendor commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator

Very interesting! If it works 🤷 But it does show us where the issue lies, if these groups running in parallel are the reason for the failure.

@Aster89

Aster89 commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator Author

The failures are due to

/home/runner/work/haskell-language-server/haskell-language-server/test/functional/Main.hs:12:28: error: [GHC-88464]
    Variable not in scope:
      dependentTestGroup
        :: t0 -> DependencyType -> [TestTree] -> TestTree
   |
12 | main = defaultTestRunner $ dependentTestGroup "haskell-language-server" AllFinish
   |                            ^^^^^^^^^^^^^^^^^^

In the same log, I read that tasty-1.5.3 is being used, and indeed dependentTestGroup was introduced in tasty-1.5.4 to replace sequentialTestGroup.

I suppose the issue is that stackage-lts23 and stackage-lts24 ship with <= tasty-1.5.3, but I'm relying on a >= tasty-1.5.4 functionality.

I wonder what happens if I enforce a lower bound of 1.5.4 for tasty, as there's currently none in haskell-language-server.cabal.


Incidentally, why that ugly spacing???

@fendor

fendor commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator

Incidentally, why that ugly spacing???

That's just the default formatting with stylish-haskell, our formatter for HLS. Is this the first time you are working outside the case-split plugin? :D

The purpose is to align it with qualified, e.g.

import qualified Data.List
import           Data.Set

I wonder what happens if I enforce a lower bound of 1.5.4 for tasty, as there's currently none in haskell-language-server.cabal.

Stack would ignore it (or throw the error earlier), you need to add that version as an extra-dep in the stack.yamls.

@Aster89

Aster89 commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator Author

Incidentally, why that ugly spacing???

That's just the default formatting with stylish-haskell, our formatter for HLS. Is this the first time you are working outside the case-split plugin? :D

No, probably it's the first time I run stylish-haskell on a file where there's no import qualified.

The purpose is to align it with qualified, e.g.

import qualified Data.List
import           Data.Set

Yeah, I know that, I just don't understand the preemptive spacing in the event that somebody will add/make an import qualified at some point in the future. It looks ugly.

I would say that suffering the diff of spaces in that event at review time is little cost. (That's the only reason I can see for the current behavior.)

@fendor

fendor commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator

It looks ugly.

The point of this is to keep git diffs small. If you reformatted all imports because one import added qualified, then it would be really annoying. Think of your plugin, where you have about 100 imports. Assume none of them used qualified, but then you add a qualified import. You'd have 100 loc for just a single import. Then you realize you don't want to import it qualified, and have another 100 loc changes.

Also, you get used to it. Best not to think about it too much.

@Aster89
Aster89 enabled auto-merge (squash) October 22, 2026 09:02

@fendor fendor left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@Aster89
Aster89 merged commit c68deb4 into haskell:master Oct 22, 2026
41 checks passed
@Aster89
Aster89 deleted the fix-5149-with-serial-tests branch October 22, 2026 10:24
Aster89 added a commit to Aster89/haskell-language-server that referenced this pull request Nov 10, 2026
Avoid `#if`-branching on the presence/absence of the plugins, and use
`idePlugins` to retrieve the currently available plugins (as determined
by the passed `Config`).

---

I think there's definitely some flakiness around, as I've very sparingly
observed the failure below. It's not unlikely that we just need to
serialize tests a bit more, which I only partly did via haskell#5152.
```
haskell-language-server
  part 1
    plugin config
      generic plugin config
        custom defaults:                                      FAIL
          Exception: ghc-internal:GHC.Internal.IO.Exception.IOException:

          /home/enrico/haskell-language-server/test/testdata/diagnostics/test/testdata/diagnostics: changeWorkingDirectory: does not exist (No such file or directory)

          HasCallStack backtrace:
            ioError, called at libraries/unix/System/Posix/PosixPath/FilePath.hsc:106:5 in unix-2.8.8.0-fd61:System.Posix.PosixPath.FilePath

          Use -p '$0=="haskell-language-server.part 1.plugin config.generic plugin config.custom defaults"' to rerun this test only.
        custom defaults and user config on some other plugin: OK (0.43s)
        custom defaults and non overlapping user config:      FAIL
          Exception: ghc-internal:GHC.Internal.IO.Exception.IOException:

          /home/enrico/haskell-language-server/test/testdata/diagnostics/test/testdata/diagnostics: changeWorkingDirectory: does not exist (No such file or directory)

          HasCallStack backtrace:
            ioError, called at libraries/unix/System/Posix/PosixPath/FilePath.hsc:106:5 in unix-2.8.8.0-fd61:System.Posix.PosixPath.FilePath

          Use -p '/custom defaults and non overlapping user config/' to rerun this test only.
        custom defaults and overlapping user plugin config:   FAIL
          Exception: ghc-internal:GHC.Internal.IO.Exception.IOException:

          /home/enrico/haskell-language-server/test/testdata/diagnostics/test/testdata/diagnostics: changeWorkingDirectory: does not exist (No such file or directory)

          HasCallStack backtrace:
            ioError, called at libraries/unix/System/Posix/PosixPath/FilePath.hsc:106:5 in unix-2.8.8.0-fd61:System.Posix.PosixPath.FilePath

          Use -p '/custom defaults and overlapping user plugin config/' to rerun this test only.
        custom defaults and non plugin user config:           OK (0.24s)
    generate schema
      vscode-extension-schema:                                OK (0.47s)
      generate-default-config:                                OK (0.48s)
      plugins-custom-config-markdown-reference:               OK (0.47s)
  part 2
    format document
      lsp formatting provider
        respects none:                                        OK (0.03s)
        can change on the fly:                                OK (0.44s)
    behaviour on malformed projects
      Missing module diagnostic:                              OK (0.86s)
      Missing module diagnostic - no matching prefix:         OK (0.54s)
    hie-bios
      loads main-is module:                                   OK (13.91s)
    window/workDoneProgress
      sends indefinite progress notifications:                OK (0.69s)
      eval plugin sends progress reports:                     OK (19.10s)
      ormolu plugin sends progress notifications:             OK (0.98s)
      fourmolu plugin sends progress notifications:           OK (0.75s)

3 out of 17 tests failed (19.58s)
```
Aster89 added a commit to Aster89/haskell-language-server that referenced this pull request Nov 10, 2026
Avoid `#if`-branching on the presence/absence of the plugins, and use
`idePlugins` to retrieve the currently available plugins (as determined
by the passed `Config`).

---

I think there's definitely some flakiness around, as I've very sparingly
observed the failure below. It's not unlikely that we just need to
serialize tests a bit more, which I only partly did via haskell#5152.
```
haskell-language-server
  part 1
    plugin config
      generic plugin config
        custom defaults:                                      FAIL
          Exception: ghc-internal:GHC.Internal.IO.Exception.IOException:

          /home/enrico/haskell-language-server/test/testdata/diagnostics/test/testdata/diagnostics: changeWorkingDirectory: does not exist (No such file or directory)

          HasCallStack backtrace:
            ioError, called at libraries/unix/System/Posix/PosixPath/FilePath.hsc:106:5 in unix-2.8.8.0-fd61:System.Posix.PosixPath.FilePath

          Use -p '$0=="haskell-language-server.part 1.plugin config.generic plugin config.custom defaults"' to rerun this test only.
        custom defaults and user config on some other plugin: OK (0.43s)
        custom defaults and non overlapping user config:      FAIL
          Exception: ghc-internal:GHC.Internal.IO.Exception.IOException:

          /home/enrico/haskell-language-server/test/testdata/diagnostics/test/testdata/diagnostics: changeWorkingDirectory: does not exist (No such file or directory)

          HasCallStack backtrace:
            ioError, called at libraries/unix/System/Posix/PosixPath/FilePath.hsc:106:5 in unix-2.8.8.0-fd61:System.Posix.PosixPath.FilePath

          Use -p '/custom defaults and non overlapping user config/' to rerun this test only.
        custom defaults and overlapping user plugin config:   FAIL
          Exception: ghc-internal:GHC.Internal.IO.Exception.IOException:

          /home/enrico/haskell-language-server/test/testdata/diagnostics/test/testdata/diagnostics: changeWorkingDirectory: does not exist (No such file or directory)

          HasCallStack backtrace:
            ioError, called at libraries/unix/System/Posix/PosixPath/FilePath.hsc:106:5 in unix-2.8.8.0-fd61:System.Posix.PosixPath.FilePath

          Use -p '/custom defaults and overlapping user plugin config/' to rerun this test only.
        custom defaults and non plugin user config:           OK (0.24s)
    generate schema
      vscode-extension-schema:                                OK (0.47s)
      generate-default-config:                                OK (0.48s)
      plugins-custom-config-markdown-reference:               OK (0.47s)
  part 2
    format document
      lsp formatting provider
        respects none:                                        OK (0.03s)
        can change on the fly:                                OK (0.44s)
    behaviour on malformed projects
      Missing module diagnostic:                              OK (0.86s)
      Missing module diagnostic - no matching prefix:         OK (0.54s)
    hie-bios
      loads main-is module:                                   OK (13.91s)
    window/workDoneProgress
      sends indefinite progress notifications:                OK (0.69s)
      eval plugin sends progress reports:                     OK (19.10s)
      ormolu plugin sends progress notifications:             OK (0.98s)
      fourmolu plugin sends progress notifications:           OK (0.75s)

3 out of 17 tests failed (19.58s)
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

flaky test type: bug Something isn't right: doesn't work as intended, documentation is missing/outdated, etc..

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ghcup run --ghc 9.10.3 -- cabal run -w ghc-9.10.3 func-test fails, but adding -- -j1 it succeeds

2 participants