Running SDK tests through VS Code fails with `Top-level main() function takes arguments`

[sigurdm]

The test issue noted above seems to be specific to using stdout.write in tests (see dart-lang/test#1749), other tests seem to work.

However, I found some other issues trying to run SDK tests through VS Code, such as:

Failed to load "D:/Dev/Google/dart-sdk/sdk/pkg/vm_service/test\add_breakpoint_rpc_kernel_test.dart": Top-level main() function takes arguments.
package:test_core/src/runner/plugin/platform_helpers.dart 93:23  deserializeSuite.<fn>
dart:async/zone.dart 1538:47                                     _rootRunUnary
dart:async/zone.dart 1429:19                                     _CustomZone.runUnary
dart:async/zone.dart 1329:7                                      _CustomZone.runUnaryGuarded

Which happens because the main function is declared as:

void main([args = const <String>[]]) => runIsolateTests(

@jakemac53 I'm not sure why it has the parameter defined like this, but should this be supported by dart test?

Originally posted by @DanTup in #56220

[jakemac53]

The test package does not support running tests that require command line arguments. It should however support this case... it is supposed to allow functions which have an optional arguments parameter.

Maybe that isn't working correctly?

[jakemac53]

Oh, it looks like the main function for that test actually looks like this Future<void> main(args) => ... - making the args optional should fix it.

[jakemac53]

It looks like this test hangs if ran through the test runner still.

I noticed its trying to read from stdin so that might be the issue.

[DanTup]

Oh, I don't know where I quoted that code from above, because it doesn't seem to match what's in the file I also quoted.

I noticed its trying to read from stdin so that might be the issue.

😬

Is there an easy way to confirm if that test is running on the bots?

[jakemac53]

Is there an easy way to confirm if that test is running on the bots?

The bots don't use the test runner - and it works fine if ran manually - so would assume the bots can run it just fine today.

The issue seems to be that it is actually returning the value of runIsolateTests, which doesn't complete until its able to read something from stdin. So it never even gets past the 'loading' phase (this doesn't complete until the future returned from main completes).

I tried just not returning the result of that function call but then I get other issues.

[jakemac53]

This whole test just needs some work to be runnable with the test runner.

The specific issue is actually here - all this code should be in a tearDownAll most likely, and all this should be in a setUpAll (assigning to late variables), although the latter part is likely not required.

That still doesn't get the tests passing with the test runner, but it is closer. I get an issue connecting to the VM service though.

[DanTup]

The bots don't use the test runner - and it works fine if ran manually

The one that reads from stdin? How do the bots satisfy that?

It's not clear to me if many of these tests are expected to work with the test runner, but I'll have a run through all the ones in the common packages I open and see if I can get them working (otherwise, we might need to reconsider defaulting to using the test runner in the SDK, which would be a shame).

[DanTup]

This whole test just needs some work to be runnable with the test runner.

Is there some difference between how tests (produced by calling test()) starts exeturing with/without the runner? I was trying to understand why the tests never run (while we're waiting on the line you linked) through the runner, but do work directly through dart.

[jakemac53]

The one that reads from stdin? How do the bots satisfy that?

I think the code path I saw there was actually for a subprocess of the test itself, or some code path that this specific test isn't hitting at all, it was a red herring.

It's not clear to me if many of these tests are expected to work with the test runner, but I'll have a run through all the ones in the common packages I open and see if I can get them working (otherwise, we might need to reconsider defaulting to using the test runner in the SDK, which would be a shame).

Previously they couldn't even try, so it isn't surprising they did things that don't work. But ideally we would get them all running that way.

Is there some difference between how tests (produced by calling test()) starts exeturing with/without the runner? I was trying to understand why the tests never run (while we're waiting on the line you linked) through the runner, but do work directly through dart.

Yes, lots of differences. Usually they don't matter but sometimes they do.

[mraleph]

@jakemac53 could you tag this with appropriate area- label? I am at loss how to categorize this.

[DanTup]

The original report here was slightly bogus, because the code I quoted wasn't the code causing the error.

There's still an issue here, but it spans multiple SDK packages - the issue is really "the SDK packages have many tests that do not work through dart test".

We could rename this issue, or we could try splitting it up for the different packages. I'm working through some of the tests (in packages I'm familiar with) but I don't know how many will be simple fixes (eg. don't use Platform.script) and how many will be like the one Jake found above (potentially requiring some reworking of the tests).