summaryReporter doesn't print main testSuite when there are nested testSuites

[gian1200]

Given:

• src/index.spec.ts

describe('a test suite', () => {
  it('should pass', () => {
    expect(true).toBeTrue();
  });
  it('should fail', () => {
    expect(true).toBeFalse();
  });

  describe('an inner test suite', () => {
    it("should pass 2", () => {
      expect(undefined).toBeUndefined();
    });
    it("should fail 2", () => {
      expect(undefined).toBeTrue();
    });

    describe('another inner test suite', () => {
      it("should pass 3", () => {
        expect(null).toBeNull();
      });
      it("should fail 3", () => {
        expect(null).toBeTrue();
      });
    });
  });
});

Expected:

Actual:

Quick Analysis:

Seems to work by calling logSuite instead of logResults, in line 97

web/packages/test-runner/src/reporter/summaryReporter.ts

Lines 94 to 101 in d5ae228

	reportTestFileResults({ logger, sessionsForTestFile }) {
	cachedLogger = logger;
	for (const session of sessionsForTestFile) {
	logResults(logger, session.testResults, '', session.browser);
	logger.log('');
	}
	reportBrowserLogs(logger, sessionsForTestFile);
	},

However, not sure if it's expected to have the browser name on all testSuite lines, or only the root one.

[nsanzimana1]

If you're testing in a single browser at a time or focusing on one at a time, it's best to include the browser name only at the root level. If you're running cross-browser tests or multiple suites for different browsers, including the browser name at each suite level may be beneficial for clarity.

[gian1200]

Good point.

However, I did further testing and this are my findings:

{
... 
  browsers: [
    playwrightLauncher({ product: 'chromium' }),
    playwrightLauncher({ product: 'firefox' }),
    playwrightLauncher({ product: 'webkit' })
  ],
... 
}

Report is independent of testing behavior. Results are always organized in this hierarchy:

 testFile (browser 1) > suite(s) > test
 testFile (browser 2) > suite(s) > test
 testFile (browser n) > suite(s) > test

 testFile 2 (browser 1) > suite(s) > test
 testFile 2 (browser 2) > suite(s) > test
 testFile 2 (browser n) > suite(s) > test

 testFile n (browser 1) > suite(s) > test
 testFile n (browser 2) > suite(s) > test
 testFile n (browser n) > suite(s) > test

Similar findings with flatten: true

PS: regardless of the browser comment, the main suite on each testFile report still needs to be fixed on the report.

Unless someone wants to do it earlier, I'll try to post a PR covering all my findings in the next weeks.

[nsanzimana1]

ok

[gian1200]

I manged to implement all changes. It works fine in regular and flatten mode.

However, there seems to be a Mocha specific behavior where all definitions are always wrapped in an "invisible suite". As stated in Mocha docs:

Top-level hooks, tests and suites are all made members of an “invisible” root suite; there is only one root suite for the entire process

This explains why summary report was implemented that way. However, is this behavior also expected from other runners (e.g. Jasmine)?

In case not, this seems to be the location where Mocha results could be unwrapped. A simple validation on root suite should be enough (no tests and only one suite child).

web/packages/test-runner-mocha/src/collectTestResults.ts

Lines 57 to 69 in aadcbea

	function getSuiteResults(suite: Mocha.Suite): TestSuiteResult {
	collectHooks((suite as any)._beforeAll as Hook[]);
	collectHooks((suite as any)._afterAll as Hook[]);
	collectHooks((suite as any)._beforeEach as Hook[]);
	collectHooks((suite as any)._afterEach as Hook[]);
	const suites = suite.suites.map(s => getSuiteResults(s));
	const tests = getTestResults(suite.tests);
	return { name: suite.title, suites, tests };
	}
	const testResults = getSuiteResults(mocha.suite);

Results when no root suite is defined:

• Regular:
  

• Flatten:
  

Results when root suite is defined:

• Regular:
  
  

• Flatten:
  
  

PS: extra indentation on failed test is associated to the usage of "aegean check mark (U+10102)". Not sure if working as expected or bug specific to PowerShell, or the usage of that character.