Fail to catch spec which fails by jasmine default timeout

[vdimikj]

Hi there!

When test fail due to
Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
Protractor-flake fails to parse the failing test and the spec is not rerun thus failing to "rerun potentially flakey protractor tests before failing."

For better recreation I have setup an example found on git-url .

• Operating system and version - MacOs - 10.13.1
• Node.js version - v6.12.0
• Protractor version - Version 5.1.1
• Protractor flake version - "3.0.1"
• Protractor configuration file

// solves `SyntaxError: Unexpected token import
require("babel-register")({
    presets: [ 'es2015' ]
});
const fs = require('fs');
const FancyReporter = require('fancy-protractor-reporter').Reporter;

const fancyReporter = new FancyReporter({
    path: 'report/fancy' + new Date().toISOString().substring(0,19),
    screenshotOnPassed: true,
    consolidateAll: true,
    // isSharded: true
});

exports.config = {
    /**
     *  Uncomment ONE of the following to connect to: seleniumServerJar OR directConnect. Protractor
     *  will auto-start selenium if you uncomment the jar, or connect directly to chrome/firefox
     *  if you uncomment directConnect.
     */
    //seleniumServerJar: "node_modules/protractor/node_modules/webdriver-manager/selenium/selenium-server-standalone-3.4.0.jar",
    directConnect: true,

    specs: ['specs/*Spec.js'],
    baseUrl: 'http://qualityshepherd.com',
    framework: 'jasmine',

    onPrepare: () => {
        // set browser size...
        browser.manage().window().setSize(1024, 800);

        if (!fs.existsSync('report')) {
                    fs.mkdirSync('report');
        }

        // better jasmine 2 reports...
        const SpecReporter = require('jasmine-spec-reporter');
        jasmine.getEnv().addReporter(new SpecReporter({displayStacktrace: 'specs'}));
        jasmine.getEnv().addReporter(fancyReporter);
    },
    afterLaunch: () => {
        fancyReporter.combineReports();
    },

    capabilities: {
        browserName: 'chrome',
        shardTestFiles: false,
        maxInstances: 1
        // chromeOptions: {
        //     args: [
        //         // disable chrome's wakiness
        //         '--disable-infobars',
        //         '--disable-extensions',
        //         'verbose',
        //         'log-path=/tmp/chromedriver.log'
        //     ],
        //     prefs: {
        //         // disable chrome's annoying password manager
        //         'profile.password_manager_enabled': false,
        //         'credentials_enable_service': false,
        //         'password_manager_enabled': false
        //     }
        // }
    },

    jasmineNodeOpts: {
        showColors: true,
        displaySpecDuration: true,
        // overrides jasmine's print method to report dot syntax for custom reports
        print: () => {},
        defaultTimeoutInterval: 50000
    }
};

• Output from your test suite

P0 - Edit Spec1
	    ✓ should test1 changes (54 secs)
	.    ✓ should test2 it (59 secs)
	FA Jasmine spec timed out. Resetting the WebDriver Control Flow.
	A Jasmine spec timed out. Resetting the WebDriver Control Flow.
	A Jasmine spec timed out. Resetting the WebDriver Control Flow.
	    ✗ should test3 it (6 mins 57 secs)
	      - Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
	          at tryOnTimeout (timers.js:232:11)
	      - Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
	          at tryOnTimeout (timers.js:232:11)
	      - Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
	          at tryOnTimeout (timers.js:232:11)

[NickTomlin]

Unfortunately this is not possible without using a custom reporter; the stacktraces for timeouts don't include the path to the source spec in them if I remember correctly.

You could write your own reporter that prints the spec files out and then checks for failures and then write a custom protractor flake parser.

There's a feature request here #72 that goes over things.

I'd love to see this implemented and then pull requested if you have time!

[theandrewlane]

@vdimikj: If you set shardTestFiles to true in protractor.conf.js and use the multi parser, you won't have this issue. Protractor logs "Specs: <(specs-to-be-executed)>" for each browser instance by default, and the parser will identify the specs with Jasmine timeout errors.

If you must execute sequentially, you can try implementing the following:

In your protractor config, get Protractor's logger & CLI args:

const cliArgs = require('yargs').argv,
  Logger = require('protractor/built/logger').Logger,
  logger = new Logger('JasmineFailLogger');

Add the onPrepare() hook to your config, and include the following:

onPrepare: () => {
      const jasmineEnv = jasmine.getEnv();
      // Hacky workaround for https://github.com/NickTomlin/protractor-flake/issues/83
      jasmineEnv.addReporter({
          specDone: (result) => {
            if (result.status === 'failed' && result.failedExpectations[0].message.startsWith('Error: ')) {
              const specName = result.fullName.replace(/ .*/, '');
              const cliArgSpecs = cliArgs.specs.split(',');
              const failedSpecPath = cliArgSpecs.filter(spec => spec.includes(specName));
              // Log this the same way protractor logs sequential specs so the parser finds it
              logger.info(`A Jasmine error occured at UserContext.it (${failedSpecPath.toString()}:)`);
            }
          }
        }
      );
    },