Access Electron APIs without IPC bridge

[goosewobbler]

It may be possible to access the Electron APIs via some other mechanism than the IPC bridge in the user-imported main / preload scripts: for instance, Playwright does not appear to use an IPC bridge. Doing this would improve user experience from the first run as a user would no longer need to amend their application code to use the service. We should investigate the use of CDP to execute code in the main process.

https://webdriver.io/docs/devtools-service
https://chromedevtools.github.io/devtools-protocol/tot/Runtime/#method-evaluate

[255kb]

Very interested by this, for one specific reason: I build the app using electron-builder which prune the devDependencies during the packaging. It means that this package will not be available at runtime.
Bundling this package only during tests is a bit cumbersome to do.

[goosewobbler]

@255kb Yes, this is likely to be a complex piece of work but I think it would be incredibly valuable for the ease of use of the service. The bundling issue is not one I have personally experienced since I routinely bundle my electron apps, but it certainly gives more confidence that finding an alternative to the IPC bridge should be the main feature in development of v7 of the service.

[goosewobbler]

BiDi is recently supported by WDIO, and has something we might leverage:

https://w3c.github.io/webdriver-bidi/#command-script-callFunction

[goosewobbler]

This is next up, will prototype with CDP first as the service isn't using BiDi yet; can follow up to enable BiDi and look to support BiDi IPC comms when it is enabled, falling back to CDP when it is not.

[goosewobbler]

Prototyping CDP comms in https://github.com/webdriverio-community/wdio-electron-service/tree/sm/cdp-ipc-bridge-replacement, using cdp-web as a starting point.

An alternative to the fallback approach above is to skip CDP completely, just use BiDi and then give users the option of either using BiDi or sticking with the existing IPC bridge (the latter is perhaps required for older Electron / Chromium versions?)

[christian-bromann]

using cdp-web as a starting point.

You should be able to call getPuppeteer to get the CDP instance for a Chromium session, no?

[goosewobbler]

You should be able to call getPuppeteer to get the CDP instance for a Chromium session, no?

I tried different combinations of .page(), .pages(), .target(), .targets() with Page.evaluate but I think it's just evaluating code in the renderer and we want access to the electron APIs in the main process. I guess we need puppeteer.cdpsession.send; getPuppeteer gives us the Browser instance which doesn't have access. The reason I didn't go for Puppeteer immediately is that my investigation showed it likely to be too high level a solution, however I'm happy to be proven wrong...

Is there another way to get CDPSession with Puppeteer in WDIO?

[christian-bromann]

Shouldn't it be:

const page = await browser.getPuppeteer()
const cdp = await page.createCDPSession()
await client.send('Animation.enable');
client.on('Animation.animationCreated', () =>
  console.log('Animation created!')
);
const response = await client.send('Animation.getPlaybackRate');
console.log('playback rate is ' + response.playbackRate);
await client.send('Animation.setPlaybackRate', {
  playbackRate: response.playbackRate / 2,
});

[goosewobbler]

Thanks for that, now I have the session I'm now looking at how to get the correct execution context - alternatively, might be able to utilise this flat mode approach. The difference with what we're looking to achieve and that SO post is that the Electron main process isn't accessible from devtools, just the isolated context which I believe is where the preload script is run.

A fair bit of trial and error with this stuff. Here is what playwright does, for reference:

https://github.com/microsoft/playwright/blob/main/packages/playwright-core/src/server/electron/electron.ts
https://github.com/microsoft/playwright/blob/main/packages/playwright-core/src/server/electron/loader.ts

[mato533]

@goosewobbler
We can get execution context by implementing following logic. But every context that can be get using Puppeteer CDPSession could not access electron. I have tried to import electron using async import('electron') or require('electron').

https://github.com/mato533/wdio-electron-service/blob/8c8a0c228a2c1d72985bd43ec2b3b3ba5b6e33e6/packages/wdio-electron-service/src/cdp.ts#L22-L42

So, I suggest to connect directly node using CDP session.
pass --inspect=localhost:<port> to the chrome driver argument. and connect web socket that would be opend by argument. then send message of CDP.
In this method, you can read the electron using require and expose it to the global scope, which allows you to reference the electron in the function passed by browser.electron.execute().
However, a limitation currently exists that prevents webdirverio from setting different capabilities for different workers.

this is implementation of the web socket approach. --> https://github.com/mato533/wdio-electron-service/tree/feature/non-ipc

If you are interested in this approach and want the above implementation to work I will give a detailed explanation.
If you move forward with realizing it, I think you should look for a better library regarding ws connectivity and create an issue/PR on WebdriverIO.
If you tell the branch rules, i will push to this repository.

[goosewobbler]

But every context that can be get using Puppeteer CDPSession could not access electron. I have tried to import electron using async import('electron') or require('electron').

This was my findings so far as well...

So, I suggest to connect directly node using CDP session. pass --inspect=localhost:<port> to the chrome driver argument. and connect web socket that would be opend by argument. then send message of CDP. In this method, you can read the electron using require and expose it to the global scope, which allows you to reference the electron in the function passed by browser.electron.execute(). However, a limitation currently exists that prevents webdirverio from setting different capabilities for different workers.

If you are interested in this approach and want the above implementation to work I will give a detailed explanation. If you move forward with realizing it, I think you should look for a better library regarding ws connectivity and create an issue/PR on WebdriverIO. If you tell the branch rules, i will push to this repository.

I'm definitely interested in this approach, what are the drawbacks? And what work needs to be done in WebdriverIO? Do you think it would be relatively easy to rework the mocking to use this? I always considered this ticket to be quite a lot of work, but admittedly never got past the "trying to get Electron access through CDPSession" stage.

We should get @christian-bromann involved too, and come up with a plan.

[christian-bromann]

However, a limitation currently exists that prevents webdirverio from setting different capabilities for different workers.

If there is anything we can change in core to make things easier here, let me know.

[mato533]

@goosewobbler
Thank you for your comment!

what are the drawbacks?

• With the current restrictions in place.
  Since communication between processes is done as before, I believe that there is still the same limitation as before, which is the need for ingenuity at build time and the limitation on the objects that can be exchanged.
• increasing complexity
  since Puppeteer does not appear to be able to connect a CDPSession to a URL, the need to add a library to the dependency that controls CDP sessions may be a drawback in terms of increasing complexity.
• Performance (Maybe?)
  I think connecting the CDPSession will use more resources than it does now, and perhaps performance will be affected. We need to check.

And what work needs to be done in WebdriverIO?

We believe that there are three things to do:

• select a mechanism to control the CDPSession
• modify the launcher so that it can pass the necessary parameters to connect to Node via CDP
• modify it so that electron.execute can be executed via the CDP session

Do you think it would be relatively easy to rework the mocking to use this?

I don't think it is difficult. I was able to use most of the current implementation in my experimental blanch, and it almost worked with the modifications regarding the CDP session and execute.
However, it is difficult to estimate the difficulty of Sync Mock with Event trigger as discussed in #905 at this time, as we are still considering how to implement it.

[mato533]

@christian-bromann
Thank you for your comment.

Please allow me to explain the issues I am aware of and the response that may be necessary on the part of WDIO.

• Requirement
  I would like to set up different capability for each worker that WDIO launches. Specifically, I want to set different port numbers for each worker.

    capabilities: [
      {
        'browserName': 'electron',
        'goog:chromeOptions': {
          args: '--inspect=localhost:XXX', // workerA: XXX=50000, workerB: XXX=50001...
        },
      },
    ],
  

• Issue
  We wanted to use onWorkerStart to achieve this. However, because the WDIO side passes a same reference of capability to all workers, if worker A sets 100 and then worker B sets 101, worker A also sets 101, resulting in the same port number being set for all workers.

  I think the relevant code is here.

  https://github.com/webdriverio/webdriverio/blob/63ac0e07bef1ac9e9075bbef24bd9c9340085bdc/packages/wdio-cli/src/launcher.ts#L516-L521

• solution
  The following modification allows different settings for each Worker.

  log.info('Run onWorkerStart hook')
  + const capsForWorker = JSON.parse(JSON.stringify(caps)) // Normally, I would like to use “lodash/cloneDeep” etc.
  - await runLauncherHook(config.onWorkerStart, runnerId, caps, specs, this._args, execArgv)
  + await runLauncherHook(config.onWorkerStart, runnerId, capsForWorker, specs, this._args, execArgv)
      .catch((error) => this._workerHookError(error))
  - await runServiceHook(this._launcher!, 'onWorkerStart', runnerId, caps, specs, this._args, execArgv)
  + await runServiceHook(this._launcher!, 'onWorkerStart', runnerId, capsForWorker, specs, this._args, execArgv)
      .catch((error) => this._workerHookError(error))
  
  // prefer launcher settings in capabilities over general launcher
  const worker = await this.runner.run({
              cid: runnerId,
              command: 'run',
              configFile: this._configFilePath,
              args: {
                  ...this._args,
                  /**
                   * Pass on user and key values to ensure they are available in the worker process when using
                   * environment variables that were locally exported but not part of the environment.
                   */
                  user: config.user,
                  key: config.key
              },
  -            caps,
  +            caps: capsForWorker,
              specs,
              execArgv,
              retries
          })

@goosewobbler
I believe that once these corrections are reflected on the WDIO side, we will officially be past the stage of "trying to get Electron access through CDPSession" .

[mato533]

@christian-bromann
I have created an issue and PR for the above comment.
I would appreciate to your review it.

webdriverio/webdriverio#14131
webdriverio/webdriverio#14132

[mato533]

Thanks to @christian-bromann, the fixes on the WDIO side have been merged.

Future Tasks on the part of this service should include

• Selecting the best libraries for handling CDP sessions
  The chrome-remote-interface may be suitable to this use-case
• Modifying launcher.ts to allow inspect to be configured
• Adding an implementation for connecting CDP sessions
• Execution of pre-processing to access electron (getting execution context, global scoping of
• Modifying the execute function so that it can be executed on the electron side using Runtime.evaluate or Runtime.callFunctionOn.

[goosewobbler]

@mato533 Great breakdown of the work required here, I'll let you drive this one. Feel free to add any sub-tasks to this seeing as Github now supports them.

[mato533]

Hi @goosewobbler, thanks so much for taking over this task to me!

I will create sub-issue if needed and and let's discuss about issue that will happen.
Or, It may be better to proceed with to create the branch for this task and create PRs toward it.

By the way, WDIO have been to the release included the changes we had hoped.
Once again, thank you, @christian-bromann .

Then, we did a pretty quick implementation to check the validity of this approach, including whether it really works and whether it worked and met the E2E scenario so far. And we've achieved that!
*) It may not be the pipeline you're used to seeing, but what we're doing is the same - I wanted to download and see log files for each OS and scenario, so I rebuild them.

https://github.com/mato533/wdio-electron-service/actions/runs/13193300710

But there are still unit tests, tentative error handling, inefficient implementations and detailed specifications to consider, so I see this as just a first step!

[goosewobbler]

I like it, also is it running faster?

[mato533]

Personally, my feeling is that it's no different, and the CI time hasn't changed significantly either.

If anything, the possibility to mock ipcMain itself and not having to include this service as a dependency in the electron app seems to be more of an advantage.

[goosewobbler]

Absolutely. I actually just meant the pipeline changes but it makes sense. I don't expect to see a significant reduction in build times until we can build everything in PNPM, remove the hacks and restructure the turborepo build pipeline.

[mato533]

@goosewobbler
Sorry for my misunderstandings...

It is true that Hack is taking longer, but it appears that the Mac-Universal build (whether forge or builder) is taking just as long.
That may be a bottleneck and make it difficult to save significant time. It's just a possibility...

[goosewobbler]

It is true that Hack is taking longer, but it appears that the Mac-Universal build (whether forge or builder) is taking just as long. That may be a bottleneck and make it difficult to save significant time. It's just a possibility...

Not sure it's a bottleneck as nothing really depends on it but it is definitely one of the larger tasks in the pipeline. It's two mac builds in one so it makes sense. The reason why I'm kind of focussed on the Forge hack as a time saver is that without the hack, we should be able to parallelise more tasks with turborepo...let's see anyway.