Prototype ESM-only package workaround for unbundled back end

[tsmaeder]

Feature Description:

There seem to be some workarounds for making require() of ESM-only modules work in pre-node-22 versions (fix-esm comes to mind immediately). Alternatively, we might see what the experimental flag does (https://nodejs.org/api/cli.html#--experimental-require-module)

[tsmaeder]

I was able to get the browser version of the Theia back end to load correctly using the flag --experimental-require-modules and some changes in how we load the modelcontext classes (in particular, import from *.js files).

[tsmaeder]

The flag above is available in node versions >= 22. Since electron brings it's own node version, we would have to update electron to a version that uses nodejs >= 22. Such a version is not available right now (https://www.electronjs.org/docs/latest/tutorial/electron-timelines), but the next version (35) should include node 22.

[tsmaeder]

I also tried to use fix-esm. For this, we need to do `require('fix-esm').register() early in the lifecyle, for example in main.js. We could add code to does this dependending on a flag.

[tsmaeder]

Earlier, I prototyped using babel-register with https://babeljs.io/docs/babel-plugin-transform-modules-commonjs in order to make our unit tests run when consuming Monaco as ESM modules and this worked, so it should also work with an unbundled back end.

[tsmaeder]

in particular, import from *.js files

By that I mean I had to change the import

import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio';

to

import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js';

[msujew]

I also tried to use fix-esm. For this, we need to do `require('fix-esm').register() early in the lifecyle, for example in main.js. We could add code to does this dependending on a flag.

@tsmaeder FYI the flag could be whether the main entry in the electron app points to a src-gen file, as that indicates that the adopter is using a non-bundled backend.

[tsmaeder]

babel/babel#14579 seems to indicate that the "babel on the fly" thing will not work, and indeed I still get ERR_REQUIRE_ESM even with requiring @babel/register in main.js

[tsmaeder]

The last idea I have is this: we could use the async import() function for all ESM modules we use and put the loaded modules into the global object (globalThis) somewhere. In order for this to play well, we'd have to let typescript know the type of all these modules somehow. Also, there would have to be some contribution architecture for the back end generator to tell it to load those modules in main.js, etc. I think prototyping such an approach would be beyond the scope of this work, though. @msujew wdyt?

[msujew]

Sure, I believe that should work well. However, this would only be a temporary solution, right?

[sdirix]

An alternative is to document the fix-esm approach in ADOPTING.md and do not do any changes on Theia side. Adopters would then need to point their Electron start script to their own one, import and use fix-esm and then call the generated one.

Once we switch to ESM modules the issue will be "fixed" without this adopter need

[tsmaeder]

From this cursory investigation, I would propose the following action items:

• add a flag that loads fix-esm in the appropriate places.
• add @babel/register or fix-esm to our mocha configs (this is not strictly related the "unbundled back end" issue, but we don't want to discourage writing of unit tests because folks run into ESM loading issues).