Transition to Yarn Workspaces

[leeyi45]

Transition to Yarn Workspaces

Yarn has supported Workspaces since Yarn 1, but it has matured over later versions and we can now use it since we have moved to using Yarn 4.

The main motivations for doing this are:

• Reduced install size. Yarn workspaces support focused installs, which means that a single team working on a single bundle or tab won't have to install the entire repository's worth of packages that won't be relevant to them
• Better Compartmentalization. There is a significant amount of code that functioned like a common library for bundles and tabs that didn't really have a proper place to live. Workspaces allow us to separate this code neatly.
• Dependency Management. Previously, if tabs relied on a specific bundle, it was down to the build scripts to determine if the tabs needed rebuilding when the bundle was rebuilt. This is more complicated than expected. Instead, workspaces allow us to rely on Yarn's dependency resolution features to determine what needs to be rebuilt and when.
• Automatic Worktree Detection. If so desired, Yarn automatically detects which bundles/tabs have changed with respect to the main git branch and won't rebuild those that haven't. This responsibility previously fell onto the build scripts, but I think Yarn's implementation is far more robust.
• More Configurability: Each bundle/tab can maintain its own set of dependencies now, separate from other bundles. This allows us to have patches local to a specific bundle (like the @jscad/modeling patch for csg). Also each bundle/tab can customize its own tsconfig etc..

Migration to vitest

This repository was migrated to vitest, which has been designed as a drop in replacement for jest. I find that getting jest to work with the wide mix of ESM and CJS syntaxes has been a major headache, and configuration for jest has always been needlessly complicated.

vitest supports TypeScript and ESM out of the box and has generally provided a far smoother experience, especially when dealing with dependencies that need to be tested (like typedoc).

Playwright Testing

We haven't exactly been able to properly test the interactive functionalities of our tabs and other React components. Now, using playwright, we can do just that.

Further work needs to be done to figure out how to better handle playwright installation.

Why I hate Jest

Basically some scripts relied on __dirname, which is only available in CJS, and so would not work if compiled to ESM. But the scripts also used top level await, so compiling to CJS wasn't an option. Thus, import.meta.dirname would have to be used instead, but that meant that jest couldn't transform these files unless it was run in its experimental ESM mode, which would then cause many of the CJS bundled dependencies to break because they don't use imports with extensions.

The alternatives would have been significantly bulkier jest configurations or major changes to the code, and even after trying both of those I still ended up with errors because jest wouldn't properly transform dependencies.

Also because jest insists on transforming to CJS, we can't use verbatimModuleSyntax: true in tsconfig, instead requiring some weird configuration as a workaround.

Upgrading Module Manifests

This PR is the beginning of the steps required to address #57. Bundles will now be able to specify a requires field in their manifest that specifies the minimum version of Source that the bundle can be used with.

A bundle manifest can now look like:

{
  "tabs": [],
  "requires": 1
}

Each bundle specifies its own manifest, which is then collated by the build tools automatically.

Bundles will also be able to specify versions to help work toward resolving #79. The version field in package.json gets merged with the manifest during compilation.

Vitepress Docs Server

The Modules Repo's wiki has been quite the disorganized mess (oops). While I was thinking about how to go about documenting the changes introduced by this repository I realized that we were probably going to need something more serious than the Github Wiki.

So I've begun porting the Wiki over to this new Vitepress server (found in ./docs) whilst trying to update any outdated documentation.

[leeyi45]

There are still many tasks left to be done, this is far from complete, but I thought I'd open this now for feedback.

I'm especially concerned that the structure of the repository looks a lot more complex now, so I'm wondering if there's any need for simplifying things.

[RichDom2185]

A lot of files do not have a trailing final empty newline which is not ideal.

Could you reformat them?

Best to also add in .vscode/settings.json:

{
  // ...other stuff
  "files.insertFinalNewline": true
}

[leeyi45]

RFC:

Effectively the modules repository has three kinds of documentation:

• Bundle Docs for Users (example)
• Bundle Docs for Developers (example)
• Repo Docs for Developers (the various READMEs in ./scripts that are being migrated)

Should we unify all of these under one documentation server? I'm not even sure how we would go about doing that.

I have also been experimenting with using Typedoc to generate Markdown that can be used with Vitepress to document things like our common React components library (which probably will need to become its own fully fledged thing in the future).

[RichDom2185]

RFC:

Effectively the modules repository has three kinds of documentation:

• Bundle Docs for Users (example)
• Bundle Docs for Developers (example)
• Repo Docs for Developers (the various READMEs in ./scripts that are being migrated)

Should we unify all of these under one documentation server? I'm not even sure how we would go about doing that.

I have also been experimenting with using Typedoc to generate Markdown that can be used with Vitepress to document things like our common React components library (which probably will need to become its own fully fledged thing in the future).

@leeyi45 I agree we should move from GitHub Wiki to source code for docs, it's been something in my TODO for a while now. I also agree Vitepress is better, having (unsuccessfully tried Docusaurus in the past.

Though I think since we are now moving it to source code, we should standardize and use something like markdownlint to format our Markdown files.

My original plan for the above-linked repo was to continually pull documentation files from the various individual repositories (added as git submodules), but I am second-guessing it now since it seems like it would be quite messy hmm (though technically doable).

[RichDom2185]

Separately, given the size of codebase, do you have any thoughts on using biome/oxlint over Prettier/ESLint for formatting/linting?

[leeyi45]

Separately, given the size of codebase, do you have any thoughts on using biome/oxlint over Prettier/ESLint for formatting/linting?

At some point I'm thinking about re-incorporating Prettier, but right now I think that would just contribute to the scope creep that this PR is becoming

I have added some configuration to ESLint for linting our markdown and JSON files. Outside of ESLint I have no experience with any other linters so I can't comment on those.

[RichDom2185]

Hmm, possible to run some of these steps in parallel using multiple jobs/a matrix strategy?

[leeyi45]

I gave this a brief go, but I'm most certainly not experienced enough with Github Actions to make this work. I've left some of my work in the repository and have just left the old workflow in place

[leeyi45]

Is the CI taking over half an hour normal since it's the first time (and we are limited to 10 concurrent jobs)?

Subsequent ones would be faster since it only runs for changed files right?

It runs for changed files that are changed relative to the master branch.

I don't know why the CI is taking so long to run. If there is some way to like cache dependencies (especially skipping the playwright install phase) that would be great

[leeyi45]

Is the CI taking over half an hour normal since it's the first time (and we are limited to 10 concurrent jobs)?

Subsequent ones would be faster since it only runs for changed files right?

I figured out the issue:
Several of the bundles/tabs rely on modules-lib, which needs playwright for testing. Because the library has to be "installed" before the bundle or tab can use it, playwright was being installed over 60 times.

I think its worth investigating how to cache the libraries, but for now I think I've finally ironed out most of the issues.