Localstorage projects using Y.js

[glastonbridge]

⚠️ Investigation project, do not merge.

This builds on the persistence module developed for the python editor, for the basic overview look here.

microbit-foundation/python-editor-v3#1236

Challenges using a CRDT-backed persistent storage in CreateAI

Zustand persistence

Having a Zustand store with interchangeable project data is a challenge, because it does not wait for React's events, but connects to the persistence store on the javascript module load. This means it's ready to go out of the box, but if you are managing multiple projects then loading and switching projects break the assumption that you know where your project is before React is ready.

I addressed this by writing my own persistent storage singleton (persist just needs a get and set that can pack json) that exists on module load, and by turning off automatic hydration of the Zustand store, instead hydrating once a database is connected. This happens in src/store-persistence.ts and store-persistence-hooks.ts has the logic to change the backing store.

An alternate pattern would be to have a Zustand store without persist middleware, and write to it from outside the store (as opposed to inside, in the persist storage), replicating the data whenever you load or save from persistence.

I have not exposed the app itself to changes in persistence state (e.g. what do you do if you try reading the store while it's loading?)

Y.js datatypes in Zustand

It made sense to me to use Actions as a spike, so that multiple users could, e.g. add actions and recordings to the same project. However, Zustand treats states as immutable wavefronts, while Y.js expects to be able to create deltas from insert and delete operations on a persistent object. I created a useActions hook that exposes the Y.js actions.

The Zustand store contains logic that manages updates to the MakeCode project, and so this needs to be able to see the actions. I have, therefore, plumbed the actions into the state provided by the custom persist store mentioned above. Zustand itself is datatype-agnostic and doesn't care that it's not a serialisable type, but it rather breaks Zustand's paradigm, and if you ever wanted to use immer on it, it might cause issues.

This would be simplified if the MakeCode project was updated from an observer, rather than inside the Zustand store, as it could then pull in the actions from other sources without being run inside a Zustand action. However this was somewhat out of scope for this project.

MakeCode projects

As discussed elsewhere, the structure of a MakeCode project is currently very difficult to align with CRDTs, and I have not had any successful attempts.

[github-actions]

Preview build will be at
https://review-createai.microbit.org/localstorage-projects/