Synchronize different stores over a single WebSocket

[waynesbrain]

Is your feature request related to a problem? Please describe.

WebSockets are expensive. Browsers allow only a few of them per domain. If I want to split my stores up and synchronize them with my server running a createWsServer then I have to have a separate WebSocket per store (I think).

Describe the solution you'd like

Expand createWsServer and createWsSynchronizer to multiplex the messaging for multiple stores. So instead of sending [clientId, message-for-store] the messages would be framed as [clientId, storeId, message-for-store].

Describe alternatives you've considered

I considered putting all reactive state into a single store, but I don't love that option.

Additional context

I am building a dev server that runs locally and I can probably open more WebSockets than the normal web app might, without degrading performance, but I would rather not have 5 sockets open if I can avoid it.

[jamesgpearce]

WOW, this is a very cool suggestion. Indeed, you can re-use the WebSocket for multiple synchronizers, but bad stuff is going to happen when it all gets mixed up.

I need to think about the API & protocol changes here to see if I can avoid a breaking change. Obviously it would be good if all the other clients don't have to listen to the chatter of other stores, so saying what they want to subscribe to is also important.

Need to sleep on it, but this is a legit requirement. Thank you.

[waynesbrain]

Thanks for considering it! I made a branch of tinybase to see if I could help out so I'll share my thinking so far.

One reason I wanted separate stores is because some tables should be persisted and others shouldn't. [1] For instance, say I have a table that contains a file tree listing. That table (fs_nodes) should not be persisted because the files on disk are the source of truth. However whenever a client wants to edit a file, we will load the file contents into a separate store with a different set of tables (possibly structured for that file type) and share that with the client. I would want to persist that store and it's tables while a client is editing because if the dev server crashes the user can pickup where they left off complete with undo history.

Thinking about this made me realize that I couldn't just hand a static map of stores by id to createWsServer because some of the stores will be ephemeral. [2] That would require a WsServer interface to add/remove stores after createWsServer is called. You can see my current solution using a separate WebSocket per store below. [3]

[1] I was also working on sharing a server store to clients without persisting it and I'm using the MemoryPersister that I mentioned here to do that.

[2] One other question I had is - How can I currently remove a persister from createWsServer once I've created a persister for it? I think that the tinybase WsServer probably just cleans up after itself if all clients disconnect from that perister but I'm not sure yet.

[3] Here's what I have right now to manage the separate stores, one per pathId.

export interface TinyController {
  pathId: string;
  persister: Persister<
    Persists.MergeableStoreOnly | Persists.StoreOrMergeableStore
  >;
}

const controllers = new Map<string, TinyController>();
let server: TinyWsServer | undefined;

export const tinyServer = {
  /** Registers a new store/persister when it comes online. */
  register(ctrl: TinyController) {
    controllers.set(ctrl.pathId, ctrl);
  },
  /** Should only be called after signaling all clients to disconnect from the associated socket. */
  unregister(ctrl: TinyController) {
    // CONSIDER: We could stop persisters here, but we're leaving it up to the creator of said persister.
    controllers.delete(ctrl.pathId);
  },
};

export function initTinyServer() {
  const wss = new WebSocketServer({ noServer: true });
  server = createWsServer(wss, getPersisterByPath);
  sockets.register({ // <-- Handles socket upgrades/emits 'connection' event...
    dispose,
    // e.g. ws://localhost:43021/sockets/v1/project
    path: /^\/sockets\/v1\/t\//,
    wss,
  });
}

function dispose() {
  if (server) {
    server.destroy();
  }
}

function getPersisterByPath(pathId: string) {
  const ctrl = controllers.get(pathId);
  console.log("TINY FIND", pathId, ctrl?.pathId);
  return ctrl?.persister;
}

[waynesbrain]

Just one last obvious thought on this - To avoid breaking changes and perhaps to avoid unnecessary complexity, it might be beneficial to create this as a completely different set of synchronizers, i.e. if you say "createWsServer + WsSynchronizer is for a single store" and a new createWsService + WsServiceSynchronizer is for multiple stores.

As a user it would be perfectly acceptable (to have to decide up front) to use createWsService (or something) for servicing multiple stores versus createWsServer for a single store.

[jamesgpearce]

One other question I had is - How can I currently remove a persister from createWsServer once I've created a persister for it? I think that the tinybase WsServer probably just cleans up after itself if all clients disconnect from that persister but I'm not sure yet.

Yes, it should (close both the persister and the local synchronizer)!

https://github.com/tinyplex/tinybase/blob/main/src/synchronizers/synchronizer-ws-server/index.ts#L145-L148

Let me know if it doesn't seem to work...

[jamesgpearce]

To avoid breaking changes

I was thinking this could just be a final optional parameter when creating the synchronizer on the client. That can then go as an optional part of the message payload. I might try a few ideas...

[waynesbrain]

when creating the synchronizer on the client

The first argument to createWsSynchronizer is currently a single store, but I am asking how to synchronize multiple stores over a single websocket. Maybe I am missing something.

I am interested in being able to use a single websocket to allow tinybase client bind to a server store the same way that I am doing it now (where I use multiple websocket connections, one per store). This allows my server, to choose different synchronizers for different tinybase stores. (e.g. memory vs file vs db, etc.)

I'm probably missing something because the code is really great but it's not the easiest code to read (but I realize why you built it this way, to keep it tiny!)

[waynesbrain]

In a related issue for me, today I am considering how much I can trust the local (LAN) web clients of my local dev server project if I use TinyBase and I wanted to share my thoughts. Which also helps me think more... thanks! 🤣

• My local dev server watches the filesystem and fills up a TinyBase store with the listing details.
  • Server does not treat TinyBase as a source of truth, only writes changes.
• The server shares the store to clients via WebSocket so they can work with the listing.
• Clients are supposed to make HTTP requests to CRUD the listing, but since they have access to the TinyBase store, they could mess up the fs listing store for other clients. [1]
• Clients will also two-way sync with other server stores at times, while editing/watching a file etc., which was the reason for this very issue Synchronize different stores over a single WebSocket #177.

[1] So because of this I am wondering if I should ask for a createWsServer feature wherein you can declare that a Store should only sync one way? This might be too much to ask and I am not sure if that fits the mission of TinyBase. I'm also not sure if it's entirely needed because I am creating the only browser client for my local dev server and if I expose anything to 3rd party devs I don't have to expose the TinyBase store.

In another scenario where I try to model and sync all activities via a single store, I was thinking of asking for tables that only sync one way (so they would be writable on the server and readable on the client). Maybe it's a bad idea though.

[waynesbrain]

One more actually related issue (for me) - How can I share a WebSocket with TinyBase? What if I want TinyBase to handle some paths and have my own connection handling for other paths?

(This is for in case I want to just do the one-way sync myself without TinyBase, it would be great if I could use the same socket.)

[jamesgpearce]

The first argument to createWsSynchronizer is currently a single store, but I am asking how to synchronize multiple stores over a single websocket. Maybe I am missing something.

Right, I was thinking the API could go from:

const store1 = createMergeableStore();
const store2 = createMergeableStore();

const webSocket1 = new WebSocket('ws://example.com/1');
const webSocket2 = new WebSocket('ws://example.com/2');

const synchronizer1 = await createWsSynchronizer(store1, webSocket1);
const synchronizer2 = await createWsSynchronizer(store2, webSocket2);

to

const store1 = createMergeableStore();
const store2 = createMergeableStore();

const webSocket = new WebSocket('ws://example.com');

const synchronizer1 = await createWsSynchronizer(store1, webSocket, 'store1');
const synchronizer2 = await createWsSynchronizer(store2, webSocket, 'store2');

That final string identifier is sent in messages andused to route messages on the server, and so only only synchronizers on other clients that specified the same string will get synced.

Does that work?

One problem is that there are a bunch of other optional parameters on this synchronizer creator API so I might have to jiggle the types/ordering around a bit to try and do it without making it a breaking change.

[jamesgpearce]

One more actually related issue (for me) - How can I share a WebSocket with TinyBase? What if I want TinyBase to handle some paths and have my own connection handling for other paths?

On the server, you probably want the noServer option which let you create different handlers for different HTTP upgrade requests. (Also a good place to do auth if needed).

HTH!

[waynesbrain]

That final string identifier is sent in messages andused to route messages on the server, and so only only synchronizers on other clients that specified the same string will get synced.

Oh IIIII seeee what you were saying now. Yeah that actually makes a lot more sense than the single synchronizer that I was imagining.

Thanks for the tip on the different handlers for HTTP upgrade requests. I believe that will create a new socket for each upgraded request, which means I can't share the socket with TinyBase...but I was probably wrong to ask for that anyway. It's not a great idea for you or me to be sharing frames on the same socket 🤣