multi-process support

[carlsverre]

Currently, Graft is single-process (per graft data dir) due to Fjall. While this keeps some things simple, it makes interacting with Graft at runtime more difficult and breaks certain user expectations. One of those is that users coming from SQLite expect to be able to open the same Graft Volume (via the Graft SQLite extension) in multiple processes.

To make Graft multi-process, we either need Fjall to support multi-process (@marvin-j97?) or to move metadata into a multi-process-friendly store like SQLite.

[marvin-j97]

need Fjall to support multi-process

Multi-process is not possible because it unfeasible to synchronize memtables between processes, amongst other problems.

[ignatz]

+1 on supporting multi-process.

Out of curiosity, what's the current failure mode? Will a concurrent connect from a separate process fail or is there a risk of corruption?

Just double checking, is it ok to hold multiple connections to the same graft dirs, volumes, tags in the same process? Naively I would assume that it should be ok, given how graft's implementation joins all operations into a shared tokio runtime 🤷‍♀️

[carlsverre]

Currently, Fjall will fail to start due to the file lock it uses to prevent multiple processes from accessing the same data directory.

Within the same process, Graft is fully thread-safe.

Ack on +1 for supporting multi-process. It's one of the big goals I have right now for the next iteration of Graft.

[ignatz]

Happy to hear that there's no risk of corruption.

Ack on +1 for supporting multi-process. It's one of the big goals I have right now for the next iteration of Graft.

I assume that the process would need to (periodically) yield the lock in phases of quiet, just like sqlite's SHARED vs EXCLUSIVE mode (https://sqlite.org/lockingv3.html).

Using this opportunity to blatantly (and egoistically) bring up #237, which may also add some flexibility and at least elevate some of the shared locking challenges :hide:

[carlsverre]

That's certainly one way it could go, though it would require an additional synchronization layer to determine which process is the writer (and probably a novel way to coordinate read access). I'm looking at a lot of different solutions right now to find the right one for Graft.

#237 is also very much on the table as part of the design space.