Offline Capabilities?

[AgarwalPragy]

Offline first apps deliver a better experience when the user goes offline. They also offer a snappier experience in general thanks to the client-side cache.

Are there any plans for supporting offline-first capabilities to surreal db, especially the features highlighted in these essays

1. https://stopa.io/post/279
2. https://stopa.io/post/296

[tobiemh]

Hi @AgarwalPragy , absolutely - this is a definite intention for SurrealDB. And that's the whole reason why SurrealDB is designed to run embedded in the browser, as well as in a server environment...

We're not fully set on the implementation details of this at the moment. We've looked at Operational Transformation, and CRDTs, and we would ideally like to find a middle ground between them (in terms of offline support / speed / being able to operate as a full database), which is an area of research which is seeing more interest at the moment!

[iyobo]

Offline-first and multi-user data online sync would make me dump my [p|c]ouchDB stack and jump on surrealDB immediately.

[Vendrieou]

agree!

[PackMeister]

Ditto!

[sp90]

💯

[davidjeba]

With tight integration with Cloudflare, the database on the edge will/can act as distributed endpoint for lower latency

[tslocke]

Any update @tobiemh ?

[mghase]

any update please

[Alexfilus]

I was looking something like that.
I need something with can be easily used as embedded database, and synchronizing with remote server witch storing data for all users.

[JasCodes]

@tobiemh is this on back-burner?

[adamski]

I arrived here after discovering SurrealDB. I am also looking for an "offline-first" data sync solution that syncs between server and local.
Current solution I've been planning is to sync JSON between SQLite and Postgres.

SurrealDB looks amazing but I can't find anything on synchronisation between embedded and server.

[JasCodes]

https://github.com/electric-sql/electric

much better for postgres -> sqlite

[adamski]

I went with RealmDB in the end. I need a C++ SDK. Theirs has reached preview stage, almost production ready.

[tantaman]

There's also https://github.com/vlcn-io/cr-sqlite which is a SQLite extension and is available anywhere SQLite is. The API is available over SQL so it'll work with any language that has SQLite bindings.

I'm working on a revised iteration that allows using any DB on the backend you'd like (e.g., postgres, mysql, surreal). That is many months out, however.

[davidlondono]

I need multi tenant solution
So on local support multi users and multi projects

I had to develop a layer on top of sqlite for that but is not native.

Can subrrealdb be the solution?

[davidhmays]

Any updates on this? @tobiemh

[JasCodes]

I did ask @tobiemh in QnA, this is not on near term horizon. I can understand, its very complicated problem.

[vmolsa]

Implementing fallback (offline) support would be a valuable feature for SurrealDB. For example, if the main db responds too slowly when writing log entries, sensor data, or any kind of data that doesn't have to be synced in realtime then embedded / local database could store these entries and once the main database is back online or functioning properly, the local db could be synced with the main one. This would enhance reliability and ensure continuous operation.

[terion-name]

As far as I can tell it can be quite easily done with live queries.. As I have thought of it, while there is no such replication native support, one can share schema for server and db and simply subscribe to live queries for all tables that put all the data to local replica respecting user's permissions. However the tricky part would be diffs syncs... For this obviously Change Feeds will be suitable, but again - then you need to import this manually and idk how permissions are handled with CF.

In any case looks doable, but with some manual work

[HansenBerlin]

Offline capability along with a synch mechanism would be a gamechanger. I tried to achieve this using tikv by running a pd/tikv cluster on multiple nodes, each containing a surreal db instance as well. One of them was "simulating" an onprem machine that at some time could go offline, local db updates can be done and after it gets connected again it snychs with the rest of the cluster. Turns out tikv doesnt work that way and whenever my pd instance got disconnected from the rest of the cluster it failed, and with it the surreal db instance. Bummer. :-/ But would have been to easy I guess. Any idea of how to achieve this (without being to hacky) using tikv?

[sp90]

We just saw the release of 2.0 are there anything on the horizon for offline support?

Specifically for my usecase it would be

• Having a Tauri app using the surrealdb wasm database for offline purpose
• Then having the wasm db sync to my cloud db when possible handling conflicts in the best way possible

[sp90]

I will take the liberty and add you @tobiemh hope to hear back from the team

[nickchomey]

Dont do that...

[sp90]

Why not it's a real concern and its been over 2 years since this was mentioned as something they wanted to do, it's an easy thing to answer it can be one of three responses:

• No we decided not to do it
• Yes we're (working on/researching) it
• Yes but not yet we're expecting to see this happen in [INSERT_TIMEFRAME]

Can't see why that is such a big problem

[nickchomey]

Sorry I was saying dont tag people - especially the ceo. Have patience, they're busy with lots of more important issues right now.

[sp90]

But i was just tagging the person who commented on it back then, I respect peoples time and I dont expect an answer but I think its only fair to tag the person who was the respondand back then

[mghase]

I think powersyn will be the bes https://www.powersync.com/

[tymon3568]

i love surrealdb, but at now, if they don't have what you want, just comback big man (Postgres for stable and sure evethings work!). Don't use SQLite, use Postgres instead.
https://electric-sql.com/ for sync,
https://pglite.dev/ for offline database (local file, indexdb in web, or ram)

[sp90]

I would love to test it for my wasm db to cloud hosted db

I mostly wanna use it as a offline first approach for desktop and mobile apps that I wanna build :)

[Luxbit]

I build it for the same purpose, an offline first desktop app with companion mobile apps. Both the client and server are written in Rust, but I haven't tried to compile it as a wasm package yet. Atm, I'm just running the local sync client in a separate process which listens to database events. My goal is to keep it completely separate from the app logic, such that I can keep using the normal surrealdb SDKs without any changes.

[sp90]

So you're running surreal from the rust implementation, just thought it was easier to use their wasm db in the app that way you have an easy way to interface with it - then have some sort of nice sync behaviour with different merge strategies

What do you then do make a whole separate API inside the rust backend of your app then?

[Luxbit]

Yes I'm running surrealdb as a separate process and connect to it using a unix socket or a named pipe . That wasn't the initial plan, but the embedded Node version currently has a bug where live events do not work properly. I actually think this is the cleaner solution now, because I can keep using all the SDKs and just rely on database events. This is the reason why it can't sync schemas right now. I resolve all conflicts server-side, not on the clients, merges are not necessary because I sync everything using CRUD change feeds, which the server retains until all clients are in sync again.

[sp90]

But what if two users changes the same thing while one is offline I guess you need to know which one to take?

Is it a patch event and then the latest incoming change win ?

[nickchomey]

I've recently been experimenting with surrealdb in an offline-first web app and have found it to be quite slow at doing queries.

I ended up using rxdb, which is vastly more performant as well as offers many extremely useful plugins such as leader election for multiple tabs, a flexible sync/replication engine, and more.

[mghase]

any status on offline sync

[mghase]

no anyfeedback

[bitnom]

found this: https://github.com/plasmatech8/surrealdb-sandbox

[tstewart-klaudhaus]

+1

Surreal looks interesting and the multi-model data support is very persuasive.

But sync is a big problem (have implemented it myself on a business-case-specific basis in multiple commercial offline-first apps) and Surreal lacks this in comparison to current alternatives in the space such as RxDB, Electric and Powersync.

Embedded runtime and live queries appear to offer a good portion of a solution, auto-submit on reconnect and CRDT or similar conflict resolution strategy would complete that. If performance / cost considerations were favourable, Surreal would then have a lot of bases covered.

[itsezc]

For sure! there is an upstream PR coming for persistence (on the browser) for Tanstack DB so just waiting on the merge there and I think they're using SQLite for it.

The request flow between Tanstack and rxdb are similar in the sense that both requests go "local-first" before syncing with the remote server, the end goal is to run with SurrealDB WASM so you get full feature parity once some optimizations land.

[limcheekin]

I'm author of the Flutter SurrealDB WASM package at https://pub.dev/packages/surrealdb_wasm, perhaps I could integrate your works of https://github.com/ForetagInc/tanstack-db-surrealdb to this package. Anyone needs this feature to be implemented?

[itsezc]

@limcheekin Of course, you're welcome to do with it as you please! Let me know if I can be of any help on my end.

[tstewart-klaudhaus]

That does look interesting @itsezc. It will definitely benefit from browser persistence being added to Tanstack DB.

The RxDB / SurrealDB replication adapter also sounds interesting @nickchomey. Is that a public project?

[nickchomey]

@tstewart-klaudhaus no it's not public, and it's not really an adapter that would be generally useful to anyone at this point.

But all you need to do is make a few backend http endpoints that handle the pull, pullstream and push endpoints for the http replication plugin. There's server.ts examples in this doc

https://rxdb.info/replication-http.html

All you need to do, for example, is change mongoCollection.find to make a query to surrealdb (or mysql or anything) and massage the response into the rxdb replication format. Likewise for the pullstream (I think I use live query) and push handler (I'm not using push).

It would be interesting to try using surrealdb's graphql api or making custom endpoints with the new DEFINE API, but I wanted to keep it simple.

It would also be neat to use jsonschema to automatically create/update/migrate schemas for both surrealdb and rxdb. I think I have it somewhat manual for now though.