Remote queries use case

[jamesnocentini]

I’ve been thinking about remote queries, your approach to caching could help do that too. There are times where you just need to perform the query on the server side so the HTTP API call does that with the query string params holding the details of the query (number of photos, search keyword…). The result is kept in the NSURLCache and Couchbase Lite. And then have the results of that query mapped to a channel the user has access to so that they are periodically updated and injected back in NSURLCache by the sync engine.

That works well for getting updates of documents in a channel, but I’m wondering about new documents that would match the query. I don’t think they would be picked up by the sync engine as they wouldn't be in the channel. What do you think?

[tfalencar]

By remote queries I believe you mean some db query that runs in the server side (and not RPC); If that is the case, you are correct; Actually what the HTTP API app server does to create the response is transparent (it could fetch data from a database query, or it could generate it however (some local calculations) etc. So you got the idea.. In the end of the article, I mentioned that there can be a little process running in the server which periodically scan all the documents (request and its info) for which a response exists, re-executes the http request against the API service, and then updates the document with the new content in case it actually changed: this will fire the sync in the gateway and the clients will receive it. Here is the "snippet" from the article I am refer to:

"To make the system update the NSURLResponses automatically (that is, without the need to someone having to record / push data from the mobile side), we can setup a server job that from time to time iterates through each response document in the database, checks whether new data is available (that is, the response changed for a certain request), and if so update the document, causing it to be pushed back to the client."

You just made me realise that this not only reduces http connections, but as well CPU cycles in the server: It can potentially reduce the number of database queries etc down to how frequent the "little service" is running to monitor for changes. This could be further optimised if you know better the behaviour of the specific application in order to avoid the "monitoring service" altogether: when you have an event which causes a database change, you take the opportunity to update the cache document from the native couchbase SDK's.

One thing I didn't mention in the text though, is that I implemented the demo thinking of public http API's. If you need to content control (data access), then its necessary to check the http headers and apply a channel to the document appropriately (say, for basic HTTP Auth). The whole thing can be automated though. In the demo of this project this is currently not implemented (server side code). I plan to implement it given enough interest in the project..

Did I understand your question correctly?

[tfalencar]

Updating the documents can be done from the client by: 1- delete the app to erase the CBL documents; 2- comment-out any pull replication; Enable push replication; 3- Now run the app to generate the docs locally again. Any requests fired with cbcache_ category will get its contents updated to the server.

This is more for demonstration purposes, in reality I recommend creating/updating the docs from the server side. I've updated the files in ServerCode folder; It includes a small 'Man in The Middle' script which intercepts all requests at certain port, forwards to the HTTP API, and upon response creates the respective CB document.

[jamesnocentini]

I think the frequency of when the server job is running can be tricky. It's good to display stale information to the user when he is offline because it's better than having nothing on the screen. But when the user is connected and is doing a "pull to refresh" he might be expecting new data. And NSURLCache will return the data pulled by sync gateway until the server job updates the response document and in turn gets replicated to the device.

It could be nice to have a way to know when a query response has changed in the same way couchbase lite knows when a document changed. Maybe by listening on the changes feed and when a document of a certain type changes it will update the response document.

[tfalencar]

You're right, but the cool thing is that you can use it when it's convenient: on your pull-to-refresh method, you can make a 'normal' request from an NSURLSession which has its configuration not overwriting the http's caching policies; This way you can assure that response is current. And upon say, a viewDidLoad, use the 'synced caching' mechanism.
On the other hand, one may think of other approaches too: say, display a "last updated at xx/xx/xxx" - using the date of a last sync completion. Or, implement a call back in the cbcache_dataTaskWithUrl so that when there's an update for that request, notify the user interface (without making the user loose notion of where he is located in the content navigation).
Yet another idea I just had is to generate a hash(e.g. md5) of the CBL's doc response body and send that to an API in the server, which in turns make a new request to the original API and responds either with new data or with a 304 (not modified). The server could then as well use the opportunity to update the respective couch's document if there was actually any change.

[tfalencar]

But I understand what you mean; I think the problem is that the 'world' is used to having this notion of non-active connection thing - which requires us to pull-to-refresh; Perhaps in the beginning one might want to add a "connection working" indicator or something like that. I know that doing a pull-to-refresh and then stopping the replication gives this 'sense' or better 'understanding' by making the app feel like the 'old ways', but in the other side we'd loose many advantages of the fact that we could always have an active connection and only get changes instead of making the request every time. Couchbase sync gateway is supposed to handle several connections anyway because of Go's concurrency. We gotta be careful though, its the same with the UITableView's LiveQuery: if we simply update the interface without the user's consent he'll loose track of what happened or the context of where he is/was in the table (say, some rows were deleted) or the scroll view area gets pushed down/up, and that can be annoying.