How to monitor both local and server changes?

[tom-andersson]

Firstly, thank you for your work on such a useful package. My issue is that I am uncertain about how to best set up a continual onedrive process that both uploads local changes and downloads changes from the OneDrive server.

For context, my use case is that I have two laptops from which I want to edit the same set of notes. I'm unlikely to be using both to edit the notes at the same time, but I want each laptop to 'pull' from the OneDrive server any previous changes made from the other laptop upon startup, and continually 'push' any local changes while I'm editing the notes. I just want to achieve this in a robust, simple way.

As I understand it, --synchronize achieves exactly the two-way interaction I want, but only at a single point in time. --monitor achieves the continual uploading, but doesn't monitor for changes from the server. So --monitor doesn't achieve the 'downloading changes from the other laptop' aspect of my use case.

I've read docs/USAGE.md#use-webhook-to-subscribe-to-remote-updates-in-monitor-mode, and it seemed to address my use case by implementing 'event-based' downloading from the server. Long story short, I tried installing nginx and setting up the webhook as-per the USAGE.md docs, but this led to an error when running onedrive -m and I gave up due to lack of experience.

I'm wondering if the following is simplest solution to my use case: Upon startup of either laptop, run onedrive --resync --synchronize --download-only; onedrive --monitor. The first line implements any config updates and downloads any updates to files from my other laptop. When that command finishes, the onedrive --monitor process is kicked off to continually upload to the OneDrive server, ready for the other laptop to download when it is next turned on. With this solution, I will just have to be careful to not edit the same notes from both laptops at the same time, because the two onedrive --monitor processes would both be sending 'pushes' and overwriting each other on the server.

I apologise for any naivety here. I really did try to work this out myself from the docs and look through previous issues, but I couldn't find the info I needed in a form I could understand. Perhaps the solution to my problem could be included in USAGE.md for people like me who are unfamiliar with webhooks but want a simple solution using the basic command line options?

[abraunegg]

@tom-andersson

As I understand it, --synchronize achieves exactly the two-way interaction I want, but only at a single point in time. --monitor achieves the continual uploading, but doesn't monitor for changes from the server. So --monitor doesn't achieve the 'downloading changes from the other laptop' aspect of my use case.

Your understanding is not correct here for --monitor method of operation.

In --monitor mode the client will monitor the 'server' for changes, but will only do this at predefined intervals - 300 seconds by default. This value is user configurable

During the 'sync' loop, changes made online are 'downloaded' to the local machine.

I'm wondering if the following is simplest solution to my use case: Upon startup of either laptop, run onedrive --resync --synchronize --download-only; onedrive --monitor.

This is one way to do things, but is also a sure fire way to screw up your local sync state. Do not use --resync unless the application tells you to use this option. What --resync does is it deletes your 'known' sync state, thus, the client has no idea if your local files are the right ones, or if the remote are the right ones - thus you can get into a data loss situation very quickly given your usage scenario.

The best solution for you that I can recommend is to have a startup script like you suggested, but containing the following:

# Upload local changes .. something may have been changed when 'offline'
onedrive --synchronize --upload-only
# Download any changes from OneDrive now that local changes have been uploaded
onedrive --synchronize --download-only
# Start systemd sync process to monitor local filesystem and OneDrive changes
systemctl --user start onedrive

The alternative here is for you to use https://github.com/jstaf/onedriver - as this implements the 'on-demand' functionality, meaning you are only ever going to be editing the file as it exists on OneDrive. I do not know how it would handle your scenario (2 systems, editing same file) and how OneDrive will interpret those edits.

[tom-andersson]

Thank you very much @abraunegg. I have set the script you suggested up and in testing I was able to confirm that the changes on either machine propagate to the other via OneDrive upon running the startup script. I haven't tested the 'changes made offline also get uploaded' aspect, but I expect that should work too. This is very helpful functionality for my workflow.

With the systemctl process running in the background, could you confirm that powering off the laptop will end the process safely? I.e. no manual systemctl --user stop onedrive command is needed?

Thanks also for the note about resync being a bad idea unless prompted. That might explain why my local files were being uploaded with an extended filename.

It does strike me as strange that --resync upon a small configuration change (which I've been doing several times an hour while setting this up) requires the whole local database to be regenerated (if that's the right terminology). I have thousands of files in my OneDrive and upon resync I get thousands of Requested file to upload exists on OneDrive - local database is out of sync for this file: <filename> after --resync, and it takes a few minutes and triggers a HTTP request throttling pause. However, I'm sure there's a good reason for this!

[abraunegg]

@tom-andersson

With the systemctl process running in the background, could you confirm that powering off the laptop will end the process safely? I.e. no manual systemctl --user stop onedrive command is needed?

This totally depends on the Linux distribution you are using. I would refer to your OS documentation on how services are stopped on shutdown - in what order, how this is triggered and so on.

This being said, if you are using Ubuntu or Ubuntu based and installed from a package - you must be critically aware of the following: https://github.com/abraunegg/onedrive/blob/master/docs/ubuntu-package-install.md#known-issues-with-installing-from-the-above-packages

It does strike me as strange that --resync upon a small configuration change (which I've been doing several times an hour while setting this up) requires the whole local database to be regenerated

The 'trigger' depends on what option is being changed. If you are changing 'skip_dir' or 'skip_file' or 'sync_list' then yes (these are the main triggers), because now your local DB contains data for files that need to be skipped, and, we do not want to be checking them or referring to them.

When making changes, you 'should' use --dry-run and test all your changes before doing a sync.

I have thousands of files in my OneDrive and upon resync I get thousands of Requested file to upload exists on OneDrive - local database is out of sync for this file: <filename> after --resync, and it takes a few minutes and triggers a HTTP request throttling pause. However, I'm sure there's a good reason for this!

Yes - this is a side effect of --resync but this there is no way to avoid this - except for using --dry-run to test your configuration changes first.

[erenoglu]

In --monitor mode the client will monitor the 'server' for changes, but will only do this at predefined intervals - 300 seconds by default. This value is user configurable

Hi @abraunegg, why doesn't the --monitor do its work by "receiving" push messages, ie when something changes on server, server shall "push" a message to onedrive client instead of client "pulling" if there's change every 300 seconds? This would make downloads from server instant or very minimal delay, since client would be notified. Is there no function in the Onedrive API or not yet implemented?
I'm used to this from Seafile which I used to use and now testing onedrive client I found that indeed it takes max 300 seconds for my web based changes to appear on my PC.

[abraunegg]

@erenoglu

why doesn't the --monitor do its work by "receiving" push messages, ie when something changes on server, server shall "push" a message to onedrive client instead of client "pulling" if there's change every 300 seconds?

Because this is how --monitor works.

Now, if you want to 'monitor' the server for changes, you need to enable a 'webhook'. When you do this, the client subscribes to all changes being performed online, and are processed within 1 second of the notification being received. Please read the documentation for further details: https://github.com/abraunegg/onedrive/blob/master/docs/USAGE.md#use-webhook-to-subscribe-to-remote-updates-in-monitor-mode

[erenoglu]

Thanks @abraunegg. Even if I can manage a reverse proxy myself when at home or even on this laptop, this probably won't help when I'm outside roaming around in a business trip, while in a hotel or other corporate with firewalls etc where I can't secure a public access to my installation.
Would this be the way the official client also works? I was hopeful if there was a two-way comms pipe when client connects to the server and server could use that to push messages about file changes to client.

[abraunegg]

@erenoglu
The aspect you are then looking for is this: #2360

[erenoglu]

yes, thanks.