update-wikidata might loose updates

[pfps]

If the QLever server has crashed, it appears that the update script keeps on trying to send updates and does not hold until the server is back. If the server comes back then my guess is that the updates that were supposed to be sent when it was down are dropped. If new update comes in my understanding is that the update times will be sent corresponding to this new update and even restarting the update script will not resend the updates that were not sent.

Given that updates are under development it might not be worthwhile to fix the script now.

[hannahbast]

Yes, qlever update-wikidata does not handle that case properly. It could handle it as follows:

1. The latest version (still a PR) is not only writing a timestamp but the exact offset of the next message to be consumed: https://qlever.dev/wikidata/Fagn2C . This is important for precise resuming and was a minor bug before (there can be multiple messages with the exact same timestamp, the timestamps have only millisecond resolution)

2. Before processing a new batch, the command could query the backend for the latest offset. When that is not equal to the offset of the next message it reads, something is wrong. In particular, when the offset from the backend is smaller, it could conclude that the server restarted and resume from there

Given the latest changes that would actually pretty easy to integrate, so maybe I will just do that

But yes, so far this script is a proof of concept and a working solution. The final product will look differently. And it's also unclear whether the Wikidata update stream will keep its current form

[pfps]

It's good enough for now and good enough for me to get some idea of performance. Right now I'm running a benchmark alongside full-out updates to get a dump up-to-date, which means that QLever is processing updates about half the time because the update script takes about the same amount of time to read and prepare the updates as QLever takes to process them. The slowdown on the queries so far (based on about 1/4 of the benchmarks) is about 10-15%, which is certainly tolerable.