Updater stands still and refuses to exit

[harej]

jh@orb-qlever:/app/qlever$ qlever update-wikidata

To enable autocompletion, run the following command, and consider adding it to your `.bashrc` or `.zshrc`:

eval "$(register-python-argcomplete qlever)" && export QLEVER_ARGCOMPLETE_ENABLED=1


Command: update-wikidata

SINCE=$(curl -s http://orb-qlever:7001 -H "Accept: text/csv" -H "Content-type: application/sparql-query" --data "PREFIX wikibase: <http://wikiba.se/ontology#> PREFIX schema: <http://schema.org/> SELECT * WHERE { { SELECT (MIN(?date_modified) AS ?updates_complete_until) { wikibase:Dump schema:dateModified ?date_modified } } UNION { wikibase:Dump wikibase:updatesCompleteUntil ?updates_complete_until } } ORDER BY DESC(?updates_complete_until) LIMIT 1" | sed 1d)
Process SSE stream from https://stream.wikimedia.org/v2/stream/rdf-streaming-updater.mutation.v2 in batches of up to 100,000 messages 

Press Ctrl+C to finish and exit gracefully

The progress bar stays frozen at 0/100000. Pressing Ctrl+C to cancel just tells me to monitor my blood pressure while refusing to actually exit.

[hannahbast]

@harej Yes, we thought it's always good to give valuable health advice.

Is http://orb-qlever:7001 actually reachable? If yes, what does the first command in blue return:

curl -s http://orb-qlever:7001 -H "Accept: text/csv" -H "Content-type: application/sparql-query" --data "PREFIX wikibase: <http://wikiba.se/ontology#> PREFIX schema: <http://schema.org/> SELECT * WHERE { { SELECT (MIN(?date_modified) AS ?updates_complete_until) { wikibase:Dump schema:dateModified ?date_modified } } UNION { wikibase:Dump wikibase:updatesCompleteUntil ?updates_complete_until } } ORDER BY DESC(?updates_complete_until) LIMIT 1" | sed 1d

[harej]

From /etc/hosts:

(venv) jh@orb-qlever:/app/qlever$ cat /etc/hosts
127.0.0.1       localhost
127.0.1.1       orb-qlever

(venv) jh@orb-qlever:/app/qlever$ curl -s http://orb-qlever:7001 -H "Accept: text/csv" -H "Content-type: application/sparql-query" --data "PREFIX wikibase: <http://wikiba.se/ontology#> PREFIX schema: <http://schema.org/> SELECT * WHERE { { SELECT (MIN(?date_modified) AS ?updates_complete_until) { wikibase:Dump schema:dateModified ?date_modified } } UNION { wikibase:Dump wikibase:updatesCompleteUntil ?updates_complete_until } } ORDER BY DESC(?updates_complete_until) LIMIT 1" | sed 1d
2025-12-15T23:00:35Z

[hannahbast]

@harej That looks good. Next question is whether the Wikidata stream works for you:

curl -s "https://stream.wikimedia.org/v2/stream/rdf-streaming-updater.mutation.v2?since=2025-12-15T23:00:35Z"

This should produce a never-ending stream of messages

And qlever --version outputs qlever 0.5.36. correct?

[harej]

The version is still 0.5.36.

(venv) jh@orb-qlever:/app/qlever$ curl -s "https://stream.wikimedia.org/v2/stream/rdf-streaming-updater.mutation.v2?since=2025-12-15T23:00:35Z"
:ok

event: message
id: [{"topic":"codfw.rdf-streaming-updater.mutation","partition":0,"timestamp":1765839635313}]
data: {"dt":"2025-12-15T23:00:34Z","entity_id":"Q124352016","meta":{"domain":"www.wikidata.org","dt":"2025-12-15T23:00:35.312647Z","request_id":"f6833d6d-9129-447f-ad27-c3977bdfedd8","stream":"rdf-streaming-updater.mutation.v2","id":"607ccde5-8a0d-45f5-adb9-2f261c038e74","topic":"codfw.rdf-streaming-updater.mutation","partition":0,"offset":963623872},"operation":"diff","rdf_added_data":{"data":"\n<http://www.wikidata.org/entity/Q124352016> <http://schema.org/dateModified> \"2025-12-15T23:00:34Z\"^^<http://www.w3.org/2001/XMLSchema#dateTime> ;\n\t<http://schema.org/version> 2442837808 ;\n\t<http://schema.org/description> \"page de catégorie d'un projet Wikimédia\"@fr .\n","mime_type":"text/turtle"},"rdf_deleted_data":{"data":"\n<http://www.wikidata.org/entity/Q124352016> <http://schema.org/description> \"page de catégorie de Wikimedia\"@fr ;\n\t<http://schema.org/version> 2431544317 ;\n\t<http://schema.org/dateModified> \"2025-11-19T12:11:25Z\"^^<http://www.w3.org/2001/XMLSchema#dateTime> .\n","mime_type":"text/turtle"},"rev_id":2442837808,"sequence":0,"sequence_length":1,"$schema":"/mediawiki/wikibase/entity/rdf_change/2.0.0"}

and so on

So what gets me is that these commands work in isolation, but then actually trying to update-wikidata results in silence.

[hannahbast]

@harej It's not easy from afar to say what's going wrong here. My hunch is that you are not getting messages from the update stream, when connecting to it via qlever update-wikidata. Here are some comments and things you could do:

1. The Wikidata update stream is sensitive to exactly which headers are being sent, and the policy of the stream has changed multiple times in the past months. With curl, different headers are sent than from qlever update-wikidata. I don't know if the problem lies there, but it's a candidate

2. The command has a --since argument, which takes a date in the format 2025-12-15T23:00:35Z. Can you try out a few dates and see if the command gets stuck for all of them?

3. The command also has a --offset argument, which takes an integer. There is also --num-messages for just processing a fixed number of messages. Can you look at the update stream and pick any offset you encounter and then try something like qlever update-wikidata --num-messages 1 --offset 1100467782

[hannahbast]

@harej Any news on this?

[harej]

Trying out different variants with --since, --num-messages 1 results in the same behavior, which is to say nothing happens.

If it's a user agent issue, wouldn't that result in a much louder error from the stream, or does the QLever updater suppress those errors? Is there any way I can try a different user agent?

[harej]

Or: you can tell me what user agent the QLever updater uses, and I can manually submit a curl request using that header.

[harej]

(venv) jh@orb-qlever:/app/qlever$ curl -H "Accept: text/event-stream" -H "User-Agent: qlever update-wikidata" https://stream.wikimedia.org/v2/stream/rdf-streaming-updater.mutation.v2                                    
:ok                                                                                                                                                                                                                       
                                                                                                                                                                                                                          
event: message                                                                                                                                                                                                            
id: [{"topic":"codfw.rdf-streaming-updater.mutation","partition":0,"timestamp":1766792909022}]                                                                                                                            
data: {"dt":"2025-12-26T23:48:28Z","entity_id":"Q137455909","meta":{"domain":"www.wikidata.org","dt":"2025-12-26T23:48:29.021701Z","request_id":"7787cc2c-5d6e-426a-9293-9cd5b7920e94","stream":"rdf-streaming-updater.mut
ation.v2","id":"54949570-48d5-4fa5-9b6a-8b3df257928d","topic":"codfw.rdf-streaming-updater.mutation","partition":0,"offset":968323748},"operation":"diff","rdf_added_data":{"data":"\n<http://www.wikidata.org/entity/Q137
455909> <http://wikiba.se/ontology#statements> 15 ;\n\t<http://www.wikidata.org/prop/direct/P495> <http://www.wikidata.org/entity/Q38> ;\n\t<http://schema.org/dateModified> \"2025-12-26T23:48:28Z\"^^<http://www.w3.org/
2001/XMLSchema#dateTime> .\n\n<http://www.wikidata.org/entity/statement/Q137455909-d50be98a-4770-35ea-389c-0e3d8cb1b948> <http://www.wikidata.org/prop/statement/P495> <http://www.wikidata.org/entity/Q38> .\n\n<http://w
ww.wikidata.org/entity/Q137455909> <http://www.wikidata.org/prop/P495> <http://www.wikidata.org/entity/statement/Q137455909-d50be98a-4770-35ea-389c-0e3d8cb1b948> .\n\n<http://www.wikidata.org/entity/statement/Q13745590
9-d50be98a-4770-35ea-389c-0e3d8cb1b948> <http://wikiba.se/ontology#rank> <http://wikiba.se/ontology#NormalRank> ;\n\ta <http://wikiba.se/ontology#BestRank> .\n\n<http://www.wikidata.org/entity/Q137455909> <http://schem
a.org/version> 2447585048 .\n","mime_type":"text/turtle"},"rdf_deleted_data":{"data":"\n<http://www.wikidata.org/entity/Q137455909> <http://schema.org/dateModified> \"2025-12-26T22:13:58Z\"^^<http://www.w3.org/2001/XML
Schema#dateTime> ;\n\t<http://schema.org/version> 2447565900 ;\n\t<http://wikiba.se/ontology#statements> 14 .\n","mime_type":"text/turtle"},"rev_id":2447585048,"sequence":0,"sequence_length":1,"$schema":"/mediawiki/wik
ibase/entity/rdf_change/2.0.0"} 

Is there any way to make the updater more explicit about what it is doing so that we can see at what point it is choking?

[hannahbast]

@harej It's just a Python script, and the code is here: https://github.com/qlever-dev/qlever-control

See https://github.com/qlever-dev/qlever-control?tab=readme-ov-file#for-developers for how to install it locally

Specifically, the code for the update-wikidata command is in https://github.com/qlever-dev/qlever-control/blob/main/src/qlever/commands/update_wikidata.py (each command has its own script). You can place log.info(...) at strategic places to see what's going on. Debugging Python scripts is relatively straightforward, since error messages provide the whole stack trace

Alternatively, if it's possible to provide guest access to the machine, I would be happy to take a look at what's going on

The code uses Python's requests_sse package to connect to the SSE stream. Maybe that package does something subtly different on your machine. But it's really hard to say as long as I can't reproduce the problem (and you are the first one to report it)

[hannahbast]

@harej Any updates [sic] on this?

[harej]

Not sure how my comments didn't post, but I would be happy to give you access to my VM. I just need a username and public key.

[hannahbast]

@harej Here you go

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICQRCINev8aEYNd6/DcL5uu4+QS5mNFIWvqOLQk1KaHH bast@informatik.uni-freiburg.de

[harej]

@hannahbast Connect with your username bast to 10.0.1.200 via station1001.usrdm1.scatter.red. In other words, ssh -J station1001.usrdm1.scatter.red bast@10.0.1.200. QLever is in /app/qlever. Let me know if there is anything you can't access.

[hannahbast]

@harej I tried it and get the following:

$ ssh -J station1001.usrdm1.scatter.red bast@10.0.1.200
bast@station1001.usrdm1.scatter.red: Permission denied (publickey).
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

EDIT: I could solve it by putting the following in my ~/.ssh/config:

Host station1001.usrdm1.scatter.red
  User bast
  IdentityFile ~/.ssh/id_ed25519_github
  IdentitiesOnly yes

Host orb-qlever-2
  HostName 10.0.1.200
  User bast
  ProxyJump station1001.usrdm1.scatter.red
  IdentityFile ~/.ssh/id_ed25519_github
  IdentitiesOnly yes