Could not open file "wikidata.meta-data.json" for reading. Possible causes: The file does not exist or the permissions are insufficient.

[harej]

I am setting up Wikidata following your instructions. It is looking for a wikidata.meta-data.json even though none of the prior steps created this file, nor does your documentation explain where this file is or how to produce one. I also do not know why it is looking in that path, unless that is the path inside of the container.

(venv) jh@orb-qlever:/app/qlever$ qlever start

To enable autocompletion, run the following command, and consider adding it to your `.bashrc` or `.zshrc`:

eval "$(register-python-argcomplete qlever)" && export QLEVER_ARGCOMPLETE_ENABLED=1


Command: start

docker run -d --restart=unless-stopped -u $(id -u):$(id -g) -v /etc/localtime:/etc/localtime:ro -v $(pwd):/index -p 7001:7001 -w /index --name qlever.server.wikidata --init --entrypoint bash adfreiburg/qlever -c 'ServerMain -i wikidata -j 8 -p 7001 -m 128G -c 100G -e 5G -k 200 -s 600s -a wikidata_g6wp2BZaNoSd > wikidata.server-log.txt 2>&1'

Follow wikidata.server-log.txt until the server is ready (Ctrl-C stops following the log, but NOT the server)

2025-12-22 15:20:36.650 - INFO: QLever Server, compiled on Sat Dec 20 02:26:59 UTC 2025 using git hash 394482
2025-12-22 15:20:36.652 - INFO: Initializing server ...
2025-12-22 15:20:36.653 - ERROR: Could not open file "wikidata.meta-data.json" for reading. Possible causes: The file does not exist or the permissions are insufficient. The absolute path is "/index/wikidata.meta-data.json".
tail: wikidata.server-log.txt: file truncated
2025-12-22 15:20:37.269 - INFO: QLever Server, compiled on Sat Dec 20 02:26:59 UTC 2025 using git hash 394482
2025-12-22 15:20:37.270 - INFO: Initializing server ...
2025-12-22 15:20:37.271 - ERROR: Could not open file "wikidata.meta-data.json" for reading. Possible causes: The file does not exist or the permissions are insufficient. The absolute path is "/index/wikidata.meta-data.json".
tail: wikidata.server-log.txt: file truncated
2025-12-22 15:20:37.910 - INFO: QLever Server, compiled on Sat Dec 20 02:26:59 UTC 2025 using git hash 394482
2025-12-22 15:20:37.912 - INFO: Initializing server ...
2025-12-22 15:20:37.913 - ERROR: Could not open file "wikidata.meta-data.json" for reading. Possible causes: The file does not exist or the permissions are insufficient. The absolute path is "/index/wikidata.meta-data.json".
tail: wikidata.server-log.txt: file truncated
2025-12-22 15:20:38.628 - INFO: QLever Server, compiled on Sat Dec 20 02:26:59 UTC 2025 using git hash 394482
2025-12-22 15:20:38.630 - INFO: Initializing server ...
2025-12-22 15:20:38.631 - ERROR: Could not open file "wikidata.meta-data.json" for reading. Possible causes: The file does not exist or the permissions are insufficient. The absolute path is "/index/wikidata.meta-data.json".
Ctrl-C pressed, exiting ...

(venv) jh@orb-qlever:/app/qlever$ 

[hannahbast]

@harej If the file is not there, then the index build did not complete. What are the last lines of the log of your .index-log.txt?

And which machine are you using (processor, number of cores, RAM, disk space, operating system)

[harej]

The VM provisioned for this project has 256 GB of RAM, 64 cores, and 8 TB of storage available (hard drive zpool with SSDs for L2ARC).

2025-12-21 18:50:58.396 - INFO: QLever IndexBuilder, compiled on Sat Dec 20 02:26:59 UTC 2025 using git hash 394482
2025-12-21 18:50:58.401 - INFO: You specified "locale = en_US" and "ignore-punctuation = 1"
2025-12-21 18:50:58.401 - INFO: You specified "ascii-prefixes-only = true", which enables faster parsing for well-behaved TTL files
2025-12-21 18:50:58.401 - INFO: You specified "num-triples-per-batch = 5,000,000", choose a lower value if the index builder runs out of memory
2025-12-21 18:50:58.401 - INFO: By default, integers that cannot be represented by QLever will throw an exception
2025-12-21 18:50:58.401 - INFO: Processing triples from 3 input streams ...
2025-12-21 18:50:58.478 - INFO: Parsing input triples and creating partial vocabularies, one per batch ...
2025-12-21 23:08:43.628 - INFO: Triples parsed: 20,809,442,995 [average speed 1.3 M/s, last batch 1.6 M/s, fastest 2.2 M/s, slowest 0.3 M/s] 
2025-12-21 23:08:45.108 - INFO: Number of triples created (including QLever-internal ones): 32,411,846,955 [may contain duplicates]
2025-12-21 23:08:45.108 - INFO: Number of partial vocabularies created: 4,071
2025-12-21 23:08:45.108 - INFO: Merging partial vocabularies ...
2025-12-21 23:10:02.766 - WARN: Total vocabulary order violated for "◌𖿰"@mul and "◌𞥊"@en
2025-12-21 23:10:02.766 - WARN: Total vocabulary order violated for "◌𖿱"@mul and "◌𞥉"@en

I am also getting the same issue on another VM that has 128 GB of RAM, 24 cores, and 3 TB of storage (1x NVMe SSD).

2025-12-22 15:06:41.270 - INFO: QLever IndexBuilder, compiled on Mon Dec 22 02:36:27 UTC 2025 using git hash 5b09c6
2025-12-22 15:06:41.270 - INFO: You specified "locale = en_US" and "ignore-punctuation = 1"
2025-12-22 15:06:41.270 - INFO: You specified "ascii-prefixes-only = true", which enables faster parsing for well-behaved TTL files
2025-12-22 15:06:41.270 - INFO: You specified "num-triples-per-batch = 5,000,000", choose a lower value if the index builder runs out of memory
2025-12-22 15:06:41.270 - INFO: By default, integers that cannot be represented by QLever will throw an exception
2025-12-22 15:06:41.270 - INFO: Processing triples from 3 input streams ...
2025-12-22 15:06:41.272 - INFO: Parsing input triples and creating partial vocabularies, one per batch ...
2025-12-22 17:01:35.037 - INFO: Triples parsed: 20,809,442,995 [average speed 3.0 M/s, last batch 3.1 M/s, fastest 3.5 M/s, slowest 2.4 M/s] 
2025-12-22 17:01:36.101 - INFO: Number of triples created (including QLever-internal ones): 32,411,846,955 [may contain duplicates]
2025-12-22 17:01:36.101 - INFO: Number of partial vocabularies created: 4,071
2025-12-22 17:01:36.101 - INFO: Merging partial vocabularies ...

[harej]

Operating system on both is Debian 13

[hannahbast]

@harej That is more than enough resources for building Wikidata with QLever. We need more information, here are three comments and two questions:

1. When the index build aborts without any error message, then the probability is high that it was killed by the operating system because it ran out of memory. Note that such events generally cannot be caught by the killed process, hence the absence of a proper error message from QLever. Sometimes you get an error message from the operating system, sometimes you don't

2. Please check if there is an error message in /var/log/syslog indicating that a process was killed because of memory. We recommend installing earlyoom. It is very easy to install (you install the package and it runs with meaningful settings and just works). Earlyoom makes sure that your machine never freezes and you get nice error messages in /var/log/syslog when a process is killed by earlyoom

3. Please update to the latest version of the qlever CLI (which is 0.5.36 at the time of this writing, see https://pypi.org/project/qlever). In earlier versions, the qlever index command sometimes swallowed error messages from the operating system, in case there were any at all

4. Were there any other processes running on the system when you were building the index? Or was the available RAM limited in other significant ways?

5. Exactly which Qleverfile did you use? Please copy and paste it. And how exactly did you build the index, with just qlever index or did you use any other arguments? If possible, paste the command line that was shown by qlever index (you did include the command line for qlever start)

[harej]

/var/log/syslog does not exist on either system.

There weren't other major processes as the VM (on each machine) was created specifically for Qlever and nothing else.

I just used qlever index without any other arguments.

Qleverfile on the VM with 128 GB of RAM:

[DEFAULT]
NAME = wikidata

[data]
GET_DATA_URL      = https://files.scatter.red/wikimedia/other/wikibase/wikidatawiki
GET_DATA_CMD      = curl -LRC - -O ${GET_DATA_URL}/20251215/wikidata-20251215-all-BETA.ttl.bz2 -O ${GET_DATA_URL}/20251219/wikidata-20251219-lexemes-BETA.ttl.bz2 2>&1 | tee wikidata.download-log.txt && curl -sL ${GET_DATA_URL}/dcatap.rdf | docker run -i --rm -v $$(pwd):/data stain/jena riot --syntax=RDF/XML --output=NT /dev/stdin > dcatap.nt
DATE_WIKIDATA     = $$(date -r wikidata-20251215-all-BETA.ttl.bz2 +%d.%m.%Y || echo "NO_DATE")
DATE_WIKIPEDIA    = $$(date -r wikipedia-abstracts.nt +%d.%m.%Y || echo "NO_DATE")
DESCRIPTION       = Complete Wikidata, from ${GET_DATA_URL} (wikidata-20251215-all-BETA.ttl.bz2 and wikidata-20251219-lexemes-BETA.ttl.bz2), version ${DATE_WIKIDATA}

[index]
INPUT_FILES      = wikidata-20251215-all-BETA.ttl.bz2 wikidata-20251219-lexemes-BETA.ttl.bz2 dcatap.nt
MULTI_INPUT_JSON = [{ "cmd": "lbzcat -n 4 wikidata-20251215-all-BETA.ttl.bz2", "format": "ttl", "parallel": "true" },
                    { "cmd": "lbzcat -n 1 wikidata-20251219-lexemes-BETA.ttl.bz2", "format": "ttl", "parallel": "false" },
                    { "cmd": "cat dcatap.nt", "format": "nt", "parallel": "false" }]
SETTINGS_JSON    = { "languages-internal": [], "prefixes-external": [""], "locale": { "language": "en", "country": "US", "ignore-punctuation": true }, "ascii-prefixes-only": true, "num-triples-per-batch": 5000000 }
STXXL_MEMORY     = 64G

[server]
PORT                        = 7001
ACCESS_TOKEN                = ${data:NAME}_TaY4nOASyOru
MEMORY_FOR_QUERIES          = 64G
CACHE_MAX_SIZE              = 50G
CACHE_MAX_SIZE_SINGLE_ENTRY = 5G
TIMEOUT                     = 600s

[runtime]
SYSTEM = docker
IMAGE  = adfreiburg/qlever

[ui]
UI_CONFIG = wikidata

On this VM I have updated Qlever to 0.5.36 and installed earlyoom. With those in place I am reattempting the index command, which expands out to the following:

echo '{ "languages-internal": [], "prefixes-external": [""], "locale": { "language": "en", "country": "US", "ignore-punctuation": true }, "ascii-prefixes-only": true, "num-triples-per-batch": 5000000 }' > wikidata.settings.json
docker run --rm -u $(id -u):$(id -g) -v /etc/localtime:/etc/localtime:ro -v $(pwd):/index -w /index --name qlever.index.wikidata --init --entrypoint bash adfreiburg/qlever -c 'ulimit -Sn 500000 && IndexBuilderMain -i wikidata -s wikidata.settings.json --vocabulary-type on-disk-compressed -f <(lbzcat -n 4 wikidata-20251215-all-BETA.ttl.bz2) -g - -F ttl -p true -f <(lbzcat -n 1 wikidata-20251219-lexemes-BETA.ttl.bz2) -g - -F ttl -p false -f <(cat dcatap.nt) -g - -F nt -p false --stxxl-memory 64G 2>&1 | tee wikidata.index-log.txt'

Resulting output:

2025-12-23 15:02:28.803 - INFO: QLever IndexBuilder, compiled on Mon Dec 22 02:36:27 UTC 2025 using git hash 5b09c6
2025-12-23 15:02:28.803 - INFO: You specified "locale = en_US" and "ignore-punctuation = 1"
2025-12-23 15:02:28.803 - INFO: You specified "ascii-prefixes-only = true", which enables faster parsing for well-behaved TTL files
2025-12-23 15:02:28.803 - INFO: You specified "num-triples-per-batch = 5,000,000", choose a lower value if the index builder runs out of memory
2025-12-23 15:02:28.803 - INFO: By default, integers that cannot be represented by QLever will throw an exception
2025-12-23 15:02:28.803 - INFO: Processing triples from 3 input streams ...
2025-12-23 15:02:28.804 - INFO: Parsing input triples and creating partial vocabularies, one per batch ...
2025-12-23 16:54:04.844 - INFO: Triples parsed: 20,809,442,995 [average speed 3.1 M/s, last batch 3.1 M/s, fastest 3.8 M/s, slowest 2.4 M/s] 
2025-12-23 16:54:05.676 - INFO: Number of triples created (including QLever-internal ones): 32,411,846,955 [may contain duplicates]
2025-12-23 16:54:05.676 - INFO: Number of partial vocabularies created: 4,070
2025-12-23 16:54:05.676 - INFO: Merging partial vocabularies ...

Still the same error.

[hannahbast]

@harej That is mysterious, to say the least. Can you try the following:

1. git clone git@github.com:ad-freiburg/qlever in a directory of your choice
2. Install the packages from line 10 of this Dockerfile: https://github.com/ad-freiburg/qlever/blob/master/Dockerfiles/Dockerfile.debian#L10
3. Create a build directory, go there, and execute the cmake command from line 18 of that Dockerfile: https://github.com/ad-freiburg/qlever/blob/master/Dockerfiles/Dockerfile.debian#L18
4. Install the remaining packages from line 24 (maybe not needed): https://github.com/ad-freiburg/qlever/blob/master/Dockerfiles/Dockerfile.debian#L24
5. Include the build directory in your PATH
6. See if you can execute ServerMain and IndexBuilderMain (QLever's main binaries)

If that works, set SYSTEM = native in your Qleverfile and try again

I am tagging @ddeboer who had a related problem on macOS, where it turned out to be a problem with the Docker image ("Illegal instruction")

[harej]

1. git clone https://github.com/ad-freiburg/qlever
2. cd qlever
3. sudo apt update
4. sudo apt-get install -y build-essential cmake libicu-dev tzdata pkg-config uuid-runtime uuid-dev git libjemalloc-dev ninja-build libzstd-dev libssl-dev libboost-dev libboost-program-options-dev libboost-iostreams-dev libboost-url-dev libboost-container-dev
5. mkdir build
6. cd build
7. cmake -DCMAKE_BUILD_TYPE=Release -DLOGLEVEL=INFO -DUSE_PARALLEL=true -D_NO_TIMING_TESTS=ON -GNinja .. && ninja
8. sudo apt-get update
9. sudo apt-get install -y wget python3-yaml unzip curl bzip2 pkg-config libicu-dev python3-icu libgomp1 uuid-runtime make lbzip2 libjemalloc-dev libzstd-dev libssl-dev libboost-dev libboost-program-options-dev libboost-iostreams-dev libboost-url-dev libboost-container-dev

Should I execute those binaries with any arguments or just as-is?

[hannahbast]

@harej Just as is. If you see some help info, they work in principle

[harej]

Does this look good to you?

(venv) jh@orb-qlever:/app/qlever-debug/qlever/build$ ./ServerMain
Error in command-line argument: the option '--index-basename' is required but missing
Options for ServerMain:
  -h [ --help ]                         Produce this help message.
...

(venv) jh@orb-qlever:/app/qlever-debug/qlever/build$ ./IndexBuilderMain
Error in command-line argument: the option '--index-basename' is required but missing
Options for IndexBuilderMain:
  -h [ --help ]                         Produce this help message.
...

[hannahbast]

@harej Perfect. So set your PATH so that they are also found with just ServerMain and IndexBuilderMain (without the ./), set SYSTEM = native in your Qleverfile, and execute qlever index again. You should then see a different command line (without Docker). And let's see what happens then. I hope that one of the following two happens:

1. The index build runs through without any problems
2. It crashes at the same point, but we get some kind of error message from the operating system now

[hannahbast]

@harej If the problem persists, please remove the "ascii-prefixes-only": true from the SETTINGS_JSON in the Qleverfile and then start qlever index again

[hannahbast]

@harej Did it work?

[harej]

jh@orb-qlever:~$ cd /app/qlever
jh@orb-qlever:/app/qlever$ ls
dcatap.nt  Qleverfile  venv  wikidata-20251215-all-BETA.ttl.bz2  wikidata-20251219-lexemes-BETA.ttl.bz2
jh@orb-qlever:/app/qlever$ which qlever
/home/jh/.local/bin/qlever
jh@orb-qlever:/app/qlever$ source venv/bin/activate
(venv) jh@orb-qlever:/app/qlever$ which ServerMain
/app/qlever-debug/qlever/build/ServerMain
(venv) jh@orb-qlever:/app/qlever$ which IndexBuilderMain
/app/qlever-debug/qlever/build/IndexBuilderMain
(venv) jh@orb-qlever:/app/qlever$ which qlever
/app/qlever/venv/bin/qlever
(venv) jh@orb-qlever:/app/qlever$ qlever --version

To enable autocompletion, run the following command, and consider adding it to your `.bashrc` or `.zshrc`:

eval "$(register-python-argcomplete qlever)" && export QLEVER_ARGCOMPLETE_ENABLED=1

qlever **0.5.36**
(venv) jh@orb-qlever:/app/qlever$ tail Qleverfile
CACHE_MAX_SIZE              = 50G
CACHE_MAX_SIZE_SINGLE_ENTRY = 5G
TIMEOUT                     = 600s

[runtime]
SYSTEM = native
IMAGE  = adfreiburg/qlever

[ui]
UI_CONFIG = wikidata

Output from qlever index:

2025-12-23 18:30:25.025 - INFO: QLever IndexBuilder, compiled on Tue Dec 23 06:03:53 PM PST 2025 using git hash 5b09c6
2025-12-23 18:30:25.025 - INFO: You specified "locale = en_US" and "ignore-punctuation = 1"
2025-12-23 18:30:25.026 - INFO: You specified "ascii-prefixes-only = true", which enables faster parsing for well-behaved TTL files
2025-12-23 18:30:25.026 - INFO: You specified "num-triples-per-batch = 5,000,000", choose a lower value if the index builder runs out of memory
2025-12-23 18:30:25.026 - INFO: By default, integers that cannot be represented by QLever will throw an exception
2025-12-23 18:30:25.026 - INFO: Processing triples from 3 input streams ...
2025-12-23 18:30:25.027 - INFO: Parsing input triples and creating partial vocabularies, one per batch ...
2025-12-23 20:21:26.589 - INFO: Triples parsed: 20,809,442,995 [average speed 3.1 M/s, last batch 3.1 M/s, fastest 3.8 M/s, slowest 2.4 M/s] 
2025-12-23 20:21:27.497 - INFO: Number of triples created (including QLever-internal ones): 32,411,846,955 [may contain duplicates]
2025-12-23 20:21:27.497 - INFO: Number of partial vocabularies created: 4,070
2025-12-23 20:21:27.497 - INFO: Merging partial vocabularies ...
2025-12-23 20:21:35.623 - WARN: Total vocabulary order violated for "◌𖿰"@mul and "◌𞥊"@en
2025-12-23 20:21:35.623 - WARN: Total vocabulary order violated for "◌𖿱"@mul and "◌𞥉"@en
2025-12-23 20:24:00.148 - WARN: Total vocabulary order violated for "Category:ﾟ"@en and "Category:？"@enstest 3.0 M/s, slowest 0.8 M/s] 
2025-12-23 20:24:04.396 - WARN: Total vocabulary order violated for "Category:帶「ः」的詞"@zh and "Category:帶「◌̃」的詞"@zhest 0.8 M/s] 
2025-12-23 20:52:22.192 - INFO: Words merged: 3,652,795,466 [average speed 2.0 M/s, last batch 1.2 M/s, fastest 3.6 M/s, slowest 0.2 M/s] 
2025-12-23 20:52:29.465 - INFO: Finished writing compressed internal vocabulary, size = 73.9 GB [uncompressed = 233.9 GB, ratio = 31%]
2025-12-23 20:52:29.468 - INFO: Number of words in external vocabulary: 3,652,795,466
2025-12-23 20:52:37.703 - INFO: Converting triples from local IDs to global IDs ...
2025-12-23 21:18:25.237 - INFO: Triples converted: 32,411,846,955 [average speed 20.9 M/s, last batch 36.0 M/s, fastest 38.1 M/s, slowest 2.4 M/s] 
2025-12-23 21:18:40.116 - INFO: Creating permutations SPO and SOP ...
2025-12-23 22:01:59.079 - INFO: Number of inputs to `uniqueView`: 20,809,442,9959 M/s, last batch 23.9 M/s, fastest 59.4 M/s, slowest 0.8 M/s] 
2025-12-23 22:01:59.079 - INFO: Number of unique elements: 20,442,721,721
2025-12-23 22:01:59.079 - INFO: Triples sorted: 20,442,721,721 [average speed 7.9 M/s, last batch 23.9 M/s, fastest 59.4 M/s, slowest 0.8 M/s] 
2025-12-23 22:02:00.006 - INFO: Statistics for SPO: #relations = 2,249,443,452, #blocks = 436,559, #triples = 20,442,721,721
2025-12-23 22:02:00.007 - INFO: Statistics for SOP: #relations = 2,249,443,452, #blocks = 436,559, #triples = 20,442,721,721
2025-12-23 22:02:07.362 - INFO: Number of distinct patterns: 10,317,789
2025-12-23 22:02:07.362 - INFO: Number of subjects with pattern: 2,249,443,452 [all]
2025-12-23 22:02:07.362 - INFO: Total number of distinct subject-predicate pairs: 12,518,654,766
2025-12-23 22:02:07.362 - INFO: Average number of predicates per subject: 5.6
2025-12-23 22:02:07.362 - INFO: Average number of subjects per predicate: 204,992
2025-12-23 22:02:23.118 - INFO: Creating permutations OSP and OPS ...
2025-12-23 22:57:36.811 - INFO: Triples sorted: 20,442,721,721 [average speed 6.2 M/s, last batch 15.1 M/s, fastest 25.3 M/s, slowest 0.1 M/s] 
2025-12-23 22:57:37.094 - INFO: Statistics for OSP: #relations = 3,693,791,940, #blocks = 571,882, #triples = 20,442,721,721
2025-12-23 22:57:37.094 - INFO: Statistics for OPS: #relations = 3,693,791,940, #blocks = 571,882, #triples = 20,442,721,721
2025-12-23 22:57:39.269 - INFO: Adding 2,249,443,452 triples to the POS and PSO permutation for the internal `ql:has-pattern` ...
2025-12-23 22:59:37.431 - INFO: Creating permutations PSO and POS ...
2025-12-23 23:25:00.965 - INFO: Number of inputs to `uniqueView`: 13,851,847,412 M/s, last batch 6.3 M/s, fastest 35.5 M/s, slowest 0.0 M/s]   
2025-12-23 23:25:00.965 - INFO: Number of unique elements: 8,967,011,012
2025-12-23 23:25:00.965 - INFO: Triples sorted: 8,967,011,012 [average speed 5.9 M/s, last batch 6.3 M/s, fastest 35.5 M/s, slowest 0.0 M/s] 
2025-12-23 23:25:01.231 - INFO: Statistics for PSO: #relations = 25,557, #blocks = 289,292, #triples = 8,967,011,012
2025-12-23 23:25:01.231 - INFO: Statistics for POS: #relations = 25,557, #blocks = 289,292, #triples = 8,967,011,012
2025-12-23 23:25:07.637 - INFO: Creating permutations PSO and POS ...
2025-12-24 00:33:44.121 - INFO: Triples sorted: 20,442,721,721 [average speed 5.0 M/s, last batch 8.7 M/s, fastest 29.7 M/s, slowest 0.0 M/s]  
2025-12-24 00:33:44.503 - INFO: Statistics for PSO: #relations = 61,069, #blocks = 658,487, #triples = 20,442,721,721
2025-12-24 00:33:44.503 - INFO: Statistics for POS: #relations = 61,069, #blocks = 658,487, #triples = 20,442,721,721
2025-12-24 00:33:47.735 - INFO: Index build completed

Looks like that worked. Why it didn't work in Docker, who knows.

Now if there is a way to run an updater that would be great.

[hannahbast]

@harej For starters, you can simply run qlever update-wikidata in the directory with your index and Qleverfile. I will write more about that later in a separate comment

Great that the index build worked. It shows that there is something going on with the Docker image. We recently switched from slow QEMU builds to faster native builds, but there seem to be problems with those @RobinTF