UpdateTool stops at 'ImdbRatingDatasetFactory.readData'

[x5nder]

UpdateTool always used to work without issues, until last week Plex notified me of a database corruption. After fixing this, UpdateTool always gets stuck at the same point:

[INFO ] - 2025-01-27 06:55:30 @ ImdbDockerImplementation$ImdbBatchJob.run: LIBRARIES => POST LIBRARY FILTERING
[INFO ] - 2025-01-27 06:55:30 @ ImdbDockerImplementation$ImdbBatchJob.lambda$run$3: Will process library Movies (ID=2) with agent: tv.plex.agents.movie and 5131 item(s).
[INFO ] - 2025-01-27 06:55:30 @ ImdbDockerImplementation$ImdbBatchJob.lambda$run$3: Will process library Movies (4K) (ID=12) with agent: tv.plex.agents.movie and 11 item(s).
[INFO ] - 2025-01-27 06:55:30 @ ImdbDockerImplementation$ImdbBatchJob.lambda$run$3: Will process library Documentaries (ID=25) with agent: tv.plex.agents.movie and 26 item(s).
[INFO ] - 2025-01-27 06:55:30 @ ImdbDockerImplementation$ImdbBatchJob.lambda$run$3: Will process library TV Shows (ID=5) with agent: tv.plex.agents.series and 28702 item(s).
[INFO ] - 2025-01-27 06:55:30 @ ImdbDatabaseSupport.testPlexSqliteBinaryVersion: Plex SQLite binary version: 3.39.4 | ATOMIC_INTRINSICS=1 | COMPILER=clang-11.0.1 | DEFAULT_AUTOVACUUM | DEFAULT_CACHE_SIZE=-2000 | DEFAULT_FILE_FORMAT=4 | DEFAULT_JOURNAL_SIZE_LIMIT=-1 | DEFAULT_MMAP_SIZE=0 | DEFAULT_PAGE_SIZE=4096 | DEFAULT_PCACHE_INITSZ=20 | DEFAULT_RECURSIVE_TRIGGERS | DEFAULT_SECTOR_SIZE=4096 | DEFAULT_SYNCHRONOUS=2 | DEFAULT_WAL_AUTOCHECKPOINT=1000 | DEFAULT_WAL_SYNCHRONOUS=2 | DEFAULT_WORKER_THREADS=0 | ENABLE_COLUMN_METADATA | ENABLE_DBPAGE_VTAB | ENABLE_EXPLAIN_COMMENTS | ENABLE_FTS3 | ENABLE_FTS3_PARENTHESIS | ENABLE_ICU | ENABLE_RTREE | ENABLE_UNLOCK_NOTIFY | MALLOC_SOFT_LIMIT=1024 | MAX_ATTACHED=10 | MAX_COLUMN=2000 | MAX_COMPOUND_SELECT=500 | MAX_DEFAULT_PAGE_SIZE=8192 | MAX_EXPR_DEPTH=2048 | MAX_FUNCTION_ARG=127 | MAX_LENGTH=1000000000 | MAX_LIKE_PATTERN_LENGTH=50000 | MAX_MMAP_SIZE=0x7fff0000 | MAX_PAGE_COUNT=1073741823 | MAX_PAGE_SIZE=65536 | MAX_SQL_LENGTH=1000000000 | MAX_TRIGGER_DEPTH=1000 | MAX_VARIABLE_NUMBER=32766 | MAX_VDBE_OP=250000000 | MAX_WORKER_THREADS=8 | MUTEX_PTHREADS | OMIT_DEPRECATED | SYSTEM_MALLOC | TEMP_STORE=1 | THREADSAFE=1 | 
[INFO ] - 2025-01-27 06:55:30 @ ImdbDatabaseSupport.<init>: NewExtraDataFormat has been identified as: true
[INFO ] - 2025-01-27 06:55:33 @ ImdbDockerImplementation$ImdbBatchJob.run: Library IDs on ignore list: [16]
[INFO ] - 2025-01-27 06:55:33 @ ImdbRatingDatasetFactory.readData: Reading data...

Any idea what could be wrong?

[mynttt]

Do you run this in a docker or on bare metal?

If it gets stuck at that point something very weird I/O related must happen on your system:

I just tested it on my machine (TM)

[INFO ] - 2025-02-08 01:21:45 @ ImdbRatingDatasetFactory.requestSet: IMDB Dataset has the timestamp: 1730459790615 and violates the update every 86400000 ms constraint. Refreshing dataset...
[INFO ] - 2025-02-08 01:21:45 @ ImdbRatingDatasetFactory.downloadData: Downloading IMDB rating set from: https://datasets.imdbws.com/title.ratings.tsv.gz
[INFO ] - 2025-02-08 01:21:45 @ ImdbRatingDatasetFactory.downloadData: Download succeeded @ ./__tmp_rating.gz
[INFO ] - 2025-02-08 01:21:45 @ ImdbRatingDatasetFactory.extractData: Extracting dataset...
[INFO ] - 2025-02-08 01:21:45 @ ImdbRatingDatasetFactory.extractData: Extraction completed.
[INFO ] - 2025-02-08 01:21:45 @ ImdbRatingDatasetFactory.readData: Reading data...
[INFO ] - 2025-02-08 01:21:47 @ ImdbRatingDatasetFactory.readData: 1531107 lines read.

and it is working well.

UpdateTool/src/main/java/updatetool/imdb/ImdbRatingDatasetFactory.java

Lines 154 to 167 in 117b2e1

	private static void readData(ImdbRatingDataset target) {
	Logger.info("Reading data...");
	try(var reader = Files.newBufferedReader(Main.PWD.resolve(RATING_SET))) {
	reader.readLine(); //Skip header
	String s;
	while((s = reader.readLine()) != null) {
	var split = s.split("\\s+");
	target.data.put(split[0], split[1]);
	}
	} catch(IOException e) {
	throw Utility.rethrow(e);
	}
	Logger.info("{} lines read.", target.data.size());
	}

Looking at the code; between Reading data... and xyz lines read is only a section that reads the lines from the file. The buffered reader would also fail with an exception in case it would fail to open the file.

So if it still appears for some reason the file is not being processed (?) - would be interesting to know what is written in the file and if it is actually even a valid TSV file.