Skip to content

IVF format fails search due to serialization failure #129551

Closed
@benwtrent

Description

@benwtrent

Elasticsearch Version

main

Installed Plugins

No response

Java Version

bundled

OS Version

any

Problem Description

ivf format fails to search due to invalid bytes in DocIdReader.

Steps to Reproduce

This occurred during a large vector indexing test, indexing about 100M+ vectors with aggressive merging.

Obviously, we have a serialization bug where we are miscounting bytes in the format.

Logs (if relevant)

[2025-06-17T02:58:50,200][WARN ][o.e.s.SearchService      ] [elasticsearch-0] [yt4RfBSmRcKGMzZS3IDE-w][msmarco-v2][0]: failed to execute search request for task [2783699]
org.elasticsearch.search.dfs.DfsPhaseExecutionException: Dfs Failed [Exception during dfs phase]
	at org.elasticsearch.search.dfs.DfsPhase.execute(DfsPhase.java:67) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.search.SearchService.executeDfsPhase(SearchService.java:654) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.search.SearchService.lambda$executeDfsPhase$5(SearchService.java:643) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:79) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:76) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619) ~[?:?]
	at java.lang.Thread.run(Thread.java:1447) ~[?:?]
Caused by: java.io.IOException: Unsupported number of bits per value: -47
	at org.elasticsearch.index.codec.vectors.DocIdsWriter.readInts(DocIdsWriter.java:242) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.index.codec.vectors.DefaultIVFVectorsReader$MemorySegmentPostingsVisitor.resetPostingsScorer(DefaultIVFVectorsReader.java:234) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.index.codec.vectors.IVFVectorsReader.search(IVFVectorsReader.java:279) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsReader.search(PerFieldKnnVectorsFormat.java:311) ~[lucene-core-10.2.1.jar:?]
	at org.apache.lucene.index.CodecReader.searchNearestVectors(CodecReader.java:272) ~[lucene-core-10.2.1.jar:?]
	at org.apache.lucene.index.FilterLeafReader.searchNearestVectors(FilterLeafReader.java:369) ~[lucene-core-10.2.1.jar:?]
	at org.apache.lucene.index.FilterLeafReader.searchNearestVectors(FilterLeafReader.java:369) ~[lucene-core-10.2.1.jar:?]
	at org.elasticsearch.search.internal.FieldUsageTrackingDirectoryReader$FieldUsageTrackingLeafReader.searchNearestVectors(FieldUsageTrackingDirectoryReader.java:233) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.search.internal.ExitableDirectoryReader$ExitableLeafReader.searchNearestVectors(ExitableDirectoryReader.java:166) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.search.vectors.IVFKnnFloatVectorQuery.approximateSearch(IVFKnnFloatVectorQuery.java:95) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.search.vectors.AbstractIVFKnnVectorQuery.getLeafResults(AbstractIVFKnnVectorQuery.java:151) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.search.vectors.AbstractIVFKnnVectorQuery.searchLeaf(AbstractIVFKnnVectorQuery.java:137) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.search.vectors.AbstractIVFKnnVectorQuery.lambda$rewrite$0(AbstractIVFKnnVectorQuery.java:123) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:328) ~[?:?]
	at org.apache.lucene.search.TaskExecutor$Task.run(TaskExecutor.java:173) ~[lucene-core-10.2.1.jar:?]
	at org.apache.lucene.search.TaskExecutor.lambda$invokeAll$1(TaskExecutor.java:98) ~[lucene-core-10.2.1.jar:?]
	... 6 more

Metadata

Metadata

Assignees

Labels

:Search Relevance/VectorsVector search>bugTeam:Search RelevanceMeta label for the Search Relevance team in Elasticsearchpriority:normalA label for assessing bug priority to be used by ES engineers

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions