Closed
Description
Elasticsearch Version
main
Installed Plugins
No response
Java Version
bundled
OS Version
any
Problem Description
ivf format fails to search due to invalid bytes in DocIdReader.
Steps to Reproduce
This occurred during a large vector indexing test, indexing about 100M+ vectors with aggressive merging.
Obviously, we have a serialization bug where we are miscounting bytes in the format.
Logs (if relevant)
[2025-06-17T02:58:50,200][WARN ][o.e.s.SearchService ] [elasticsearch-0] [yt4RfBSmRcKGMzZS3IDE-w][msmarco-v2][0]: failed to execute search request for task [2783699]
org.elasticsearch.search.dfs.DfsPhaseExecutionException: Dfs Failed [Exception during dfs phase]
at org.elasticsearch.search.dfs.DfsPhase.execute(DfsPhase.java:67) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
at org.elasticsearch.search.SearchService.executeDfsPhase(SearchService.java:654) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
at org.elasticsearch.search.SearchService.lambda$executeDfsPhase$5(SearchService.java:643) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
at org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:79) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
at org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:76) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
at org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619) ~[?:?]
at java.lang.Thread.run(Thread.java:1447) ~[?:?]
Caused by: java.io.IOException: Unsupported number of bits per value: -47
at org.elasticsearch.index.codec.vectors.DocIdsWriter.readInts(DocIdsWriter.java:242) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
at org.elasticsearch.index.codec.vectors.DefaultIVFVectorsReader$MemorySegmentPostingsVisitor.resetPostingsScorer(DefaultIVFVectorsReader.java:234) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
at org.elasticsearch.index.codec.vectors.IVFVectorsReader.search(IVFVectorsReader.java:279) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
at org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsReader.search(PerFieldKnnVectorsFormat.java:311) ~[lucene-core-10.2.1.jar:?]
at org.apache.lucene.index.CodecReader.searchNearestVectors(CodecReader.java:272) ~[lucene-core-10.2.1.jar:?]
at org.apache.lucene.index.FilterLeafReader.searchNearestVectors(FilterLeafReader.java:369) ~[lucene-core-10.2.1.jar:?]
at org.apache.lucene.index.FilterLeafReader.searchNearestVectors(FilterLeafReader.java:369) ~[lucene-core-10.2.1.jar:?]
at org.elasticsearch.search.internal.FieldUsageTrackingDirectoryReader$FieldUsageTrackingLeafReader.searchNearestVectors(FieldUsageTrackingDirectoryReader.java:233) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
at org.elasticsearch.search.internal.ExitableDirectoryReader$ExitableLeafReader.searchNearestVectors(ExitableDirectoryReader.java:166) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
at org.elasticsearch.search.vectors.IVFKnnFloatVectorQuery.approximateSearch(IVFKnnFloatVectorQuery.java:95) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
at org.elasticsearch.search.vectors.AbstractIVFKnnVectorQuery.getLeafResults(AbstractIVFKnnVectorQuery.java:151) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
at org.elasticsearch.search.vectors.AbstractIVFKnnVectorQuery.searchLeaf(AbstractIVFKnnVectorQuery.java:137) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
at org.elasticsearch.search.vectors.AbstractIVFKnnVectorQuery.lambda$rewrite$0(AbstractIVFKnnVectorQuery.java:123) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:328) ~[?:?]
at org.apache.lucene.search.TaskExecutor$Task.run(TaskExecutor.java:173) ~[lucene-core-10.2.1.jar:?]
at org.apache.lucene.search.TaskExecutor.lambda$invokeAll$1(TaskExecutor.java:98) ~[lucene-core-10.2.1.jar:?]
... 6 more