Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add jmh benchmarks #396

Merged
merged 6 commits into from
Feb 25, 2025
Merged

Add jmh benchmarks #396

merged 6 commits into from
Feb 25, 2025

Conversation

sam-herman
Copy link
Contributor

@sam-herman sam-herman commented Feb 22, 2025 •
edited
Loading

Add JMH Benchmarking

Overview

This PR introduces a dedicated JMH benchmarking module (benchmark-jmh) to provide basic microbenchmarking for performance measurements for jvector operations. The module enables systematic performance testing and comparison of variants of the vector search such as:

  • Search OnHeap/OnDisk
  • Random datasets vs static datasets.
  • Search latency increase with dataset size
  • Compare different scorer implementations

Changes

  • Add new benchmark-jmh module
  • Implement two sample JMH benchmarks for basic tests RandomVectorsBenchmark and StaticSetVectorsBenchmark
  • Add logging configuration for benchmark execution
  • Include benchmark documentation and usage instructions

Implementation Details

  • Uses JMH 1.37 for microbenchmarking with blackhole compiler settings
  • Includes both throughput and average time measurements
  • Implements proper warmup and measurement cycles

Usage

mvn clean install
java --enable-native-access=ALL-UNNAMED \
  --add-modules=jdk.incubator.vector \
  -XX:+HeapDumpOnOutOfMemoryError \
  -Xmx14G -Djvector.experimental.enable_native_vectorization=true \
  -jar benchmarks-jmh/target/benchmarks-jmh-4.0.0-beta.2-SNAPSHOT.jar

Output

The output summary will look like this:

Benchmark                                                   (numBaseVectors)  (numQueryVectors)  Mode  Cnt   Score   Error  Units
RandomVectorsBenchmark.testOnHeapRandomVectors                          1000                 10  avgt    5   9.454 ± 0.329  ms/op
RandomVectorsBenchmark.testOnHeapRandomVectors                         10000                 10  avgt    5  12.852 ± 0.512  ms/op
RandomVectorsBenchmark.testOnHeapRandomVectors                        100000                 10  avgt    5  13.037 ± 0.352  ms/op
RandomVectorsBenchmark.testOnHeapRandomVectors                       1000000                 10  avgt    5  15.759 ± 0.610  ms/op
StaticSetVectorsBenchmark.testOnHeapWithRandomQueryVectors               N/A                N/A  avgt    5   8.253 ± 0.172  ms/op

The full output will look something like this:

WARNING: Using incubator modules: jdk.incubator.vector
# JMH version: 1.37
# VM version: JDK 23.0.2, OpenJDK 64-Bit Server VM, 23.0.2+7-58
# VM invoker: /Library/Java/JavaVirtualMachines/jdk-23.0.2.jdk/Contents/Home/bin/java
# VM options: --enable-native-access=ALL-UNNAMED --add-modules=jdk.incubator.vector -XX:+HeapDumpOnOutOfMemoryError -Xmx14G -Djvector.experimental.enable_native_vectorization=true
# Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 2 iterations, 10 s each
# Measurement: 5 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors
# Parameters: (numBaseVectors = 1000, numQueryVectors = 10)

# Run progress: 0.00% complete, ETA 00:05:50
# Fork: 1 of 1
WARNING: Using incubator modules: jdk.incubator.vector
# Warmup Iteration   1: 2025-02-22T07:05:19.037924Z io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors-jmh-worker-1 INFO Starting configuration XmlConfiguration[location=jar:file:/Users/samuelherman/projects/jvector/benchmarks-jmh/target/benchmarks-jmh-4.0.0-beta.2-SNAPSHOT.jar!/log4j2.xml, lastModified=2025-02-22T07:05:12.069Z]...
2025-02-22T07:05:19.044574Z io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors-jmh-worker-1 INFO Start watching for changes to jar:file:/Users/samuelherman/projects/jvector/benchmarks-jmh/target/benchmarks-jmh-4.0.0-beta.2-SNAPSHOT.jar!/log4j2.xml every 0 seconds
2025-02-22T07:05:19.045170Z io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors-jmh-worker-1 INFO Configuration XmlConfiguration[location=jar:file:/Users/samuelherman/projects/jvector/benchmarks-jmh/target/benchmarks-jmh-4.0.0-beta.2-SNAPSHOT.jar!/log4j2.xml, lastModified=2025-02-22T07:05:12.069Z] started.
2025-02-22T07:05:19.047856Z io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors-jmh-worker-1 INFO Stopping configuration org.apache.logging.log4j.core.config.DefaultConfiguration@4865f20c...
2025-02-22T07:05:19.048282Z io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors-jmh-worker-1 INFO Configuration org.apache.logging.log4j.core.config.DefaultConfiguration@4865f20c stopped.
Feb 21, 2025 11:05:19 PM io.github.jbellis.jvector.vector.VectorizationProvider lookup
WARNING: Native vector API was not enabled. Failed to load supporting native library.
Feb 21, 2025 11:05:19 PM io.github.jbellis.jvector.vector.PanamaVectorizationProvider <init>
INFO: Preferred f32 species is 128
Feb 21, 2025 11:05:19 PM io.github.jbellis.jvector.vector.VectorizationProvider lookup
INFO: Java incubating Vector API enabled. Using PanamaVectorizationProvider.
9.426 ms/op
# Warmup Iteration   2: 9.415 ms/op
Iteration   1: 9.379 ms/op
Iteration   2: 9.417 ms/op
Iteration   3: 9.526 ms/op
Iteration   4: 9.564 ms/op
Iteration   5: 9.384 ms/op


Result "io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors":
  9.454 ±(99.9%) 0.329 ms/op [Average]
  (min, avg, max) = (9.379, 9.454, 9.564), stdev = 0.085
  CI (99.9%): [9.125, 9.783] (assumes normal distribution)


# JMH version: 1.37
# VM version: JDK 23.0.2, OpenJDK 64-Bit Server VM, 23.0.2+7-58
# VM invoker: /Library/Java/JavaVirtualMachines/jdk-23.0.2.jdk/Contents/Home/bin/java
# VM options: --enable-native-access=ALL-UNNAMED --add-modules=jdk.incubator.vector -XX:+HeapDumpOnOutOfMemoryError -Xmx14G -Djvector.experimental.enable_native_vectorization=true
# Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 2 iterations, 10 s each
# Measurement: 5 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors
# Parameters: (numBaseVectors = 10000, numQueryVectors = 10)

# Run progress: 20.00% complete, ETA 00:05:35
# Fork: 1 of 1
WARNING: Using incubator modules: jdk.incubator.vector
# Warmup Iteration   1: 2025-02-22T07:06:43.016487Z io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors-jmh-worker-1 INFO Starting configuration XmlConfiguration[location=jar:file:/Users/samuelherman/projects/jvector/benchmarks-jmh/target/benchmarks-jmh-4.0.0-beta.2-SNAPSHOT.jar!/log4j2.xml, lastModified=2025-02-22T07:05:12.069Z]...
2025-02-22T07:06:43.026214Z io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors-jmh-worker-1 INFO Start watching for changes to jar:file:/Users/samuelherman/projects/jvector/benchmarks-jmh/target/benchmarks-jmh-4.0.0-beta.2-SNAPSHOT.jar!/log4j2.xml every 0 seconds
2025-02-22T07:06:43.027019Z io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors-jmh-worker-1 INFO Configuration XmlConfiguration[location=jar:file:/Users/samuelherman/projects/jvector/benchmarks-jmh/target/benchmarks-jmh-4.0.0-beta.2-SNAPSHOT.jar!/log4j2.xml, lastModified=2025-02-22T07:05:12.069Z] started.
2025-02-22T07:06:43.029484Z io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors-jmh-worker-1 INFO Stopping configuration org.apache.logging.log4j.core.config.DefaultConfiguration@4865f20c...
2025-02-22T07:06:43.030070Z io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors-jmh-worker-1 INFO Configuration org.apache.logging.log4j.core.config.DefaultConfiguration@4865f20c stopped.
Feb 21, 2025 11:06:43 PM io.github.jbellis.jvector.vector.VectorizationProvider lookup
WARNING: Native vector API was not enabled. Failed to load supporting native library.
Feb 21, 2025 11:06:43 PM io.github.jbellis.jvector.vector.PanamaVectorizationProvider <init>
INFO: Preferred f32 species is 128
Feb 21, 2025 11:06:43 PM io.github.jbellis.jvector.vector.VectorizationProvider lookup
INFO: Java incubating Vector API enabled. Using PanamaVectorizationProvider.
13.040 ms/op
# Warmup Iteration   2: 12.795 ms/op
Iteration   1: 13.074 ms/op
Iteration   2: 12.798 ms/op
Iteration   3: 12.746 ms/op
Iteration   4: 12.771 ms/op
Iteration   5: 12.870 ms/op


Result "io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors":
  12.852 ±(99.9%) 0.512 ms/op [Average]
  (min, avg, max) = (12.746, 12.852, 13.074), stdev = 0.133
  CI (99.9%): [12.340, 13.363] (assumes normal distribution)


# JMH version: 1.37
# VM version: JDK 23.0.2, OpenJDK 64-Bit Server VM, 23.0.2+7-58
# VM invoker: /Library/Java/JavaVirtualMachines/jdk-23.0.2.jdk/Contents/Home/bin/java
# VM options: --enable-native-access=ALL-UNNAMED --add-modules=jdk.incubator.vector -XX:+HeapDumpOnOutOfMemoryError -Xmx14G -Djvector.experimental.enable_native_vectorization=true
# Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 2 iterations, 10 s each
# Measurement: 5 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors
# Parameters: (numBaseVectors = 100000, numQueryVectors = 10)

# Run progress: 40.00% complete, ETA 00:08:13
# Fork: 1 of 1
WARNING: Using incubator modules: jdk.incubator.vector
# Warmup Iteration   1: 2025-02-22T07:10:48.184211Z io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors-jmh-worker-1 INFO Starting configuration XmlConfiguration[location=jar:file:/Users/samuelherman/projects/jvector/benchmarks-jmh/target/benchmarks-jmh-4.0.0-beta.2-SNAPSHOT.jar!/log4j2.xml, lastModified=2025-02-22T07:05:12.069Z]...
2025-02-22T07:10:48.189528Z io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors-jmh-worker-1 INFO Start watching for changes to jar:file:/Users/samuelherman/projects/jvector/benchmarks-jmh/target/benchmarks-jmh-4.0.0-beta.2-SNAPSHOT.jar!/log4j2.xml every 0 seconds
2025-02-22T07:10:48.190082Z io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors-jmh-worker-1 INFO Configuration XmlConfiguration[location=jar:file:/Users/samuelherman/projects/jvector/benchmarks-jmh/target/benchmarks-jmh-4.0.0-beta.2-SNAPSHOT.jar!/log4j2.xml, lastModified=2025-02-22T07:05:12.069Z] started.
2025-02-22T07:10:48.191953Z io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors-jmh-worker-1 INFO Stopping configuration org.apache.logging.log4j.core.config.DefaultConfiguration@4865f20c...
2025-02-22T07:10:48.192352Z io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors-jmh-worker-1 INFO Configuration org.apache.logging.log4j.core.config.DefaultConfiguration@4865f20c stopped.
Feb 21, 2025 11:10:48 PM io.github.jbellis.jvector.vector.VectorizationProvider lookup
WARNING: Native vector API was not enabled. Failed to load supporting native library.
Feb 21, 2025 11:10:48 PM io.github.jbellis.jvector.vector.PanamaVectorizationProvider <init>
INFO: Preferred f32 species is 128
Feb 21, 2025 11:10:48 PM io.github.jbellis.jvector.vector.VectorizationProvider lookup
INFO: Java incubating Vector API enabled. Using PanamaVectorizationProvider.
13.037 ms/op
# Warmup Iteration   2: 12.935 ms/op
Iteration   1: 13.127 ms/op
Iteration   2: 12.943 ms/op
Iteration   3: 13.040 ms/op
Iteration   4: 12.948 ms/op
Iteration   5: 13.129 ms/op


Result "io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors":
  13.037 ±(99.9%) 0.352 ms/op [Average]
  (min, avg, max) = (12.943, 13.037, 13.129), stdev = 0.091
  CI (99.9%): [12.685, 13.389] (assumes normal distribution)


# JMH version: 1.37
# VM version: JDK 23.0.2, OpenJDK 64-Bit Server VM, 23.0.2+7-58
# VM invoker: /Library/Java/JavaVirtualMachines/jdk-23.0.2.jdk/Contents/Home/bin/java
# VM options: --enable-native-access=ALL-UNNAMED --add-modules=jdk.incubator.vector -XX:+HeapDumpOnOutOfMemoryError -Xmx14G -Djvector.experimental.enable_native_vectorization=true
# Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 2 iterations, 10 s each
# Measurement: 5 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors
# Parameters: (numBaseVectors = 1000000, numQueryVectors = 10)

# Run progress: 60.00% complete, ETA 00:24:03
# Fork: 1 of 1
WARNING: Using incubator modules: jdk.incubator.vector
# Warmup Iteration   1: 2025-02-22T07:41:24.577770Z io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors-jmh-worker-1 INFO Starting configuration XmlConfiguration[location=jar:file:/Users/samuelherman/projects/jvector/benchmarks-jmh/target/benchmarks-jmh-4.0.0-beta.2-SNAPSHOT.jar!/log4j2.xml, lastModified=2025-02-22T07:05:12.069Z]...
2025-02-22T07:41:24.585937Z io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors-jmh-worker-1 INFO Start watching for changes to jar:file:/Users/samuelherman/projects/jvector/benchmarks-jmh/target/benchmarks-jmh-4.0.0-beta.2-SNAPSHOT.jar!/log4j2.xml every 0 seconds
2025-02-22T07:41:24.586486Z io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors-jmh-worker-1 INFO Configuration XmlConfiguration[location=jar:file:/Users/samuelherman/projects/jvector/benchmarks-jmh/target/benchmarks-jmh-4.0.0-beta.2-SNAPSHOT.jar!/log4j2.xml, lastModified=2025-02-22T07:05:12.069Z] started.
2025-02-22T07:41:24.588383Z io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors-jmh-worker-1 INFO Stopping configuration org.apache.logging.log4j.core.config.DefaultConfiguration@4865f20c...
2025-02-22T07:41:24.588705Z io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors-jmh-worker-1 INFO Configuration org.apache.logging.log4j.core.config.DefaultConfiguration@4865f20c stopped.
Feb 21, 2025 11:41:24 PM io.github.jbellis.jvector.vector.VectorizationProvider lookup
WARNING: Native vector API was not enabled. Failed to load supporting native library.
Feb 21, 2025 11:41:24 PM io.github.jbellis.jvector.vector.PanamaVectorizationProvider <init>
INFO: Preferred f32 species is 128
Feb 21, 2025 11:41:24 PM io.github.jbellis.jvector.vector.VectorizationProvider lookup
INFO: Java incubating Vector API enabled. Using PanamaVectorizationProvider.
15.937 ms/op
# Warmup Iteration   2: 15.460 ms/op
Iteration   1: 15.563 ms/op
Iteration   2: 15.842 ms/op
Iteration   3: 15.633 ms/op
Iteration   4: 15.804 ms/op
Iteration   5: 15.952 ms/op


Result "io.github.jbellis.jvector.bench.RandomVectorsBenchmark.testOnHeapRandomVectors":
  15.759 ±(99.9%) 0.610 ms/op [Average]
  (min, avg, max) = (15.563, 15.759, 15.952), stdev = 0.158
  CI (99.9%): [15.149, 16.369] (assumes normal distribution)


# JMH version: 1.37
# VM version: JDK 23.0.2, OpenJDK 64-Bit Server VM, 23.0.2+7-58
# VM invoker: /Library/Java/JavaVirtualMachines/jdk-23.0.2.jdk/Contents/Home/bin/java
# VM options: --enable-native-access=ALL-UNNAMED --add-modules=jdk.incubator.vector -XX:+HeapDumpOnOutOfMemoryError -Xmx14G -Djvector.experimental.enable_native_vectorization=true
# Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 2 iterations, 10 s each
# Measurement: 5 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: io.github.jbellis.jvector.bench.StaticSetVectorsBenchmark.testOnHeapWithRandomQueryVectors

# Run progress: 80.00% complete, ETA 01:26:12
# Fork: 1 of 1
WARNING: Using incubator modules: jdk.incubator.vector
# Warmup Iteration   1: 2025-02-22T12:50:08.942348Z io.github.jbellis.jvector.bench.StaticSetVectorsBenchmark.testOnHeapWithRandomQueryVectors-jmh-worker-1 INFO Starting configuration XmlConfiguration[location=jar:file:/Users/samuelherman/projects/jvector/benchmarks-jmh/target/benchmarks-jmh-4.0.0-beta.2-SNAPSHOT.jar!/log4j2.xml, lastModified=2025-02-22T07:05:12.069Z]...
2025-02-22T12:50:08.950436Z io.github.jbellis.jvector.bench.StaticSetVectorsBenchmark.testOnHeapWithRandomQueryVectors-jmh-worker-1 INFO Start watching for changes to jar:file:/Users/samuelherman/projects/jvector/benchmarks-jmh/target/benchmarks-jmh-4.0.0-beta.2-SNAPSHOT.jar!/log4j2.xml every 0 seconds
2025-02-22T12:50:08.951460Z io.github.jbellis.jvector.bench.StaticSetVectorsBenchmark.testOnHeapWithRandomQueryVectors-jmh-worker-1 INFO Configuration XmlConfiguration[location=jar:file:/Users/samuelherman/projects/jvector/benchmarks-jmh/target/benchmarks-jmh-4.0.0-beta.2-SNAPSHOT.jar!/log4j2.xml, lastModified=2025-02-22T07:05:12.069Z] started.
2025-02-22T12:50:08.956372Z io.github.jbellis.jvector.bench.StaticSetVectorsBenchmark.testOnHeapWithRandomQueryVectors-jmh-worker-1 INFO Stopping configuration org.apache.logging.log4j.core.config.DefaultConfiguration@5fb60c32...
2025-02-22T12:50:08.956917Z io.github.jbellis.jvector.bench.StaticSetVectorsBenchmark.testOnHeapWithRandomQueryVectors-jmh-worker-1 INFO Configuration org.apache.logging.log4j.core.config.DefaultConfiguration@5fb60c32 stopped.
Feb 22, 2025 4:50:08 AM io.github.jbellis.jvector.vector.VectorizationProvider lookup
WARNING: Native vector API was not enabled. Failed to load supporting native library.
Feb 22, 2025 4:50:09 AM io.github.jbellis.jvector.vector.PanamaVectorizationProvider <init>
INFO: Preferred f32 species is 128
Feb 22, 2025 4:50:09 AM io.github.jbellis.jvector.vector.VectorizationProvider lookup
INFO: Java incubating Vector API enabled. Using PanamaVectorizationProvider.
04:50:09.095 [io.github.jbellis.jvector.bench.StaticSetVectorsBenchmark.testOnHeapWithRandomQueryVectors-jmh-worker-1] INFO  io.github.jbellis.jvector.bench.StaticSetVectorsBenchmark - base vectors size: 10000, query vectors size: 100, loaded, dimensions 128
8.378 ms/op
# Warmup Iteration   2: 8.282 ms/op
Iteration   1: 8.274 ms/op
Iteration   2: 8.254 ms/op
Iteration   3: 8.181 ms/op
Iteration   4: 8.254 ms/op
Iteration   5: 8.301 ms/op


Result "io.github.jbellis.jvector.bench.StaticSetVectorsBenchmark.testOnHeapWithRandomQueryVectors":
  8.253 ±(99.9%) 0.172 ms/op [Average]
  (min, avg, max) = (8.181, 8.253, 8.301), stdev = 0.045
  CI (99.9%): [8.081, 8.425] (assumes normal distribution)


# Run complete. Total time: 05:46:25

REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.

NOTE: Current JVM experimentally supports Compiler Blackholes, and they are in use. Please exercise
extra caution when trusting the results, look into the generated code to check the benchmark still
works, and factor in a small probability of new VM bugs. Additionally, while comparisons between
different JVMs are already problematic, the performance difference caused by different Blackhole
modes can be very significant. Please make sure you use the consistent Blackhole mode for comparisons.

Benchmark                                                   (numBaseVectors)  (numQueryVectors)  Mode  Cnt   Score   Error  Units
RandomVectorsBenchmark.testOnHeapRandomVectors                          1000                 10  avgt    5   9.454 ± 0.329  ms/op
RandomVectorsBenchmark.testOnHeapRandomVectors                         10000                 10  avgt    5  12.852 ± 0.512  ms/op
RandomVectorsBenchmark.testOnHeapRandomVectors                        100000                 10  avgt    5  13.037 ± 0.352  ms/op
RandomVectorsBenchmark.testOnHeapRandomVectors                       1000000                 10  avgt    5  15.759 ± 0.610  ms/op
StaticSetVectorsBenchmark.testOnHeapWithRandomQueryVectors               N/A                N/A  avgt    5   8.253 ± 0.172  ms/op

Samuel Herman and others added 4 commits February 21, 2025 22:01
add jmh skeleton
c9aa09d 
Signed-off-by: Samuel Herman <[email protected]>
@sam-herman
add random query test
1fc9fea 
Signed-off-by: Samuel Herman <[email protected]>
@sam-herman
minor fix
8403c3d 
Signed-off-by: Samuel Herman <[email protected]>
@sam-herman
added a random benchamrk with parameter grid
4100bf8 
Signed-off-by: Samuel Herman <[email protected]>
@tlwillke tlwillke self-requested a review February 24, 2025 18:55
tlwillke
tlwillke approved these changes Feb 25, 2025
View reviewed changes
Copy link
Collaborator

@tlwillke tlwillke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 with the rat fix and change to instructions

@tlwillke tlwillke merged commit 1f6c5b1 into jbellis:main Feb 25, 2025
0 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tlwillke tlwillke tlwillke approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@sam-herman @tlwillke