CNDB-12456: Add ANN_OPTIONS to CQL SELECT queries #1525

adelapena · 2025-01-22T12:15:22Z

Add ANN_OPTIONS to CQL SELECT queries, as described in this document: https://docs.google.com/document/d/1Q2SkE1aBkcy25DnCWkdUqpCf8rZA6R_AOU6bbORbarI

The proposed syntax is:

SELECT * FROM t 
   ORDER BY v ANN OF [1.0] 
   LIMIT 10 
   WITH ann_options = {'rerank_k': 10};

The new options and not something specific to SAI, same as the ORDER BY col ANN OF val part.

Internally, the ANN options are an attribute of the RowFilter included in every ReadCommand. Every index implementation receives them as part of the ReadCommand passed in methods such as Index.validate(ReadCommand), Index.Group.queryPlanFor(RowFilter) or Index.QueryPlan.searcherFor(ReadCommand).

Currently SAI rejects queries with ANN options. I think consuming those options is something we can do in a separate ticket, keeping this one focused on the CQL and internode messaging part.

This patch increases the internode messaging version because the ANN options have to be included in the serialization of the ReadCommand they belong to. Hopefully we won't need to bump the messaging version again if we add new ANN options.

I have chosen to include the options in the RowFilter because it seemed the cleaner approach, and so possibly less problematic when rebasing on Apache. The downside is that it adds a 32-bit int to the serialization of every SELECT query. An alternative, possibly more convoluted approach to save us those 4 extra bytes could be placing the ANN options inside the relevant ANN RowFilter.Expression. @jbellis @ekaterinadimitrova2 should I give a go to this alternative approach?

Checklist before you submit for review

Make sure there is a PR in the CNDB project updating the Converged Cassandra version
Use NoSpamLogger for log lines that may appear frequently in the logs
Verify test results on Butler
Test coverage for new/modified code is > 80%
Proper code formatting
Proper title for each commit staring with the project-issue number, like CNDB-1234
Each commit has a meaningful description
Each commit is not very long and contains related changes
Renames, moves and reformatting are in distinct commits

adelapena · 2025-01-24T16:24:16Z

The last commit tries the alternative approach mentioned above, placing the new ANNOptions in the ANN RowFilter.Expression rather than in RowFilter to save us some serialization. It turns out it's not as noisy as I expected. Also, it more or less follows the steps of CNDB-10731 so it won't make us diverge from Apache much more than we already have.

src/java/org/apache/cassandra/index/ANNOptions.java

ekaterinadimitrova2 · 2025-01-24T23:23:05Z

Currently SAI rejects queries with ANN options. I think consuming those options is something we can do in a separate ticket, keeping this one focused on the CQL and internode messaging part.

Agreed

This patch increases the internode messaging version because the ANN options have to be included in the serialization of the ReadCommand they belong to. Hopefully we won't need to bump the messaging version again if we add new ANN options.

The flag makes sense to me

The last commit tries the alternative approach mentioned above, placing the new ANNOptions in the ANN RowFilter.Expression rather than in RowFilter to save us some serialization. It turns out it's not as noisy as I expected. Also, it more or less follows the steps of CNDB-10731 so it won't make us diverge from Apache much more than we already have.

I like the last version - placing the new ANNOptions in the ANN RowFilter.Expression rather than in RowFilter. I skimmed on a high level, though, and I did not get into details. Let me know when you are ready for review.

adelapena · 2025-01-25T00:47:10Z

Thanks for the feedback. The patch is ready for review.

ekaterinadimitrova2

Looks great, left some tiny comments/questions

src/java/org/apache/cassandra/hints/HintsDescriptor.java

src/java/org/apache/cassandra/cql3/statements/SelectOptions.java

src/java/org/apache/cassandra/db/filter/RowFilter.java

test/unit/org/apache/cassandra/db/filter/ANNOptionsTest.java

src/java/org/apache/cassandra/db/filter/RowFilter.java

test/unit/org/apache/cassandra/db/filter/ANNOptionsTest.java

src/java/org/apache/cassandra/db/commitlog/CommitLogDescriptor.java

src/java/org/apache/cassandra/net/MessagingService.java

adelapena · 2025-01-29T17:56:44Z

Thanks for the feedback. I've addressed some of the suggestions, I hope to address the rest of them tomorrow.

ekaterinadimitrova2 · 2025-01-29T18:31:44Z

I've addressed some of the suggestions,

They all look good to me, thanks!

I hope to address the rest of them tomorrow.

Just ping me when you are ready, thanks

ekaterinadimitrova2 · 2025-01-30T14:55:26Z

src/java/org/apache/cassandra/db/filter/ANNOptions.java

        Set<InetAddressAndPort> badNodes = MessagingService.instance().endpointsWithVersionBelow(MessagingService.VERSION_DS_11);
+        if (MessagingService.current_version < MessagingService.VERSION_DS_11)
+            badNodes.add(FBUtilities.getBroadcastAddressAndPort());


Oh, so we were missing the current one, good catch!

ekaterinadimitrova2 · 2025-01-30T15:23:31Z

src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java

@@ -238,7 +238,7 @@ else if (fromHeader != null)
                    descriptor = fromHeader;
                else descriptor = fromName;

-                if (descriptor.version > CommitLogDescriptor.current_version)
+                if (descriptor.version > CommitLogDescriptor.CURRENT_VERSION)


nit: Sonar complains that now we need to remove the Visible for testing annotation. Feel free to do it on commit

That seems unrelated to the changes, but removing anyway.

ekaterinadimitrova2

Looks great, except after the latest changes shouldRejectANOptions is failing with the error below. Please feel free to commit once you update that test. Thanks

java.lang.AssertionError: 
Expecting throwable message:
  "ANN options are not supported in clusters below DS 11."
to contain:
  "SAI doesn't support ANN options yet."
but did not.

rangeRestrictedTest - seems unrelated flaky; I just linked a ticket from Butler quickly.

And one more small comment that could be addressed on commit.

Side note: It would have been nice code coverage for new classes to report 100%, which I am surprised it does not for ANNOptions, but looking into the lines it seems ok?

ekaterinadimitrova2 · 2025-01-30T15:45:37Z

The test results on the other PR seem to show some issue. Please hold on committing this one until we clarify them. Thanks

adelapena · 2025-01-30T16:04:48Z

Looks great, except after the latest changes shouldRejectANOptions is failing with the error below.

That fails now because we have changed the current version. I'm simply removing the test because ANNOptionsDistributedTest covers this very same case with better tooling. Also, it's probably better not to add messaging service noise to the already crowded NativeIndexDDLTest.

Side note: It would have been nice code coverage for new classes to report 100%, which I am surprised it does not for ANNOptions, but looking into the lines it seems ok?

I guess changing the casing of the current_version constants has increased the scope of the changes quite a bit, so now it's including a bunch of code that we are not really changing. Same thing as with that @VisibleForTesting issue.

ekaterinadimitrova2 · 2025-01-30T16:27:53Z

I guess changing the casing of the current_version constants has increased the scope of the changes quite a bit, so now it's including a bunch of code that we are not really changing. Same thing as with that @VisibleForTesting issue.

Agreed about @VisibleForTesting (I asked for it on commit to be changed as it takes one second and we say "let's leave things better than we found them"; this was LHF not requiring additional review, etc). But ANNOptions is a completely new class, my side note there stands :-)

ekaterinadimitrova2 · 2025-01-30T16:29:00Z

covers this very same case with better tooling. Also, it's probably better not to add messaging service noise to the already crowded NativeIndexDDLTest.

+1

adelapena · 2025-01-30T16:39:32Z

But ANNOptions is a completely new class, my side note there stands :-)

Coverage report of ANNOptions, with a coverage of 95.1% that seems to meet our PR gate requirements, is missing some branches of the standard equals and hashcode implementations and part of a line in Serializer.flags. I think we are fine. Do you want me to spend more time on it to reach 100% coverage?

ekaterinadimitrova2 · 2025-01-30T16:49:39Z

but looking into the lines it seems ok?

missing some branches of the standard equals and hashcode implementations and part of a line in Serializer.flags. I think we are fine.

It seems we are on the same page

Do you want me to spend more time on it to reach 100% coverage?

No

ekaterinadimitrova2 · 2025-01-31T21:32:44Z

My +1 still stands with the latest changes, on clean CI here and the other branch. Thanks

* Rename MessagingService.VERSION_SG_10 to MessagingService.VERSION_DS_10. * Add the new MessagingService.VERSION_DS_11 messaging version, which is not used as current yet. * Add a new `ds.current_messaging_version` system property to specify the current messaging version. To use the new messaging version, CC should be started with -Dds.current_messaging_version=101.

Add new CQL syntax to specify ANN options in ANN queries. The only available ANN option at the moment is `rerank_k`, which is the amplified limit for the ANN query to get more accurate results. This doesn't add support for ANN options to SAI, which is planned to be done separately. The new options are included in the ANN expression of the `RowFilter` of the `ReadCommand` that represents the query. Thus, they are included in the serialization of the command in the coordiantor and delivered to the replicas. The new ANN options require that the messaging version of all the nodes is at least VERSION_DS_11. That version is not yet used by default, to use it CC should be started with -Dds.current_messaging_version=101.

sonarqubecloud · 2025-02-03T14:58:16Z

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
92.0% Coverage on New Code
0.3% Duplication on New Code

See analysis details on SonarQube Cloud

cassci-bot · 2025-02-03T15:05:38Z

✔️ Build ds-cassandra-pr-gate/PR-1525 approved by Butler

Approved by Butler
See build details here

adelapena self-assigned this Jan 22, 2025

adelapena marked this pull request as draft January 22, 2025 12:16

adelapena marked this pull request as ready for review January 22, 2025 14:10

adelapena force-pushed the CNDB-12456-main branch from 7c35d1e to 42a25dc Compare January 22, 2025 14:10

adelapena marked this pull request as draft January 22, 2025 14:10

adelapena marked this pull request as ready for review January 24, 2025 01:09

adelapena force-pushed the CNDB-12456-main branch from 42a25dc to fbc3e21 Compare January 24, 2025 01:09

adelapena marked this pull request as draft January 24, 2025 01:10

adelapena marked this pull request as ready for review January 24, 2025 12:19

adelapena force-pushed the CNDB-12456-main branch from fbc3e21 to 1c76376 Compare January 24, 2025 12:20

adelapena commented Jan 24, 2025

View reviewed changes

src/java/org/apache/cassandra/index/ANNOptions.java Outdated Show resolved Hide resolved

adelapena force-pushed the CNDB-12456-main branch from 16af975 to 25a8223 Compare January 27, 2025 15:14

ekaterinadimitrova2 reviewed Jan 28, 2025

View reviewed changes

adelapena force-pushed the CNDB-12456-main branch from 25a8223 to 8d68bf4 Compare January 29, 2025 17:40

ekaterinadimitrova2 reviewed Jan 30, 2025

View reviewed changes

ekaterinadimitrova2 approved these changes Jan 30, 2025

View reviewed changes

adelapena force-pushed the CNDB-12456-main branch from 1a158e3 to 2776f01 Compare January 30, 2025 16:49

adelapena force-pushed the CNDB-12456-main branch from 7c28f40 to 601a78b Compare January 31, 2025 17:53

adelapena added 2 commits February 3, 2025 14:11

adelapena force-pushed the CNDB-12456-main branch from 601a78b to 08aef9d Compare February 3, 2025 14:12

adelapena merged commit 5821fea into main Feb 3, 2025
462 of 477 checks passed

adelapena deleted the CNDB-12456-main branch February 3, 2025 17:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CNDB-12456: Add ANN_OPTIONS to CQL SELECT queries #1525

CNDB-12456: Add ANN_OPTIONS to CQL SELECT queries #1525

adelapena commented Jan 22, 2025 •

edited

Loading

adelapena commented Jan 24, 2025

ekaterinadimitrova2 commented Jan 24, 2025

adelapena commented Jan 25, 2025

ekaterinadimitrova2 left a comment

adelapena commented Jan 29, 2025

ekaterinadimitrova2 commented Jan 29, 2025

ekaterinadimitrova2 Jan 30, 2025

ekaterinadimitrova2 Jan 30, 2025

adelapena Jan 30, 2025

ekaterinadimitrova2 left a comment •

edited

Loading

ekaterinadimitrova2 commented Jan 30, 2025

adelapena commented Jan 30, 2025

ekaterinadimitrova2 commented Jan 30, 2025

ekaterinadimitrova2 commented Jan 30, 2025

adelapena commented Jan 30, 2025

ekaterinadimitrova2 commented Jan 30, 2025

ekaterinadimitrova2 commented Jan 31, 2025

sonarqubecloud bot commented Feb 3, 2025

cassci-bot commented Feb 3, 2025

CNDB-12456: Add ANN_OPTIONS to CQL SELECT queries #1525

CNDB-12456: Add ANN_OPTIONS to CQL SELECT queries #1525

Conversation

adelapena commented Jan 22, 2025 • edited Loading

Checklist before you submit for review

adelapena commented Jan 24, 2025

ekaterinadimitrova2 commented Jan 24, 2025

adelapena commented Jan 25, 2025

ekaterinadimitrova2 left a comment

Choose a reason for hiding this comment

adelapena commented Jan 29, 2025

ekaterinadimitrova2 commented Jan 29, 2025

ekaterinadimitrova2 Jan 30, 2025

Choose a reason for hiding this comment

ekaterinadimitrova2 Jan 30, 2025

Choose a reason for hiding this comment

adelapena Jan 30, 2025

Choose a reason for hiding this comment

ekaterinadimitrova2 left a comment • edited Loading

Choose a reason for hiding this comment

ekaterinadimitrova2 commented Jan 30, 2025

adelapena commented Jan 30, 2025

ekaterinadimitrova2 commented Jan 30, 2025

ekaterinadimitrova2 commented Jan 30, 2025

adelapena commented Jan 30, 2025

ekaterinadimitrova2 commented Jan 30, 2025

ekaterinadimitrova2 commented Jan 31, 2025

sonarqubecloud bot commented Feb 3, 2025

Quality Gate passed

cassci-bot commented Feb 3, 2025

✔️ Build ds-cassandra-pr-gate/PR-1525 approved by Butler

adelapena commented Jan 22, 2025 •

edited

Loading

ekaterinadimitrova2 left a comment •

edited

Loading