Skip to content

Conversation

fsk119
Copy link
Member

@fsk119 fsk119 commented Sep 25, 2025

What is the purpose of the change

Support to parse VECTOR_SEARCH function.

Brief change log

  • Add SqlVectorSearchTableFunction
  • Add type inference for the newly added function
  • Set the correct scope for VECTOR_SEARCH function

Verifying this change

This change added tests and can be verified as follows:

  • Added plan tests for VECTOR_SEARCH function

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
  • The serializers: (yes / no / don't know)
  • The runtime per-record code paths (performance sensitive): (yes / no / don't know)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
  • The S3 file system connector: (yes / no / don't know)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@flinkbot
Copy link
Collaborator

flinkbot commented Sep 25, 2025

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

import java.io.Serializable;

/**
* A {@link DynamicTableSource} that search rows of an external storage system by one or more
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: search -> searches

* vectors during runtime.
*
* <p>Compared to {@link ScanTableSource}, the source does not have to read the entire table and can
* lazily fetch individual values from a (possibly continuously changing) external table when
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be interesting to compare it to a LookupTableSource also.

PARAM_SEARCH_TABLE,
PARAM_COLUMN_TO_SEARCH,
PARAM_COLUMN_TO_QUERY,
PARAM_TOP_K));
Copy link
Contributor

@davidradl davidradl Sep 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I an see what TOP_K is from googling. It would be useful to add in the documentation describing the parameters with this change - including the default. I wonder if we should add paging, to be able to handle a large number of results or is this not done with vector databases.

@github-actions github-actions bot added the community-reviewed PR has been reviewed by the community. label Sep 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
community-reviewed PR has been reviewed by the community.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants