Skip to content

Question: how to register a FTS3/FTS4 custom tokenizer?  #1227

Closed

Description

@Luminqi

From the document of SQLite: https://www.sqlite.org/fts3.html, the way to register a custom tokenizer is executing a sql:

SELECT fts3_tokenizer(<tokenizer-name>, <sqlite3_tokenizer_module ptr>)

I am stuck on how to provide the second parameter, which is a buffer storing the pointer to the tokenizer. How can this be done in Node.js?

Activity

  1. changed the title [-]Question: how to register a custom FTS3/FTS4 custom tokenizer? [/-] [+]Question: how to register a FTS3/FTS4 custom tokenizer? [/+] on Jul 18, 2024
  2. Luminqi commented on Aug 6, 2024

    @Luminqi
    Author

    Find a solution:

    1. write tokenizer as a sqlite extension and call SELECT fts3_tokenizer(<tokenizer-name>, <sqlite3_tokenizer_module ptr>) in extension init function
    2. load the extension using loadExtension api

    I am not sure if this is the best solution.

  3. locked and limited conversation to collaborators on Sep 12, 2024
  4. converted this issue into a discussion #1258 on Sep 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions