Skip to content

Support for embedding documents#122

Merged
jnu merged 8 commits intomainfrom
embedding
Mar 28, 2025
Merged

Support for embedding documents#122
jnu merged 8 commits intomainfrom
embedding

Conversation

@jnu
Copy link
Copy Markdown
Contributor

@jnu jnu commented Mar 27, 2025

We will begin to support the (opt-in) feature of retaining embeddings of input documents. Changes included in this PR:

  • Upgrade bc2 module to pick up built-in support for generating embeddings on redacted text
  • Database table for storing embeddings + metadata related to generation
  • Infrastructure to support embeddings in Terraform (openai model deployment)
  • Updated app config to support embedding model definition
  • Logic to inject embedding generation request to pipeline as needed
  • Code to save generated embeddings when they exist

@jnu jnu changed the title [WIP] Support for embedding documents Support for embedding documents Mar 28, 2025
@jnu jnu merged commit f4a76c3 into main Mar 28, 2025
1 check failed
@jnu jnu deleted the embedding branch March 28, 2025 19:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant