RFC: Switch to rust backend for performance

[chonknick]

• Add ci.yml for Rust tests and Python build verification
• Add release.yml for publishing to crates.io and PyPI
• Update sync-check.yml for new src/data path
• Remove old lint.yml, publish.yml, test.yml (consolidated into ci.yml)

[Copilot]

Pull request overview

This PR transitions the catsu embeddings client from a pure Python implementation to a Rust-based backend with Python bindings, aimed at improving performance. The changes consolidate multiple GitHub Actions workflows, update the data path for model catalog, and introduce Rust implementations for 11 embedding providers.

Key changes:

• Replaces Python provider implementations with Rust equivalents
• Adds Python bindings via PyO3 for the Rust core
• Consolidates GitHub Actions workflows (test.yml, lint.yml, publish.yml → ci.yml, release.yml)
• Updates model catalog path from src/catsu/data/models.json to src/data/models.json

Reviewed changes

Copilot reviewed 81 out of 84 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
website/public/models.json	Updates symlink path to new data location
src/providers/*.rs	Implements embedding providers in Rust (VoyageAI, Together, OpenAI, Nomic, Mixedbread, Mistral, Jina, Gemini, DeepInfra, Cohere, Cloudflare)
src/catalog.rs	Adds Rust model catalog with embedded JSON data
packages/python/*	Introduces Python bindings and packaging for Rust library
examples/*.{rs,py}	Adds example usage files for each provider
.github/workflows/*	Consolidates and updates CI/CD workflows
Cargo.toml	Adds Rust workspace and dependencies

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Import uses CatsuClient but other example files import Client. The exposed Python API should use Client for consistency.

Suggested change

	from catsu import CatsuClient
	def main():
	client = CatsuClient()
	from catsu import Client
	def main():
	client = Client()

[Copilot]

Import uses CatsuClient but other example files import Client. The exposed Python API should use Client for consistency.

Suggested change

	from catsu import CatsuClient
	def main():
	client = CatsuClient()
	from catsu import Client
	def main():
	client = Client()

To fix the problem, explicitly restrict the GITHUB_TOKEN permissions in this workflow. Since these jobs only check out code and run Rust/Python builds and tests, they only need read access to repository contents. The minimal and appropriate fix is to add a permissions: block at the top level of the workflow (so it applies to all jobs) with contents: read.

Concretely, edit .github/workflows/ci.yml and insert a new permissions: section between the name: CI and the on: block. No other changes are required, because none of the jobs needs write permissions to GitHub resources. The resulting header of the file will be:

name: CI

permissions:
  contents: read

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

This keeps existing functionality intact while enforcing least‑privilege access for the workflow’s GITHUB_TOKEN.

In general, the fix is to explicitly set a permissions block so that GITHUB_TOKEN has only the minimal scopes required. Since this workflow only checks out code and runs local build/test commands, it only needs read access to repository contents. The simplest and safest change is to add a top-level permissions section with contents: read, which will apply to all jobs (test-rust and test-python) because they do not have their own permissions blocks.

Concretely, in .github/workflows/ci.yml, add a new top-level permissions: mapping near the top of the file (e.g., after name: CI and before on:). Set contents: read as the only permission. No steps need to be modified and no additional imports or actions are required. This preserves existing functionality while constraining the GITHUB_TOKEN to read-only repository contents.

In general, the fix is to explicitly restrict GITHUB_TOKEN permissions in this workflow to the minimum required. Since the jobs only need to check out code, build, upload artifacts, and publish to external registries using separate secrets, they only require read access to repository contents. We can define a single root‐level permissions block that applies to all jobs in the workflow.

Concretely, in .github/workflows/release.yml, add a permissions: section near the top of the file (after name: or after on:) specifying contents: read. This will ensure that all jobs use a GITHUB_TOKEN limited to read‑only access to repository contents. No other code or steps need to change, and no additional imports or definitions are required.

The general fix is to explicitly declare a permissions block that limits the GITHUB_TOKEN to the least privileges needed. Here, the jobs only need to read repository contents to check out code; they do not push commits, modify releases, or interact with issues/PRs. Therefore, adding permissions: contents: read at the workflow root is sufficient and will apply to all jobs in this workflow unless overridden.

Concretely, in .github/workflows/release.yml, add a top-level permissions: section after the name: (or before on:) with contents: read. This single change constrains the default token permissions for all jobs (publish-crate, publish-python, publish-python-sdist, and upload-pypi) without changing any of the existing steps or their behavior. No additional imports, methods, or definitions are required, as this is purely a configuration change in the workflow YAML.

In general, the problem is fixed by adding an explicit permissions: block that restricts the GITHUB_TOKEN to the minimal scopes required. For this workflow, none of the jobs need to write to the repository, or access issues, pull requests, etc. They only need to read the repository contents to build and then use separate registry tokens to publish. So we can safely set contents: read at the workflow level, which applies to all jobs and satisfies CodeQL’s recommendation.

The best fix without changing functionality is to add a single top-level permissions: block right after the on: section in .github/workflows/release.yml:

• Set permissions: contents: read at the root level.
• This will automatically apply to all jobs (publish-crate, publish-python, publish-python-sdist, upload-pypi) since they do not define their own permissions.
• No job requires additional scopes (such as packages or id-token) given the current snippet, because publishing is done with explicit secrets, not GITHUB_TOKEN, and there’s no OIDC usage shown.

No imports or additional methods are needed because this is a YAML workflow configuration change only.

In general, the fix is to add an explicit permissions block that grants the least privileges required by the jobs. Since the shown jobs only need to read repository contents (for checkout) and do not modify issues, PRs, or contents, they can safely run with contents: read. Define this at the top (root) of the workflow so it applies to all jobs, including upload-pypi, which CodeQL highlighted.

Concretely, edit .github/workflows/release.yml and insert a root-level permissions block after the name: Release line (before on:). Use:

permissions:
  contents: read

This keeps GITHUB_TOKEN limited to read-only repository contents for all jobs, without changing the current functionality, because none of the steps need write access. No additional imports, methods, or definitions are required.

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	catsu-docs	fd98d47	Commit Preview URL Branch Preview URL	Jan 07 2026, 08:52 AM

[cloudflare-workers-and-pages]

Deploying catsu with    Cloudflare Pages

Latest commit:	fd98d47
Status:	✅  Deploy successful!
Preview URL:	https://a624cc3e.catsu-3ib.pages.dev
Branch Preview URL:	https://refactor-rust.catsu-3ib.pages.dev

View logs