Sentiment Engine

A lightweight sentiment analytics and topic clustering service for support-style conversations. It accepts transcripts, queues them for processing, stores structured records in Postgres, writes embeddings to Qdrant, clusters emerging themes, labels them with an LLM, and exposes query endpoints for insights and drill-down.

What Is Included

• FastAPI ingestion endpoint at POST /conversations
• Kafka-backed consumer that persists conversations and sentiment
• Batch embedding into Qdrant
• UMAP + HDBSCAN clustering
• Cluster labeling with OpenAI
• Query endpoints for insights, drill-down, and semantic search
• Docker Compose stack for local end-to-end runs

Prerequisites

• Python 3.11+
• Docker and Docker Compose for the full stack
• OPENAI_API_KEY

Local Python Setup

python -m pip install -e .[dev]
pytest -q

The editable install makes the repo packages importable for tests and local scripts.

Docker Setup

1. Copy .env.example to .env if you want a separate local config file.
2. Fill in the OpenAI API key.
3. Start the stack:

docker compose up

4. Seed sample data once the API is healthy:

python scripts/seed.py

The pipeline service consumes messages continuously, runs an embed pass on startup, and then follows the configured cron schedules for ongoing embedding and clustering.

For the default local config in .env.example:

• embeddings run every 30 minutes
• clustering runs nightly at 2am

For a faster demo loop, reduce those cron intervals in .env before starting the stack.

Useful Endpoints

• GET /health
• POST /conversations
• GET /insights
• GET /clusters/{cluster_id}/conversations
• GET /search?q=refund

GET /insights returns only the latest clustering run by default. You can page with offset and limit, or pass a run_id to inspect a historical run explicitly.

Demo Flow

1. Start the stack with docker compose up.
2. Open http://localhost:8000/docs.
3. Seed the sample dataset with python scripts/seed.py.
4. Validate semantic retrieval first with GET /search?q=refund.
5. Validate PM-facing summaries with GET /insights.
6. Open one cluster through GET /clusters/{cluster_id}/conversations to show transcript drill-down.

GET /search is the best first signal that ingestion, embedding, and Qdrant are all working. GET /insights may take longer because it depends on the clustering and labeling jobs.

Notes

• The Docker stack installs the local package at container startup for simplicity.
• The clustering libraries are imported lazily so the rest of the project can still be imported in environments that do not yet have the full ML stack installed.
• Embedding and cluster labeling now both use OpenAI; LABEL_MODEL defaults to gpt-4o-mini.
• The clustering configuration is tuned to stay stable on the included 50-conversation demo fixture as well as on larger batches.
• If you change application code while the stack is running, restart the relevant container with docker compose restart api or docker compose restart pipeline.