Master

.gitignore

@@ -79,3 +79,6 @@ npm-debug.log*
 yarn-debug.log*
 yarn-error.log*web/.next/
 CLAUDE.md
+web/.next/
+web/tsconfig.tsbuildinfo
+scenarios-market-analysis.md

AGENTS.md

@@ -0,0 +1,40 @@
+# Repository Guidelines
+
+## Project Structure & Module Organization
+- `api/`: FastAPI backend (Alembic migrations, Celery, tests under `api/tests`).
+- `web/`: Next.js 15 + TypeScript frontend (TailwindCSS, ESLint).
+- `scripts/`: Helper scripts for seeding and demos.
+- `sample_dataset_templates/`: CSV templates for datasets.
+- `docker-compose*.yml`: Local dev and service orchestration.
+- `Makefile`: Common developer commands.
+
+## Build, Test, and Development Commands
+- Backend (Dockerized): `make dev` — start API, DB, Redis for local dev.
+- Full demo: `make demo` — migrate, start, seed, quick API checks.
+- DB migrations: `make migrate` — run Alembic upgrades.
+- Seed data: `make seed` — insert demo evaluators/scenarios/datasets.
+- Inspect: `make logs` (API logs), `make status` (container status).
+- Stop/clean: `make clean` — down + prune volumes.
+- Frontend: `cd web && npm run dev | build | start | lint`.
+- Backend tests: `cd api && pytest -v` (use `-m unit|integration|auth|api`).
+
+## Coding Style & Naming Conventions
+- Python (api): 4-space indentation; PEP 8; modules `snake_case.py`; classes `PascalCase`; functions/vars `snake_case`.
+- TypeScript/React (web): components export `PascalCase`; files in `web/src/components` use kebab-case (e.g., `scenario-preview-modal.tsx`).
+- Imports: prefer relative within package boundaries; avoid circular deps.
+- Formatting: use project defaults (ESLint for `web/` via `npm run lint`).
+
+## Testing Guidelines
+- Framework: Pytest (`api/pytest.ini` configured). Test files: `test_*.py`, classes `Test*`, functions `test_*`.
+- Run: `cd api && pytest -v` or targeted (e.g., `pytest tests/test_auth.py -m auth`).
+- Optional coverage: `pytest --cov=.` if `pytest-cov` is available.
+- Add minimal, focused tests near the code under test; prefer unit over integration unless necessary.
+
+## Commit & Pull Request Guidelines
+- Commits: concise imperative subject (≤72 chars). Example: `feat(api): add PII regex evaluator`.
+- Include rationale and scope in body; reference issues (`Closes #123`).
+- PRs: clear description, steps to test, screenshots for UI changes, and linked issue(s). Keep changes scoped.
+
+## Security & Configuration Tips
+- Never commit secrets. Use `.env.example` to document new variables. Local envs: root `.env`, `api/.env`, `web/.env.local`.
+- Default local URLs: API `http://localhost:8000`, Web `http://localhost:3000`.

CONTRIBUTING.md

@@ -0,0 +1,37 @@
+# Contributing
+
+Thanks for your interest in improving EvalWise! This guide summarizes how to get set up, develop, and submit changes. For detailed project conventions, see `AGENTS.md`.
+
+## Getting Started
+- Prereqs: Docker + Docker Compose, Make, Node.js (for `web/`).
+- Environment: copy `.env.example` to `.env` (and `api/.env`, `web/.env.local` as needed). Never commit secrets.
+
+## Local Development
+- Start services: `make dev` (API, Postgres, Redis) or `make demo` (migrate + seed + smoke checks).
+- Useful targets: `make migrate`, `make seed`, `make logs`, `make status`, `make clean`.
+- Frontend: `cd web && npm run dev` (and `npm run lint` before PRs).
+
+## Testing
+- Backend tests: `cd api && pytest -v`.
+- Markers: `-m unit|integration|auth|api` (e.g., `pytest -m auth`).
+- Optional coverage: `pytest --cov=.` if `pytest-cov` is available.
+
+## Coding Standards
+- Python: PEP 8, 4-space indentation, `snake_case` for files/functions/variables, `PascalCase` for classes.
+- TypeScript/React: `PascalCase` components, kebab-case component filenames in `web/src/components`.
+- See `AGENTS.md` for structure, naming, and additional tips.
+
+## Commit & PR Process
+- Branch: `feature/<short-name>`, `fix/<short-name>`, or `chore/<short-name>`.
+- Commits: imperative and scoped. Example: `feat(api): add PII regex evaluator`.
+- PRs should include:
+  - Purpose and summary of changes
+  - Testing notes (commands, expected outputs)
+  - Screenshots for UI changes
+  - Linked issue references (e.g., `Closes #123`)
+
+## Resources
+- Project guidelines: `AGENTS.md`
+- Makefile commands: `make help`
+- API docs (local): `http://localhost:8000/docs`
+- Web app (local): `http://localhost:3000`

api/adapters/factory.py

@@ -3,6 +3,9 @@
 from .openai_adapter import OpenAIAdapter
 from .ollama_adapter import OllamaAdapter
 import os
+from sqlalchemy.orm import Session
+from sqlalchemy import create_engine
+from database import get_db
 
 class ModelAdapterFactory:
     """Factory for creating model adapters"""
@@ -14,19 +17,52 @@ class ModelAdapterFactory:
         "ollama": OllamaAdapter,
     }
 
+    @classmethod
+    def get_organization_api_key(cls, provider: str, organization_id: str, db: Session) -> Optional[str]:
+        """Retrieve and decrypt API key for provider from organization's stored keys"""
+        try:
+            from auth.models import EncryptedApiKey
+            from utils.encryption import encryption
+
+            # Find the API key for this provider and organization
+            encrypted_key_record = db.query(EncryptedApiKey).filter(
+                EncryptedApiKey.provider == provider,
+                EncryptedApiKey.organization_id == organization_id,
+                EncryptedApiKey.is_active == True
+            ).first()
+
+            if not encrypted_key_record:
+                return None
+
+            # Decrypt the API key
+            return encryption.decrypt_api_key(encrypted_key_record.encrypted_key)
+
+        except Exception as e:
+            print(f"Failed to retrieve API key for {provider}: {str(e)}")
+            return None
+
     @classmethod
     def create_adapter(
-        self,
+        cls,
         provider: str,
         api_key: Optional[str] = None,
-        base_url: Optional[str] = None
+        base_url: Optional[str] = None,
+        organization_id: Optional[str] = None,
+        db: Optional[Session] = None
     ) -> BaseModelAdapter:
         """Create adapter for the specified provider"""
 
-        if provider not in self._adapters:
+        if provider not in cls._adapters:
             raise ValueError(f"Unsupported provider: {provider}")
 
-        adapter_class = self._adapters[provider]
+        adapter_class = cls._adapters[provider]
+
+        # Try to get API key from organization's stored keys first
+        if not api_key and organization_id and db:
+            stored_api_key = cls.get_organization_api_key(provider, organization_id, db)
+            if stored_api_key:
+                api_key = stored_api_key
+                print(f"Using stored API key for {provider} from organization {organization_id}")
 
         # Handle provider-specific configurations
         if provider == "openai":

api/alembic/versions/b8f5c9a3d2e1_remove_usage_tracking_from_api_keys.py

@@ -0,0 +1,28 @@
+"""remove_usage_tracking_from_api_keys
+
+Revision ID: b8f5c9a3d2e1
+Revises: 9b8d8990d1c0
+Create Date: 2025-01-05 10:00:00.000000
+
+"""
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision = 'b8f5c9a3d2e1'
+down_revision = '9b8d8990d1c0'
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    # Remove usage tracking columns from encrypted_api_keys table
+    op.drop_column('encrypted_api_keys', 'last_used')
+    op.drop_column('encrypted_api_keys', 'usage_count')
+
+
+def downgrade() -> None:
+    # Add the columns back if we need to rollback
+    op.add_column('encrypted_api_keys', sa.Column('usage_count', sa.Integer(), nullable=True, default=0))
+    op.add_column('encrypted_api_keys', sa.Column('last_used', sa.DateTime(), nullable=True))

api/alembic/versions/c9d4e8f7a6b5_add_llm_providers_table.py

@@ -0,0 +1,44 @@
+"""add_llm_providers_table
+
+Revision ID: c9d4e8f7a6b5
+Revises: b8f5c9a3d2e1
+Create Date: 2025-01-05 10:30:00.000000
+
+"""
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision = 'c9d4e8f7a6b5'
+down_revision = 'b8f5c9a3d2e1'
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    # Create llm_providers table
+    op.create_table('llm_providers',
+        sa.Column('id', sa.UUID(), nullable=False),
+        sa.Column('organization_id', sa.UUID(), nullable=False),
+        sa.Column('name', sa.String(), nullable=False),
+        sa.Column('provider_type', sa.String(), nullable=False),
+        sa.Column('encrypted_api_key', sa.Text(), nullable=True),
+        sa.Column('base_url', sa.String(), nullable=True),
+        sa.Column('default_model_name', sa.String(), nullable=False),
+        sa.Column('default_temperature', sa.Float(), nullable=False),
+        sa.Column('default_max_tokens', sa.Integer(), nullable=False),
+        sa.Column('is_default', sa.Boolean(), nullable=True),
+        sa.Column('is_active', sa.Boolean(), nullable=True),
+        sa.Column('created_at', sa.DateTime(), nullable=True),
+        sa.Column('updated_at', sa.DateTime(), nullable=True),
+        sa.Column('created_by', sa.UUID(), nullable=False),
+        sa.ForeignKeyConstraint(['created_by'], ['users.id'], ),
+        sa.ForeignKeyConstraint(['organization_id'], ['organizations.id'], ),
+        sa.PrimaryKeyConstraint('id')
+    )
+
+
+def downgrade() -> None:
+    # Drop the table if we need to rollback
+    op.drop_table('llm_providers')

api/auth/models.py

@@ -1,4 +1,4 @@
-from sqlalchemy import Column, String, DateTime, Boolean, Text, ForeignKey, Integer
+from sqlalchemy import Column, String, DateTime, Boolean, Text, ForeignKey, Integer, Float
 from sqlalchemy.dialects.postgresql import UUID
 from sqlalchemy.orm import relationship
 from datetime import datetime
@@ -24,6 +24,7 @@ class Organization(Base):
     # Relationships
     user_organizations = relationship("UserOrganization", back_populates="organization")
     encrypted_api_keys = relationship("EncryptedApiKey", back_populates="organization")
+    llm_providers = relationship("LLMProvider", back_populates="organization")
 
     def __repr__(self):
         return f"<Organization(name='{self.name}')>"
@@ -97,17 +98,43 @@ class EncryptedApiKey(Base):
     updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
     created_by = Column(UUID(as_uuid=True), ForeignKey("users.id"), nullable=False)
 
-    # Usage tracking
-    last_used = Column(DateTime, nullable=True)
-    usage_count = Column(Integer, default=0)
-
     # Relationships
     organization = relationship("Organization", back_populates="encrypted_api_keys")
 
     def __repr__(self):
         return f"<EncryptedApiKey(provider='{self.provider}', name='{self.key_name}')>"
 
 
+class LLMProvider(Base):
+    """LLM Provider configurations for organizations"""
+    __tablename__ = "llm_providers"
+
+    id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
+    organization_id = Column(UUID(as_uuid=True), ForeignKey("organizations.id"), nullable=False)
+    name = Column(String, nullable=False)  # User-friendly name like "My OpenAI Account"
+    provider_type = Column(String, nullable=False)  # openai, ollama, azure_openai, etc.
+    encrypted_api_key = Column(Text, nullable=True)  # AES encrypted API key (nullable for local providers)
+    base_url = Column(String, nullable=True)  # Custom base URL
+
+    # Model defaults
+    default_model_name = Column(String, nullable=False)
+    default_temperature = Column(Float, nullable=False, default=0.7)
+    default_max_tokens = Column(Integer, nullable=False, default=1000)
+
+    # Metadata
+    is_default = Column(Boolean, default=False)
+    is_active = Column(Boolean, default=True)
+    created_at = Column(DateTime, default=datetime.utcnow)
+    updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
+    created_by = Column(UUID(as_uuid=True), ForeignKey("users.id"), nullable=False)
+
+    # Relationships
+    organization = relationship("Organization", back_populates="llm_providers")
+
+    def __repr__(self):
+        return f"<LLMProvider(name='{self.name}', type='{self.provider_type}')>"
+
+
 class LoginAttempt(Base):
     """Track failed login attempts for rate limiting"""
     __tablename__ = "login_attempts"