This PR implements the first two phases of the core ML integration and prepares the backend for the upcoming Admin Dashboard.Feature/core integration - #26
Closed
RanudulaW wants to merge 7 commits into
Conversation
Introduce one-time media JWTs, Postgres media tables, Cloudflare R2 storage, session-gated token routes, and TDD coverage for upload/download. Co-authored-by: Cursor <cursoragent@cursor.com>
Resolve conflicts with dev auth/cases work, renumber media migrations, and add MEDIA_LOCAL_STORAGE to save uploads on disk instead of R2. Co-authored-by: Cursor <cursoragent@cursor.com>
feat(api): add media upload download manager with R2 and JWTs
- Built FastAPI server for LangGraph workflow execution - Refactored ML nodes to use structured output for categorizations - Implemented Go HTTP client to trigger ML API - Created Echo webhook handler in Go to receive async classification updates - Added full unit test coverage for ML layer and backend clients
- Added SQL JOIN logic to group reported cases by category name - Added logic to calculate average resolution times for issues - Structured JSON response for the upcoming Next.js frontend dashboard
There was a problem hiding this comment.
Pull request overview
This PR lays groundwork for “core integration” by adding an initial Python FastAPI/LangGraph classification API, expanding the Go /cases/stats analytics response, and introducing a new backend media upload/download subsystem (JWT-based, backed by Postgres and local disk or Cloudflare R2).
Changes:
- Added
agentic-systemFastAPI/classifyendpoint and a simplified LangGraph workflow + tests. - Expanded Go
CaseStatsanalytics to include counts grouped by category and average resolution time. - Added backend media storage (local/R2), one-time JWT token flows, migrations, router endpoints, and integration-style HTTP tests.
Reviewed changes
Copilot reviewed 54 out of 67 changed files in this pull request and generated 13 comments.
Show a summary per file
| File | Description |
|---|---|
| project_plan.md | Adds a high-level implementation plan for platform phases. |
| backend-api/tests/api/session_test.go | Adds integration test for dev session token issuance. |
| backend-api/tests/api/media_test.go | Adds integration tests for upload/download token flow and media operations. |
| backend-api/tests/api/helpers_test.go | Adds shared test stack helper wiring router + in-memory media services. |
| backend-api/tests/api/health_test.go | Refactors health tests to use shared test stack. |
| backend-api/tests/api/auth_test.go | Updates handler construction signature for auth tests. |
| backend-api/sqlc/sqlc.yaml | Minor sqlc config cleanup (comment removal). |
| backend-api/sqlc/schema/media.sql | Adds sqlc schema for media tables. |
| backend-api/sqlc/schema/.gitkeep | Removes placeholder schema .gitkeep content. |
| backend-api/sqlc/queries/media.sql | Adds sqlc queries for media objects/tokens and token claiming. |
| backend-api/sqlc/queries/.gitkeep | Removes placeholder query .gitkeep content. |
| backend-api/README.md | Documents media feature env vars/usage and migration workflow. |
| backend-api/Makefile | Adds migrate-up/migrate-down targets. |
| backend-api/internal/router/router.go | Wires dev session token and media routes into Echo router. |
| backend-api/internal/ml/client.go | Introduces initial Go HTTP client stub for ML classify API. |
| backend-api/internal/ml/client_test.go | Adds placeholder ML client test (currently stub-based). |
| backend-api/internal/middleware/auth.go | Adds session-JWT middleware for media token issuance routes. |
| backend-api/internal/media/token.go | Implements one-time media JWT issuance/parsing. |
| backend-api/internal/media/token_test.go | Tests one-time media JWT round-trips and error cases. |
| backend-api/internal/media/store.go | Adds in-memory object store implementation. |
| backend-api/internal/media/store_r2.go | Adds Cloudflare R2 (S3 API) object store implementation. |
| backend-api/internal/media/store_local.go | Adds local filesystem object store implementation. |
| backend-api/internal/media/store_local_test.go | Tests local filesystem store behavior and factory selection. |
| backend-api/internal/media/store_factory.go | Adds factory to select local vs R2 store from config. |
| backend-api/internal/media/session.go | Implements session JWT issuance/parsing for media token endpoints. |
| backend-api/internal/media/service.go | Adds media service orchestrating token issuance, upload, download. |
| backend-api/internal/media/service_test.go | Tests media service upload/download flows and constraints. |
| backend-api/internal/media/repository.go | Adds repository abstraction + in-memory repository for tests. |
| backend-api/internal/media/repository_postgres.go | Implements Postgres repository using sqlc-generated queries. |
| backend-api/internal/media/policy.go | Adds scope policy (size/MIME/TTL) for uploads. |
| backend-api/internal/media/policy_test.go | Tests scope policy behaviors. |
| backend-api/internal/media/errors.go | Defines media-specific domain errors. |
| backend-api/internal/handler/webhook.go | Adds ML webhook handler for async classification updates. |
| backend-api/internal/handler/webhook_test.go | Adds unit test for ML webhook handler. |
| backend-api/internal/handler/media.go | Adds HTTP handlers for media token issuance + upload/download. |
| backend-api/internal/handler/handler.go | Extends handler dependency injection to include media service + tokens. |
| backend-api/internal/handler/case.go | Expands /cases/stats response with by-category counts + avg resolution seconds. |
| backend-api/internal/db/sqlcdb/users.sql.go | Regenerates sqlc code (version bump). |
| backend-api/internal/db/sqlcdb/roles.sql.go | Regenerates sqlc code (version bump). |
| backend-api/internal/db/sqlcdb/role_permissions.sql.go | Regenerates sqlc code (version bump). |
| backend-api/internal/db/sqlcdb/querier.go | Adds media query methods to sqlc Querier interface. |
| backend-api/internal/db/sqlcdb/models.go | Adds sqlc models for MediaObject/MediaToken. |
| backend-api/internal/db/sqlcdb/media.sql.go | Adds sqlc-generated media queries implementation. |
| backend-api/internal/db/sqlcdb/entities.sql.go | Regenerates sqlc code (version bump). |
| backend-api/internal/db/sqlcdb/db.go | Regenerates sqlc code (version bump). |
| backend-api/internal/db/sqlcdb/cases.sql.go | Regenerates sqlc code (version bump). |
| backend-api/internal/db/sqlcdb/auth.sql.go | Regenerates sqlc code (version bump). |
| backend-api/internal/db/migrations/000004_create_media_tokens.up.sql | Adds migration for media_tokens table. |
| backend-api/internal/db/migrations/000004_create_media_tokens.down.sql | Adds down migration for media_tokens. |
| backend-api/internal/db/migrations/000003_create_media_objects.up.sql | Adds migration for media_objects table. |
| backend-api/internal/db/migrations/000003_create_media_objects.down.sql | Adds down migration for media_objects. |
| backend-api/internal/db/migrations/.gitkeep | Removes placeholder migrations .gitkeep content. |
| backend-api/internal/config/config.go | Adds media storage config + validates R2 vs local toggle. |
| backend-api/internal/config/config_test.go | Extends config tests for media config validation paths. |
| backend-api/go.sum | Updates module sums for new deps and sqlc regen. |
| backend-api/go.mod | Adds new module dependencies for media/R2 and migrations tooling. |
| backend-api/cmd/server/main.go | Wires media store/repo/service into server startup. |
| backend-api/.env.example | Documents media env vars and R2 settings in example env file. |
| agentic-system/tests/test_graph.py | Adds LangGraph workflow test for classification node output. |
| agentic-system/tests/test_api.py | Adds FastAPI endpoint test for /classify. |
| agentic-system/src/graph/workflow.py | Replaces previous example workflow with a classification-only graph. |
| agentic-system/src/api/server.py | Adds FastAPI server exposing /classify. |
| agentic-system/src/agents/classification_agent.py | Adds classification agent using structured output. |
| agentic-system/pyproject.toml | Adds FastAPI/uvicorn/httpx + pytest tooling to Python project deps. |
Files not reviewed (9)
- backend-api/internal/db/sqlcdb/auth.sql.go: Generated file
- backend-api/internal/db/sqlcdb/cases.sql.go: Generated file
- backend-api/internal/db/sqlcdb/db.go: Generated file
- backend-api/internal/db/sqlcdb/entities.sql.go: Generated file
- backend-api/internal/db/sqlcdb/media.sql.go: Generated file
- backend-api/internal/db/sqlcdb/models.go: Generated file
- backend-api/internal/db/sqlcdb/querier.go: Generated file
- backend-api/internal/db/sqlcdb/role_permissions.sql.go: Generated file
- backend-api/internal/db/sqlcdb/roles.sql.go: Generated file
Suppressed comments (1)
backend-api/README.md:236
- The migration file list in this README section doesn’t match the actual files in
internal/db/migrations/(media migrations are 000003/000004, and there are existing 000001/000002 migrations). This makes the migration instructions confusing for contributors.
Migrations live in `internal/db/migrations/`. Use sequential numbered files and [golang-migrate](https://github.com/golang-migrate/migrate):
internal/db/migrations/
├── 000001_create_media_objects.up.sql
├── 000001_create_media_objects.down.sql
├── 000002_create_media_tokens.up.sql
└── 000002_create_media_tokens.down.sql
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+22
to
+30
| func (c *Client) ClassifyIssue(caseID, description, imageURL string, lat, lng float64) error { | ||
| payload := map[string]interface{}{ | ||
| "description": description, | ||
| "image_url": imageURL, | ||
| "gps_coords": map[string]float64{ | ||
| "lat": lat, | ||
| "lng": lng, | ||
| }, | ||
| } |
Comment on lines
+11
to
+20
| // We'll write the actual Client later | ||
| type MLClient struct { | ||
| BaseURL string | ||
| } | ||
|
|
||
| func (c *MLClient) ClassifyIssue(caseID, description, imageURL string, lat, lng float64) error { | ||
| // A simple HTTP client implementation that sends POST to BaseURL | ||
| // This will be implemented in the main package | ||
| return nil | ||
| } |
Comment on lines
33
to
+47
| // Citizen case intake and category discovery are intentionally public. | ||
| v1.GET("/case-categories", h.ListCaseCategories) | ||
| v1.POST("/cases", h.SubmitCase) | ||
|
|
||
| // Development helper for media session JWTs | ||
| if cfg != nil && cfg.Environment == "development" { | ||
| v1.POST("/dev/session-token", h.DevSessionToken) | ||
| } | ||
|
|
||
| // Media upload/download (one-time JWT for upload/download; session JWT for token issue) | ||
| mediaGroup := v1.Group("/media") | ||
| mediaGroup.POST("/tokens/upload", h.IssueUploadToken, middleware.RequireSessionJWT(tokens)) | ||
| mediaGroup.POST("/tokens/download", h.IssueDownloadToken, middleware.RequireSessionJWT(tokens)) | ||
| mediaGroup.POST("/upload", h.UploadMedia) | ||
| mediaGroup.GET("/download", h.DownloadMedia) |
Comment on lines
+27
to
+38
| func (h *MLWebhookHandler) HandleWebhook(c echo.Context) error { | ||
| var req MLWebhookRequest | ||
| if err := c.Bind(&req); err != nil { | ||
| return c.JSON(http.StatusBadRequest, echo.Map{"error": err.Error()}) | ||
| } | ||
|
|
||
| if err := h.Updater.UpdateCaseMLData(req.CaseID, req.Classification, req.InstitutionID); err != nil { | ||
| return c.JSON(http.StatusInternalServerError, echo.Map{"error": "failed to update case"}) | ||
| } | ||
|
|
||
| return c.JSON(http.StatusOK, echo.Map{"message": "success"}) | ||
| } |
| InstitutionID string `json:"institution_id"` | ||
| } | ||
|
|
||
| func (h *MLWebhookHandler) HandleWebhook(c echo.Context) error { |
Comment on lines
+1
to
+10
| import os | ||
| from dotenv import load_dotenv | ||
|
|
||
| # Load environment variables from .env file before anything else | ||
| load_dotenv() | ||
|
|
||
| from fastapi import FastAPI, HTTPException | ||
| from pydantic import BaseModel | ||
| from typing import Dict, Any, Optional | ||
| from src.graph.workflow import create_workflow |
Comment on lines
+41
to
+42
| except Exception as e: | ||
| raise HTTPException(status_code=500, detail=str(e)) |
Comment on lines
+2
to
+5
| from src.graph.workflow import create_workflow | ||
| from unittest.mock import patch, MagicMock | ||
| from langchain_core.messages import AIMessage | ||
| from src.agents.classification_agent import ClassificationOutput |
Comment on lines
+1
to
+4
| import pytest | ||
| from fastapi.testclient import TestClient | ||
| from unittest.mock import patch | ||
| from src.api.server import app |
Comment on lines
+156
to
+163
| func bearerToken(c echo.Context) (string, error) { | ||
| header := c.Request().Header.Get(echo.HeaderAuthorization) | ||
| if header != "" { | ||
| const prefix = "Bearer " | ||
| if !strings.HasPrefix(header, prefix) { | ||
| return "", echo.NewHTTPError(http.StatusUnauthorized, "Authorization must be Bearer token") | ||
| } | ||
| token := strings.TrimSpace(strings.TrimPrefix(header, prefix)) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes Included
agentic-system): Built a FastAPI server containing a structured LangGraph AI agent to automatically categorize citizen issues based on descriptions and GPS coordinates.backend-api): Implemented an HTTP client to trigger the Python ML API, and added an Echo webhook handler to receive asynchronous classification updates./api/v1/cases/statsendpoint with SQL JOINs to dynamically group case counts by category name and calculate average resolution times.