Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
207 changes: 92 additions & 115 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Use `bd` for ALL tasks/issues (no markdown TODO lists).

## Project Overview

**Perception** is an AI news intelligence platform using Google's Agent Development Kit (ADK). 8 specialized agents on Vertex AI Agent Engine coordinate to collect, analyze, and synthesize news into executive briefs.
**Perception** is an AI news intelligence platform using Google's Agent Development Kit (ADK). 9 specialized agents on Vertex AI Agent Engine coordinate to collect, analyze, and synthesize news into executive briefs.

**GCP Project:** `perception-with-intent`
**Dashboard:** https://perception-with-intent.web.app
Expand All @@ -25,87 +25,118 @@ Use `bd` for ALL tasks/issues (no markdown TODO lists).
## Architecture

```
Firebase Dashboard (React)
Firebase Dashboard (React + Vite + TypeScript)
↓ Firestore real-time listeners + auto-ingestion trigger
Vertex AI Agent Engine (9 Agents via A2A Protocol)
↓ ADK sub_agents + config_path references
Cloud Run MCP Service (FastAPI, 7 tool endpoints + /trigger/ingestion)
Vertex AI Agent Engine (8 Agents via A2A Protocol)
Cloud Run MCP Service (7 tool endpoints)
Firestore (articles, daily_summaries, topics_to_monitor)
Firestore (named database: "perception-db")
└── Collections: articles, daily_summaries, topics_to_monitor
```

**Key principle:** Firebase = humans, Agents = thinking, MCPs = doing, Firestore = storage.

### Two Entrypoints

The agent system has two different entrypoints:

1. **Local dev** (`perception_app/main.py`) - Imports `jvp_agent.build_a2a_agent()` and runs via uvicorn. The `jvp_agent` package is a **top-level module** (not inside `perception_app/`), providing the A2A protocol wrapper.

2. **Production** (`perception_app/agent_engine_app.py`) - Loads the orchestrator YAML config for Vertex AI Agent Engine deployment. Currently stubbed with TODOs for ADK loader integration.

### Agent System (perception_app/perception_agent/agents/)

| Agent | YAML Config | Role |
|-------|-------------|------|
| 0 | `agent_0_orchestrator.yaml` | Editor-in-Chief, coordinates workflow |
| 1 | `agent_1_source_harvester.yaml` | Fetches news from RSS/APIs |
| 2 | `agent_2_topic_manager.yaml` | Manages tracked keywords |
| 3 | `agent_3_relevance_ranking.yaml` | Scores articles 1-10 |
| 4 | `agent_4_brief_writer.yaml` | Generates summaries |
| 5 | `agent_5_alert_anomaly.yaml` | Detects alerts |
| 6 | `agent_6_validator.yaml` | Data quality checks |
| 7 | `agent_7_storage_manager.yaml` | Firestore persistence |
| 8 | `agent_8_tech_editor.yaml` | Technology section editor |
Agent 0 (orchestrator) loads all sub-agents via `config_path` references in its YAML. Each agent has a corresponding tools file (`perception_app/perception_agent/tools/agent_N_tools.py`).

| Agent | Role |
|-------|------|
| 0 - Orchestrator | Editor-in-Chief, coordinates the full ingestion workflow |
| 1 - Source Harvester | Fetches news from RSS/APIs |
| 2 - Topic Manager | Manages tracked keywords |
| 3 - Relevance Ranking | Scores articles 1-10 |
| 4 - Brief Writer | Generates executive summaries |
| 5 - Alert & Anomaly | Detects alerts worth surfacing |
| 6 - Validator | Data quality checks |
| 7 - Storage Manager | Firestore persistence |
| 8 - Tech Editor | Technology section curation |

### MCP Service (perception_app/mcp_service/)

FastAPI endpoints at `/mcp/tools/*`:
- `fetch_rss_feed` - RSS fetching (real implementation)
- `fetch_api_feed`, `fetch_webpage` - Content fetching (stubbed)
- `store_articles`, `generate_brief` - Storage/synthesis (stubbed)
- `log_ingestion_run`, `send_notification` - Ops (stubbed)
FastAPI app with routers at `/mcp/tools/*`. Real implementation: `fetch_rss_feed`. Stubbed: `fetch_api_feed`, `fetch_webpage`, `store_articles`, `generate_brief`, `log_ingestion_run`, `send_notification`. Also exposes `/trigger/ingestion` for dashboard auto-ingestion.

### Dashboard (dashboard/)

React 18 SPA with Firebase Auth (protected routes), Firestore real-time data, and an auto-ingestion hook (`useAutoIngestion`) that fires once per session on Dashboard mount. Uses shadcn/ui components (`dashboard/src/components/ui/`), Framer Motion animations, and Tremor charts.

Routes: `/` (Articles feed), `/dashboard` (command center), `/briefs`, `/authors`, `/topics`, `/about` (landing), `/login`.

**Firestore named database:** The dashboard connects to `perception-db` (not the default database): `getFirestore(app, 'perception-db')`.

### Functions (functions/)

Firebase Cloud Functions (Node.js 22). Minimal setup — `firebase-admin` + `firebase-functions` only. No TypeScript source files yet.

## Commands

### Setup
```bash
make install # Create venv and install deps
make install # Create venv + install requirements.txt
pip install -r requirements-test.txt # Additional test deps (separate file)
gcloud auth application-default login
gcloud config set project perception-with-intent
```

### Development
```bash
# Agent system (local)
make dev # Run local ADK server (localhost:8080)
# Or directly:
python -m perception_app.main --host 127.0.0.1 --port 8080

# Verify agent configs
# Dashboard
cd dashboard && npm install && npm run dev # Vite dev server

# Verify agent YAML configs load
python perception_app/agent_engine_app.py
```

### Testing
```bash
# Run all tests
make test
# Or: pytest tests/ -v --cov=perception_app
make test # Note: Makefile uses --cov=app (legacy path)
pytest tests/ -v --cov=perception_app # Correct coverage target

# Run specific test suites (as CI does)
# By test directory (matches CI)
pytest tests/unit/ -v --no-cov
pytest tests/api/ -v --no-cov
pytest tests/mcp/ -v --no-cov
pytest tests/agent/ -v --no-cov
pytest tests/security/ -v --no-cov

# Run single test file
# By marker (defined in pyproject.toml)
pytest -m unit # Fast, no external deps
pytest -m smoke # Quick smoke tests
pytest -m "not slow" # Skip slow tests
pytest -m e2e # Playwright browser tests (needs: playwright install)

# Single file
pytest tests/unit/test_rss_parsing.py -v

# Full production suite (~5,196 tests)
# Full suite (~5,196 tests)
./scripts/run_all_tests.sh
```

### Linting
```bash
make lint # Run all linters
make lint # Note: Makefile targets app/ (legacy path)
make format # Auto-format with black

# Individual tools
# Run against correct paths (both app/ and perception_app/ exist)
black --check perception_app/
ruff check perception_app/
mypy perception_app/

# Dashboard
cd dashboard && npm run lint
```

### Deployment
Expand All @@ -121,94 +152,43 @@ gcloud run deploy perception-mcp \
# Dashboard
cd dashboard && npm run build && firebase deploy --only hosting

# Terraform
cd infra/terraform/envs/dev
tofu init && tofu plan && tofu apply
```

### Verify MCP is Alive
```bash
curl https://perception-mcp-w53xszfqnq-uc.a.run.app/health

curl -X POST https://perception-mcp-w53xszfqnq-uc.a.run.app/mcp/tools/fetch_rss_feed \
-H "Content-Type: application/json" \
-d '{"feed_url": "https://news.ycombinator.com/rss", "max_items": 10}'
```

## Project Structure
# Functions
firebase deploy --only functions

# Terraform
cd infra/terraform/envs/dev && tofu init && tofu plan && tofu apply
```
perception/
├── perception_app/
│ ├── main.py # Dev entrypoint (uvicorn)
│ ├── agent_engine_app.py # Production entrypoint (Vertex AI)
│ ├── perception_agent/
│ │ ├── agents/ # Agent YAML configs (agent_0-8)
│ │ ├── tools/ # Python tools per agent (agent_*_tools.py)
│ │ └── config/
│ │ └── rss_sources.yaml # RSS feed sources
│ ├── jvp_agent/ # A2A protocol wrapper
│ │ ├── agent.yaml # JVP agent config
│ │ ├── a2a.py # A2A protocol implementation
│ │ └── memory.py # Context caching
│ └── mcp_service/
│ ├── main.py # FastAPI app
│ └── routers/ # Tool endpoints (rss.py, storage.py, etc.)
├── dashboard/ # React + Vite + TypeScript
├── infra/terraform/ # OpenTofu IaC
├── scripts/
│ ├── dev_run_adk.sh # Local dev server
│ ├── deploy_agent_engine.sh # Deploy to Vertex AI
│ ├── run_all_tests.sh # Full test suite
│ └── run_ingestion_once.py # Manual ingestion trigger
├── tests/
│ ├── unit/ # Unit tests
│ ├── api/ # API tests
│ ├── mcp/ # MCP router tests
│ ├── agent/ # Agent tests
│ ├── security/ # Security tests
│ └── factories/ # Test data factories
└── 000-docs/ # Technical documentation
```

## Key Files

- `perception_app/agent_engine_app.py:36` - Agent loading function
- `perception_app/perception_agent/agents/agent_0_orchestrator.yaml:1` - Root orchestrator config
- `perception_app/mcp_service/main.py:39` - MCP FastAPI app
- `.github/workflows/ci.yml` - CI pipeline (lint, test, security, tofu)

## CI/CD

Push to `main` triggers GitHub Actions:
1. **lint** - black, ruff, mypy
2. **test** - pytest (unit, api, mcp, agent, security)
3. **security** - safety check dependencies
4. **tofu** - Terraform/OpenTofu validate
Push to `main` or `develop` triggers `.github/workflows/ci.yml`:
1. **lint** - black, ruff (both `app/` and `perception_app/` paths)
2. **test** - pytest by directory (unit, api, mcp, agent, security) using `requirements-test.txt`
3. **security** - safety check on requirements.txt
4. **tofu** - OpenTofu init/fmt/validate on `infra/terraform/envs/dev`

Deployment workflows:
- `.github/workflows/deploy-mcp.yml` - MCP to Cloud Run
- `.github/workflows/deploy-agent-engine.yml` - Agents to Vertex AI
- `.github/workflows/deploy-dashboard.yml` - Dashboard to Firebase
Deployment workflows (separate files):
- `deploy-mcp.yml` - MCP to Cloud Run
- `deploy-agent-engine.yml` / `deploy-agents.yml` - Agents to Vertex AI
- `deploy-dashboard.yml` / `deploy-firebase-dashboard.yml` - Dashboard to Firebase

Uses Workload Identity Federation (WIF) for keyless GCP auth.

## Tech Stack
## Cloud-Only MCP Policy

- **Agents:** Google ADK, Vertex AI Agent Engine, A2A Protocol
- **AI Model:** Gemini 2.0 Flash
- **MCP Service:** FastAPI on Cloud Run
- **Database:** Firestore (us-central1, `perception-db`)
- **Dashboard:** React 18 + Vite + TypeScript + TailwindCSS
- **IaC:** OpenTofu (Terraform-compatible)
- **Observability:** Cloud Logging, OpenTelemetry (partial)
MCPs run ONLY on Cloud Run. No localhost MCP servers.
- Agents: Local development OK (`make dev`)
- MCP: Cloud Run only — never run locally
- Tests: Use staging Cloud Run URLs
- Dashboard auto-ingestion hits the Cloud Run MCP directly

## Cloud-Only MCP Policy
## Codebase Gotchas

MCPs run ONLY on Cloud Run. No localhost MCP servers:
- ✅ Agents: Local development OK (`make dev`)
- ❌ MCP: Cloud Run only (no local testing)
- ✅ Tests: Use staging Cloud Run URLs
- **Dual source paths**: Both `app/` and `perception_app/` exist. The Makefile lint/test targets reference `app/` (legacy), while active code is in `perception_app/`. CI lints both.
- **`jvp_agent` is top-level**: Not inside `perception_app/`. It's imported directly by `perception_app/main.py`.
- **Named Firestore database**: `perception-db`, not `(default)`. Must pass this name when initializing Firestore clients.
- **pytest asyncio**: `asyncio_mode = "auto"` in pyproject.toml — no need for `@pytest.mark.asyncio`.
- **Line length**: 120 chars (black + ruff both configured to 120).

## Environment Variables

Expand All @@ -219,13 +199,10 @@ VERTEX_LOCATION=us-central1
VERTEX_AGENT_ENGINE_ID=4241324322704064512
```

Dashboard uses `VITE_FIREBASE_*` env vars (with hardcoded fallbacks in `dashboard/src/firebase.ts`).

## Documentation

All docs in `000-docs/` with naming convention: `PREFIX-TYPE-CATEGORY-description.md`
- `6767-*` = Evergreen (architecture, guides, plans)
- `0XX-*` = Sequential (AARs, phase reports)

Key docs:
- `6767-AT-ARCH-observability-and-monitoring.md` - Monitoring guide
- `6767-OD-GUID-agent-engine-deploy.md` - Deployment guide
- `6767-PP-PLAN-release-log.md` - Version history
14 changes: 14 additions & 0 deletions dashboard/src/components/IngestionButton.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,20 @@ export default function IngestionButton() {
[stopPolling, handleComplete]
)

// Listen for auto-ingestion started by useAutoIngestion hook
useEffect(() => {
const handler = (e: Event) => {
const runId = (e as CustomEvent<{ run_id: string }>).detail.run_id
if (phase === 'idle' && runId) {
setPhase('starting')
startTimeRef.current = Date.now()
pollStatus(runId)
}
}
window.addEventListener('auto-ingestion-started', handler)
return () => window.removeEventListener('auto-ingestion-started', handler)
}, [phase, pollStatus])
Comment on lines +148 to +160

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

2. Missed auto-ingestion event 🐞 Bug ⛯ Reliability

Auto-ingestion progress can be missed because a one-shot CustomEvent is dispatched from one
component effect while the listener is registered in another component effect, with no guaranteed
ordering. This can leave the IngestionButton idle even though an ingestion run is active (more
likely under React StrictMode dev double-mount).
Agent Prompt
### Issue description
Auto-ingestion communicates the run id via a one-shot window event. Because listener registration happens in a different component effect, the event can be dispatched before the listener exists, so the UI never starts polling.

### Issue Context
- `useAutoIngestion()` dispatches `auto-ingestion-started` after receiving 202.
- `IngestionButton` only polls if it receives that event.
- There is no fallback to discover an active run if the event is missed.

### Fix Focus Areas
- dashboard/src/hooks/useAutoIngestion.ts[7-38]
- dashboard/src/components/IngestionButton.tsx[148-160]
- dashboard/src/pages/Dashboard.tsx[35-56]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


const handleRunIngestion = async () => {
setPhase('starting')
startTimeRef.current = Date.now()
Expand Down
40 changes: 40 additions & 0 deletions dashboard/src/hooks/useAutoIngestion.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
import { useEffect } from 'react'
import { toast } from 'sonner'

const MCP_URL = 'https://perception-mcp-w53xszfqnq-uc.a.run.app'

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The MCP_URL is hardcoded to a specific production endpoint and is duplicated in IngestionButton.tsx. It is better to use an environment variable to allow for different environments (local, staging, production) and to centralize the configuration.

Suggested change
const MCP_URL = 'https://perception-mcp-w53xszfqnq-uc.a.run.app'
const MCP_URL = import.meta.env.VITE_MCP_URL || 'https://perception-mcp-w53xszfqnq-uc.a.run.app'

const SESSION_KEY = 'perception-auto-ingestion-fired'

export default function useAutoIngestion() {
useEffect(() => {
if (sessionStorage.getItem(SESSION_KEY)) return

sessionStorage.setItem(SESSION_KEY, '1')

fetch(`${MCP_URL}/trigger/ingestion`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
trigger: 'auto',
time_window_hours: 24,
max_items_per_source: 50,
}),
})
.then(async (res) => {
if (res.status === 202) {
const data = await res.json()
toast.info('Refreshing your feed...', {
description: `Run ID: ${data.run_id}`,
})
Comment on lines +22 to +27
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

data.run_id is untyped; toast can render "Run ID: undefined" if the response shape changes.

res.json() returns any, so data.run_id silently falls through as undefined if the API response doesn't include that field.

🛡️ Proposed fix
-        if (res.status === 202) {
-          const data = await res.json()
-          toast.info('Refreshing your feed...', {
-            description: `Run ID: ${data.run_id}`,
-          })
-          window.dispatchEvent(
-            new CustomEvent('auto-ingestion-started', {
-              detail: { run_id: data.run_id },
-            })
-          )
-        }
+        if (res.status === 202) {
+          const data = await res.json() as { run_id?: string }
+          if (data.run_id) {
+            toast.info('Refreshing your feed...', {
+              description: `Run ID: ${data.run_id}`,
+            })
+            window.dispatchEvent(
+              new CustomEvent('auto-ingestion-started', {
+                detail: { run_id: data.run_id },
+              })
+            )
+          }
+        }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@dashboard/src/hooks/useAutoIngestion.ts` around lines 22 - 27, The response
from res.json() is untyped so data.run_id may be undefined and toast could show
"Run ID: undefined"; in useAutoIngestion, define an interface (e.g.,
AutoIngestionResponse { run_id?: string }) or cast the JSON to that type, then
guard or normalize the value before passing to toast (e.g., const runId =
data?.run_id ?? 'unknown' or only include the Run ID description when present)
so the toast always receives a safe, meaningful string and you avoid rendering
"undefined".

window.dispatchEvent(
new CustomEvent('auto-ingestion-started', {
detail: { run_id: data.run_id },
})
)
}
// 409 = already running, silently ignore
})
.catch(() => {
// Fire-and-forget — don't block dashboard load
})
Comment on lines +13 to +38

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

1. useautoingestion() swallows fetch errors 📘 Rule violation ⛯ Reliability

The new auto-ingestion request silently ignores network/JSON/server failures and unexpected HTTP
statuses, making production issues hard to detect and debug. This can leave users with a stale
dashboard with no actionable feedback or telemetry.
Agent Prompt
## Issue description
`useAutoIngestion()` currently ignores non-`202` HTTP responses and swallows all fetch/parse errors in an empty `.catch()`, creating silent failures.

## Issue Context
Auto-ingestion is a critical freshness mechanism for the dashboard. When it fails, we need at least (a) safe user-facing feedback (generic), and (b) internal diagnostics (logging/telemetry) with enough context (status code, request path, correlation/run id if available).

## Fix Focus Areas
- dashboard/src/hooks/useAutoIngestion.ts[13-38]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Comment on lines +9 to +38

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

3. Session guard blocks retry 🐞 Bug ⛯ Reliability

The auto-ingestion sessionStorage guard is set before the POST succeeds and errors are swallowed. If
the request fails, auto-ingestion won’t retry for the rest of the browser session, leaving the feed
stale.
Agent Prompt
### Issue description
Auto-ingestion sets the session guard before the network request completes. If the request fails, the guard remains and prevents retries for the whole session.

### Issue Context
Current implementation intentionally swallows errors, but should not permanently disable auto-ingestion due to transient failures.

### Fix Focus Areas
- dashboard/src/hooks/useAutoIngestion.ts[8-38]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

}, [])
}
3 changes: 3 additions & 0 deletions dashboard/src/pages/Articles.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -234,18 +234,21 @@ export default function Articles() {

try {
const articlesRef = collection(db, 'articles')
const cutoff = new Date(Date.now() - 48 * 60 * 60 * 1000).toISOString()
let q

if (selectedCategory === 'all') {
q = query(
articlesRef,
where('published_at', '>=', cutoff),
orderBy('published_at', 'desc'),
limit(100)
)
} else {
q = query(
articlesRef,
where('category', '==', selectedCategory),
where('published_at', '>=', cutoff),
orderBy('published_at', 'desc'),
limit(100)
Comment on lines 248 to 253

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This query combines an equality filter (category) with a range filter and ordering on another field (published_at). In Firestore, this specific combination requires a composite index. If this index hasn't been created yet in the Firebase Console, this query will fail at runtime with a failed-precondition error. Please ensure the index for category: ASC, published_at: DESC is deployed.

)
Comment on lines 247 to 254
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verify that this composite query works without additional Firestore indexes. The query combines category ==, published_at >=, and orderBy published_at desc. While the existing index (category ASC, published_at DESC) should handle this, test in production to confirm no index errors occur.

Expand Down
3 changes: 3 additions & 0 deletions dashboard/src/pages/Dashboard.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import SystemActivityCard from '../components/SystemActivityCard'
import AuthorsCard from '../components/AuthorsCard'
import FooterBranding from '../components/FooterBranding'
import IngestionButton from '../components/IngestionButton'
import useAutoIngestion from '../hooks/useAutoIngestion'

// Stagger animation variants for cards
const containerVariants = {
Expand All @@ -32,6 +33,8 @@ const itemVariants = {
}

export default function Dashboard() {
useAutoIngestion()

return (
<motion.div
initial="hidden"
Expand Down
Loading