Merge pull request #28 from navapbc/fg/playwright-artifacts-schema

feat: migrate traces and artifacts storage to GCP deployed db

Author: fg-nava

mastra-test-app/docs/DEPLOYMENT_NOTES.md

@@ -6,7 +6,7 @@ This guide provides an exact, copy‑pasteable sequence to deploy `mastra-test-a
 - Google Cloud project with billing enabled
 - `gcloud` CLI installed and authenticated
 - API keys: OpenAI, Anthropic, Exa
-- PostgreSQL connection URL (currently Neon)
+- GCP Cloud SQL PostgreSQL database (see [GCP_POSTGRESQL_SETUP.md](./GCP_POSTGRESQL_SETUP.md))
 
 ---
 
@@ -55,14 +55,32 @@ cd labs-asp-experiments/mastra-test-app
 pnpm install
 ```
 
-### 5) Configure environment
+### 5) Authorize VM IP for database access
+```bash
+# Get the VM's external IP
+EXTERNAL_IP=$(gcloud compute instances describe mastra-app \
+  --zone=us-west1-a \
+  --format='get(networkInterfaces[0].accessConfigs[0].natIP)')
+
+echo "VM External IP: $EXTERNAL_IP"
+
+# Add VM IP to Cloud SQL authorized networks (replace existing IPs as needed)
+gcloud sql instances patch app-dev --authorized-networks=$EXTERNAL_IP
+```
+
+### 6) Configure environment
 Create `.env` in the project root with required variables:
 ```bash
 cat > .env << 'EOF'
 OPENAI_API_KEY=your_openai_key
 ANTHROPIC_API_KEY=your_anthropic_key
 EXA_API_KEY=your_exa_key
-DATABASE_URL=postgresql://user:pass@host:5432/db?sslmode=require
+
+# GCP Cloud SQL PostgreSQL (update with your actual values)
+DATABASE_URL="postgresql://app_user:your_password@your_db_ip/app_db?sslmode=disable"
+
+# CORS Configuration (replace YOUR_VM_IP with actual external IP)
+CORS_ORIGINS="http://localhost:4111,http://0.0.0.0:4111,http://YOUR_VM_IP:4111,*"
 
 # Required for auth middleware
 MASTRA_JWT_SECRET=replace_with_a_strong_random_secret
@@ -72,20 +90,47 @@ NODE_ENV=production
 EOF
 ```
 
-### 7) Build and (optionally) apply migrations
+**Note:** For production, consider using `sslmode=require` with proper SSL certificates instead of `sslmode=disable`.
+
+### 7) Build and apply migrations
 ```bash
 # Build the app and prepare Prisma client in the output
 pnpm build
 
-# If your database is new or migrations need to be applied
+# Apply database migrations
 npx prisma migrate deploy
+
+# Generate Prisma client
+npx prisma generate
 ```
 
 ### 8) Start the application
+
+#### Option A: Development/Testing (foreground)
 ```bash
 pnpm dev
 ```
 
+#### Option B: Production/Persistent (with PM2)
+```bash
+# Install PM2 globally
+sudo npm install -g pm2
+
+# Start the app with PM2
+pm2 start "pnpm dev" --name app-playground
+
+# Useful PM2 commands:
+pm2 list                    # View all processes
+pm2 logs app-playground     # View logs
+pm2 restart app-playground  # Restart the app
+pm2 stop app-playground     # Stop the app
+pm2 delete app-playground   # Remove from PM2
+
+# Auto-restart PM2 processes on system reboot
+pm2 startup
+pm2 save
+```
+
 The server listens on `0.0.0.0:4111`. In another terminal:
 ```bash
 EXTERNAL_IP=$(gcloud compute instances describe mastra-app \
@@ -102,6 +147,13 @@ echo "http://$EXTERNAL_IP:4111/auth/login"
 - Auth-protected UI is served at `/auth/login` and the Web Automation Agent playground at `/agents/webAutomationAgent/chat/` after login.
 
 ### Troubleshooting
-- Prisma client errors: ensure you ran `pnpm build`. If needed, run `npx prisma generate` once, then `pnpm build` again.
-- Firewall: verify with `gcloud compute firewall-rules list --filter="name~allow-mastra-app"`.
-- Logs: if using PM2, run `pm2 logs mastra-app`; otherwise, observe terminal output from `pnpm dev`.
+- **Prisma client errors:** ensure you ran `pnpm build`. If needed, run `npx prisma generate` once, then `pnpm build` again.
+- **Firewall issues:** verify with `gcloud compute firewall-rules list --filter="name~allow-mastra-app"`.
+- **Database connection:** check that VM IP is authorized in Cloud SQL and `.env` has correct DATABASE_URL.
+- **Logs:** 
+  - PM2: `pm2 logs app-playground`
+  - Foreground: observe terminal output from `pnpm dev`
+- **PM2 process management:**
+  - Check status: `pm2 status`
+  - Restart if needed: `pm2 restart app-playground`
+  - Clear logs: `pm2 flush`

mastra-test-app/docs/GCP_POSTGRESQL_SETUP.md

@@ -0,0 +1,180 @@
+## Overview
+
+We migrated to Google Cloud SQL PostgreSQL due to memory and storage limitations on our previous database. The setup uses a cost-effective shared-core instance suitable for development environments.
+
+## Prerequisites
+
+- Google Cloud SDK (`gcloud`) installed and authenticated
+- Access to the `nava-labs` GCP project
+- Appropriate IAM permissions for Cloud SQL
+
+## Database Configuration
+
+### Instance Details
+- **Instance Name:** `app-dev`
+- **Database Engine:** PostgreSQL 15
+- **Tier:** `db-g1-small` (1.7 GB RAM, shared-core)
+- **Region:** `us-central1`
+- **Storage:** 100GB SSD with automatic backups enabled
+- **IP Address:** `your_ip_address`
+
+### Database & User
+- **Database Name:** `app_db`
+- **Username:** `app_user`
+- **Password:** `your_password`
+
+## Setup Instructions
+
+### 1. Verify GCP Project Configuration
+
+```bash
+# Check current project
+gcloud config get-value project
+# Should return: nava-labs
+```
+
+### 2. Create PostgreSQL Instance
+
+```bash
+gcloud sql instances create app-dev \
+  --database-version=POSTGRES_15 \
+  --tier=db-g1-small \
+  --region=us-central1 \
+  --storage-type=SSD \
+  --storage-size=100GB \
+  --backup
+```
+
+**Note:** PostgreSQL on Cloud SQL only supports custom tiers or shared-core tiers (not the standard predefined tiers like `db-n1-standard-*`).
+
+### 3. Create Database
+
+```bash
+gcloud sql databases create app_db --instance=app-dev
+```
+
+### 4. Create Database User
+
+```bash
+gcloud sql users create app_user \
+  --instance=app-dev \
+  --password=your_password
+```
+
+### 5. Get Connection Information
+
+```bash
+# Get the instance IP address
+gcloud sql instances describe app-dev \
+  --format="value(ipAddresses[0].ipAddress)"
+```
+
+### 6. Authorize Your IP Address
+
+Before you can connect to the database, you need to add your current IP address to the authorized networks:
+
+```bash
+# Get your current public IP address
+curl -s https://ipinfo.io/ip
+
+# Add your IP to the authorized networks (replace YOUR_IP with the actual IP)
+gcloud sql instances patch app-dev --authorized-networks=YOUR_IP
+```
+
+**Important:** When adding a new IP address, make sure to include any previously authorized IPs, otherwise they will be overwritten.
+
+### 7. Update Environment Configuration
+
+Update your `.env` file with the new database connection string:
+
+```env
+# New GCP Cloud SQL PostgreSQL
+DATABASE_URL="postgresql://app_user:your_password@your_ip_address/app_db?sslmode=require"
+```
+
+## Connection String Format
+
+```
+postgresql://[username]:[password]@[host]:[port]/[database]?[parameters]
+```
+
+For our setup:
+- **Username:** `app_user`
+- **Password:** `your_password`
+- **Host:** `your_ip_address`
+- **Port:** `5432` (default, omitted)
+- **Database:** `app_db`
+- **SSL Mode:** `require` (mandatory for Cloud SQL)
+
+## Database Migration
+
+After setting up the new database and authorizing your IP, run Prisma migrations to set up the schema:
+
+```bash
+# Deploy existing migrations to the new database
+npx prisma migrate deploy
+
+# Generate the Prisma client
+npx prisma generate
+
+# Verify the schema is synchronized (optional)
+npx prisma db push --accept-data-loss
+```
+
+**Note:** If you get a connection error, make sure you've completed step 6 (IP authorization) above.
+
+## Environment Naming Convention
+
+Following our multi-environment strategy:
+- **Development:** `app-dev` (current setup)
+- **Staging:** `app-staging` (future)
+- **Production:** `app-prod` (future)
+
+## Cost Considerations
+
+- **Tier:** `db-g1-small` is cost-effective for development
+- **Storage:** 100GB SSD provides good performance
+- **Backups:** Enabled for data safety
+- **Region:** `us-central1` offers good pricing
+
+## Troubleshooting
+
+### Common Issues
+
+1. **"Only custom or shared-core instance Billing Tier type allowed for PostgreSQL"**
+   - Use `db-g1-small` or `db-custom-X-Y` tiers, not `db-n1-standard-*`
+
+2. **"Can't reach database server" or connection timeouts**
+   - Ensure your IP is authorized: `gcloud sql instances patch app-dev --authorized-networks=YOUR_IP`
+   - Verify SSL mode is set to `require`
+   - Check that the instance is running: `gcloud sql instances describe app-dev`
+
+3. **Prisma migration fails**
+   - Make sure IP authorization is complete before running migrations
+   - Verify the DATABASE_URL in your `.env` file is correct
+
+### Useful Commands
+
+```bash
+# List all SQL instances
+gcloud sql instances list
+
+# Get instance details
+gcloud sql instances describe app-dev
+
+# List databases in instance
+gcloud sql databases list --instance=app-dev
+
+# List users in instance
+gcloud sql users list --instance=app-dev
+
+# Connect via gcloud (for testing)
+gcloud sql connect app-dev --user=app_user --database=app_db
+```
+
+## Security Notes
+
+- Database password is stored in `.env` file (ensure it's in `.gitignore`)
+- SSL is required for all connections
+- Plan to use Cloud SQL Proxy for better security in production
+- IP whitelisting should be configured for production environments

mastra-test-app/prisma/migrations/20250814205016_add_playwright_artifacts/migration.sql

@@ -0,0 +1,29 @@
+-- CreateTable
+CREATE TABLE "public"."mastra_artifacts" (
+    "id" TEXT NOT NULL,
+    "sessionId" TEXT NOT NULL,
+    "fileName" TEXT NOT NULL,
+    "fileType" TEXT NOT NULL,
+    "mimeType" TEXT NOT NULL,
+    "size" INTEGER NOT NULL,
+    "content" BYTEA NOT NULL,
+    "metadata" JSONB NOT NULL DEFAULT '{}',
+    "traceId" TEXT,
+    "threadId" TEXT,
+    "createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    "updatedAt" TIMESTAMP(3) NOT NULL,
+
+    CONSTRAINT "mastra_artifacts_pkey" PRIMARY KEY ("id")
+);
+
+-- CreateIndex
+CREATE INDEX "mastra_artifacts_sessionId_idx" ON "public"."mastra_artifacts"("sessionId");
+
+-- CreateIndex
+CREATE INDEX "mastra_artifacts_fileType_idx" ON "public"."mastra_artifacts"("fileType");
+
+-- CreateIndex
+CREATE INDEX "mastra_artifacts_traceId_idx" ON "public"."mastra_artifacts"("traceId");
+
+-- CreateIndex
+CREATE INDEX "mastra_artifacts_createdAt_idx" ON "public"."mastra_artifacts"("createdAt");

mastra-test-app/prisma/schema.prisma

@@ -66,3 +66,24 @@ model HouseholdDependent {
   @@map("household_dependents")
 }
 
+model PlaywrightArtifact {
+  id        String   @id @default(cuid())
+  sessionId String
+  fileName  String
+  fileType  String   // 'screenshot' | 'trace' | 'session' | 'other'
+  mimeType  String
+  size      Int
+  content   Bytes
+  metadata  Json     @default("{}")
+  traceId   String?
+  threadId  String?
+  createdAt DateTime @default(now())
+  updatedAt DateTime @updatedAt
+
+  @@index([sessionId])
+  @@index([fileType])
+  @@index([traceId])
+  @@index([createdAt])
+  @@map("mastra_artifacts")
+}
+

mastra-test-app/src/mastra/agents/data-ops-agent.ts

@@ -4,21 +4,30 @@ import { anthropic } from '@ai-sdk/anthropic';
 import { google } from '@ai-sdk/google';
 import { databaseTools } from '../tools/database-tools';
 import { storageTools } from '../tools/storage-tools';
+import { artifactTools } from '../tools/artifact-tools';
 
 export const dataOpsAgent = new Agent({
   name: 'Data Ops Agent',
-  description: 'Agent specialized in database and Mastra storage queries (participants, threads, messages, traces).',
+  description: 'Agent specialized in database and Playwright artifacts and Mastra storage queries (participants, threads, messages, traces).',
   instructions: `
     You are a concise data operations assistant. Use the provided tools to:
     - Query and manage participants/household records
     - Inspect Mastra threads, messages, and traces
+    - Store and retrieve Playwright artifacts (screenshots, traces, session data)
     - Return small, readable result sets
+    
+    The artifact tools allow you to:
+    - Store files from disk as artifacts in the database
+    - List and search stored artifacts
+    - Retrieve artifacts by ID or session
+    - Delete artifacts when no longer needed
   `,
 //   model: google('gemini-2.5-pro'),
   model: anthropic('claude-sonnet-4-20250514'),
   tools: {
     ...Object.fromEntries(databaseTools.map(t => [t.id, t])),
     ...Object.fromEntries(storageTools.map(t => [t.id, t])),
+    ...Object.fromEntries(artifactTools.map(t => [t.id, t])),
   },
 
   defaultStreamOptions: {