diff --git a/CLAUDE.md b/CLAUDE.md index c3f91bd..af06c57 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,18 +6,21 @@ This document defines code style guidelines, review criteria, project-specific r VoidRunner is a distributed task execution platform designed for secure, scalable code execution. The project follows an incremental development approach through well-defined Epic milestones. -### Current Implementation Status (Epic 1 βœ… Complete) +### Current Implementation Status (Epic 1-2 βœ… Complete) - **Backend**: Go + Gin framework + PostgreSQL (pgx driver) - **API**: RESTful API with JWT authentication and comprehensive task management - **Database**: PostgreSQL with optimized schema and cursor pagination +- **Container Execution**: Docker executor with comprehensive security controls +- **Queue System**: Redis-based task queuing with retry logic and dead letter handling +- **Worker Management**: Embedded worker pool with concurrency controls and health monitoring - **Testing**: 80%+ code coverage with unit and integration tests - **Documentation**: OpenAPI/Swagger specs with comprehensive examples -### Planned Architecture (Epic 2-4 πŸ“‹ Roadmap) -- **Container Execution**: Docker + gVisor security for safe code execution +### Planned Architecture (Epic 3-4 πŸ“‹ Roadmap) +- **Distributed Services**: Separate API and worker services for horizontal scaling - **Frontend**: Svelte + SvelteKit + TypeScript web interface - **Infrastructure**: Kubernetes (GKE) deployment with microservices -- **Queue System**: Redis for task scheduling and real-time updates +- **Log Streaming**: Real-time log collection and streaming - **Monitoring**: Real-time metrics, logging, and alerting systems ## Go Code Standards @@ -29,8 +32,7 @@ voidrunner/ β”œβ”€β”€ cmd/ # Application entrypoints β”‚ β”œβ”€β”€ api/ # βœ… API server main (implemented) β”‚ β”œβ”€β”€ migrate/ # βœ… Database migration tool (implemented) -β”‚ β”œβ”€β”€ scheduler/ # πŸ“‹ Task scheduler main (planned - Epic 2) -β”‚ └── executor/ # πŸ“‹ Task executor main (planned - Epic 2) +β”‚ └── scheduler/ # βœ… Scheduler service main (implemented - for future distributed mode) β”œβ”€β”€ internal/ # Private application code β”‚ β”œβ”€β”€ api/ # βœ… API handlers and routes (implemented) β”‚ β”œβ”€β”€ auth/ # βœ… Authentication logic (implemented) @@ -38,13 +40,13 @@ voidrunner/ β”‚ β”œβ”€β”€ database/ # βœ… Database layer (implemented) β”‚ β”œβ”€β”€ models/ # βœ… Data models (implemented) β”‚ β”œβ”€β”€ services/ # βœ… Business logic services (implemented) -β”‚ β”œβ”€β”€ executor/ # πŸ“‹ Task execution engine (planned - Epic 2) -β”‚ β”œβ”€β”€ queue/ # πŸ“‹ Message queue integration (planned - Epic 2) -β”‚ └── security/ # πŸ“‹ Security utilities (planned - Epic 2) +β”‚ β”œβ”€β”€ executor/ # βœ… Task execution engine (implemented) +β”‚ β”œβ”€β”€ queue/ # βœ… Redis queue integration (implemented) +β”‚ └── worker/ # βœ… Worker management (implemented) β”œβ”€β”€ pkg/ # Public libraries β”‚ β”œβ”€β”€ logger/ # βœ… Structured logging (implemented) -β”‚ β”œβ”€β”€ metrics/ # πŸ“‹ Prometheus metrics (planned - Epic 2) -β”‚ └── utils/ # πŸ“‹ Shared utilities (planned) +β”‚ β”œβ”€β”€ utils/ # βœ… Shared utilities (implemented) +β”‚ └── metrics/ # πŸ“‹ Prometheus metrics (planned - Epic 4) β”œβ”€β”€ api/ # βœ… API specifications (OpenAPI) (implemented) β”œβ”€β”€ migrations/ # βœ… Database migrations (implemented) β”œβ”€β”€ tests/ # βœ… Integration tests (implemented) @@ -63,11 +65,12 @@ voidrunner/ - Comprehensive testing suite - OpenAPI documentation -**Epic 2: Container Execution Engine** πŸ”„ **In Development** -- Docker client integration with security +**Epic 2: Container Execution Engine** βœ… **Complete** +- Docker client integration with security controls - Task execution workflow and state management -- Real-time log collection and streaming -- Error handling and cleanup mechanisms +- Embedded worker pool with concurrency management +- Redis-based queue system with retry logic +- Health monitoring and cleanup mechanisms **Epic 3: Frontend Interface** πŸ“‹ **Planned** - Svelte project setup and architecture @@ -76,11 +79,54 @@ voidrunner/ - Real-time task status updates **Epic 4: Advanced Features** πŸ“‹ **Planned** +- Distributed services architecture (Issue #46) +- Real-time log collection and streaming (Issue #11) +- Enhanced error handling mechanisms (Issue #12) - Collaborative features and sharing - Advanced search and filtering - Real-time dashboard and system metrics - Advanced notifications and alerting +## GitHub Issues Progress Tracking + +### Epic 1: Core API Infrastructure βœ… **Complete** +- **Issue #3**: PostgreSQL Database Setup and Schema Design βœ… **Closed** +- **Issue #4**: JWT Authentication System Implementation βœ… **Closed** +- **Issue #5**: Task Management API Endpoints βœ… **Closed** +- **Issue #6**: API Documentation and Testing Infrastructure βœ… **Closed** + +### Epic 2: Container Execution Engine βœ… **Complete** +- **Issue #9**: Docker Client Integration and Security Configuration βœ… **Closed** +- **Issue #10**: Task Execution Workflow and State Management βœ… **Closed** + +**Epic 2 Enhancements** (Non-blocking improvements) +- **Issue #11**: Log Collection and Real-time Streaming πŸ“‹ **Open** (Priority 1) +- **Issue #12**: Error Handling and Cleanup Mechanisms πŸ“‹ **Open** (Priority 2) + +> **Note**: Issues #11-12 are enhancements to the completed Epic 2 functionality, not blockers. The core container execution engine with embedded workers is fully operational. + +### Epic 3: Frontend Interface πŸ“‹ **Ready to Start** +- **Issue #22**: Frontend Interface πŸ“‹ **Open** +- **Issue #23**: Svelte Project Setup and Architecture πŸ“‹ **Open** (Priority 0) +- **Issue #24**: Authentication UI and User Management πŸ“‹ **Open** (Priority 0) +- **Issue #25**: Task Creation and Management Interface πŸ“‹ **Open** (Priority 0) +- **Issue #26**: Real-time Task Status Updates πŸ“‹ **Open** (Priority 0) +- **Issue #27**: Real-time Features πŸ“‹ **Open** + +### Epic 4: Advanced Features πŸ“‹ **Future Work** +- **Issue #28**: Real-time Dashboard and System Metrics πŸ“‹ **Open** (Priority 0) +- **Issue #29**: Advanced Notifications and Alerting πŸ“‹ **Open** (Priority 1) +- **Issue #30**: Advanced Search and Filtering πŸ“‹ **Open** (Priority 1) +- **Issue #31**: Collaborative Features and Sharing πŸ“‹ **Open** (Priority 2) + +### Future Enhancements +- **Issue #46**: Separate API and Worker Services for Horizontal Scaling πŸ“‹ **Open** + - This epic will transition from embedded workers to distributed services + - Currently tracked for future implementation when scaling requirements emerge + +### Current Focus +With Epic 1-2 complete, the project has a fully functional task execution platform with embedded workers. The next logical step is Epic 3 (Frontend Interface) to provide a web-based user interface, followed by Epic 4 advanced features and eventual transition to distributed services (Issue #46). + ### Coding Standards #### 1. Naming Conventions diff --git a/README.md b/README.md index 29860c4..bd88e15 100644 --- a/README.md +++ b/README.md @@ -11,11 +11,11 @@ VoidRunner is designed to safely execute user-submitted code in isolated contain - **REST API**: Comprehensive HTTP API with 16+ endpoints for complete task lifecycle management - **JWT Authentication**: Secure user authentication with access and refresh tokens - **Task Management**: Full CRUD operations for code tasks with metadata support -- **Task Execution**: Asynchronous task execution with embedded or distributed worker architecture +- **Task Execution**: Asynchronous task execution with embedded worker architecture - **Container Security**: Docker-based execution with seccomp, resource limits, and isolation - **Queue System**: Redis-based task queuing with priority support and retry logic +- **Worker Management**: Embedded worker pool with concurrency controls and health monitoring - **Real-time Monitoring**: Health checks, worker status endpoints, and execution metrics -- **Flexible Deployment**: Embedded workers for development or separate services for production - **Database Integration**: PostgreSQL with optimized schema and cursor pagination - **Security**: Input validation, rate limiting, and secure request handling - **Testing**: 80%+ code coverage with unit and integration tests @@ -23,7 +23,9 @@ VoidRunner is designed to safely execute user-submitted code in isolated contain ## System Architecture -### Embedded Workers (Development/Small Scale) βœ… Complete +### Current Implementation (Embedded Workers) +Single-process architecture with embedded worker pool for development and production + ``` β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ Web Clients β”‚ β”‚ API Server β”‚ β”‚ PostgreSQL β”‚ @@ -37,33 +39,16 @@ VoidRunner is designed to safely execute user-submitted code in isolated contain β”‚ β”‚ - Task Processing β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ - Docker Execution β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ - Health Monitoring β”‚ β”‚ + β”‚ β”‚ - Concurrency Control β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜β—„β”€β”€β–Ίβ”‚ Docker β”‚ - β”‚ (Containers) β”‚ - β”‚ β”‚ - β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ (Containers) β”‚ + β”‚ β”‚ + β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` -### Distributed Services (Production/Scale) βœ… Complete -``` -β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ Web Clients β”‚ β”‚ API Server β”‚ β”‚ PostgreSQL β”‚ β”‚ Redis β”‚ -β”‚ (Postman/curl) │◄──►│ - HTTP API │◄──►│ Database │◄──►│ (Queues) β”‚ -β”‚ β”‚ β”‚ - Auth β”‚ β”‚ β”‚ β”‚ β”‚ -β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ - Task Mgmt β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ - β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β–² - β”‚ - β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ - β”‚ Scheduler Service│◄──►│ Docker β”‚ β”‚ - β”‚ - Worker Pools β”‚ β”‚ (Containers) β”‚ β”‚ - β”‚ - Task Executionβ”‚ β”‚ β”‚ β”‚ - β”‚ - Auto-scaling β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ - β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ - β–² β”‚ - β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ -``` - -### Future Extensions (πŸ“‹ Roadmap) +### Future Roadmap +- **Distributed Services**: Separate API and worker services for horizontal scaling - **Svelte Web UI**: Modern frontend interface - **Kubernetes Deployment**: Container orchestration - **Multi-language Support**: Additional runtime environments @@ -71,7 +56,7 @@ VoidRunner is designed to safely execute user-submitted code in isolated contain ## Quick Start -### Development Mode (Embedded Workers) +### Development Mode The fastest way to get started with VoidRunner: @@ -91,28 +76,30 @@ This starts VoidRunner with embedded workers on http://localhost:8080 - Worker Status: http://localhost:8080/health/workers - API Documentation: http://localhost:8080/docs -### Production Mode (Distributed Services) +### Production Mode -For production deployments with horizontal scaling: +For production deployments: ```bash # Using Docker Compose -./scripts/docker-deploy.sh -m prod up +docker-compose -f docker-compose.yml up -d -# Or manually -SERVER_ENV=production EMBEDDED_WORKERS=false ./bin/api -SERVER_ENV=production ./bin/scheduler +# Or build and run manually +make build +SERVER_ENV=production ./bin/api ``` ### Configuration Modes -VoidRunner supports three deployment configurations: +VoidRunner currently supports embedded worker architecture: + +> **Note**: Distributed services (Issue #46) planned for future horizontal scaling needs -| Mode | Use Case | Workers | Scaling | Setup | -|------|----------|---------|---------|-------| -| **Development** | Local development | Embedded | Single process | `EMBEDDED_WORKERS=true` | -| **Production** | Enterprise deployment | Separate service | Horizontal | `EMBEDDED_WORKERS=false` | -| **Test** | CI/CD and testing | Embedded | Single process | `SERVER_ENV=test` | +| Mode | Use Case | Workers | Configuration | +|------|----------|---------|---------------| +| **Development** | Local development | Embedded | Relaxed security, debug logging | +| **Production** | Production deployment | Embedded | Enhanced security, structured logging | +| **Test** | CI/CD and testing | Embedded | Mock executors, test database | ## API Endpoints @@ -138,7 +125,8 @@ VoidRunner supports three deployment configurations: - `DELETE /api/v1/executions/{id}` - Cancel execution ### System Health -- `GET /health` - Health check endpoint +- `GET /health` - API health check endpoint +- `GET /health/workers` - Embedded worker status and metrics - `GET /ready` - Readiness check endpoint ## Quick Start @@ -147,7 +135,8 @@ VoidRunner supports three deployment configurations: - Go 1.24.4+ installed - PostgreSQL 15+ (for database operations) -- Docker (for containerization and testing) +- Redis 7+ (for task queuing) +- Docker (for containerization and task execution) ### Setup @@ -245,14 +234,14 @@ make docs-serve ### Phase 1: Core Infrastructure βœ… Complete Task management API with authentication, database integration, and comprehensive testing. -### Phase 2: Container Execution Engine πŸ”„ In Development -Secure Docker-based code execution with resource limiting, real-time log streaming, and safety controls. +### Phase 2: Container Execution Engine βœ… Complete +Secure Docker-based code execution with embedded workers, resource limiting, and safety controls. ### Phase 3: Web Interface πŸ“‹ Planned Modern Svelte-based frontend with real-time task monitoring, code editor, and user dashboard. ### Phase 4: Advanced Features πŸ“‹ Planned -Collaborative features, advanced search and filtering, system metrics dashboard, and notification system. +Distributed services, collaborative features, advanced search and filtering, and real-time log streaming. ## Configuration @@ -276,4 +265,4 @@ Please ensure all tests pass and maintain code coverage above 80%. ## License -This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. \ No newline at end of file +This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.