Feature: Voice Input/Output Support via OpenAI Realtime API

[cyl19970726]

Summary

Add comprehensive voice input and output capabilities to the MiniAgent framework using OpenAI's Realtime API (gpt-4o-realtime-preview).

Background

The current framework supports text-based interactions with LLM providers. This feature request aims to extend support for real-time voice conversations, enabling users to interact with agents through speech input and receive audio responses.

OpenAI Realtime API Overview

• Model: gpt-4o-realtime-preview-2024-12-17
• Transport: WebRTC (preferred) or WebSocket
• Capabilities: End-to-end speech-to-speech with <1s latency
• Modalities: Configurable text/audio input/output combinations
• Pricing: $20/1M tokens (cached audio in), $40/1M tokens (audio out)

Architecture Changes Required

1. New Voice Provider Interface

Create a new voice-enabled chat provider alongside existing GeminiChat and OpenAIChatResponse:

// src/chat/openaiVoiceChat.ts
export class OpenAIVoiceChat implements IVoiceChat {
  private realtime: OpenAIRealtimeWS;
  private audioProcessor: AudioProcessor;
  
  async sendAudioStream(
    audioStream: MediaStream,
    config: VoiceConfig
  ): AsyncGenerator<VoiceResponse>;
  
  async sendTextStream(
    text: string,
    config: VoiceConfig
  ): AsyncGenerator<VoiceResponse>;
}

2. Voice Configuration Extensions

Extend existing configuration to support voice parameters:

interface VoiceConfig {
  voice: 'alloy' | 'echo' | 'fable' | 'onyx' | 'nova' | 'shimmer';
  modalities: ('text' | 'audio')[];
  turnDetection: 'server_vad' | 'manual';
  audioFormat: 'pcm16' | 'opus';
  sampleRate: 24000;
}

3. Audio Processing Layer

Add audio processing utilities:

// src/audio/audioProcessor.ts
class AudioProcessor {
  encodeAudio(buffer: Float32Array): string; // Base64 for API
  decodeAudio(base64Data: string): Float32Array;
  resample(input: Float32Array, fromRate: number, toRate: number): Float32Array;
}

4. Event System Extensions

Extend AgentEventType to include voice-specific events:

enum AgentEventType {
  VoiceStart = 'voice.start',
  VoiceAudioDelta = 'voice.audio.delta',
  VoiceTextDelta = 'voice.text.delta',
  VoiceComplete = 'voice.complete',
  VoiceError = 'voice.error'
}

5. Browser/Node Compatibility

• Browser: Use Web Audio API + WebRTC
• Node: Use native audio modules or external libraries
• Streaming: Implement chunked audio processing

Implementation Plan

Phase 1: Core Voice Integration (Priority: High)

• Create OpenAIVoiceChat class implementing voice streaming
• Add voice configuration to AllConfig interface
• Implement basic audio encoding/decoding
• Add voice-specific event types

Phase 2: Audio Processing (Priority: High)

• Implement audio stream handling
• Add resampling utilities for different audio formats
• Create audio buffer management
• Handle audio device access (microphone/speakers)

Phase 3: Provider Integration (Priority: Medium)

• Update StandardAgent to support voice providers
• Add voice mode detection in agent factory
• Implement session management for voice sessions
• Add voice-specific error handling

Phase 4: Testing & Examples (Priority: Medium)

• Create voice-enabled example applications
• Add comprehensive test suite for audio processing
• Documentation for voice integration
• Performance benchmarks for audio streaming

Phase 5: Advanced Features (Priority: Low)

• Voice activity detection (VAD) customization
• Multi-language voice support
• Voice interruption handling
• Audio effects and voice customization

API Usage Examples

Basic Voice Chat

import { StandardAgent } from '@continue-reasoning/mini-agent';

const agent = new StandardAgent(tools, {
  chatConfig: {
    provider: 'openai-voice',
    voice: 'alloy',
    modalities: ['text', 'audio']
  }
});

// Process voice input
for await (const event of agent.processWithVoice(userAudioStream)) {
  if (event.type === 'voice.audio.delta') {
    playAudio(event.data.audio);
  }
}

Text-to-Speech Mode

const agent = new StandardAgent(tools, {
  chatConfig: {
    provider: 'openai-voice',
    modalities: ['text'] // input, audio output
  }
});

// Text input, audio output
for await (const event of agent.process('Hello, how are you?')) {
  if (event.type === 'voice.audio.delta') {
    playAudio(event.data.audio);
  }
}

Technical Considerations

Security & Privacy

• Audio data encryption in transit
• Local audio processing where possible
• User consent for microphone access
• Audio data retention policies

Performance

• Audio chunk size optimization (20ms opus frames)
• Buffer management for low latency
• Connection lifecycle management (30min session limit)
• Fallback to text mode on connection issues

Browser Compatibility

• WebRTC support matrix
• Audio API compatibility
• Mobile browser considerations
• Progressive enhancement for non-voice environments

Dependencies

• openai package (already included)
• Web Audio API (browser)
• Node.js audio modules (optional)
• WebRTC libraries (optional)

Breaking Changes

None - this is additive functionality that maintains backward compatibility with existing text-based interfaces.

Testing Strategy

1. Unit tests for audio processing utilities
2. Integration tests with OpenAI Realtime API
3. Browser compatibility testing
4. Performance benchmarks for latency
5. End-to-end voice conversation tests

Documentation Updates

• Voice integration guide
• API reference for voice configuration
• Browser setup instructions
• Troubleshooting audio issues
• Performance optimization tips

Success Criteria

• Sub-second voice response latency
• Support for all OpenAI voice models
• Cross-browser compatibility
• Comprehensive test coverage (>80%)
• Production-ready examples
• Complete documentation

Related Issues

• May require updates to session management for audio session state
• Consider integration with existing token tracking for voice usage
• Future enhancement: Multi-provider voice support (Google, Azure, etc.)

[cyl19970726]

📋 Detailed Technical Specification

🎯 Design Principles for Voice Integration

1. Streaming-First Architecture: All voice operations support real-time streaming with minimal latency
2. Cross-Platform Compatibility: Works seamlessly in browser (WebRTC/Web Audio) and Node.js environments
3. Modular Extension: Voice capabilities are optional add-ons that don't break existing text interfaces
4. Type-Safe Pipeline: Strong TypeScript typing throughout the entire audio processing chain
5. Progressive Enhancement: Graceful fallback to text mode when voice is unavailable

🏗️ New Interface Architecture

Core Voice Interfaces Hierarchy

// Base voice capability layer
IVoiceChat (extends IChat)
├── IAudioDeviceManager    // Device management
├── IAudioProcessor       // Audio processing pipeline  
├── IVoiceEventEmitter    // Voice event system
└── IHybridAgent         // Text + voice unified agent

// Configuration interfaces
IVoiceConfig
├── IVoiceChatConfig     // Extended chat config
├── AudioStreamConfig    // Audio encoding settings
└── VoiceSessionConfig   // Session-specific settings

Key Interface Definitions

// Primary voice chat interface
export interface IVoiceChat {
  startSession(config: IVoiceChatConfig): Promise<string>;
  sendAudio(sessionId: string, audio: AudioData): Promise<void>;
  sendText(sessionId: string, text: string): Promise<void>;
  streamVoiceResponse(sessionId: string): AsyncGenerator<VoiceEvent>;
  interrupt(sessionId: string): Promise<void>;
}

// Audio processing pipeline
export interface IAudioProcessor {
  encodeAudio(audio: AudioData, format: AudioFormat): Promise<EncodedAudio>;
  decodeAudio(encoded: EncodedAudio): Promise<AudioData>;
  resampleAudio(audio: AudioData, targetRate: SampleRate): Promise<AudioData>;
  preprocessAudio(audio: AudioData, options: AudioPreprocessOptions): Promise<AudioData>;
}

// Device management
export interface IAudioDeviceManager {
  getInputDevices(): Promise<MediaDeviceInfo[]>;
  getOutputDevices(): Promise<MediaDeviceInfo[]>;
  requestMicrophoneAccess(): Promise<MediaStream>;
  setInputDevice(deviceId: string): Promise<void>;
  checkPermissions(): Promise<PermissionState>;
}

🔄 Voice Processing Flow

1. Session Initialization Flow

User Agent → IHybridAgent.setMode('voice')
    → IVoiceChat.startSession(config)
        → IAudioDeviceManager.requestMicrophoneAccess()
            → MediaStream established
                → OpenAI Realtime API session created
                    → Session ID returned

2. Real-time Audio Processing Flow

Microphone → AudioProcessor.encodeAudio() 
    → WebSocket/RTC transmission
        → OpenAI Realtime API processing
            → Audio response chunks
                → AudioProcessor.decodeAudio()
                    → Speaker output

3. Text-to-Speech Flow

Text Input → IVoiceChat.sendText()
    → OpenAI TTS synthesis
        → Audio stream chunks
            → AudioProcessor.decodeAudio()  
                → Speaker output

📁 New Directory Structure

src/
├── chat/
│   ├── interfaces.ts              # Existing interfaces
│   ├── openaiChat.ts             # Existing OpenAI chat
│   ├── geminiChat.ts             # Existing Gemini chat
│   ├── voiceInterfaces.ts        # NEW: Voice interfaces
│   ├── openaiVoiceChat.ts        # NEW: OpenAI Realtime implementation
│   └── voice/                    # NEW: Voice sub-module
│       ├── audioProcessor.ts     # Audio encoding/decoding
│       ├── deviceManager.ts      # Device management
│       ├── streamHandler.ts      # Real-time streaming
│       └── vadDetector.ts        # Voice activity detection
├── audio/
│   ├── browser/                  # Browser-specific implementations
│   │   ├── webAudioProcessor.ts
│   │   ├── webrtcHandler.ts
│   │   └── mediaDeviceManager.ts
│   ├── node/                     # Node.js-specific implementations  
│   │   ├── nodeAudioProcessor.ts
│   │   └── nodeDeviceManager.ts
│   └── shared/                   # Cross-platform utilities
│       ├── audioUtils.ts
│       ├── resampler.ts
│       └── formatConverter.ts
├── agents/
│   ├── hybridAgent.ts            # NEW: Text+voice unified agent
│   └── voiceEnhancedAgent.ts     # NEW: Voice-specific agent
└── examples/
    ├── voice/
    │   ├── basicVoiceChat.ts     # Simple voice conversation
    │   ├── textToSpeech.ts       # Text input, voice output
    │   ├── voiceToText.ts        # Voice input, text output
    │   └── hybridMode.ts         # Dynamic mode switching
    └── browser/
        └── voiceDemo.html        # Browser voice demo

🗺️ Architecture Diagram

High-Level Voice Architecture

┌─────────────────────────────────────────────────────────────┐
│                     Voice-Enhanced Agent                    │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────┐ │
│  │   Text Mode     │  │   Voice Mode    │  │ Hybrid Mode │ │
│  │   (Existing)    │  │    (New)        │  │   (New)     │ │
│  └─────────────────┘  └─────────────────┘  └─────────────┘ │
├─────────────────────────────────────────────────────────────┤
│                  Voice Interface Layer                      │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────┐ │
│  │ IVoiceChat      │  │ IAudioProcessor │  │ IDeviceMgr  │ │
│  │ Implementation  │  │   Pipeline      │  │             │ │
│  └─────────────────┘  └─────────────────┘  └─────────────┘ │
├─────────────────────────────────────────────────────────────┤
│                  Platform Adapters                          │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────┐ │
│  │ Browser APIs    │  │ Node.js APIs    │  │ WebRTC/WS   │ │
│  │ Web Audio API   │  │ Native Modules  │  │ Transport   │ │
│  └─────────────────┘  └─────────────────┘  └─────────────┘ │
└─────────────────────────────────────────────────────────────┘

Voice Processing Pipeline

┌─────────────┐     ┌──────────────┐     ┌─────────────────┐
│   Audio     │────▶│   Encode     │────▶│   Realtime      │
│   Input     │     │   Compress   │     │   API           │
│ (Microphone)│     │  (Opus/PCM)  │     │                 │
└─────────────┘     └──────────────┘     └─────────────────┘
                                                  │
┌─────────────┐     ┌──────────────┐             │
│   Speaker   │◀────│   Decode     │◀────────────┘
│   Output    │     │   Decompress │     
│             │     │  (Opus/PCM)  │     
└─────────────┘     └──────────────┘     

⚙️ Configuration Schema

Voice Provider Configuration

interface VoiceProviderConfig {
  openai: {
    provider: 'openai-realtime';
    model: 'gpt-4o-realtime-preview-2024-12-17';
    voice: VoiceName;
    modalities: VoiceModality[];
    turnDetection: TurnDetectionMode;
    apiKey: string;
    audioFormat: {
      input: 'pcm16';
      output: 'opus';
      sampleRate: 24000;
    };
  };
}

Runtime Configuration Options

interface VoiceRuntimeConfig {
  // Session management
  sessionTimeout: 1800000; // 30 minutes
  maxSessionDuration: 3600000; // 1 hour
  
  // Audio processing
  chunkSize: 20; // milliseconds
  bufferSize: 1024;
  enablePreprocessing: true;
  
  // Interruption handling
  enableInterruptions: true;
  interruptionThreshold: 0.5;
  
  // Error handling
  maxRetries: 3;
  fallbackToText: true;
}

🚀 Implementation Priority Matrix

Component	Priority	Complexity	Dependencies
IVoiceChat interface	High	Medium	OpenAI SDK
AudioProcessor	High	High	Web Audio API
DeviceManager	High	Medium	Browser APIs
HybridAgent	Medium	High	Existing agents
Browser examples	Medium	Low	All above
Node.js support	Low	High	Native modules
Advanced features	Low	Medium	Core complete

🧪 Testing Architecture

Test Categories

1. Unit Tests: Audio processing, encoding/decoding
2. Integration Tests: OpenAI Realtime API communication
3. Browser Tests: WebRTC, device access, cross-browser
4. Performance Tests: Latency benchmarks, memory usage
5. E2E Tests: Complete voice conversation flows

Test Structure

test/
├── unit/
│   ├── audio/
│   │   ├── processor.test.ts
│   │   ├── encoder.test.ts
│   │   └── deviceManager.test.ts
│   └── voice/
│       ├── interfaces.test.ts
│       └── openaiVoice.test.ts
├── integration/
│   ├── openaiRealtime.test.ts
│   └── voiceSession.test.ts
├── browser/
│   ├── webrtc.test.ts
│   └── deviceAccess.test.ts
└── e2e/
    ├── voiceConversation.test.ts
    └── hybridMode.test.ts

📊 Performance Targets

• Latency: <500ms end-to-end (audio input → audio output)
• Audio Quality: 24kHz, 16-bit PCM
• Session Duration: Up to 30 minutes continuous
• Memory Usage: <100MB for audio buffers
• CPU Usage: <10% on modern devices

🔒 Security Considerations

• Audio data encrypted in transit (TLS 1.3)
• No audio data stored locally by default
• User consent required for microphone access
• Session tokens expire after 30 minutes
• Audio processing isolated from main thread