feat: add comprehensive audio model benchmarking infrastructure

[maryamtahhan]

Add complete test suite for audio models (ASR, translation, audio chat) on vLLM CPU deployments.

Audio Benchmark Playbook (audio-benchmark.yml):

• Three-play structure: Setup/Validation, Start vLLM Server, Execute Benchmarks
• Supports managed mode (Ansible starts vLLM) and external endpoint mode
• CPU auto-calculation with control plane reservation (2 cores for vLLM scheduler, KV cache, async ops)
• Audio-specific Tensor Parallel defaults: TP=1 for <64 cores, TP=2 for 64+ cores
• OMP thread calculation: (requested_cores - 2) / tensor_parallel
• Integrated vLLM metrics collection via common task files
• Generates test-metadata.json for Streamlit dashboard integration
• Containerized execution for both vLLM and GuideLLM

GuideLLM Audio Benchmark Role (benchmark_guidellm_audio):

• GuideLLM v0.6.0 container execution with audio support
• Builds command as argument list to avoid JSON escaping issues
• Audio preprocessing via --data-preprocessors encode_media
• HuggingFace cache volume mount for disk space management
• Supports audio_transcriptions, audio_translations, chat_completions request formats
• Comprehensive error handling and log capture

Created reusable task files used by both LLM and audio benchmarks:

• tasks/start-vllm-metrics-collection.yml: Unified vLLM /metrics endpoint collection
• tasks/stop-vllm-metrics-collection.yml: Unified metrics teardown
• tasks/wait-for-vllm-ready.yml: Health check logic for vLLM readiness
• tasks/setup-results-directory.yml: Results directory creation

Refactored llm-benchmark-auto.yml to use common task files (~55 lines reduced).

Five comprehensive test scenarios plus quick validation test:

1. transcription-throughput.yaml: PRIMARY TEST - "How long to transcribe N files?"

   • Sequential baseline (100 files)
   • Concurrent processing (2, 4, 8 workers)
   • Maximum throughput test
   • Metrics: total time, files/sec, audio_seconds/sec, real-time factor

2. transcription-latency.yaml: Per-request latency under load

   • Light/medium/heavy load tests
   • P50/P95/P99 latency percentiles
   • Latency degradation analysis

3. audio-duration-scaling.yaml: Performance vs audio length

   • Short (1-5s), medium (5-15s), long (15-30s), full-length clips
   • Linear vs non-linear scaling analysis
   • Per-audio-second processing cost

4. constant-rate-stress.yaml: Sustained load stability

   • Sustained rates: 2, 5, 10 req/s (5 min each)
   • Extended duration test (15 min)
   • Memory leak detection

5. format-comparison.yaml: Audio format impact

   • MP3 (64kbps, 128kbps)
   • WAV (uncompressed, 8/16/48kHz)
   • FLAC (lossless)
   • Stereo vs mono comparison

6. quick-test.yaml: Fast validation (5 files)

model-matrix.yaml:

• Whisper models: tiny (39M), small (244M), medium (769M)
• Audio chat: ultravox-v0_5-llama-3_2-1b
• Audio preprocessing presets: mp3_64k, mp3_128k, wav_16k, flac_16k
• Recommended vLLM settings per model (dtype, max_model_len, kvcache)
• Dataset configurations (LibriSpeech, Common Voice)

tests/audio-models/README.md (435 lines):

• Quick start guide (managed and external modes)
• Detailed usage for all test scenarios
• CPU allocation breakdown with control plane explanation
• Container mode documentation (no host installation required)
• Advanced configuration (manual CPU tuning, custom datasets)
• Results analysis and interpretation
• Troubleshooting guide

models/audio-models/audio-models.md:

• Model selection rationale
• Supported endpoints (ASR, translation, chat)
• Dataset descriptions (LibriSpeech, Common Voice)
• Audio preprocessing configurations
• CPU optimization parameters
• Performance expectations
• Test scenario mappings

Updated models/models.md:

• Added audio models section
• Links to audio-specific documentation
• Integration with existing model matrix

CPU Allocation Formula:

Container Allocation:  cores 0-31 (32 total)
├── Control Plane:     2 cores (scheduler, KV cache, async ops)
└── Worker Threads:    30 cores (OMP_NUM_THREADS=30)
    └── Per TP rank:   30/TP cores

OMP_NUM_THREADS = (requested_cores - 2) / tensor_parallel

Container Execution:

• vLLM container: ghcr.io/vllm-project/vllm-cpu-env:latest
• GuideLLM container: ghcr.io/vllm-project/guidellm:v0.6.0
• HuggingFace cache volume: host mount to avoid disk space issues
• SELinux context: :z flag for proper permissions
• Cache directory: 0777 mode for container write access

Metrics Collection:

• vLLM server metrics from /metrics endpoint
• GuideLLM client-side metrics (latency, throughput, audio-specific)
• Audio-specific metrics: audio_seconds, audio_samples, audio_bytes, audio_tokens
• Real-time factor: processing_time / audio_duration

Successfully validated with quick-test scenario:

• Container image pull and execution
• HuggingFace dataset download (LibriSpeech)
• Audio preprocessing (MP3 encoding)
• vLLM server startup and health checks
• Metrics collection
• Results JSON generation

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: ac447a02-4fe3-4c64-a964-c693636cf343

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}