Audio Scene Caption Lab

A small research-engineering sandbox for audio, speech, and visual scene captioning workflows.

Audio Scene Caption Lab is intentionally modest: it does not train or release a large model. Instead, it focuses on the scaffolding around multimodal experiments — manifest validation, deterministic demo adapters, lightweight metrics, and readable Markdown reports.

The project is useful when you want to sketch an audio/visual captioning workflow before connecting expensive models or larger datasets.

What this repo is

• A runnable Python package with a CLI-first demo pipeline
• A clean JSONL schema for small multimodal samples
• Demo adapters for audio tags, speech transcript handling, and visual caption hints
• Simple metrics that make failure modes visible
• A report generator for quick experiment notes

What this repo is not

• It is not a newly trained speech or multimodal foundation model
• It is not a benchmark claiming state-of-the-art results
• It is not tied to any private dataset or unavailable service

Install

git clone https://github.com/psychopathdev/audio-scene-caption-lab.git
cd audio-scene-caption-lab
python -m pip install -e ".[dev]"

For a no-install local run from the repository root, set:

export PYTHONPATH=src

Quick start

python -m audio_scene_caption_lab.cli run   --manifest examples/sample_manifest.jsonl   --out reports/demo_report.md

Inspect the generated report:

python -m audio_scene_caption_lab.cli describe examples/sample_manifest.jsonl

Manifest format

Each line is one JSON object:

{
  "sample_id": "sz-campus-001",
  "audio_path": "audio/rainy_walk.wav",
  "image_path": "frames/rainy_walk.jpg",
  "transcript": "light rain and footsteps near the library",
  "reference_caption": "A rainy campus walkway with footsteps and distant voices.",
  "metadata": {"scene": "campus", "weather": "rain"}
}

The demo adapters use filenames, transcripts, and metadata to produce deterministic outputs. This makes tests stable and keeps the repository easy to run on a laptop.

Pipeline

JSONL manifest
   │
   ├── schema validation
   ├── audio tag adapter
   ├── speech transcript adapter
   ├── vision caption adapter
   └── markdown report + simple metrics

Metrics

The current metrics are deliberately transparent:

• keyword overlap between predicted and reference captions
• caption length statistics
• missing modality counts
• transcript coverage ratio

These are not substitutes for human evaluation or task-specific benchmark metrics, but they are useful for catching obvious issues in early experiments.

Development

export PYTHONPATH=src
python -m pytest
python -m compileall src

Roadmap

• JSONL manifest loader
• Deterministic demo adapters
• Markdown report generation
• Unit tests for schema and metrics
• Optional Hugging Face adapter interfaces
• Dataset card template
• More audio-specific diagnostics

License

MIT License.