Image Recognition & Description System - 300,000 GURU

[evahteev]

Description

Add image recognition and automatic description capabilities to luka_bot. The feature enables the bot to understand, describe, and index images sent in DMs and groups, providing context-aware responses about visual content.

Key Features:

• Automatic image analysis in DMs and groups
• Vision LLM integration (GPT-4 Vision → Claude 3 → Gemini Vision)
• OCR for text extraction from images
• Image description indexing in Elasticsearch KB
• Multi-image conversation support
• Image search by content description

Impact:

• Enhanced user experience with visual content understanding
• Improved KB content with image descriptions
• Better context for multi-modal conversations

Reward

🎉 300,000 GURU tokens

Getting Started

1. Review design document: bot/docs/features/image-recognition-design.md
2. Explore image handling: Search for photo/document handlers
3. Check LLM service integration points
4. Review Elasticsearch KB structure for image metadata

Implementation Phases

Phase 1: Core Vision Integration (150,000 GURU)

• Vision LLM provider integration
• Image download and processing
• Basic description generation

Phase 2: KB Integration & OCR (100,000 GURU)

• Image description indexing
• OCR text extraction
• Multi-image conversation support

Phase 3: Advanced Features (50,000 GURU)

• Image search by content
• Cost limits and optimization
• Mobile optimization

Acceptance Criteria

• Automatic image analysis in DMs
• Optional automatic analysis in groups (configurable)
• Vision LLM provider failover working
• Image descriptions indexed in KB
• OCR text extraction functional
• Support for multiple image formats
• Cost limits per user/group implemented