LLM Response Cache for Guest Users - 250,000 GURU

[evahteev]

Description

Optimize perceived response speed for predictable, high-frequency guest user requests by caching LLM responses and streaming them as if they were generated in real-time. This dramatically improves first-impression user experience for common greetings and workflow initiations.

Key Features:

• Cache deterministic guest user responses
• Stream cached responses as if real-time
• Smart cache key generation (normalized messages)
• Support for common greetings and workflow initiations
• Redis-based caching with TTL

Impact:

• Near-instant responses for cached queries (<50ms)
• Improved first-impression user experience
• Reduced LLM API costs for common queries
• Better scalability for guest users

Reward

🎉 250,000 GURU tokens

Getting Started

1. Review cache design: bot/docs/feature/response-cache-design.md
2. Check Redis integration points
3. Explore LLM service: luka_bot/services/llm_service.py
4. Review guest user handling logic

Implementation Phases

Phase 1: Cache Infrastructure (125,000 GURU)

• Redis cache setup
• Cache key generation logic
• Message normalization

Phase 2: Streaming & Integration (125,000 GURU)

• Stream cached responses
• LLM service integration
• Cache invalidation logic

Acceptance Criteria

• Cache hits return responses in <50ms
• Cached responses stream as if real-time
• Cache keys properly normalized
• Common greetings cached correctly
• Cache invalidation working
• No regression for authenticated users