Fix Redis support in manual development setup and consolidate environment configuration

[starbops]

Summary

Fixes the manual development setup where Redis was missing, causing API server startup failures. This PR adds comprehensive Redis support and consolidates environment configuration across all files.

Key Changes

• Add Redis to manual development setup: Added Redis service to docker-compose.test.yml alongside PostgreSQL
• Rename Makefile targets: db-* → services-* to reflect that both PostgreSQL and Redis are managed together
• New service management scripts: Created start/stop/reset-test-services.sh to replace database-only scripts
• Fix development workflow: Updated scripts/start-dev.sh to use new services script
• Environment consolidation: Unified environment variables across Docker Compose, scripts, and env files

Problem Solved

The manual development setup (make services-start, make migrate-up, make dev) was failing because:

1. Only PostgreSQL was started (no Redis)
2. API server requires Redis for queue management
3. Environment variables were inconsistent across configuration files

Testing

• Manual development workflow now works: make services-start → make migrate-up → make dev
• Both PostgreSQL (port 5432) and Redis (port 6379) are available for development
• All environment files are consistent with implementation requirements

Files Changed

• Scripts: 3 new service scripts, 3 old database scripts removed, start-dev.sh updated
• Docker Compose: Added Redis service, updated environment variables
• Makefile: Renamed targets, updated help text and references
• Documentation: Updated README, CLAUDE.md, config/README.md, tests/README.md
• Environment: Consolidated variables in .env.example, Docker Compose files

🤖 Generated with Claude Code

[Copilot]

Pull Request Overview

Fix Redis support in manual development and test setups and unify environment configuration across the project.

• Add Redis alongside PostgreSQL in Docker Compose, scripts, and Makefile targets
• Rename db-* targets to services-* and replace standalone DB scripts with combined service scripts
• Expand and consolidate environment variables in .env.example, dev/prod scripts, and Compose files

Reviewed Changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/README.md	Updated integration test instructions to mention Redis and services-*
scripts/start-test-services.sh	New script to start both PostgreSQL and Redis for testing
scripts/stop-test-services.sh	New script to stop both PostgreSQL and Redis test services
scripts/reset-test-services.sh	New script to reset (stop/remove/start) test services
scripts/start-dev.sh	Switched dev startup to use services scripts and optional .env.dev
scripts/prod-env.sh	Expanded production environment variables for Redis, queue, worker, executor, etc.
scripts/dev-env.sh	Expanded development environment variables for Redis, queue, worker, executor, etc.
internal/database/connection.go	Added NewConnectionWithRetry with retry logic
docker-compose.yml	Added Redis and queue/worker/executor settings; consolidated env vars
docker-compose.test.yml	Added redis-test service and volume
docker-compose.dev.yml	Expanded Redis and related settings for development
config/README.md	Updated test env instructions to make services-start
cmd/api/main.go	Switched to NewConnectionWithRetry and removed redundant log line
README.md	Updated manual setup steps to use make services-start
Makefile	Renamed db-* targets to services-* and updated help text
Dockerfile	Improved layer caching (go.mod first) and adjusted healthcheck
CLAUDE.md	Updated test commands section to use services-*
.env.example	Expanded example for Redis, queue, worker, and executor variables

Comments suppressed due to low confidence (2)

tests/README.md:377

• The indentation of this list item is inconsistent with the surrounding markdown, which may break the numbered list formatting. Align the bullet level to match the other top-level items.

4. **Service setup issues**

.env.example:86

• The example file now includes Redis, queue, worker, and executor settings but is missing JWT, CORS, logging, and embedded-worker variables that are referenced elsewhere. Consider adding those env variables to keep the example complete.

EXECUTOR_EXECUTION_USER=1000:1000

[Copilot]

The grep -q "Up" check only verifies that at least one container is up, which may miss if Redis or PostgreSQL individually aren’t running. Consider checking both postgres-test and redis-test container statuses explicitly or using docker-compose ps --services --filter "status=running" and confirming both services are listed.

Suggested change

	if docker-compose -f docker-compose.test.yml ps | grep -q "Up"; then
	if docker-compose -f docker-compose.test.yml ps --services --filter "status=running" | grep -q "postgres-test" && \
	docker-compose -f docker-compose.test.yml ps --services --filter "status=running" | grep -q "redis-test"; then

[Copilot]

[nitpick] Using docker volume prune -f will remove all dangling volumes in the Docker host, which may affect other projects. It’s safer to remove only the named test volumes (e.g., redis_test_data and postgres_test_data).

Suggested change

	# Remove any dangling volumes
	echo "Cleaning up dangling volumes..."
	docker volume prune -f
	# Remove specific test volumes
	echo "Removing specific test volumes..."
	docker volume rm redis_test_data postgres_test_data || true

[Copilot]

This error message omits the underlying error details, making debugging harder. Capture and wrap the last error (e.g., store err and return fmt.Errorf("... after %d attempts: %w", maxRetries, lastErr)).

Suggested change

	return nil, fmt.Errorf("failed to establish database connection after %d attempts", maxRetries)
	return nil, fmt.Errorf("failed to establish database connection after %d attempts: %w", maxRetries, lastErr)