Skip to content

Add database connection resilience improvements (thread-safety and health checks) #275

@Ygohr

Description

@Ygohr

Description

Implement resilience improvements for database connections in lib-commons library, addressing thread-safety issues in Redis and automatic connection pool recovery after container restarts for PostgreSQL and MongoDB, while maintaining full backward compatibility with existing code.

Problem

The current implementation has resilience gaps in database connections:

  • Redis GetClient() and Connect() are not thread-safe for concurrent initialization - multiple goroutines calling GetClient() simultaneously can cause data races
  • PostgreSQL connection pool does not automatically recover after container restart - stale connections remain in the pool
  • MongoDB connection pool has the same issue - does not recover after container restart
  • None of the database connectors expose a Ping() method for external health checkers

These patterns prevent graceful recovery from infrastructure failures and make application health monitoring difficult.

Proposed Solution

1. Redis GetClient() Thread-Safety (HIGH IMPACT)

Implement double-check locking pattern using the existing mu mutex in the struct to ensure thread-safe lazy initialization of the Redis client.

Affected Files:

  • commons/redis/redis.go
  • commons/redis/redis_test.go

2. PostgreSQL Connection Pool Health Check (MEDIUM IMPACT)

Add Ping(ctx context.Context) error method for health checking and configure ConnMaxIdleTime to automatically close stale connections.

Affected Files:

  • commons/postgres/postgres.go
  • commons/postgres/postgres_test.go

3. MongoDB Connection Pool Health Check (MEDIUM IMPACT)

Add Ping(ctx context.Context) error method and configure ServerSelectionTimeout and HeartbeatInterval for faster detection of dead nodes.

Affected Files:

  • commons/mongo/mongo.go
  • commons/mongo/mongo_test.go

4. Redis Connection Health Check (LOW IMPACT)

Add Ping(ctx context.Context) error method for consistency with PostgreSQL and MongoDB, exposing health check capability for external health checkers.

Affected Files:

  • commons/redis/redis.go
  • commons/redis/redis_test.go

Acceptance Criteria

  • Redis GetClient() is thread-safe with double-check locking pattern
  • Ping(ctx context.Context) error method implemented for Redis, PostgreSQL, and MongoDB
  • All existing function signatures remain unchanged (backward compatible)
  • Build passes (go build ./...)
  • Tests pass (go test ./...) including race detector (-race flag)

Metadata

Metadata

Assignees

Labels

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions