Description

Introduce Celery as the background task manager for the project to enable scalable, reliable, and maintainable execution of asynchronous and scheduled jobs. This will provide a solid foundation for current and future background tasks, such as periodic data cleanup, sending email notifications, processing analytics, and more.

Current Behavior

• Background or scheduled tasks are not managed by a dedicated task queue system.
• Any periodic or asynchronous jobs must be implemented ad hoc, which can lead to complexity and maintenance challenges as the project grows.
• The project currently runs on an EC2 instance, which works for now, but may require scaling up or transitioning to managed services (like ElastiCache for Redis) as the number of tasks and data volume increases.

Expected Behavior

• Integrate Celery into the backend for managing background and scheduled tasks.
• Use Redis as the message broker (with consideration for scaling: starting with Redis on EC2, with the option to move to ElastiCache as needed).

Open Questions

• Do we need to scale the EC2 instance or use a solution like ElastiCache?