Skip to content
BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
backend
backend
 
 
docs
docs
 
 
frontend
frontend
 
 
.gitignore
.gitignore
 
 
README.md
README.md
 
 

Repository files navigation

TaskFlow

Reliable background jobs. Powerful scheduling. Built for developers.

A production-inspired distributed job scheduling platform with a real-time operator dashboard — Clean Architecture, BullMQ workers, PostgreSQL as the system of record, and Redis as the queue fabric.

Node.js TypeScript React Next.js Redis BullMQ PostgreSQL Docker License: MIT

Overview

TaskFlow is a control plane for background work. Instead of scattering setTimeout, ad-hoc cron, and one-off worker scripts across services, you define jobs once, schedule or enqueue them through an API, and let a worker pool execute them with retries, crash recovery, and a dead letter queue.

It is designed as a learning-and-portfolio system that mirrors how production schedulers are structured — not a toy todo app — while staying small enough to run on a laptop.

What you get out of the box

  • Background jobs with named handlers (noop, log, slow, fail)
  • Delayed enqueue (delayMs / runAt)
  • Cron and interval scheduling via a visual Cron Builder
  • Queue monitoring and worker heartbeats
  • Retry policies with exponential or fixed backoff
  • Dead letter queue for exhausted attempts
  • Real-time dashboard (Socket.IO)
  • JWT authentication and RBAC (admin, operator, viewer)

Architecture

┌──────────────┐
│    Client    │
└──────┬───────┘
       │
       ▼
┌──────────────────┐
│ React Dashboard  │  Next.js · TanStack Query · Socket.IO client
│   (port 3000)    │
└────────┬─────────┘
         │ REST + WebSocket (JWT)
         ▼
┌──────────────────┐
│    REST API      │  Express · Clean Architecture · Zod · Pino
│   (port 3001)    │
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│     BullMQ       │  Runs · Schedules · DLQ
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│      Redis       │  Queue broker + locks
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│   Worker Pool    │  Concurrency · Heartbeats · Stale recovery
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│   PostgreSQL     │  Jobs · Runs · Attempts · Logs · Users
└──────────────────┘

Design principles

  • Domain and use cases never import Express, pg, or BullMQ
  • Manual dependency injection in a single composition root
  • PostgreSQL owns job/run truth; Redis/BullMQ carries runId payloads
  • Optional embedded workers (START_WORKERS=true) or a separate npm run worker process

Features

Feature Description
Delayed jobs Enqueue with delayMs or runAt; status moves through delayed → queued → running
Cron scheduler Visual Cron Builder (minutes / hourly / daily / weekly / monthly / custom) → BullMQ repeatables
Multiple workers Configurable concurrency; separate worker process supported
Retry policies Fixed or exponential backoff; domain-owned retry (BullMQ attempts = 1)
Dead letter queue Failed runs after maxAttempts land on taskflow-dlq
Dashboard Live counters, charts, worker cards, recent runs
Queue monitoring Pending counts, queue position, scheduled countdown
Job history Per-job execution table with filters and CSV export
Authentication Register / login, JWT bearer tokens, role guards
Execution logs Terminal-style viewer with search, copy, download
Worker health Heartbeats every 10s; online/offline in the UI
Crash recovery Stale running runs are recovered on worker start
Realtime updates Socket.IO events: run.updated, run.log, worker.heartbeat

Tech Stack

Backend

Layer Choice
Runtime Node.js 20+
Framework Express
Language TypeScript (strict)
Validation Zod
Logging Pino
Auth JWT + bcrypt
Realtime Socket.IO
Architecture Clean Architecture · Repository pattern · Manual DI

Frontend

Layer Choice
Framework Next.js 15 (App Router)
UI React 19 · Tailwind CSS · Lucide
Data TanStack Query · Zustand
Charts Recharts
Realtime socket.io-client

Data & queue

Concern Choice
System of record PostgreSQL (pg)
Broker Redis (ioredis)
Jobs BullMQ

Dev tools

Tool Purpose
tsx Dev server / migrations
ESLint + Prettier Lint & format
node:test Domain unit tests

Screenshots

Add screenshots under docs/ and they will render here:

Dashboard Jobs
Dashboard Jobs
Login Job details
Login Job details

Tip: capture the running UI at http://localhost:3000 and save PNGs to those paths.

Installation

Prerequisites

  • Node.js 20+
  • PostgreSQL 14+
  • Redis 7+

Clone

git clone https://github.com/<your-username>/Task-Flow.git
cd Task-Flow

Backend

cd backend
cp .env.example .env
npm install

Frontend

cd frontend
cp .env.local.example .env.local
npm install

Redis

# macOS (Homebrew)
brew install redis
brew services start redis
redis-cli ping   # PONG

Note: Homebrew Redis 8 may ship loadmodule lines for modules that are not installed. If Redis exits immediately, comment out the loadmodule entries in /opt/homebrew/etc/redis.conf and restart the service.

PostgreSQL

# Create role + database (adjust for your local Postgres user)
psql -d postgres -c "CREATE ROLE taskflow LOGIN PASSWORD 'taskflow';" 2>/dev/null || true
psql -d postgres -c "CREATE DATABASE taskflow OWNER taskflow;" 2>/dev/null || true
psql -d taskflow -c "GRANT ALL ON SCHEMA public TO taskflow; ALTER SCHEMA public OWNER TO taskflow;"

Environment variables

backend/.env

NODE_ENV=development
PORT=3001
HOST=0.0.0.0
LOG_LEVEL=info
DATABASE_URL=postgres://taskflow:taskflow@127.0.0.1:5432/taskflow
REDIS_URL=redis://127.0.0.1:6379
JWT_SECRET=dev-insecure-jwt-secret-change
JWT_EXPIRES_IN=7d
CORS_ORIGIN=http://localhost:3000
WORKER_CONCURRENCY=5
START_WORKERS=true

frontend/.env.local

NEXT_PUBLIC_API_URL=http://localhost:3001
NEXT_PUBLIC_WS_URL=http://localhost:3001

Database migration

cd backend
npm run migrate

Start development

# Terminal 1 — API + embedded workers
cd backend && npm run dev

# Terminal 2 — Dashboard
cd frontend && npm run dev
Service URL
Dashboard http://localhost:3000
API http://localhost:3001
Health http://localhost:3001/health

First user — open Register in the UI, or:

curl -s http://localhost:3001/api/v1/auth/register \
  -H 'content-type: application/json' \
  -d '{"email":"admin@taskflow.dev","password":"password123"}'

The first registered account becomes admin.

Optional: separate worker process

cd backend
START_WORKERS=false npm run dev   # API only
npm run worker                    # workers only

Folder Structure

Task-Flow/
├── backend/
│   ├── migrations/              # SQL migrations
│   ├── src/
│   │   ├── application/         # Use cases + ports
│   │   ├── config/              # Zod-validated env
│   │   ├── di/                  # Manual composition root
│   │   ├── domain/              # Entities, errors, retry rules
│   │   ├── infrastructure/      # pg, BullMQ, JWT, handlers, Socket.IO
│   │   ├── presentation/http/   # Routes, middleware, controllers
│   │   ├── workers/             # BullMQ worker bootstrap
│   │   ├── scripts/migrate.ts
│   │   ├── app.ts
│   │   ├── server.ts
│   │   └── worker.ts
│   ├── package.json
│   └── tsconfig.json
├── frontend/
│   ├── src/
│   │   ├── app/                 # Next.js App Router pages
│   │   ├── components/          # UI shell, cron builder, logs viewer
│   │   ├── lib/                 # API client, formatters, cron helpers
│   │   └── stores/              # Zustand auth
│   └── package.json
├── docs/                        # Specs + screenshot placeholders
└── README.md

API Endpoints

Base URL: http://localhost:3001
Auth: Authorization: Bearer <jwt> (except health, register, login)

Operational

Method Path Description
GET /health Liveness
GET /ready Postgres + Redis readiness

Auth

Method Path Description
POST /api/v1/auth/register Create user (first → admin)
POST /api/v1/auth/login Returns { token, user }
GET /api/v1/me Current principal

Jobs

Method Path Roles Description
GET /api/v1/jobs any List / search (q, status, limit, offset)
POST /api/v1/jobs admin, operator Create definition
GET /api/v1/jobs/:jobId any Job + schedule
PATCH /api/v1/jobs/:jobId admin, operator Update handler / retry
POST /api/v1/jobs/:jobId/pause admin, operator Pause
POST /api/v1/jobs/:jobId/resume admin, operator Resume
PUT /api/v1/jobs/:jobId/schedule admin, operator Set cron / interval
DELETE /api/v1/jobs/:jobId/schedule admin, operator Remove schedule
POST /api/v1/jobs/:jobId/runs admin, operator Enqueue (Idempotency-Key optional)

Runs & ops

Method Path Description
GET /api/v1/runs List / filter runs
GET /api/v1/runs/:runId Run + attempts + logs
POST /api/v1/runs/:runId/cancel Cancel pending run
POST /api/v1/runs/:runId/retry Retry failed / DLQ run
GET /api/v1/analytics Status counts + daily series
GET /api/v1/workers Worker heartbeats

Queue Lifecycle

Status Meaning
Queued Waiting in BullMQ for a worker
Delayed / Scheduled Waiting for delay or next cron tick
Running Worker has leased the run and is executing a handler
Succeeded Handler completed; attempt recorded
Failed Attempt failed; will retry if attempts remain
Dead lettered maxAttempts exhausted; message also written to DLQ
Cancelled Cooperative cancel before / instead of execution 
Enqueue / Cron tick
        │
        ▼
   ┌─────────┐     delay/cron      ┌───────────┐
   │ Queued  │◄────────────────────│ Delayed   │
   └────┬────┘                     └───────────┘
        │ worker lease
        ▼
   ┌─────────┐
   │ Running │
   └────┬────┘
        │
   ┌────┴────┬──────────────┐
   ▼         ▼              ▼
Succeeded  Failed ──retry──► Queued/Delayed
              │
              └─ exhausted ──► Dead lettered

Worker Lifecycle

  1. Worker process starts (embedded in API or via npm run worker).
  2. Registers a heartbeat row and begins pulsing every 10s.
  3. Runs stale recovery for running jobs past their timeout.
  4. Consumes taskflow-runs with concurrency = WORKER_CONCURRENCY (default 5).
  5. For each message: load run → mark running → create attempt → execute handler → succeed or apply retry/DLQ.
  6. Schedule consumer turns cron ticks into new runs.
  7. DLQ consumer logs terminal failures for operators.

Handlers today are in-process (noop, log, slow, fail). Swap implementations under infrastructure/handlers without changing use cases.

Dashboard

Widget What it shows
Welcome header Date, worker health indicator, API uptime
Quick actions Create job · Enqueue · View runs
Stat cards Queued, running, scheduled, succeeded, failed, active workers
Queue activity Success vs failed over the last 7 days
Jobs per minute Throughput from recent runs
Worker utilization Running slots vs idle capacity
Worker cards Name, status, concurrency, last heartbeat
Recent runs Status, trigger type, duration, retries

Job detail pages add next/last run, countdown when scheduled, execution timeline, terminal logs, and history with export.

Future Improvements

  • Rate limiting on public API routes
  • Priority queues
  • Horizontally scaled worker fleets with placement affinity
  • Prometheus / OpenTelemetry metrics
  • Outbound webhooks on run terminal states
  • Email notifications for DLQ and schedule failures
  • Audit log of mutating operator actions
  • Docker Compose for Postgres + Redis + API + UI
  • OpenAPI / generated client SDK

Contributing

  1. Fork the repo and create a feature branch.
  2. Keep Clean Architecture boundaries (no framework imports in domain/).
  3. Run backend checks before opening a PR:
cd backend && npm test && npm run lint && npx tsc --noEmit
cd ../frontend && npx tsc --noEmit
  1. Prefer small, focused PRs with a clear description of behavior changes.

License

MIT — see LICENSE when present, or treat this repository as MIT-licensed for portfolio use.

Author

Yug Chaudhary

Built as a production-style systems project: distributed scheduling, queue semantics, and an operator UX you would expect from modern SaaS platforms.

If this README helped you run TaskFlow, a star on the repo is appreciated.

About

A scalable background job scheduling platform with cron jobs, queue monitoring, retries, and real-time dashboards.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages