CONCURRENT_LIMITING_SUMMARY.md

Concurrent Limiting Feature Summary

What Was Added

The resque-throttler gem now supports concurrent job limiting in addition to rate limiting. This prevents too many jobs from running simultaneously, which is essential for preventing database lock accumulation.

How It Works

1. Configuration: Add :concurrent option to rate_limit

   Resque.rate_limit(:my_queue, at: 5, per: 5, concurrent: 3)

2. Tracking: The gem tracks active jobs using Redis counters

   • Increments when job starts
   • Decrements when job completes (success or failure)

3. Enforcement: Workers check concurrent limit before starting new jobs

   • If at limit, queue is skipped
   • Job remains in queue for next worker cycle

Key Methods Added

# Check active job count
Resque.active_job_count(:my_queue)

# Check if at concurrent limit
Resque.queue_at_or_over_concurrent_limit?(:my_queue)

# Check if queue has concurrent limit configured
Resque.queue_has_concurrent_limit?(:my_queue)

Implementation Details

1. Redis Keys:

   • Active jobs counter: throttler:active_jobs:{queue_name}
   • Automatically cleaned up (no expiration needed)

2. Worker Hooks:

   • Overrides perform method to track job lifecycle
   • Uses alias_method to preserve original behavior
   • Only tracks rate-limited queues

3. Thread Safety:

   • Uses existing lock mechanism from rate limiting
   • Atomic Redis operations (INCR/DECR)

For Your Lead Uploader Issue

Your issue: Database-intensive jobs accumulate, causing lock contention.

Solution:

# In config/initializers/resque.rb
Resque.rate_limit(:lsq_rl_pro, at: 5, per: 5, concurrent: 3)

This ensures:

• Rate limit: Max 5 jobs start per 5 seconds
• Concurrent limit: Max 3 jobs run at once
• Even if jobs take 10+ seconds, only 3 run concurrently

Testing in Your Application

1. Mount the local gem in Docker:

   volumes:
     - /Users/parikshitsingh/Desktop/opensource/resque-throttler:/resque-throttler

2. Update Gemfile:

   gem 'resque-throttler', path: '/resque-throttler'

3. Run tests:

   docker exec -it ninjastool rake throttler:test:all

4. Monitor:

   docker exec -it ninjastool ruby script/throttler_live_monitor.rb

Production Deployment

After testing:

1. Push gem updates to your fork
2. Update Gemfile to point to new version
3. Deploy with new concurrent limits
4. Monitor database metrics for improvement

The feature is production-ready and backward compatible.