Legacy Scheduler Design Notes

Scheduler Design

1. Requirements
   1. Be able to permanently sideline jobs for dead repos
   2. Periodically review dead repos to see if they in fact “came back” or “really exist'”. I think this is necessary to replace our “constant rechecking”, which would have to be disabled if we are permanently sidelining jobs
   3. Enable administrator override of the queue to manually put some repos at the top.
      1. Put repos in the front of the line
      2. Pause collection on specific repos for a set period of time (i.e., put the pause in, but do not require the admin to remember to come back and release it because they won’t remember far too often)
   4. Monitoring API
      1. Component metric ideas:
         1. flower/celery task completion rate (and maybe over time too with a running average?)
         2. postgres checkpoint lag
      2. Collection metric ideas:
         1. task completion rate by task
         2. API key expiry (with censored keys)
         3. oldest last collected date for each worker/task?

implementation scheme: some kind of scoring or points based framework/mental model maybe? then maybe we could allow admins to drill down and look at where each point has been assigned from/what source it came from and/or adjust how many points each task has

• maybe this could be used to allow admins enough control to be able to tune/move the bottlenecks of augur given the system its on. i.e. if its maxing out the available disk IO, maybe the admin could tune it to either do that less, or adjust the scheduling mix to include more cpu, memory, or network intensive tasks so the system is as fully utilized as it can be (or maybe intentionally bottleneck another resource thats cheaper or less problematic)