Factory should emit a warning if max connections is insufficient

[brandoncarl]

[mirekm]

Warning doesn't sound sufficient especially in case when the number of procedures already exceeds the pool capacity. I'm not sure if this is what this issue also refers to but at the moment any team does consume a single connection by calling getNextJob routine in each tick (checking for the new jobs to appear). In turn, if the max allowed pool resources (5 by default) is not greater than the number of procedures at least by one, none of the jobs will ever execute and efficiently the queue will stuck on the resource waiting (Redis transport). Taken that this sounds like a requirement to have a working queue at the moment I think the pool max setting should only refer to the allowed connections on top of the resources taken by the getNextJob queries.

[brandoncarl]

Your point is well-taken and referring to the correct issue. My apologies as I haven't had time to get to the documentation yet.

A little explanation on the getNextJob routine (these all refer to using Redis):

1. As currently stands, each team issues a blocking pop on the particular job type it references. It reserves a connection for this.
2. If all connections are consumed, no jobs can get through.
3. An additional connection is needed by the worker to retrieve the job.

I think the issue we need to decide on is technical case versus business case. Most hosted Redis solutions have a maximum number of permitted connections. In this case, you'd really like to know the number of "maxed" out connections a TaskCo instance can have. Meanwhile, from a technical standpoint, we need to make sure that the jobs can be done.

With the current small defaults, this is definitely an issue. In reality, and in production, the number of job "types" is likely to be substantially smaller than the maximum connections. We should know, prior to deployment, how many job "types" exist, and to be able to set the max to a number much greater than that.

I think that leaves us with a few options:

1. Make max a required input during setup. This forces people to consider their connection needs early.
2. Have max refer to additional connections.
3. Emit a warning, which potentially causes a lot of headache.
4. Emit an error, which, without listeners, causes problems.
5. Throw an error when a team is added if connections aren't available.
6. Reduce connections needed by using pub/sub within the Dispatcher for the Teams under watch.

Let me know your feedback on which option you think suits. Something definitely needs to be done and if we can determine something sensible, I'll try to get it implemented this week (with the exception of number 6, which would take more work). I tend to think number 5 with either 3 or 4 makes the most sense, but I'm open to 1 and 2.

[mirekm]

Thanks for this comprehensive feedback. I agree that this is a technical vs business issue and that having a fine control over the connections pool is a requirement in production environment.
I wouldn't touch 6 at the moment - it feels like a premature optimization and an issue in question is slightly different. I think having number 5 in place along with some warning in the documentation would do the job and it would be a nice and clean solution for now. We would avoid "hiding" the real number of connections as I originally suggested and at the same time user would be warned as early as possible that the queue requires more connections in order to be able to process the tasks.

[brandoncarl]

Ok, this pull request puts number 5 in place. I think think we should add number 3 or 4 after implementing error-emitting functionality and a logging system.