Open
Description
Motivations
- Observed Impacts
- When a new router starts, if it is under initial and continuous load, the response times will be much longer than they should be
- For example, a nominal 800 ms response time will take 2-8 seconds for a minute or more
- The response times will start dropping towards the nominal response time
- If the load is removed, briefly stopped, then restored, the response time immediately drops to the nominal response time
- Scenario
- A request arrives but an instance has just disconnected
- The average requests per instance is exactly equal to the targeted number of requests per instance
- Instead of waiting a few milliseconds for an instance to connect, the request is immediately pushed into an over-busy instance, negatively impacting the response time more than waiting for a new instance to connect would be
- Challenge
- As requests on the overloaded instances complete, will they grab the queued request is trying to wait for a new instance?
Setup
- MaxConcurrentRequests: 10
- InstanceCountMultiplier: 5
- TargetRequestsPerLambda: 2
- NominalResponseTime: 800 ms
- ColdStartTime: 12,000 ms
Example
- T0: 2 requests arrive every second
- No instances available for 12 seconds (
oha -z 10m -q 2) - T12: 24 requests pile up, 1st instance connects, 10 requests are sent to 1st instance, leaving 14 pending
- T13: 2 more requests arrive, 16 pending, 2nd instance connects, 10 requests are sent to 2nd instance, leaving 6 pending
- T14: 2 more requests arrive, 8 pending, 3rd instance connects, 8 requests are sent to 3rd instance, leaving 0 pending
- T15: 2 more requests arrive, 4th instance connects, 2 requests sent to 4th instance, leaving 0 pending
- T16: 2 more requests arrive, 5th instance connects, 2 requests sent to 5th instance, leaving 0 pending
- ...
- T20: 10 requests from 1st instance finish
- This explains slow response times until ~30 seconds