This issue is going to help address the two-queue problem.
We will address the "two queue problem" with a sidecar design. Fluence with send out a probe pod, a pod that is a subset of some potential N that can dispatch work. The sidecar will be able to watch the queue depth and ungate the rest of the classical pods when resources are available. Since we need to get the job/task ID reliably, we will do a simple patch of the call to braket to add tags that are linked with the job to the task. The probe pod can then easily discover it, and along with reporting on its place in the queue, be able to retrieve the ARN and ensure the ungated pods are provided with it (this would allow access to the results for braket, which are stored in AWS S3).
And to be clear, this is a sidecar that acts as a probe for the application, and exists in that state. It uses the same standard envars that fluence sends to the pods about the scheduling choices. Fluence itself, as a scheduler, does not know anything about credentials, etc.
We will need to think through similar approaches for other vendors, but for now, this is a great start!
Detailed design notes: https://gist.github.com/vsoch/a619ede2bd051877b1c079ca6bfa8727
This issue is going to help address the two-queue problem.
We will address the "two queue problem" with a sidecar design. Fluence with send out a probe pod, a pod that is a subset of some potential N that can dispatch work. The sidecar will be able to watch the queue depth and ungate the rest of the classical pods when resources are available. Since we need to get the job/task ID reliably, we will do a simple patch of the call to braket to add tags that are linked with the job to the task. The probe pod can then easily discover it, and along with reporting on its place in the queue, be able to retrieve the ARN and ensure the ungated pods are provided with it (this would allow access to the results for braket, which are stored in AWS S3).
And to be clear, this is a sidecar that acts as a probe for the application, and exists in that state. It uses the same standard envars that fluence sends to the pods about the scheduling choices. Fluence itself, as a scheduler, does not know anything about credentials, etc.
We will need to think through similar approaches for other vendors, but for now, this is a great start!
Detailed design notes: https://gist.github.com/vsoch/a619ede2bd051877b1c079ca6bfa8727