Max timeout global configuration for tasks

[ceco556]

Summary

Concourse currently supports CONCOURSE_DEFAULT_TASK_TIMEOUT, which provides default timeout for tasks that don't have a specified one. This is very useful for ensuring tasks don't run forever.

However, there is no similar configuration for maximum allowed timeout for tasks. Users can set pipelines with big timeouts..

Proposal

Introduce a new configuration:

CONCOURSE_MAX_TASK_TIMEOUT: <max_timeout>

This property will define the maxium allowed timeout for a task, regardless of what is specified in the pipeline.

Expected Behaviour

If a task specifies a timeout larger than CONCOURSE_MAX_TASK_TIMEOUT:

• The timeout will be capped to CONCOURSE_MAX_TASK_TIMEOUT
• Probably a validation to fail set of pipeline if it exceeds the global max timeout.

[taylorsilva]

Would definitely want the validation part, just to eliminate any confusion from pipeline writers who might wonder why their timeouts aren't being respected and end up submitting bug reports 😅

[ceco556]

I've created a PR for this.
#9485

[alicul]

I am curious, why not use OPA to achieve the same? One can have much better granularity, if required. Also, the pipeline operators would have very clear and detail messages.

Personally, I am not convinced that it is Concourse job to impose these kind of restrictions...

UPDATE: don't get me wrong. I understand the need for this very well. In our Concourse deployment, we impose the limit of 2h for all tasks.. but from experience, I can tell you, you will need a few exceptions for this. If you configure one global CONCOURSE_MAX_TASK_TIMEOUT, you will not be able to do so, with OPA you decide...

[ceco556]

Hi @alicul,

Just to clarify that i understand it right. Why don't use OPA to achieve the validation part? (fly rejects a set-pipeline if the timeout exceeds the max allowed?)

If that's the point, i agree with you. It would be better to use OPA for better control.

[alicul]

Hi @ceco556

fly rejects a set-pipeline if the timeout exceeds the max allowed?

yes, exactly. But not only fly; also set-pipeline step. Furthermore, it is not only that it will get rejected when setting for the first time, it will also reject when updating the existing pipeline. Note, also that the OPA is not limited to seting the pipeline; but can control any action (API request towards Concourse) basically.

Here is the introduction to OPA with Concourse: https://concourse-ci.org/docs/operation/opa-integration

I really became a fan of the OPA integration with Concourse. Its my 2nd favorite tehnology in last few years (Concourse being the first one :) ). It really brought huge value to our Concourse deployment. Controling the task timeout is just one of the many many policies that you can enforce. It simply gives you, as a Concourse opeator, possibility to enforce rules on how to write a good pipeline. e.g. it enabled us control the workload by reducing the number of containers in our deployment from >6k to <400, just by enforcing a few policies on how to write a efficient pipeline.

The error message user will see will look something like:

$ fly -t local set-pipeline -p test -c test.yml
error: forbidden: policy check failed: 
 * <actual deny reason from OPA policy>

Some of the main advantages of OPA integration with Concourse:

1. deny and soft deny. You introduce a rule as a soft deny, inform your pipeline owners about the new rule and the deadline (e.g. 3 months); after 3 months you switch the rule to deny.
2. it allows to craft your rules as you see fit. If we look at your example with task timeout: lets say you implement CONCOURSE_MAX_TASK_TIMEOUT feature as in your PR and you set the value to 2h. It will enforce this for ALL tasks, accross ALL pipelines, accross ALL teams. There might be valid cases where you would like to have a task running for 2h and 1minute. With CONCOURSE_MAX_TASK_TIMEOUT your hands are tied. With OPA policy, its just about extending the rule a bit, by allowing a few exceptions.
3. OPA is written in Go,so its easy to develop CLI tools that in seconds inspects ALL pipelines configuration in your Concourse deployment (hundreds of pipeline configurations) and generate reports that helps you understand your pipelines and projects accross teams.

The only dowside:

• The Rego language is at the same time increadibly powerful, but a bit harder to grasp at the start. One needs to invest some time to learn how to write Rego syntax, but at the end its worthwhile to do it.... walk function is amazing :)

Not to mention that OPA completly solves the issue mentioned by @taylorsilva

just to eliminate any confusion from pipeline writers who might wonder why their timeouts aren't being respected and end up submitting bug reports

Anyway, I just wanted to share my fascination with OPA and its integration with Concourse. I hope this helps.

If you decide to use the OPA, just ping me back and I can share a few tips on how to get you kickstarted more quickly... e.g. power of metadata etc...

[taylorsilva]

Great anecdote Aleksandar! OPA is really awesome. I sadly haven't had a chance to use it in a production setting more. I do recall now that this was one of the main motivations for integrating it into Concourse, so operators can make whatever rules they need to without adding a new config to Concourse.

Thank you for jumping into this thread and reminding us about OPA :)

[ceco556]

Sorry for the delayed response, i had other things to do lately.

Thanks, @alicul, for the explanation. I’ve worked with OPA, but didn’t think of it to use it this way for controling the max timeout. I agree with your point, and I’ve closed the PR.

Thanks again!