[RFC] Schedule build/job retry

[JenySadadia]

• Add field is_retried to Node model to detect retried jobs
• Add pipeline service job_retry.py to listen to node events with node.state=done and result in ("fail", "incomplete")
• Publish cloud event for scheduler (maybe introduce a new pubsub channel retry?)
• Decide frequency of retries and retry job based on is_retried flag

[nuclearcat]

Maybe worth to add retry counter? some jobs might need to be retried several times.

[JenySadadia]

Maybe worth to add retry counter? some jobs might need to be retried several times.

Yes, makes sense.

[JenySadadia]

PRs:

• Enable job retry kernelci-core#2927
• Implement service to submit job retry #1257
• Enable build job retry #1275