Add vagrant boxes caching to CI

[VannTen]

What type of PR is this?
/kind feature

What this PR does / why we need it:
We're hitting 429 responses when running lots of jobs, presumably
because we are downloading lots of boxes again and again.
This should cache them instead.

Does this PR introduce a user-facing change?:

NONE

/ok-to-test

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: VannTen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [VannTen]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[VannTen]

This is apparently not enough because the CI runners can't extract the cache because they don't have gitlab-runner :
https://gitlab.com/kargo-ci/kubernetes-sigs-kubespray/-/jobs/7698277164#L124

[VannTen]

@ant31 if you have an idea

[ant31]

It's 429 from which service ?

[VannTen]

https://gitlab.com/kargo-ci/kubernetes-sigs-kubespray/-/jobs/7698277164#L524 I think it's app.vagrantup.com for the vagrant boxes used by molecule.

[ant31]

/retest

[k8s-ci-robot]

@VannTen: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kubespray-yamllint	2803946	link	true	/test pull-kubespray-yamllint

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[VannTen]

/hold This needs change in the CI cluster (configure either an S3 or volume cache for the gitlab-runner) before it can work.

[VannTen]

I feel like this is happening a lot recently, but maybe we just have more activity in the CI...
Maybe we could juste include the box in the image ? 🤔

[ant31]

Yes, we're probably rate limit quicker we are running more job in parallel and also since it's quicker so we probably can DL more at time.

can we host the box ourself somewhere ?

[VannTen]

can we host the box ourself somewhere ?

Probably, I guess vagrant get boxes over http. But I think using using the runner caching mechanism would have a broader use (pip packages, ansible collections, pre-commit hooks etc). I've never deployed a self-hosted minio S3 on k8s, but I was skimming their docs last week and it does not look too hard.

[tico88612]

Is there anything I can do to help with HTTP 429 CI issues?

[VannTen]

I haven't finished the minio installation+ gitlab-runner S3 integration on the CI cluster unfortunately and I'm on holidays :/ One thing that can help is doing the gitlab-ci side, this pr is a very rough draft. (In particular, it seems the gitlab-ci semantics have cache policy push/pull/pull-and-push which we should leverage to pull once per pipelines / have a default cache common to all pipelines) I'm not sure how practical that is without the cache working in the first place though. Gitlab-ci has a local executer I think, but I don't know if it can help for that kind of stuff.

[VannTen]

Another possibility I've thought about (but not worked on) is to convert everything (molecule, vagrant jobs) to use the "packet" provisioning method. Molecule for instance has an example in their docs to use kubevirt that we could convert/apply to our setup. This would also have the advantage long-term that we could ditch the custom gitlab-ci executors ***@***.***)

[ant31]

Yes, droping vagrant could be a good idea if we can replace it with kubevirt

Instead of s3-cache we could also host the vagrant box ourself too (if it's easier) in the cluster itself. would be kind of equivalent

[tico88612]

Self hosted Vagrant server unless we have the resources to do permissions control (and maintain the custom gitlab-ci executors), I think it would be better to transfer to packet.

To summarize, maybe I can try to give some of the vagrant work to packet first.
Should we need to open a new issue to track this?

[ant31]

I don't know it seems a lot of work without immediate upside.
There's one symptom of using vagrant is being rate limited to fetch boxes, which can be more easily solved with providing the boxes ourself or with cache.
But if someone want to try, yes it's welcome

[ant31]

Minio is working.

Still missing configuration is the custom kubevirt gitlab executor (looks like need to add gitlab-runner binary inside the container)

[VannTen]

Minio is working.

Thanks ! Imo the benefits of moving to "packet" (btw, maybe we should have a more descriptive name ?) are also to reduce the need for custom executors and the associated maintenance burden

[ant31]

/close replaced by : #11671