Split apart web server and build server

[jyn514]

Currently, all web requests are served from the same server that performs the crate builds. This has given us trouble in the past, since builds consume a lot of resources, driving up response times for web requests. It would be useful to instead have different servers for the website and builds.

Possible implementation:

• The web server continues having primary access to the database. Builders don't have access at all, but instead receive webhooks (or similar) telling them to start a build. When the build finishes, they report back to the main server.
• We have a 'heartbeat' mechanism similar to crater: if a build server doesn't send a message to the web server in 5 minutes, we mark it as crashed
• We have a new state for crates in the queue: queued, finished, and assigned, where assigned means there is a build server currently building that crate (Add crates to the database before building them #1011).

Benefits:

• We could build multiple crates at a time much more easily (alleviating some of the trouble from stop building crates on all targets #343) just by having more build servers. This requires Add crates to the database before building them #1011 first.
• We could potentially target more architectures without cross-compiling, by having build servers that are e.g. hosted on ARM.
• We could eventually share some build resources with crater.

@pietroalbini has experience with implementing this for crater and is helping a lot with the design aspect.

[syphar]

Just a thought, wouldn't it a possible intermediate step to just add a second server that just uses the same database and bucket?
Also, the local index repo could move to the build-server.

Then the webserver could benefit from having less load, and we can slowly work on the webhook logic

[Nemo157]

IIRC the blocker last time I touched this was the metrics, currently the builder metrics are merged into the web server metrics to get to prometheus. We would either need the metrics from the builder to be stored into the database to then be served by the web server, or pushed direct from the builder to prometheus somehow.

Once those are sorted running two processes for this should be trivial.

[pietroalbini]

A possible approach for the metrics would be to run a really small webserver out of the builder that just exposes a /metrics endpoint.

[syphar]

Just a thought, not sure if we should do it in the first version already:

when the goal is to be able to just rebuild instances via ansible then we perhaps should start

• moving state from local files to the database (for example the last seen commit of the index?)
• moving scripts from the server to the repo (or any repo)

IMO as long as we only have one build server (that also manages the queue & crates-index-diff) we don't need #1011 and we can just reboot / recreate the machine.

There would be a initial performance hit on every new build-server (downloading the toolchain, docker image etc), and also for every new webserver (local archive indexes come to mind, not sure if there is more).

For webservers it would be theoretically possible to run multiple machines with a ELB, while I actually prefer better caching on the CDN (#1552 and/or #1560 )

[syphar]

Some additional thoughts on this, steps to be able to horizontally scale the build-server:

separate registry watcher

We move the registry-watcher into a separate container / process that only exists once, saving the last seen ref additionally in the database to survive restarts, and to be able to just recreate the machine any time we want.

allow multiple build servers

1. move the queue-lock into the database
2. allow multiple builders by using SELECT FOR UPDATE SKIP LOCKED, and wrapping the build in a transaction

2 should scale well enough for a low number of build servers (<50) and wouldn't need much change to the current logic.
This would even allow autoscaling scaling build-servers, of course the initial setup time for the first build would take some time.

[pietroalbini]

We move the registry-watcher into a separate container / process that only exists once, saving the last seen ref additionally in the database to survive restarts, and to be able to just recreate the machine any time we want.

That's a great idea, that could be deployed to ECS.

[syphar]

In addition to the separate registry watcher we also have a separate job for the repository stats updater. strike that, we already have cratesfyi database update-repository-fields which does exactly what the daemon triggers once an hour

I assume ECS has something that can run a command every hour? @pietroalbini

[pietroalbini]

I would create a single binary that includes the registry watcher and update-repository-fields all together tbh. ECS has something to run periodic tasks but if we can avoid more pieces of infra I'd be happier.

[syphar]

I would create a single binary that includes the registry watcher and update-repository-fields all together tbh. ECS has something to run periodic tasks but if we can avoid more pieces of infra I'd be happier.

that's possible, thank you for the input!