Persistent process

[KoltesDigital]

Hi,

I have a use case and I fear Concourse was not designed for it. But I'd like to discuss to what extend, alternatives, etc.

I'm using Bazel for quite some times, and I'm now trying to decrease CI duration. Usually with CI, builds and tests are isolated from each other through ephemeral directories, processes... in a word: containers. However, Bazel manages isolation itself, and has a rather long warmup. Therefore, it is recommended to use persistent workers. Each machine would only have one daemon, build requests are dequeued one by one, and because they all look similar, cache hits speed up builds. Machines would be added or removed depending on the load.

FTR, I don't have enough money to rent a cluster of instances for each repo I work on. Instead, I have an old computer in the broom cupboard, and it's the worker for multiple projects, this means there's a daemon for each project. I'm just running a few builds per day.

Of course, I want to keep Concourse, it already implements a lot of concepts. How can it deal with that one? I'm only thinking about a task that just connects to an external service that itself runs Bazel, and streams back outputs. External to the ephemeral container, so it's long-lived. But that's not ideal, a container would only be running only to stream logs from another process. They would act like a queue, where each item eats some resources.

Do you have any thought on that?

[suhlig]

The closest I can think of are services, for which an RFC exists. But it's in open state, and it doesn't look like there is a lot of progress on it.

[KoltesDigital]

Agreed. My need would be a service that is launched when a build first uses it, but stays after the build, with a defined expiration when unused after which it stops. Alas, it won't be ready anytime soon, but that's another topic.

[KoltesDigital]

Although, that's not really true for Bazel. It's not a service like a database. Rather, think of it like a tool that stores date in memory across invocations. In order to do that, the first invocation spawns a server, and the CLI tool actually is a client. But the server process lives on the same machine, under the same user account, with access to the same filesystem. So where the workflow for usual CI is to first clone the repo and execute some tasks in an ephemeral container, for Bazel it's like the repo is already here, at a unique location, the Bazel server process may be started as well, the workflow is to checkout some ref and execute some tasks, and hopefully the server process was started so that the analysis cache is already available.

[KoltesDigital]

Therefore I change my answer: I think services don't address my case. But apart from that, I also have a need for services, e.g. for databases.