Support ignite backend

[BenTheElder]

What would you like to be added: Support for using ignite as a docker-like backend with improved isolation.

Why is this needed: This provides a relatively fast but much more isolated option for nodes. On the surface the UX is very similar to the docker CLI, and it operates around standard docker images, with the requirement that you use an /sbin/init (which we do, though wrapped for docker in docker workarounds).

Some thoughts:

• This likely wouldn't ever support extraMounts (or any mounts for that matter, firecracker doesn't support this). It might however support some form of persistent storage in the future, which we'd want to consider how to support as well... Add support for adding volumes to ignite run weaveworks/ignite#76
• We'll probably want to resolve the following issues upstream first:
  • missing /etc/resolv.conf weaveworks/ignite#164 Missing /etc/resolv.conf (we could work around this ourselves easily, but probably upstream will want to do something as well)
  • support configuring listen address and protocol weaveworks/ignite#142 support configuring listen address and protocol (we don't need this, but it would bring the port forwarding up to parity)
  • Consider adding ignite exec [vm] [command] weaveworks/ignite#65 Consider adding ignite exec [vm] [command] (we either need this, or we need to implement something similar with either attach or ssh, it would be least leaky to just implement exec upstream if they're willing, I reached out...)
• I already intend to decouple the internals of kind from docker better (and in fact, am working on that right now...) to support a mock node provider and podman, adding ignite seems like a natural next step, and will cause us to think a little more about things that are not exactly docker.

/priority backlog
/assign

[aojea]

Just for clarification, the goal is to replace current provider or to allow kind to use multiple providers?
By the way, I was able to play with ignite and sounds and awesome idea, this also allow to test different kernels , ...

[BenTheElder]

Definitely not replace (!) There's too many environments that won't have access to this, realistically, and we have our existing users etc. 🙃

As an additional backend this would be pretty useful and mostly can map pretty closely to what we do already 😁

[tao12345666333]

SGTM.
I think this will be fun.
The main reason for choosing ignite is due to its UX?

[danielfbm]

I am working to add kubernetes as a new provider (using kubectl to manage pods, etcs), is this a viable/desirable goal for kind ? If yes, I would like to contribute (and probably would need some guidance)

[BenTheElder]

Hi, I've actually already written this as a joke, it came up at our kubecon deep dive too :+) At this time we're not ready for that, it might be amusing someday, but we aren't prepared to actually support it and wouldn't recommend it, and need to land other changes.

…

On Tue, Dec 10, 2019, 05:47 Daniel Filipe Bruehmueller Morinigo < ***@***.***> wrote: I am working to add kubernetes as a new provider (using kubectl to manage pods, etcs), is this a viable/desirable goal for kind ? If yes, I would like to contribute (and probably would need some guidance) — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#705?email_source=notifications&email_token=AAHADKY6UMB2DZTMAHDYKVLQX6MVFA5CNFSM4IDTHVCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGPI47Q#issuecomment-564039294>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHADK35U5FZBI35QRZ2DELQX6MVFANCNFSM4IDTHVCA> .

[danielfbm]

Could you kindly share the code if possible? In our team kind fits as a perfect development environment, but not really possible to run directly from our laptops, so considering a kubernetes cluster to handle the orchestration seems fair. Thanks On Wed, Dec 11, 2019, 1:13 AM Benjamin Elder <[email protected]> wrote:

…

Hi, I've actually already written this as a joke, it came up at our kubecon deep dive too :+) At this time we're not ready for that, it might be amusing someday, but we aren't prepared to actually support it and wouldn't recommend it, and need to land other changes. On Tue, Dec 10, 2019, 05:47 Daniel Filipe Bruehmueller Morinigo < ***@***.***> wrote: > I am working to add kubernetes as a new provider (using kubectl to manage > pods, etcs), is this a viable/desirable goal for kind ? If yes, I would > like to contribute (and probably would need some guidance) > > — > You are receiving this because you were assigned. > Reply to this email directly, view it on GitHub > < #705?email_source=notifications&email_token=AAHADKY6UMB2DZTMAHDYKVLQX6MVFA5CNFSM4IDTHVCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGPI47Q#issuecomment-564039294 >, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/AAHADK35U5FZBI35QRZ2DELQX6MVFANCNFSM4IDTHVCA > > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#705?email_source=notifications&email_token=ABGPZ7UFVIN4TZOFNQEBRDDQX7E4HA5CNFSM4IDTHVCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGQATCY#issuecomment-564136331>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGPZ7RM2UAH26ML25W4U33QX7E4HANCNFSM4IDTHVCA> .

[BenTheElder]

it's not in a good state, however there are many comments here about running kind in kubernetes clusters 😬

I highly recommend avoiding that if at all possible, it's tricky to avoid the various footguns, in general I don't recommend creating priviliged containers in kubernetes clusters unless you are working on a system component like kube-proxy.

[danielfbm]

that is our main issue. Our team is mostly working with operators/CRDS/aggregations and the need to share resources (cluster of servers) is big deal for us, so we are willing to give it a shot to avoid creating another orchestration tool 😅, but really appreciate the heads up.

I looked through most of the issues that reference this kind of use case, I think #303 is the most important one here and I will post the issues I found there if it is ok for you.

Also, from what I gathered, kubernetes, istio and other projects use case is actually more like:
Pod using docker-in-docker then running kind inside, which is also a docker-in-docker solution. Would that approach be recommended instead of directly running a pod as a k8s node?

[BenTheElder]

I don't recommend it, unfortunately we must do it anyhow. This is all deeply on my backlog, currently focused on some kubernetes testing needs and #148