Running an in-cluster image registry on localhost

[nicks]

What would you like to be added:

A lot of local kubernetes solutions let you run an image registry in-cluster, and push images directly to that registry.

You port-forward the registry to localhost:32000, so that images tagged "localhost:32000" get pushed to the in-cluster registry. The cluster is configured to trust images from that registry.

This document from the MicroK8s team has a good summary of the feature under "Working with MicroK8s’ registry add-on"
https://microk8s.io/docs/working

Why is this needed:

This makes loading images into the cluster orders of magnitude faster. Even faster than side-loading with image import/export, because the registry protocol can skip layers that have already been pushed.

Even documentation on how to do this with KIND would help a lot!

[BenTheElder]

Does it actually make it faster? Note that image loading in docker at least also skips layers that are already loaded.

[BenTheElder]

The biggest problem is docker for mac / windows, where the host won't have access to the node IPs so we'd need additional port forwarding for the registry.

Otherwise, deploying a registry shouldn't be much different on kind than any Kubernetes host, there's just no built in add-ons. 🤔

[nicks]

re: making it faster: here are some numbers on my Linux machine with a warm registry (where i've already pushed the image once, and want to push it a second time)

with KIND:

$ time kind load docker-image golang:1.12

real	0m39.225s
user	0m0.438s
sys	0m2.159s

with MicroK8s:

$ time docker push localhost:32000/golang:1.12
The push refers to repository [localhost:32000/golang]
ed644461d094: Layer already exists 
39e987d2d021: Layer already exists 
510e5f32af35: Layer already exists 
2c8d31157b81: Layer already exists 
7b76d801397d: Layer already exists 
f32868cde90b: Layer already exists 
0db06dff9d9a: Layer already exists 
1.12: digest: sha256:35e22d07fcf5fee472630b821f655bb1973ded0f64a9a4f7d2db0287179ecb3c size: 1796

real	0m0.178s
user	0m0.073s
sys	0m0.066s

so yes, this is 100x slower on KIND

[nicks]

re: port-forwarding: yes! It might not be a lot of engineering work. An end-user can do a lot of this manually. I think the big pieces are:

1. Deploying the registry
2. Configuring the cluster to trust the registry over http and/or setting up the certs (this is the big part I'm not sure how to do)
3. Setting up port-forwarding to the host machine

It might be as simple as documentation on how to put the pieces together. I would be happy to try to contribute something like this but might need help.

[aojea]

🤔 those layers already exist in the repository, is this a fair comparison?

[nicks]

@aojea Yes, I'm interested in "warm" performance where the layers are already in the registry.

The kind load numbers I posted above are for the second load. The first load is even slower at about 60s.

Let me know if this is just a misunderstanding, or if there's a reason why you think that's not fair

[BenTheElder]

Thanks for the data, that additional slowness is likely due to #382 not being in yet (so we're still doing an unnecessary docker save call).

[BenTheElder]

1.) And 2.) Should be relatively standard.
3.) Well, search for the other open issues here, Good port forwarding UX for kind is trickier than it sounds on Mac / Windows. On Linux none is necessary.

[nicks]

after searching port-forwards -> 😅

ooh, #382 looks exciting, i will check it out and see how it compares! thanks for that PR @aojea , I owe you a beverage!

[nicks]

yes, #382 helps a lot!

$ time kind load docker-image golang:1.12

real	0m0.481s
user	0m0.196s
sys	0m0.095s

slightly slower than using a registry, but i'll take the bird in the hand.

[nicks]

hmmm...I think using a registry will still be a lot faster in the case where you have two different images that share most of their base layers, right?

[BenTheElder]

It might be a lot faster with lots of nodes or frequently reusing a layer but not the whole image purely because of the save side of the operation on the host, the load side is layer based internally and skips existing layers.

[BenTheElder]

A guide for doing this would be a nice addition to the docs.
/help

[nicks]

I put together a small proof of concept on how to do this:
https://github.com/windmilleng/kind-local

It's not super robust, but all the pieces are there.

[BenTheElder]

Thanks! looks like we'll need #110, soon we'll have port forwards to the host that could help get traffic to the registry from mac etc.

[BenTheElder]

#382 is now in as well 😅

[aojea]

@nicks should we close this and follow up on #110 ?

[nicks]

@aojea I think #110 is a prerequisite for what I'm asking for here, but isn't the complete story. But I also trust the KIND team on however you all want want to draw the boundaries around issue requests and/or judge them as dupes

[aojea]

@aojea I think #110 is a prerequisite for what I'm asking for here, but isn't the complete story. But I also trust the KIND team on however you all want want to draw the boundaries around issue requests and/or judge them as dupes

fair enough, let's keep this open then

[BenTheElder]

#110 is progressing.
/assign
/remove-help

[BenTheElder]

#1107

[BenTheElder]

https://kind.sigs.k8s.io/docs/user/local-registry/

not the most elegant, but it's a starting point. thinking about the UX of a friendlier option

[nicks]

Yes, that's exactly what I wanted! I can work with that! Thanks!!

This can be closed (unless you wanted to leave it open to talk about friendlier UX)

[nicks]

also -- if there's anything we can do to help to make this friendlier, we'd be happy to contribute engineering/documentation work. I was thinking a UI more like

kind create registry --name=my-registry
kind create cluster --local-registry=my-registry

but would trust your judgment on the right thing here!

[BenTheElder]

#1213 makes for a broader issue, this one is older but more specific on one particular approach, going to dedupe for that one I suppose ...