When only modifying the port configuration, is it necessary to restart the gateway instance?

[kebe7jun]

We are currently working on EnvoyGateway in hopes of bring it into our product and have encountered this issue.

Scenario: We will deploy a public EnvoyGateway instance for a cluster to expose the services in the cluster to the outside world. Here, we may design Layer 4 and Layer 7 traffic at the same time.

Problem: For layer 4 traffic, such as SSH ports, we may often add/remove instances on the backend that provide SSH services. In these cases, we need to configure Gateway and TCPRoute to ensure that traffic can reach the actual Pod from the user side. But once I modify the Gateway's listeners configuration, the Deployment of the Gateway will be restarted, which is unacceptable for a public gateway component. However, we cannot create a new Gateway instance for each port/instance, which will greatly increase our operating costs.

This may be caused by re-rendering Container's ports? But in fact this field is not used in the normal traffic path (Service's targetPort is a directly defined port, not a name). Maybe there are other reasons, but I didn't delve into it.

So, can EnvoyGateway solve this problem? Or provide an optional parameter for the user to decide? I've looked through the documentation and can't seem to find a similar configuration.

AFAIK, some other similar gateway implementations do not seem to have similar problems.

[arkodg]

hey @kebe7jun can you share the specific steps to repro this issue

[kebe7jun]

It is easy to reproduce.

root@worker-a800-1:~# kubectl get gtw
NAME                CLASS                   ADDRESS        PROGRAMMED   AGE
eg                  eg                      10.20.20.243   True         2d23h
root@worker-a800-1:~# kubectl -n envoy-gateway-system get po
NAME                                         READY   STATUS    RESTARTS   AGE
envoy-default-eg-e41e7b31-677d68cdb9-hwsfp   2/2     Running   0          17h
root@worker-a800-1:~# kubectl edit gtw eg
# add the following content:
+  - allowedRoutes:
+      namespaces:
+        from: Same
+    name: http2
+    port: 81
+    protocol: HTTP
root@worker-a800-1:~# kubectl -n envoy-gateway-system get po
NAME                                         READY   STATUS              RESTARTS   AGE
envoy-default-eg-e41e7b31-677d68cdb9-hwsfp   2/2     Running             0          17h 
envoy-default-eg-e41e7b31-78d57d556f-kfdf6   0/2     ContainerCreating   0          1s   # <--- here, new pod created.

[zirain]

looks like related to

gateway/internal/infrastructure/kubernetes/proxy/resource.go

Line 93 in 3e8730f

for _, listener := range infra.Listeners {

[kebe7jun]

Yes, Container ports are not used in Service. maybe we don’t need to render it, or add a parameter to control it? (The metrics port can be reserved to ensure that functions such as PodMonitor can be used normally.)

We really need this feature and I'd love to implement it if the community wants it.

[kebe7jun]

I've submitted a PR #4714 to address this issue, I tested it in my own environment and I can fix it to resolve the issue, which will have no impact on existing gateways.
Feel free to discuss.

[arkodg]

if we dont need the container ports, then should we just rm it ?
cc @envoyproxy/gateway-maintainers