Use single NLB for multi-domain configuration

[ikarlashov]

Is your enhancement request related to a problem? Please describe.
Trying to use a single AWS NLB for a multi-tenant cluster.

In our current ingress-nginx controller setup, we use one NLB for dozens of services, and it works perfectly fine.
A Kubernetes Service with aws-lb-controller annotations provisions an NLB with a TCP listener on port 443 and forwards traffic to the cluster. TLS termination is handled at the ingress-nginx controller level via the Ingress spec.

I haven’t found a straightforward way to do the same with the nginx-gateway controller.
App developers in the cluster can use a variety of domains, which is why creating a Gateway for *.domain.com is not an option. Developers don’t have access to the AWS account where the cluster is hosted, which is why we can’t consider any ACM functionality.

Thanks in advance!

[nginx-bot]

Hi @ikarlashov! Welcome to the project! 🎉

Thanks for opening this issue!
Be sure to check out our Contributing Guidelines and the Issue Lifecycle while you wait for someone on the team to take a look at this.

[sjberman]

Hey @ikarlashov, just to be clear, you need to be able to add an annotation to the Gateway Services in order to achieve your goal? If that's accurate, you can do so in the spec.infrastructure section of the Gateway spec: https://docs.nginx.com/nginx-gateway-fabric/install/deploy-data-plane/#set-annotations-and-labels-on-provisioned-resources

[ikarlashov]

@sjberman hey,
Not really. I think I need to explain a bit more in terms of Gateway API CRDs.

I’d like to have a single Gateway managed by the platform team, which:

• Provisions a single AWS NLB (via a Service of type LoadBalancer and annotations specified in spec.infrastructure)
• NLB proxy-passes TCP 443 to the cluster (no TLS termination)

Developer teams create multiple HTTPRoutes pointing to this Gateway, each controlling their own TLS certificate (via Kubernetes secrets).

Goal: All developer domains share the same NLB; TLS is terminated only at the nginx-gateway, not at the NLB layer.

In the Gateway API controller, there is a design limitation that I don’t know how to overcome: each Gateway resource creates a dedicated Service of type LoadBalancer (which means one NLB per Gateway object).

I’m 100% sure that using a single Gateway won’t work for my scenario. Maybe you guys have some bright ideas on how I can overcome this limitation?

[sjberman]

Just trying to fully understand you're architecture, because you say you want a single Gateway (and therefore a single NLB in front of it), but you don't want that single Gateway to serve a wildcard domain, so instead you think you need multiple Gateways?

Is it not possible to have a single Gateway with *.domain.com, and then each HTTPRoute sets a more specific hostname(s)? Is this because you have different TLS configurations for each HTTPRoute domain?

In your ingress-nginx setup, when you say dozens of services, are these your backend applications? Or are these ingress services?

If you have to have multiple Gateways, then in my head I just pictured a wild scenario that maybe would work, or maybe be completely wrong:

• Have a single entrypoint Gateway of type LoadBalancer that creates the NLB
• Create Gateways of type ClusterIP (set in the NginxProxy resource) for all your domains (so you don't end up with more NLBs)
• Configure TLS Passthrough on your entrypoint Gateway and point to the "internal" Gateways. The entrypoint Gateway just acts as a passthrough router, and the internal Gateway performs the TLS termination.

It's a lot of hops, because in this scenario you'd have client -> NLB -> Entrypoint Gateway -> internal Gateway -> backend.

Our support team could likely help with a scenario like this as well, I'm sort of thinking off the top of my head here and I haven't tested this out.

[ikarlashov]

I can't use a wildcard domain because all the domains are different and can have various levels of subdomains as well. That’s why I need to configure TLS for every domain in a dedicated Gateway resource.

By “dozens of domains” I meant Ingress objects.

At first glance, your approach sounds feasible to me! That could potentially fix my problem. Thanks for your suggestion :)

[sjberman]

By “dozens of domains” I meant Ingress objects.

Ah ok, so that's one difference with the architecture of Ingress vs. Gateway API. One ingress controller deployment can configure multiple Ingresses under one Service, but with Gateway API, a Gateway is meant to be a single Service/entrypoint into the cluster, and is independent of other Gateways. But each Gateway handles multiple Routes.

And now that I think about it, you should be able to configure multiple listeners in your Gateway, each with distinct hostnames and tls configurations (but should be able to run on the same port). That could be a much simpler solution, if that works as well.

[ikarlashov]

Single Gateway with multiple listeners could have worked, indeed. But since all developers need to modify it - and it should be done dynamically, sometimes in parallel - I don’t think it’s feasible. If there’s a typo or mistake in the Gateway spec, then the Gateway will become invalid and all teams will lose incoming traffic.

In the scenario where the platform team manages a single Gateway for developers, I think it would work, but then it’s still not dynamic since every single change would need our approval.

It would be nice to have a sub-entity of the Gateway - something like a GatewayRoute - that could be managed by the teams and then validated and compiled by the gateway-api-controller into a single Gateway. Kind of like what the nginx ingress controller does by fetching all Ingress objects and generating a single nginx.conf. Although I can imagine that this goes against the philosophy of the Gateway API and overlaps with the functionality of HTTPRoute/TLSRoute.

[mpstefan]

There is a concept of "ListenerSets" in the Gateway API at the moment to address this kind of thing. You'd have your app teams control the ListenerSets and configuration, but not the Gateway itself... though I'm not sure what real value the Gateway would have when you strip away the listeners.

This was also a mechanism to logically merge gateways (very similar to what NGINX Ingress Controller does), so you could have multiple gateways using the same data plane deployment (i.e. the same nginx deployment in our case).

There's a few problems with this though, as the Gateway API is already quite complicated already. This would also invalidate the role orientation of the API since you're mixing cluster operator and app dev roles. But the use case, as you point out, is still very valid.

At this point, you could either create separate Gateway objects, and thus separate infrastructure, for each team/hostname for access control and isolation or allow access to a single Gateway. If the spec is invalid, it will be rejected and not applied BUT that doesn't prevent all mistakes that are technically valid.

In the future, the Gateway API is definitely looking at how to tackle the logical merging problem, though I'm not sure ListenerSets will make it. I feel like it could just be a field in the Gateway object that groups up the config for the same infrastructure, but that's something that has been evolving in the community.