Target Groups Leaked

[ellistarn]

Target Groups are not automatically deleted when the corresponding Service object is deleted. It looks like there's a garbage collection mechanism that executes every 10 minutes, but it requires an HTTPRoute object to work correctly, Gateway, etc.

This object interdependence violates the Kubernetes API model (flat and independent), and breaks the fundamental product experience of kubectl apply -f myapp.yaml && kubectl delete -f myapp.yaml, where myappl.yaml contains:

• Gateway
• HTTPRoute
• Service
• Deployment

[ellistarn]

One design option to consider:

When we reconcile an object (httproute, service, deployment, etc) that's owned by our gateway implementation, we can add the finalizer lattice.gateway.k8s.aws to the object. Whenever we observe a resource that we reconcile with this finalizer, we can take action, regardless of whether or not there corresponding httproute or gateway or gatewayclass still exists.

[ellistarn]

Related: #15

[liwenwu-amazon]

TLDR

Issue

If customer have deleted ALL HTTPRoute(s) in the cluster, lattice target group(s) created by HTTPRoute can be leaked.

Workaround

Assuming customer will not create HTTPRoute(s) anymore, they need to use VPC lattice API to clean these leaked target group(s)

Notes

If customer have any HTTPRoute(s) belong to a gateway of VPC Lattice Gateway Class, now or future, the lattice controller will reconcile and clean up those leaked target group(s)

[liwenwu-amazon]

Added large size due to testing effort(s) and may require some refactor to trigger cleanup leaked target groups

• Test(s) must make sure there is NO bug which can wipe out valid target group(s). Need to come up all combinations for testing and validation.
• Test(s) must make sure there is NO hung on deleting HTTPRoute,
• The design should make sure it is not too expensive, e.g. controller should not iterate through all lattice target group(s) and K8S gateway(s), httproute(s), service(s) objects all the times to figure if a target group can be garbage collected

[liwenwu-amazon]

A workaround for this:

Provide a script or program to clean up staled target-groups