Always create AAAA alias records in route53 (attempt 2) #5111

rlees85 · 2025-02-20T16:27:35Z

Description

This change creates AAAA and A records for AWS Route53. It is heavily based on this pull request that seems to have sadly been abandoned: #3605

This is now based on the current master branch. I've tried to add some more thorough test cases so that when I let this loose in a Kubernetes cluster hopefully not too many bad things will happen.

Fixes: #3707
Fixes: #4509

All credits to @johngmyers for the vast majority of the leg work.

Checklist

Unit tests updated
End user documentation updated

Checklist to drop WIP

Try this in a real cluster and see if it actually works
Fully address concerns from maintainers in other pull request

k8s-ci-robot · 2025-02-20T16:27:44Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign szuecs for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-02-20T16:27:46Z

Welcome @rlees85!

It looks like this is your first PR to kubernetes-sigs/external-dns 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/external-dns has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2025-02-20T16:27:47Z

Hi @rlees85. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

rlees85 · 2025-02-20T17:06:30Z

I just let it rip in our development cluster. At first look, seems to work well. AAAA records created and nothing has broken, yet.

One observation, is that the TXT records that are created seem a bit odd:

We get: _extdns.aaaa-example.host.name for the AAAA record but the A record is translated somewhere, so we get _extdns.cname-example.host.name. This feels like carry-over from the caching thing or some intentional backwards compatibility? I don't know if we care about this, will need guidance here.

We also get: _extdns.example.host.name as normal.

If anyone has any good example test cases I can carry out I will do so.

rlees85 · 2025-02-20T17:19:22Z

I found the concerns @mloiseleur raised:

It should be documented in doc/tutorials/aws.md, since it works following this tutorial on Service with annotations

I think I've addressed this now.

When using --managed-record-types="A,CNAME", it did NOT update records at all (Same behavior between this PR and v0.13.5). So it does not seem to be a working opt-out.

This works, we were using it wrong:

        - --managed-record-types=A
        - --managed-record-types=CNAME

When changing the hostname, for instance from test2.example.com to test3.example.com, it did not delete the AAAA record:

I can confirm the last one is now working. It cleans up AAAA records when both changing the name of an ingress as well as when deleting it.

ivankatliarchuk · 2025-02-20T17:39:22Z

When you ready, please share all relevant manifest, kubectl and AWS commands. I would like to test it as well.

One concern here, not sure if covered, why would I need AAAA records alongside A records? This seems more like a flag to me, as why would I need AAAA records if I never asked for it, plus it's easy to remove a flag and make it default behaviour at some point?

ivankatliarchuk · 2025-02-20T17:43:15Z

/ok-to-test

rlees85 · 2025-02-20T19:49:53Z

@ivankatliarchuk AAAA records are the same as A records, but for IPv6. So for example, if a load balancer in AWS has dualstack networking enabled (both IPv4 and IPv6) we should add both an A record with the IPv4 address and AAAA record for IPv6.

The previous pull request found that even with IPv4 only load balancers, adding AAAA records causes no harm, since simply no IPv6 address is returned. This makes the code much simpler.

There is not really much point in turning this off beyond not liking AAAA records being in the console when having a look there. I think there were concerns about the extra requests contributing to rate limiting, but a caching feature has gone in recently to mitigate that (I believe?). If a way to turn this off is a deal breaker, I will try to implement something.

With manifests sadly its all a bit rough right now.

Compile the binary, with make
Create this (terrible) Dockerfile next to the binary, in build:

FROM registry.k8s.io/external-dns/external-dns:v0.15.0
COPY "external-dns" "/ko-app/external-dns"
ENTRYPOINT [ "/ko-app/external-dns" ]

Manually push the resulting image to a container registry. I've had to use a private one for now since its a company cluster.
Manually edit the deployment for external DNS, to use the new image.

I will have a look to see if I can host it in a public repository somewhere for testing.

ivankatliarchuk · 2025-02-20T20:24:07Z

Just share manifests here. Normally, you don't need to create a container. You could do something like this https://github.com/kubernetes-sigs/external-dns/blob/master/docs/contributing/dev-guide.md#execute-code-without-building-binary, as long as external-dns is aware about the cluster context.

I got your point, sounds reasonable. What about dualstack-without-public-ipv4 ?

Probaly worth to validate cases when ALB switch is happening from ipv4 -> dualstack -> ipv4 where or not it works without issues

ivankatliarchuk · 2025-02-20T20:27:10Z

docs/tutorials/kube-ingress-aws.md

-
-Example:
-
-```yaml


what happens to this example?

Since this change creates both A & AAAA records (when ALIAS records are enabled) regardless of the loadbalancer being dual stack, this example isn't really relevant to external DNS anymore.

I'm not sure if agree here. This is a tutorial how to create Ingress/ALB on AWS with dualstack. Not related to this pull request.

ivankatliarchuk · 2025-02-20T20:27:25Z

docs/tutorials/aws-load-balancer-controller.md

-
-Example:
-
-```yaml


same, why this example is removed?

Same as above, in scope of External DNS having the load balancer "dualstack" makes no difference to the behaviour.

Same, this is a tutoral how to create aws-load-balancer ingress. Not sure how it relates to changes in behaviour.

What this pull request solves is actually

The above Ingress object will result in the creation of an ALB with a dualstack
interface. ExternalDNS will create both an A echoserver.example.org record and
an AAAA record of the same name, that each are aliases for the same ALB.

I would suggest having a look through the discussion on the original abandoned pull request around this. The existing solution that looked at the "dualstack" annotation worked for the ALB ingress controller ONLY. The idea is that this will work for any ingress controller (with a service of type loadbalancer). At least that is my understanding. I am using ingress nginx with AWS load balancer controller and external DNS is not making any AAAA records, regardless of annotations.

I did. It's not abbout anootations or anything else. The TUTORIAL represent to me, how to create an AWS Load balancer.

This tutorial describes how to use ExternalDNS with the aws-load-balancer-controller.
Setting up ExternalDNS and aws-load-balancer-controller

I might missing the point, why it should be deleted. The tutorial explains how to configure manifests to have a loadbalancer bootstrapped with aws-load-balancer controller

From the tutorial

Example: ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/ip-address-type: dualstack name: echoserver spec: ingressClassName: alb rules: - host: echoserver.example.org http: paths: - path: / backend: service: name: echoserver port: number: 80 pathType: Prefix ``` The above Ingress object will result in the creation of an ALB with a dualstack interface. ExternalDNS will create both an A `echoserver.example.org` record and an AAAA record of the same name, that each are aliases for the same ALB.

So to me, no matter where or not A and AAAA created, if you apply this example to a cluster, it will create an AWS LoadBalancer with DualStack and should have A and AAAA records. I understand that we all read this tutorials differently.

And this pull request actually fixes a behaviour that is documented, aka should have A and AAAA records in Route53

ivankatliarchuk · 2025-02-20T20:37:22Z

Am I understood correctly

Before

piVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/ip-address-type: dualstack
  name: echoserver
spec:
  ingressClassName: alb
  rules:
  - host: echoserver.example.org
    http:
      paths:
      - path: /
        backend:
          service:
            name: echoserver
            port:
              number: 80
        pathType: Prefix

With this change, the proposed behaviour is not to rely on alb.ingress.kubernetes.io/ip-address-type: dualstack but all the records that have IPv6 to actually add them to DNS registry?

This statement is correct

So for example, if a load balancer in AWS has dualstack networking enabled (both IPv4 and IPv6) we should add both an A record with the IPv4 address and AAAA record for IPv6.

But now, on top of that, for example we have dozens of nodes with IPv6 that we do not want to expose over route53, suddenly will be exposed?

rlees85 · 2025-02-20T21:56:23Z

But now, on top of that, for example we have dozens of nodes with IPv6 that we do not want to expose over route53, suddenly will be exposed?

I can't think of any case where you'd have a load balancer with dualstack enabled where you'd want to expose IPv4 but not IPv6. The simple solution here would be to make it not a dualstack load balancer, then it won't be exposed on IPv6.

We create an AAAA record for it (as long as ALIAS is enabled), but it just resolves to blank/empty, so nothing is exposed.

rlees85 · 2025-02-20T22:24:31Z

You can exclude this by using the following flags:

        - --managed-record-types=A
        - --managed-record-types=CNAME

Tested working just now. I think that covers everything required to get this moving

ivankatliarchuk · 2025-02-21T08:30:52Z

Could you share e2e test results and manifests you were using? Example test cases #5085 (comment)

I believe in this case, we are looking for something like create NLB and validate different things, like change it from ipv4 to dualstack and back.

ivankatliarchuk · 2025-02-21T08:39:31Z

You can exclude this by using the following flags:
        - --managed-record-types=A
        - --managed-record-types=CNAME
Tested working just now. I think that covers everything required to get this moving

This most likely should go to documentation somewhere.

ivankatliarchuk · 2025-02-21T08:56:07Z

Any chance to rebase with master? I have make cover-html failing locally

ivankatliarchuk

The PR is quite big. I'll try to go over changes in sources over a weekend.

Could you review docs, there are some references to an annotation external-dns.alpha.kubernetes.io/dualstack. Just reference to annotations need to be removed, the actual manifest we better keep for , as it's still applies

Example

external-dns/docs/sources/gateway.md

Line 87 in 58ac76e

    
           External DNS Controller uses the `external-dns.alpha.kubernetes.io/dualstack` annotation to determine this. If this annotation is

ivankatliarchuk · 2025-02-21T08:42:57Z

docs/tutorials/kube-ingress-aws.md

-
-Example:
-
-```yaml


I'm not sure if agree here. This is a tutorial how to create Ingress/ALB on AWS with dualstack. Not related to this pull request.

ivankatliarchuk · 2025-02-21T08:44:42Z

docs/tutorials/aws-load-balancer-controller.md

-
-Example:
-
-```yaml


Same, this is a tutoral how to create aws-load-balancer ingress. Not sure how it relates to changes in behaviour.

ivankatliarchuk · 2025-02-21T08:45:13Z

docs/tutorials/aws-load-balancer-controller.md

-
-Example:
-
-```yaml


What this pull request solves is actually

The above Ingress object will result in the creation of an ALB with a dualstack
interface. ExternalDNS will create both an A echoserver.example.org record and
an AAAA record of the same name, that each are aliases for the same ALB.

ivankatliarchuk · 2025-02-21T09:09:50Z

provider/aws/aws.go

@@ -788,18 +783,29 @@ func (p *AWSProvider) AdjustEndpoints(endpoints []*endpoint.Endpoint) ([]*endpoi
 			} else {
 				ep.SetProviderSpecificProperty(providerSpecificEvaluateTargetHealth, strconv.FormatBool(p.evaluateTargetHealth))
 			}
+
+			if ep.RecordType == endpoint.RecordTypeCNAME {


is it not safer to create a new Endpoint? To separate the creation of the AAAA record and the modification of the original record type.

if ep.RecordType == endpoint.RecordTypeCNAME { // This needs to match two records from Route53, one alias for 'A' (IPv4) // and one alias for 'AAAA' (IPv6). aliasCnameAaaaEndpoints = append(aliasCnameAaaaEndpoints, &endpoint.Endpoint{ DNSName: ep.DNSName, Targets: ep.Targets, RecordType: endpoint.RecordTypeAAAA, RecordTTL: ep.RecordTTL, Labels: ep.Labels, ProviderSpecific: ep.ProviderSpecific, SetIdentifier: ep.SetIdentifier, }) ep.RecordType = endpoint.RecordTypeA }

I'll be first to admit that whilst I enjoy Go I rarely get to use it day to day. What would the advantage be in creating a new endpoint as opposed to cloning the existing one?

So we not accidently change values, as you passing it by reference in current implementation

I see, that could be a misunderstanding on my part. If there is a way of cloning it (without just referencing it) would that not be better? Since if the structure of "endpoint.Endpoint" ever changes, we need to update the fields in this fairly obscure place also?

I shared an example that could be described as cloning.

ivankatliarchuk · 2025-02-21T09:14:16Z

provider/aws/aws.go

@@ -1142,7 +1142,7 @@ func useAlias(ep *endpoint.Endpoint, preferCNAME bool) bool {
 // and (if so) returns the target hosted zone ID
 func isAWSAlias(ep *endpoint.Endpoint) string {
 	isAlias, exists := ep.GetProviderSpecificProperty(providerSpecificAlias)
-	if exists && isAlias == "true" && ep.RecordType == endpoint.RecordTypeA && len(ep.Targets) > 0 {
+	if exists && isAlias == "true" && slices.Contains([]string{endpoint.RecordTypeA, endpoint.RecordTypeAAAA}, ep.RecordType) && len(ep.Targets) > 0 {


I see the same logic aka slices.Contains shell we consider to create a method in endpoints go?

something like

func (ep *Endpoint) IsSupportedRecordTypes(r ...string) bool

Just an idea

ivankatliarchuk · 2025-02-21T09:25:35Z

source/ingress_test.go

@@ -44,12 +44,11 @@ func (suite *IngressSuite) SetupTest() {
 	fakeClient := fake.NewSimpleClientset()


In this file missing ipv4 with ipv6 test

ivankatliarchuk · 2025-02-21T09:26:30Z

source/gateway_test.go

@@ -246,47 +245,3 @@ func TestIsDNS1123Domain(t *testing.T) {
 		})
 	}
 }
-
-func TestDualStackLabel(t *testing.T) {


The test is removed, which is understandable, but what about a test where ipv4 with ipv6 ?

ivankatliarchuk · 2025-02-21T09:26:39Z

source/kong_tcpingress.go

@@ -192,16 +191,6 @@ func (sc *kongTCPIngressSource) filterByAnnotations(tcpIngresses []*TCPIngress)
 	return filteredList, nil
 }

-func (sc *kongTCPIngressSource) setDualstackLabel(tcpIngress *TCPIngress, endpoints []*endpoint.Endpoint) {


The test is removed, which is understandable, but what about a test where ipv4 with ipv6 ?

ivankatliarchuk · 2025-02-21T09:26:57Z

source/skipper_routegroup.go

@@ -324,16 +323,6 @@ func (sc *routeGroupSource) endpointsFromTemplate(rg *routeGroup) ([]*endpoint.E
 	return endpoints, nil
 }

-func (sc *routeGroupSource) setRouteGroupDualstackLabel(rg *routeGroup, eps []*endpoint.Endpoint) {


The test is removed, which is understandable, but what about a test where ipv4 with ipv6 ?

ivankatliarchuk · 2025-02-21T09:27:29Z

source/traefik_proxy.go

@@ -269,7 +269,6 @@ func (ts *traefikSource) ingressRouteEndpoints() ([]*endpoint.Endpoint, error) {
 		}


Shell we have a test fir ipv4 and ipv6?

rlees85 · 2025-02-21T09:52:28Z

Thanks for the review (so far). There is a lot to go at here, I will try and come back with some changes.
edit: I will also come back with some manifests and methods of testing that I've used.

extra tests.

ivankatliarchuk · 2025-02-21T10:02:16Z

We most likely need to be think more about our users as well.

As in this pull request we are removing the support for external-dns.alpha.kubernetes.io/dualstack most likely, instead of ditching it completely, we should first have a warning message in every place where the annotation was expected. And mention that external-dns.alpha.kubernetes.io/dualstack annotation is no longer supported when found on a source or provider. So end users aware and no longer rely on it.

@mloiseleur wdyt?

rlees85 · 2025-02-21T10:11:00Z

@ivankatliarchuk

I wasn't aware of an external-dns.alpha.kubernetes.io/dualstack annotation, just a alb.ingress.kubernetes.io/ip-address-type: dualstack which is primarily affecting the behaviour of the AWS load balancer controller.

Following this change, ExternalDNS does not "care" if this third party annotation is set or not, hence why I don't really want to get bogged down with documenting it in ExternalDNS.

It looks like external-dns.alpha.kubernetes.io/dualstack is only documented on the gateway API source. I will have a look to see what that is actually doing and if it is affected by this PR.

edit: yes it looks like gateway source used it. The label was only ever picked up by the AWS provider, which still now take the "with dualstack annotation" behaviour by default. To users, who are using this annotation, will see no change in behaviour. Users who don't like seeing AAAA records in the Route53 console will need the exclusion flag, as per push docs. Sorry I hope this helps!

ivankatliarchuk · 2025-02-21T11:35:57Z

@ivankatliarchuk

I wasn't aware of an external-dns.alpha.kubernetes.io/dualstack annotation, just a alb.ingress.kubernetes.io/ip-address-type: dualstack which is primarily affecting the behaviour of the AWS load balancer controller.

Following this change, ExternalDNS does not "care" if this third party annotation is set or not, hence why I don't really want to get bogged down with documenting it in ExternalDNS.

It looks like external-dns.alpha.kubernetes.io/dualstack is only documented on the gateway API source. I will have a look to see what that is actually doing and if it is affected by this PR.

edit: yes it looks like gateway source used it. The label was only ever picked up by the AWS provider, which still now take the "with dualstack annotation" behaviour by default. To users, who are using this annotation, will see no change in behaviour. Users who don't like seeing AAAA records in the Route53 console will need the exclusion flag, as per push docs. Sorry I hope this helps!

Got it. I've tried to understand why this annotations was added, but not too sure. Some relevant pull requests https://github.com/kubernetes-sigs/external-dns/pulls?q=is%3Apr+dualstack+annotation+is%3Aclosed. Seems like is safe to remove references to this annotation

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 20, 2025

k8s-ci-robot requested review from mloiseleur and szuecs February 20, 2025 16:27

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Feb 20, 2025

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Feb 20, 2025

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Feb 20, 2025

rlees85 mentioned this pull request Feb 20, 2025

WIP Always create AAAA alias records in route53 #3605

Open

2 tasks

rlees85 force-pushed the always-aaaa branch from a7faa8e to 698c883 Compare February 20, 2025 16:32

rlees85 changed the title ~~WIP: Always create AAAA alias records in route53 (attempt 2)~~ Always create AAAA alias records in route53 (attempt 2) Feb 20, 2025

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 20, 2025

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Feb 20, 2025

ivankatliarchuk suggested changes Feb 20, 2025

View reviewed changes

k8s-ci-robot assigned ivankatliarchuk Feb 20, 2025

ivankatliarchuk suggested changes Feb 21, 2025

View reviewed changes

rlees85 added 3 commits February 21, 2025 10:00

First pass based on existing PR, what is currently on master and some

04edb1a

extra tests.

Try to resolve AWS service documentation

51d49a7

Add documentation on how to opt-out of AAAA record creation

23dd7ea

rlees85 force-pushed the always-aaaa branch from f091b7d to 23dd7ea Compare February 21, 2025 10:00

		@@ -44,12 +44,11 @@ func (suite *IngressSuite) SetupTest() {
		fakeClient := fake.NewSimpleClientset()

		@@ -269,7 +269,6 @@ func (ts traefikSource) ingressRouteEndpoints() ([]endpoint.Endpoint, error) {
		}

Always create AAAA alias records in route53 (attempt 2) #5111

Are you sure you want to change the base?

Always create AAAA alias records in route53 (attempt 2) #5111

Conversation

rlees85 commented Feb 20, 2025 • edited Loading

k8s-ci-robot commented Feb 20, 2025

k8s-ci-robot commented Feb 20, 2025

k8s-ci-robot commented Feb 20, 2025

rlees85 commented Feb 20, 2025

rlees85 commented Feb 20, 2025 • edited Loading

ivankatliarchuk commented Feb 20, 2025

ivankatliarchuk commented Feb 20, 2025

rlees85 commented Feb 20, 2025

ivankatliarchuk commented Feb 20, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ivankatliarchuk Feb 21, 2025 • edited Loading

Choose a reason for hiding this comment

ivankatliarchuk commented Feb 20, 2025

rlees85 commented Feb 20, 2025

rlees85 commented Feb 20, 2025

ivankatliarchuk commented Feb 21, 2025 • edited Loading

ivankatliarchuk commented Feb 21, 2025

ivankatliarchuk commented Feb 21, 2025

ivankatliarchuk left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rlees85 commented Feb 21, 2025 • edited Loading

ivankatliarchuk commented Feb 21, 2025

rlees85 commented Feb 21, 2025 • edited Loading

ivankatliarchuk commented Feb 21, 2025

rlees85 commented Feb 20, 2025 •

edited

Loading

rlees85 commented Feb 20, 2025 •

edited

Loading

ivankatliarchuk Feb 21, 2025 •

edited

Loading

ivankatliarchuk commented Feb 21, 2025 •

edited

Loading

ivankatliarchuk left a comment •

edited

Loading

rlees85 commented Feb 21, 2025 •

edited

Loading

rlees85 commented Feb 21, 2025 •

edited

Loading