-
Notifications
You must be signed in to change notification settings - Fork 640
✨ IPv6 support for self-managed clusters #5603
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Welcome @tthvo! |
|
Hi @tthvo. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/cc @nrb @sadasu @patrickdillon I am not yet sure what to do with e2e tests or if there are any existing ones for IPv6 clusters...I leave it as an pending TODO. |
|
@tthvo: GitHub didn't allow me to request PR reviews from the following users: sadasu, patrickdillon. Note that only kubernetes-sigs members and repo collaborators can review this PR, and authors cannot review their own PRs. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
A quick preview of |
|
/ok-to-test |
|
/assign @mtulio Asking you for a review Marco as I know you have been working on this downstream |
(AWSCluster resource)
AWS requires that when registering targets by instance ID for an IPv6 target group, the targets must have an assigned primary IPv6 address. Note: The default subnets managed by CAPA are already set up to assign IPv6 addresses to newly created ENIs.
The httpProtocolIPv6 field enables or disables the IPv6 endpoint of the instance metadata service. The SDK only applies this field if httpEndpoint is enabled. When running on single-stack IPv6, pods only have IPv6, thus requiring an IPv6 endpoint to query IMDS as IPv4 network is unreachable.
In the case where egress-only-internet-gateway is deleted, CAPA reconcilation loop will create a new one. Thus, CAPA needs to modify the routes to point to the new eigw ID.
This allows IPv6-only workloads to reach IPv4-only services. AWS supports this via NAT64/DNS64. More details: https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-nat64-dns64.html
CAPA handles icmpv6 as a protocol number 58. AWS accepts protocol number when creating rules. However, describing a rule from AWS API returns the protocol name, thus causing CAPA to not recognize it and fail.
…ices For IPv4, we have field NodePortIngressRuleCidrBlocks that specifies the allowed source IPv4 CIDR for node NodePort services on port 30000-32767. This extends that field to also accept IPv6 source CIDRs.
We need an option to configure IPv6 source CIDRs for SSH ingress rule of the bastion host. This extends the field allowedCIDRBlocks to also accepts IPv6 CIDR blocks.
When creating a bastion host for an IPv6 cluster, the instance has both public IPv4 and IPv6. Thus, we need to report them in the cluster status if any. This also adds an additional print column to display that bastion IPv6.
This is a minimal template set to install an IPv6-enabled cluster. Both the controlplane and worker nodes must use nitro-based instance type (with IPv6 support).
This is a set of customized calico CNI manifests to support single-stack IPv6 cluster. Note that VXLAN is used since IP-in-IP currently only supports IPv4. References: - https://docs.tigera.io/calico/latest/networking/ipam/ipv6#ipv6 - https://docs.tigera.io/calico/latest/getting-started/kubernetes/self-managed-onprem/config-options#switching-from-ip-in-ip-to-vxlan - https://docs.tigera.io/calico/latest/networking/configuring/vxlan-ipip
This combines existing docs for IPv6 EKS clusters with non-EKS ones, and also properly register the topic page into the documentation TOC.
Validation for specified VPC and subnet CIDRs is added for early feedback from the webhook. There are already existing checks for bastion and nodePort CIDRs.
The following is added: - [BYO VPC] Mention the required route when enabling DNS64. - [BYO VPC] Mention that CAPA only utilizes the IPv6 aspect of the dual stack VPC.
There is a brief period where the IPv6 CIDR is not yet associated with the subnets. Thus, when CAPA creates the default dualstack subnets, it should wait until the IPv6 CIDR is associated before proceeding. If not, CAPA will misinterprete the subnet as non-IPv6 and proceed its reconcilation. The consequence is that CAPA will skip creating a route to eigw. Route to eigw for destination "::/0" to eigw is required for EC2 instance time sync on start-up.
…ined When AWSCluster.spec.network.vpc.ipv6 is non-nil, most handlers in CAPA treats it as "adding" IPv6 capabilities on top of IPv4 infrastructure. Except security group ingress rules for API LB. This commit aligns the API LB SG handler with the rest of the code base. These rules can be overriden in the AWSCluster LB spec to allow only IPv6 CIDRs if needed.
The field isIpv6 is set to true if and only if the subnet has an associated IPv6 CIDR. This means the VPC is also associated with an IPv6 CIDR.
The field targetGroupIPType is added to the loadbalancer spec to allow configuring ip address type of target group for API load balancers. This field is not applicable to Classic Load Balancers (CLB). This commit also defines a new network status field to determine the ip type of API load balancers.
When creating subnets in a managed VPC with IPv6 enabled, automatically assign IPv6 CIDR blocks to subnets that have isIPv6=true but no explicit IPv6CidrBlock specified. This simplifies subnet configuration by allowing users to enable IPv6 without manually calculating and assigning individual subnet IPv6 CIDRs, for example, in case where VPC IPv6 CIDR is unknown pre-provisioning and AWS will assign one during VPC creation. Note: This logic only applies when spec.network.vpc.ipv6 is non-nil, subnets are managed and non-existing.
The field awscluster.spec.network.vpc.ipv6.ipamPool defines the IPAM pool to allocate an IPv6 CIDR for the VPC. Previously, CAPA only considers field awscluster.spec.network.vpc.ipamPool, which is used only for VPC IPv4 CIDR allocation. Additionally, CAPA should preserve the ipv6 spec fields, provided by the users, for example, the ipv6 ipamPool. Previously, these spec fields are lost during vpc reconcilation.
NAT64/DNS64 is meant to be enabled for IPv6-only subnets, in which instances do not have IPv4 [0]. If we enable DNS64 for dualstack subnets, instances will receive both A/AAAA records for IPv4-only services. In most cases, OS-level settings will prefer IPv6, leading to traffic to flow via NAT gateway instead of using IPv4 directly. Reference [0] https://aws.amazon.com/blogs/networking-and-content-delivery/dual-stack-architectures-for-aws-and-hybrid-networks-part-2/
Add new dualstack cluster template and documentation updates for IPv6 and dualstack cluster configurations. Additionally, docs for configuring API LB's target group IP type is also added. New cluster templates and calico manifest are included for creating dualstack clusters.
2fb4f55 to
160c6b1
Compare
|
The latest rebase is to resolve conflicts with |
mtulio
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Excelent feature, @tthvo ! A few questions related to the matrix of supported scenarios, as well the backward compatibility.
| // Valid values are ipv4 and ipv6. If not specified, defaults to ipv4 unless | ||
| // the VPC has IPv6 enabled, in which case it defaults to ipv6. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
unless the VPC has IPv6 enabled
Is that statement correct even when preferred is IPv4 which has IPv6 enabled in the VPC?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reading the comment I understood that users would have the path where the VPC supports IPv6 but target would be Ip4, is that correct my understanding?
|
|
||
| ## Overview | ||
|
|
||
| CAPA enables you to create IPv6 and dualstack (IPv4 + IPv6) Kubernetes clusters on Amazon Web Services (AWS) on a dualstack network infrastructure. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What about this following how AWS refers to IPv6 deployment on their docs ("IPv6 only" or "IPv6-only")?
| CAPA enables you to create IPv6 and dualstack (IPv4 + IPv6) Kubernetes clusters on Amazon Web Services (AWS) on a dualstack network infrastructure. | |
| CAPA enables you to create IPv6 only and dualstack (IPv4 + IPv6) Kubernetes clusters on Amazon Web Services (AWS) on a dualstack network infrastructure. |
I was also wondering about the "single-stack IPv6" verbiage, but I didn't find many references in AWS docs.
|
|
||
| ## Enabling IPv6 capabilities | ||
|
|
||
| To instruct CAPA to configure IPv6 capabilities for the network infrastructure, you must explicitly define `spec.network.vpc.ipv6` in either `AWSCluster` (for self-managed clusters) or `AWSManagedControlPlane` (for EKS clusters). See [IPv6 CIDR Allocations](#ipv6-cidr-allocations) for different IPv6 CIDR configuration options. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
or
AWSManagedControlPlane(for EKS clusters)
Is ROSA, managed OCP, also reacts to this field?
| kubectl -n kube-system edit cm/coredns | ||
| ``` | ||
|
|
||
| **Note**: This CoreDNS workaround is NOT required for dualstack clusters where pods have both IPv4 and IPv6 addresses. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am note sure if I followed completely this statement. I see the CoreOS limitation, but are we going to support IPv6 only self-managed clusters (without requiring NAT64)? I am wondering scenarios where users wants to rid off public IPv4 costs, which includes NAT Gateway (required by NAT64);
| <aside class="note warning"> | ||
| <h1>Warning</h1> | ||
|
|
||
| The AWS Cloud Controller Manager (CCM) does **not** currently support dualstack Load Balancers. When creating Services of type LoadBalancer in a dualstack cluster, the Load Balancers will be created with **only** IPv4. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What about to mention that CLB does not support IPv6 by AWS too? as well point to the CCM issue that will be supported when resolved: kubernetes/cloud-provider-aws#1219
| ## Target Group IP Address Type | ||
| The secondary load balancer supports the same `targetGroupIPType` configuration as the primary load balancer. By default, the target group IP address type is set based on the VPC configuration: | ||
| - If the VPC has IPv6 enabled, the target group uses `ipv6` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what is the behavior in dualstack preferred ipv4? Is that the default only (discovered ipv6) and user must manually set to ipv4 when wants preferred?
| func (s *Service) getNat64PrivateRoute(natGatewayID string) *ec2.CreateRouteInput { | ||
| return &ec2.CreateRouteInput{ | ||
| NatGatewayId: aws.String(natGatewayID), | ||
| DestinationIpv6CidrBlock: aws.String(services.NAT64CidrBlock), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is NAT64 required to make an operational cluster on AWS? Can we optionally enable it without impact, specially in preferred ipv6?
One of concerns of users willing to use IPv6 is the IPv4 costs, on AWS we can't fully eliminate it as public LBs still need it, but I wonder if we can reduce the dependency of Nat Gateways with that new proposal. What do you think?
With the preferred IPv6 topology, do we need to allocate one-per-az Nat Gateway (which is a expensive resource)? I wonder if we can create minimum topology (single, or dual, instead per-AZ) of NAT Gateways on IPv6 clusters, helping users to eliminate public IPv4 address from their environment.
topology
Something like: NATGatewayTopologyStrategy: default|single|dual
Where:
- default: one per az, not changing the default (zonal redundant)
- single: one NGW for entire cluster (SPOF)
- dual: dual NGW with HA in two AZs (if cluster is deployed on >1 AZs)
| const ( | ||
| LoadBalancerIPAddressTypeIPv4 = LoadBalancerIPAddressType("ipv4") | ||
| LoadBalancerIPAddressTypeDualstack = LoadBalancerIPAddressType("dualstack") | ||
| LoadBalancerIPAddressTypeDualstackWithoutPublicIPv4 = LoadBalancerIPAddressType("dualstack-without-public-ipv4") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are we using this const / is it a valid value for AWS API?
| if subnet.ZoneType != nil && subnet.IsEdge() { | ||
| if subnet.ParentZoneName == nil { | ||
| allErrs = append(allErrs, field.Invalid(field.NewPath("subnets"), r.Spec.NetworkSpec.Subnets, "ParentZoneName must be set when ZoneType is 'local-zone'.")) | ||
| allErrs = append(allErrs, field.Invalid(subnetField.Index(i).Child("parentZoneName"), subnet.ParentZoneName, "ParentZoneName must be set when ZoneType is 'local-zone'.")) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I known it would be outside this PR, but let me document it - this comment must be something like this as isEdge() evaluates both Local Zone and Wavelength locations:
| allErrs = append(allErrs, field.Invalid(subnetField.Index(i).Child("parentZoneName"), subnet.ParentZoneName, "ParentZoneName must be set when ZoneType is 'local-zone'.")) | |
| allErrs = append(allErrs, field.Invalid(subnetField.Index(i).Child("parentZoneName"), subnet.ParentZoneName, "ParentZoneName must be set when ZoneType is 'local-zone' or 'wavelength-zone.")) |
| // Enable DNS64 so that the Route 53 Resolver returns DNS records for IPv4-only services | ||
| // containing a synthesized IPv6 address prefixed 64:ff9b::/96. | ||
| // This is needed alongside NAT64 to allow IPv6-only workloads to reach IPv4-only services. | ||
| // We only need to enable on private subnets. | ||
| if !sn.IsPublic { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Two comments on this:
1:
This is needed alongside NAT64 to allow IPv6-only workloads to reach IPv4-only services... We only need to enable on private subnets.
May this prevent ip6-only workloads running in nodes created in "public subnets" accessing resources in private subnets?
- I am not seeing this route in any of private subnets of clusters I've created both with primarily ipv4 and ipv6, I need to investigate closely in the output manifests or logs if I missed some error
What type of PR is this?
/kind feature
What this PR does / why we need it:
As of today, CAPA supports IPv6 on EKS, but not self-managed clusters. Thus, these changes bring IPv6 support for self-managed clusters, both single-stack IPv6 and dualstack.
Which issue(s) this PR fixes:
Fixes #2420
Fixes #3381
Special notes for your reviewer:
test/e2e/data/cni. Calico does not support IPv6 with "IP-in-IP" so we need to use VXLAN.Checklist:
Release note: