feat: implement IPVLAN Subinterface configuration and provision

[ngcxy]

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR introduces support for subinterface provisioning in the DRANET driver, currently implementing IPVLAN as the subinterface type. The feature addresses two networking requirements:

• Support RDMA NIC sharing #63: It provides the implementing foundation for sharing a physical interface among multiple pods by allowing multiple subinterfaces to be created on a single NIC.
• Support LACP bonded network interfaces #239: It satisfies the requirement for LACP-bonded interfaces by creating a new subinterface inside the pod network namespace without moving the host’s parent interface.

The key changes include:

• Add SubInterfaceConfig and IPVlanConfig to NetworkConfig, including IPRanges — a list of allocation ranges, each expressed as a CIDR or an explicit start/end pair.
• During ResourceClaim preparation, assign the subinterface name, allocate its IP(s), and configure source-based routing. User-configured routes/rules are preserved — automatic source-based routing is skipped when the user already specified them.
• Add a dedicated pkg/ipam package with a concurrency-safe node-local LocalIPAM that owns the set of in-use addresses and hands out one address per IP family from the configured ranges; allocated IPs are released on claim teardown.
• Integrate subinterface create/delete (IPVLAN) inside the pod network namespace via the NRI run/stop sandbox hooks.
• Derive the subinterface range from GCE VM metadata in GetDeviceConfig, using a shared, cloud-agnostic cloudutil.IPRangeFromCIDR helper (network and broadcast always excluded; providers specify any additional reserved counts).

Which issue(s) this PR is related to:

Fixes #63, #239

Special notes for your reviewer:

The code was tested by requesting a subinterface in ResourceClaim. The pod runs successfully with the IPVlan interface (connectivity verified via ping test), and the parent interface remains in the host’s namespace.
ResourceClaimTemplate yaml example:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: one-ipvlan
spec:
  spec:
    devices:
      requests:
      - name: one-ipvlan
        exactly:
          deviceClassName: netdev.google.com
          allocationMode: ExactCount
          count: 1
      config:
      - opaque:
          driver: dra.net
          parameters:
            interface:
              subInterface:
                type: "ipvlan"

Does this PR introduce a user-facing change?

Enable the support for IPVlan subinterfaces provisioning on parent NICs, configured with dynamic IP allocation and source-based routing which support both IPv4 and IPv6.
Subinterface creation can be triggered either by the cloud provider or by a user request in a ResourceClaim.
`Type` (only `ipvlan` is supported for now) is required to enable subinterface creation, and `IPRanges` should be provided when no generic IPAM is enabled.

[netlify]

✅ Deploy Preview for dranet ready!

Name	Link
🔨 Latest commit	9517532
🔍 Latest deploy log	https://app.netlify.com/projects/dranet/deploys/6a68e965f666080008446841
😎 Deploy Preview	https://deploy-preview-241--dranet.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[kubernetes-prow]

Hi @ngcxy. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[Copilot]

Pull request overview

This draft PR adds IPVLAN-based subinterface support to the DRANET driver to enable sharing a single NIC across multiple pods, including allocation-time naming/IPAM, netns plumbing for subinterface creation/teardown, and GCE-derived subinterface ranges.

Changes:

• Added SubInterface API types/defaults/validation and integrated subinterface-aware behavior into DRA + NRI hooks.
• Introduced a LocalIPAM allocator and PodConfigStore tracking to allocate addresses from subinterface ranges.
• Implemented GCE-specific subinterface IPRange derivation (IPv6 /80 from node /64) plus bare-metal routing/neighbor config.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
pkg/driver/subinterfaces.go	Creates/deletes IPVLAN subinterfaces in the container netns and reports NetworkDeviceData.
pkg/driver/pod_device_config.go	Adds in-memory allocated-IP tracking to back LocalIPAM allocation decisions.
pkg/driver/nri_hooks.go	Switches Run/Stop hooks to create/delete subinterfaces when configured; factors common netns config.
pkg/driver/local_ipam.go	New local sequential allocator for IP selection from a CIDR range.
pkg/driver/driver.go	Wires LocalIPAM into driver startup.
pkg/driver/dra_hooks.go	Adds subinterface-aware naming + optional IP assignment from IPRange + source-based rule insertion.
pkg/cloudprovider/gce/gce.go	Adds GCE-derived subinterface IPRange plus bare-metal route/neighbor configuration.
pkg/cloudprovider/gce/gce_test.go	Adds tests for the new GCE device-config behavior and IPv6 range derivation.
pkg/apis/types.go	Introduces SubInterfaceConfig, SubInterfaceType, and IPVlanConfig API types.
pkg/apis/constants.go	Adds the SubInterfaceTypeIPVlan constant.
pkg/apis/defaults.go	Defaults SubInterface type/mode/flag.
pkg/apis/validation.go	Validates SubInterface and IPVLAN config and IPRange format.
pkg/apis/validation_test.go	Adds test cases for subinterface validation.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[tamilmani1989]

Does this consider multi nic scenario? There can be multiple shared nics in host.. in that case, it might required to have separate vrf for each nic..

[ngcxy]

Does this consider multi nic scenario? There can be multiple shared nics in host.. in that case, it might required to have separate vrf for each nic..

Hi @tamilmani1989 , could you elaborate more on the requirement of separate vrf? This feature will cover the multi-NIC scenario as more generic use cases, but within the current change, the vrf part modification isn't considered. Would like to hear your suggestion on this.

[Bowser1704]

No way to inherit routes/neighbors from the parent interface. The GCE provider hardcodes a virtual gateway MAC, but bond gateways are dynamic (NDP-learned), not available in cloud metadata. Consider copyRoutesFromParent / copyNeighborsFromParent fields so the driver can read from the parent at runtime.

[ngcxy]

I have updated addSourceBasedRouting in dra_hooks to dynamically detects the gateway from the parent, and create routes based on it. The permanent neighbors are directly copied from the host during prepareResourceClaim.

[Bowser1704]

Thanks for the work!

addSourceBasedRouting clears user-configured routes, which silently discards any routes/rules the user specified in the ResourceClaim. Users can and do write custom static routes.

Should this skip auto source-based routing when the user already configured routes/rules, rather than overwriting them?

[ngcxy]

Good catch! Updated to keep user config routes and skip source-based routing in dra hooks.

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ngcxy
Once this PR has been reviewed and has the lgtm label, please ask for approval from gauravkghildiyal. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[gauravkghildiyal]

Thanks @ngcxy. Leaving some feedback for mainly the 2nd commit. Will revisit the others later.

[tamilmani1989]

Does this consider multi nic scenario? There can be multiple shared nics in host.. in that case, it might required to have separate vrf for each nic..

Hi @tamilmani1989 , could you elaborate more on the requirement of separate vrf? This feature will cover the multi-NIC scenario as more generic use cases, but within the current change, the vrf part modification isn't considered. Would like to hear your suggestion on this.

if multiple nic share same address space, then it might need vrf to route traffic via appropriate interface

[Bowser1704]

Use same logger like line 343.

[Bowser1704]

Should this exclusion applies to IPv6 too?

[Bowser1704]

LGTM. I prototyped Aliyun's LACP-bond use case (issue #239) on top of this PR, and it only took about 90 lines in our cloud provider — zero changes to pkg/driver.

[gauravkghildiyal]

Another round focusing mostly on API.

[gauravkghildiyal]

readability: Please format the config across multiple lines.

[gauravkghildiyal]

readability: It's acceptable to have simpler things be in one line, but I think all these configs together seem to warrant being spread over multiple lines.

[gauravkghildiyal]

Why would be permit this use case of allowing both a CIDR and a Start+End together?

[gauravkghildiyal]

I think our validation tests should also take defaulting into account.

[gauravkghildiyal]

Any specific reason to have this divergence in behaviour for IPRanges? If not, let's have consistent behaviour

[gauravkghildiyal]

Interface names have a 15 char limit. Adding something like "-ipvlan" consumes 7 characters itself and it's possible that the "<parent_name>-ipvlan" may easily exceed the 15 char limit.

I would say let's not add any prefix/suffix at all and keep the original name as the default. We can leave the option of choosing a custom name

[gauravkghildiyal]

I've been giving this some more thought and I think merging this together with the existing Interface should be possible and avoid a lot of duplication.

Any thoughts on roughly something like this?

// 1. Remove SubInterface from NetworkConfig
type NetworkConfig struct {
	Profile   string          `json:"profile,omitempty"`
	Interface InterfaceConfig `json:"interface"`
	
	// REMOVED: SubInterface *SubInterfaceConfig `json:"subInterface,omitempty"`
	
	Routes    []RouteConfig   `json:"routes,omitempty"`
	Rules     []RuleConfig    `json:"rules,omitempty"`
	// ... (Neighbors, Ethtool)
}

// 2. Add the 3 new fields to your existing InterfaceConfig
type InterfaceConfig struct {
	// ADDED: Type determines how the interface is provided to the Pod.
	// "passthrough" (default) or "ipvlan"
	Type InterfaceType `json:"type,omitempty"`
	
	// ADDED: IPVlan holds IPVLAN-specific settings; only evaluated when Type is "ipvlan".
	IPVlan *IPVlanConfig `json:"ipvlan,omitempty"`

	// ... (Keep existing fields: Name, Addresses, MTU, VRF, etc.) ...

	// ADDED: IPRanges allows node-local IPAM to generate addresses for this interface.
	IPRanges []IPRangeConfig `json:"ipRanges,omitempty"`
}

// 3. Rename SubInterfaceType to InterfaceType
type InterfaceType string

const (
	InterfaceTypePassthrough InterfaceType = "passthrough"
	InterfaceTypeIPVlan      InterfaceType = "ipvlan"
)

// 4. Keep your existing IPVlanConfig (no changes needed here)
type IPVlanConfig struct {
	Mode string `json:"mode,omitempty"`
	Flag string `json:"flag,omitempty"`
}

There's some good general guidance on API design here https://github.com/kubernetes/community/blob/main/contributors/devel/sig-architecture/api-conventions.md#automatic-resource-allocation-and-deallocation, including things like why we should define things like an InterfaceType

[Bowser1704]

PrepareResourceClaims only calls LocalIPAM.Allocate() when SubInterface.Addresses is empty. When a user statically specifies Addresses instead of IPRanges, that address is used as-is and never registered into LocalIPAM.allocatedIPs.

This means IPAM has no visibility into statically-assigned addresses during the current driver session. If another claim's IPRanges happens to overlap with a manually-specified static address, Allocate() has no way to know it's taken and can hand it out to a different pod — a real collision, not just a hypothetical one.

(After a driver restart, GetAllocatedIPs() reseeds LocalIPAM from every persisted SubInterface.Addresses, static or not — pod_device_config.go:337 — so static addresses become "known" post-restart.)

Suggested direction: make address resolution go through one path regardless of source, instead of two paths that only sometimes touch shared state.

• If the config specifies Addresses, LocalIPAM should still record/reserve them (and can validate they don't collide with something already allocated), instead of skipping IPAM entirely.
• If the config specifies IPRanges, LocalIPAM fills in Addresses from the pool, exactly as it does today.

Either way, once Addresses is resolved, everything downstream (addIPVlan, addSourceBasedRouting) is already source-agnostic — it just iterates whatever ended up in Addresses. So this isn't a call to rewrite the consuming side; it's specifically about making the resolution step (static vs. IPAM-derived) go through the same bookkeeping, so LocalIPAM is actually the single source of truth for "what's in use" rather than only knowing about addresses it personally handed out.

[aojea]

We should not make IPAM part of the interface API, https://github.com/kubernetes-sigs/dranet/pull/241/changes#r3710308724

An Interface or Subinterface define its attributes, IPAM is not an interface attribute, that process has to happen before we get here and populate the existing Addresses field, so each cloud provider can decide how they do IPAM

[aojea]

we already have Addresses why do we want to leak IPRanges here? Are we not able to do the IPAM without leaking it into the API? we are already doing it for the ones that are not subinterfaces

[aojea]

I really want to avoid leaking IPAM config in the API ... this is about subinterfaces properties, IPAM is network configuration

[aojea]

/assign @aojea

let's discuss the ipam addition and why it can not be part of the provider logic

[aojea]

/hold

Unless there's some additional limitation I can't see from the current diff, I don't see the need for a second way of handling IPAM here that's different from what we already do for the primary interface.
Profile + ProfileProvider.GetProfileConfig/ReleaseProfileConfig already gives any provider a stateful, per-claim hook to allocate/release addresses and hand back Addresses.

Adding IPRanges/IPRangeConfig to SubInterfaceConfig with a driver-owned LocalIPAM invoked directly from dra_hooks.go, reintroduces IPAM as a first-class API/driver that is something that goes against the philosophy of the project, the API represents Interface properties, not Interface configuration properties , see more in opencontainers/runtime-spec#1271 (comment) about how the design of the container networking netdvice is specified. If there's a concrete requirement that GetProfileConfig can't achieve, let's make it clear, but I honestly don't see one yet.

GetDeviceConfig is for static, cacheable, per-device metadata — called once, merged unconditionally, regardless of which claim ends up using the device, is the declarative state of the Network Interface. GetProfileConfig is the per-claim, stateful hook — the one that's supposed to actually allocate and provide dynamic configuration. In this PR, GCE's GetDeviceConfig hands back IPRanges (fine as device metadata) but then the actual per-claim address allocation happens generically inside driver, against that field, instead of inside GetProfileConfig. That's what's requiring IPRanges to be leaked in the API and the extra merge/dedup logic in config_merge.go.

I think all the use cases can be solved just with SubInterface.Addresses:

1. Cloud provider with its own address pool (GCE, and AWS/Azure/Alibaba later)

config:
- opaque:
    driver: dra.net
    parameters:
      profile: gce-subinterface   # any non-empty value triggers GetProfileConfig
      interface:
        subInterface:
          type: ipvlan

// pkg/cloudprovider/gce/gce.go
type GCEInstance struct {
    // ...
    subIPAM *ipam.LocalIPAM // internal allocator; never (de)serialized
}

func (g *GCEInstance) GetProfileConfig(id cloudprovider.DeviceIdentifiers, claimUID types.UID, config *apis.NetworkConfig) (*apis.NetworkConfig, error) {
    if config.SubInterface == nil || config.SubInterface.Type == "" {
        return nil, nil // not a subinterface request
    }
    ranges, err := g.subInterfaceRangesFor(id) // derived from instance metadata, stays internal
    if err != nil {
        return nil, err
    }
    addrs, err := g.subIPAM.Allocate(ranges)
    if err != nil {
        return nil, err
    }
    return &apis.NetworkConfig{SubInterface: &apis.SubInterfaceConfig{Addresses: addrs}}, nil
}

func (g *GCEInstance) ReleaseProfileConfig(id cloudprovider.DeviceIdentifiers, claimUID types.UID, config *apis.NetworkConfig) error {
    if config.SubInterface == nil {
        return nil
    }
    for _, a := range config.SubInterface.Addresses {
        g.subIPAM.Release(a)
    }
    return nil
}

`pkg/ipam.LocalIPAM (this PR's allocator) is still 100% reusable here as a library.

2. User wants a static address, no IPAM involved

parameters:
  interface:
    subInterface:
      type: ipvlan
      addresses: ["192.168.1.50/24"]

No profile field set, no provider round-trip — identical to how static addresses already work for the primary interface today.

3. Bare metal / no cloud provider: for Support RDMA NIC sharing #63 and Support LACP bonded network interfaces #239

This is where "should we ship a local-IPAM profile webhook" matters, and the answer is: we already ship this with cmd/webhook-whereabouts and returns Interface.Addresses. It's already wired end-to-end today via --profile-provider=webhook --webhook-url=....

So I can see two ways to reuse that pattern instead of inventing a new one:

(a) Extend existing webhook-whereabouts to also populate SubInterface.Addresses when req.Config.SubInterface != nil that is very simple.

(b) Create a new cmd/webhook-local-ipam built directly on this PR's pkg/ipam.LocalIPAM library, for operators who don't want the CNI/whereabouts dependency:

/ cmd/webhook-local-ipam/main.go (sketch)
type Server struct {
    ipam   *ipam.LocalIPAM
    ranges []ipam.Range // from a flag/ConfigMap, e.g. --ranges=10.24.3.0/24,2001:db8::/64
}

func (s *Server) GetProfileConfig(w http.ResponseWriter, r *http.Request) {
    var req webhook.ProfileRequest
    json.NewDecoder(r.Body).Decode(&req)
    if req.Config == nil || req.Config.SubInterface == nil {
        json.NewEncoder(w).Encode(apis.NetworkConfig{})
        return
    }
    addrs, err := s.ipam.Allocate(s.ranges)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
    json.NewEncoder(w).Encode(apis.NetworkConfig{SubInterface: &apis.SubInterfaceConfig{Addresses: addrs}})
}

func (s *Server) ReleaseProfileConfig(w http.ResponseWriter, r *http.Request) {
    var req webhook.ProfileRequest
    json.NewDecoder(r.Body).Decode(&req)
    if req.Config != nil && req.Config.SubInterface != nil {
        for _, a := range req.Config.SubInterface.Addresses {
            s.ipam.Release(a)
        }
    }
    w.WriteHeader(http.StatusOK)
}

4. Unnumbered / L2-only subinterface — no profile, no addresses; dranet just creates the ipvlan link with no source-based routing.

Summary

Scenario	profile set?	addresses in claim?	Who allocates	New pkg/apis surface needed
Cloud-provider pool (GCE/AWS/Azure)	yes	no	provider's GetProfileConfig	none
Static user IP	no	yes	user	none
Bare-metal pool, simple UX	yes	no	webhook-whereabouts (extended) or new webhook-local-ipam	none
Unnumbered / L2 only	no	no	n/a	none

In every case Addresses is the only thing that ever crosses the API boundary, exactly like the primary interface.

Can we keep pkg/ipam.LocalIPAM as a library consumed by provider implementations (in-tree cloud providers or a standalone webhook), instead of adding IPRanges to SubInterfaceConfig and running allocation inside driver?

[kubernetes-prow]

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ngcxy]

Thanks @aojea, this is really helpful and reframes the whole API comprehension for me. I will refactor the API to drop the config-oriented IPRanges and let Cloud Provider to produce the static Addresses, avoiding driver-side configuration logic for the device.

I see three ways as the next plan, in increasing scope:

Option 1 (minimum change)
Relocate the invocation of LocalIPAM into cloud provider specific profile config. Keep the current interaction GetAllocatedIPs with PodConfigStore persistent storage for initial allocated IPs seeding during the startup.

Option 2 (separate persistent storage)
Besides the relocation, also remove GetAllocatedIPs that connects between LocalIPAM and driver's PodConfigStore, creating a dedicated persistent storage for ipam state to fully detach from the driver.

Option 3 (webhook for bare metal)
Exactly your point 3(b), a standalone webhook built on LocalIPAM, reusing the webhook-whereabouts pattern. The webhook will require its own checkpoint for persistent storage as well.

I believe for the current PR option 1 is sufficient to unblock, unless such interaction with persistent storage is not preferred. For option 3, I can follow up in future work for bare metal support.

Please let me know if this plan sounds right to you.

[aojea]

Option 1 sounds reasonable to me, once the IPs are allocated we assume they are part of the Interface until they are released

[OguzPastirmaci]

Very excited about this work. This will help us at OCI avoid moving RDMA PFs.

I used this PR as the base for OKE tests with Nvidia IPAM. IPv4 worked with IPvlan children.

IPv6 also worked, but our fabric uses SLAAC. The Nvidia IPAM address was not used for the data path. Support for an IPvlan child without an explicit address would make the IPv6 configuration much simpler.