usdm: expose IOVA mapping APIs for user memory #131

gcabiddu · 2026-01-06T17:46:44Z

Introduce two new public APIs to allow users to map their own pre-allocated memory buffers to IOVA addresses for zero-copy DMA operations:

qaeMemMapContiguousIova(): Allocates a contiguous IOVA region and maps user-provided virtual memory for DMA operations. Returns the IOVA address on success.
qaeMemUnmapContiguousIova(): Unmaps a previously mapped IOVA region and releases the IOVA address space for reuse.

These APIs enable users to avoid unnecessary memory copies when the QAT DMA engine transfers data to/from user-allocated buffers. The user is responsible for ensuring the virtual memory remains valid while the IOVA mapping is active.

Copilot

Pull request overview

This PR introduces two new public APIs (qaeMemMapContiguousIova and qaeMemUnmapContiguousIova) to enable zero-copy DMA operations by allowing users to map their pre-allocated memory buffers directly to IOVA addresses for QAT hardware access.

Key changes:

Added qaeMemMapContiguousIova() to allocate contiguous IOVA regions and map user-provided virtual memory for DMA operations
Added qaeMemUnmapContiguousIova() to unmap and release previously mapped IOVA regions
Both APIs include proper error handling for no-IOMMU mode and unregistered VFIO containers

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
quickassist/utilities/libusdm_drv/user_space/vfio/qae_mem_utils_vfio.c	Implements the two new IOVA mapping/unmapping functions with validation checks and error handling
quickassist/utilities/libusdm_drv/qae_mem.h	Adds comprehensive API documentation for the new public functions including parameters, return values, and usage notes

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-06T17:47:45Z

quickassist/utilities/libusdm_drv/user_space/vfio/qae_mem_utils_vfio.c

+    }
+
+    /* Allocate IOVA address space */
+    iova = allocate_iova(size, alignment);


Type mismatch: allocate_iova expects uint32_t size but is being passed size_t size. This could lead to truncation on 64-bit systems where size_t is larger than uint32_t. Consider either casting the parameter with appropriate validation or modifying the function signature to accept size_t.

Copilot · 2026-01-06T17:47:45Z

quickassist/utilities/libusdm_drv/user_space/vfio/qae_mem_utils_vfio.c

+    }
+
+    /* Release the IOVA address space */
+    iova_release(iova, size);


Type mismatch: iova_release expects uint32_t size but is being passed size_t size. This could lead to truncation on 64-bit systems where size_t is larger than uint32_t. Consider either casting the parameter with appropriate validation or modifying the function signature to accept size_t.

gcabiddu · 2026-01-06T17:48:42Z

Adding @Wangl56.

Wangl56 · 2026-01-07T11:24:03Z

Adding @Wangl56.

Thanks @gcabiddu , this change looks good to me.

fionatrahe · 2026-01-07T13:32:38Z

quickassist/utilities/libusdm_drv/qae_mem.h

+ *
+ * @param[in]  virt       Pointer to the user-allocated virtual memory to map.
+ *                        Must be a valid, accessible virtual address.
+ * @param[in]  size       Size in bytes of the memory region to map. Will be


Should we add that it cannot exceed QAE_MAX_ALLOC_SIZE? And check for that internally. Same limit as for qaeMemAllocNUMA(). Else it could be difficult to find a contiguous IOVA. And we'd need testing to see if internal usdm code handles larger sizes ok.

The same allocation limitation in qaeMemAllocNUMA() does not apply to this API. qaeMemMapContiguousIova() maps pre-existing virtual memory to IOVA space without allocating new memory.

Finding contiguous IOVA space should not be an issue given the following:

IOVA address space: 32-bit (4GB) on 32-bit systems, 39-bit (512GB) on x86_64

IOVA allocation granularity: 2MB slabs (SLAB_BITS is 21)

I tested this API and it works up to 4GB - 2MB.

BTW, I realized there is a small issue. The API signature accepts size_t (8 bytes on 64-bit systems), but internally calls allocate_iova(uint32_t size, ...) which limits the maximum to UINT32_MAX (~4GB). Currently, there's no overflow protection when casting size_t to uint32_t.

We can limit the input size. I was thinking 2GB. This will prevent overflows and limit the usage of the address space. What do you think?

fionatrahe · 2026-01-07T13:34:27Z

quickassist/utilities/libusdm_drv/qae_mem.h

+ *                        Must be a valid, accessible virtual address.
+ * @param[in]  size       Size in bytes of the memory region to map. Will be
+ *                        rounded up to the nearest page boundary internally.
+ * @param[in]  alignment  Alignment requirement for the IOVA allocation in


Do we really need this param? Could we replace it with a comment saying the iova will be aligned to IOVA_SLAB_SIZE (2MB) boundary? Wondering what's the use-case for someone wanting to set a higher value alignment?

If we keep it I'd suggest calling it iova_alignment (just to avoid any confusion in comparisons with the phys_alignment_byte param in qaeMemAllocNUMA() )

Makes sense. We can drop this parameter and hardcode it to IOVA_SLAB_SIZE. It was done this way since allocate_iova() takes as input an alignment.

fionatrahe · 2026-01-07T13:44:16Z

quickassist/utilities/libusdm_drv/qae_mem.h

+ *                        bytes. Must be a power of 2. Will be rounded up to
+ *                        at least IOVA_SLAB_SIZE (2MB) internally.
+ *
+ * @retval >0  The allocated IOVA address on success. This value should be


General comment: Is the expectation that this API is used for a large block of memory, subsequently managed by the user for multiple buffers or is it something done for every buffer used by the appl?
i.e. can they later call qatVirtToPhysNUMA(pVirt) for some offset into the mapped memory to get the corresponding iova ? If not and the application calls the new API for every buffer, is there any danger of running out of IOVAs as each is aligned to 2MB? 262,144 iova slabs possible

Is the expectation that this API is used for a large block of memory, subsequently managed by the user for multiple buffers or is it something done for every buffer used by the appl?

I don't know. @Wangl56 can you comment on this?

Regarding qatVirtToPhysNUMA(), good catch. At the moment we are not storing the mappings into the usdm page table. This is needed in order to use the memory mapped through this new api using QATlib.

fionatrahe · 2026-01-07T13:50:06Z

quickassist/utilities/libusdm_drv/qae_mem.h

+ *      allocated through qaeMemAllocNUMA() - use qaeMemFreeNUMA() instead.
+ *
+ ****************************************************************************/
+int qaeMemUnmapContiguousIova(const uint64_t iova, const size_t size);


I don't see any protection if the application unmap something not allocated by the corresponding API. WE just rely on the comment and common sense. What are the consequences of incorrect use, e.g.
if a user calls qaeMemUnmapContiguousIova() on something allocated by qaeMemAllocNuma() do we get a memory leak as the memory would not be freed? Or the opposite - if they call qaeMemFreeNUMA() on memory mapped by the new API will it work - will it free the IOVA and free the memory the appl allocated? Doesn't sound too risky - but has it been tested?

If qaeMemUnmapContiguousIova() is used on an iova from qaeMemAllocNuma(), that address will be unmapped, but not freed. The device won't be able to access it causing the IOMMU to report a DMAR error.

gcabiddu · 2026-01-08T17:08:01Z

quickassist/utilities/libusdm_drv/user_space/vfio/qae_mem_utils_vfio.c

+                  __func__, __LINE__);
+        return 1;
+    }
+


Add a check

size > QAE_IOVA_MAP_MAX_SIZE

Introduce two new public APIs to allow users to map their own pre-allocated memory buffers to IOVA addresses for zero-copy DMA operations: - qaeMemMapContiguousIova(virt, size): Allocates a contiguous IOVA region and maps user-provided virtual memory for DMA operations. The IOVA is aligned to IOVA_SLAB_SIZE (2MB) internally. Maximum supported size is 2GB. The mapping is registered in the USDM page table, enabling qaeVirtToPhysNUMA() lookups on the mapped region. Returns the IOVA address on success, 0 on failure. - qaeMemUnmapContiguousIova(virt, size): Unmaps a previously mapped IOVA region and releases the IOVA address space for reuse. The IOVA is automatically looked up from the page table using the provided virtual address. These APIs enable users to avoid unnecessary memory copies when the QAT DMA engine transfers data to/from user-allocated buffers. The user is responsible for ensuring the virtual memory remains valid while the IOVA mapping is active. Note: These APIs are only available in VFIO mode and will fail if the system is configured in no-IOMMU mode. Memory allocated through qaeMemAllocNUMA() should not be mapped or unmapped using these APIs. Signed-off-by: Giovanni Cabiddu <[email protected]>

gcabiddu requested review from Ahsan-Atta, Copilot and fionatrahe January 6, 2026 17:46

gcabiddu changed the title ~~usdm: Expose IOVA mapping APIs for user memory~~ usdm: expose IOVA mapping APIs for user memory Jan 6, 2026

Copilot AI reviewed Jan 6, 2026

View reviewed changes

gcabiddu mentioned this pull request Jan 6, 2026

Expose iova management and dma_map api in vfio mem utils to users #124

Closed

fionatrahe force-pushed the iova_apis branch from b6383fa to e4f0b42 Compare January 7, 2026 10:59

fionatrahe reviewed Jan 7, 2026

View reviewed changes

gcabiddu force-pushed the iova_apis branch 2 times, most recently from 8482930 to cb71853 Compare January 8, 2026 13:11

gcabiddu commented Jan 8, 2026

View reviewed changes

usdm: expose IOVA mapping APIs for user memory #131

Are you sure you want to change the base?

usdm: expose IOVA mapping APIs for user memory #131

Uh oh!

Conversation

gcabiddu commented Jan 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

gcabiddu commented Jan 6, 2026

Uh oh!

Wangl56 commented Jan 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants