UPM-DMA: An Efficient Userspace DMA-Pinned Memory Management Strategy for NVMe SSDs

Fast storage devices (e.g., NVMe SSDs), which integrate new storage medium and parallel architecture, provide promising storage solutions for cloud data center loads. However, they make the lengthy IO stack that is negligible in the past become a new performance bottleneck. Current user-mode solutions can eliminate context switches between kernel and user space by moving partial kernel IO stack into userspace. Unfortunately, they introduce additional overhead for pinning memory used by DMA in userspace which is quite time-consuming for IO-intensive workloads. To address this issue, we present UPM-DMA, an alternative and efficient memory management strategy for NVMe SSDs to lighten the IO stack by amortizing per-request latency. UPM-DMA can dynamically select the appropriate pinned memory mode for different IO requests, especially the pinned memory pool for medium data IO requests. We further explore different parameter settings to optimize system performance and memory space efficiency. Finally, we implement the overall strategy as a memory library named UPM libs and integrate it into the SPDK framework. The official benchmarks, SPDK perf, are adopted to evaluate our solution. The experimental results show that compared with the default processing method in SPDK, there is an improvement of at least 17% under various test data sizes.