[SPDK] Using SPDK IOAT driver to drive data to PCIe space

[SPDK] Using SPDK IOAT driver to drive data to PCIe space

[nufosmatic]

[-- Attachment #1: Type: text/plain, Size: 1038 bytes --]

Leaving aside for the moment that the IOAT example (build/examples/ioat_perf) is using SPDK library calls that do not differentiate between the memory that is used for managing a mempool and the data that serves as buffers in a mempool, why would the SPDK IOAT perform very, very badly  addressing memory in PCIe space.

I have a device that I know will run 2-6 GB/s on a x8 interface from other DMA engines in this system. And the IOAT example will run as fast as 5 GB/s  into the default DPDK memory reserves. And I know that the IOAT engine (at least the Crystal Beach IOAT engine on an Intel Cascade Lake SuperMicro X11DPG-QT with Sky Lake-E CBDMA) using another commercial middleware will run as much as 9 GB/s in a x16 interface.

But the IOAT example, modified to separate the memory management from the data (see rte_mempool_empty() and rte_mempool_populate_virt()), runs  <<0.1 GB/s into x8 and x4 PCIe spaces. Is there some other feature that needs to be enabled in the IOAT driver to allow it to drive to PCIe space fastly?