[SPDK] Re: Using SPDK IOAT driver to drive data to PCIe space

[SPDK] Re: Using SPDK IOAT driver to drive data to PCIe space

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 1598 bytes --]

Nothing that I can think of… maybe you can expand a little on what modifications you made?  Slack might be a faster way to get some conversations going too but this works as well ☺

From: nufosmatic(a)nufosmatic.com <nufosmatic(a)nufosmatic.com>
Date: Monday, June 14, 2021 at 3:12 AM
To: spdk(a)lists.01.org <spdk(a)lists.01.org>
Subject: [SPDK] Using SPDK IOAT driver to drive data to PCIe space
Leaving aside for the moment that the IOAT example (build/examples/ioat_perf) is using SPDK library calls that do not differentiate between the memory that is used for managing a mempool and the data that serves as buffers in a mempool, why would the SPDK IOAT perform very, very badly  addressing memory in PCIe space.

I have a device that I know will run 2-6 GB/s on a x8 interface from other DMA engines in this system. And the IOAT example will run as fast as 5 GB/s  into the default DPDK memory reserves. And I know that the IOAT engine (at least the Crystal Beach IOAT engine on an Intel Cascade Lake SuperMicro X11DPG-QT with Sky Lake-E CBDMA) using another commercial middleware will run as much as 9 GB/s in a x16 interface.

But the IOAT example, modified to separate the memory management from the data (see rte_mempool_empty() and rte_mempool_populate_virt()), runs  <<0.1 GB/s into x8 and x4 PCIe spaces. Is there some other feature that needs to be enabled in the IOAT driver to allow it to drive to PCIe space fastly?
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org