Re: [SPDK] spdk peer2peer dma fio latency

[PR PR <prprpr7819 at gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 1205 bytes --]

Hi, I am trying some experiments to evaluate performance of peer2peer dma.
I am using spdk to control the nvme drives and fio-plugin compiled with
spdk. I am seeing a weird behavior where when I run 4K IOs with IO-Depth of
1 peer2peer DMA from nvme drive to some pci device (which exposes memory
via Bar1) in a different numa node has a 50th percentile latency of 17
usecs. The same  experiment but where nvme device and pcie device in same
numa node (node 0) has a latency of 38 usecs. In both cases fio was running
in node 0 cpu core and pci device (which exposes memory via Bar1) is
connected to node 1. DMA from nvme device to host memory also takes 38
usecs.

To summarize the cases below

1. nvme (numa node 0) - pci device (numa node 1)   --- 18 usecs
2. nvme (numa node 1) - pci device (numa node 1)   --- 38 usecs
3. nvme (numa node 0) - host memory   --- 38 usecs

fio running in numa node 0 cpu core in all cases.

For higher IO Depth values cross numa case (case 1 above), latency
increases steeply and performs poorly than case 2 and case 3.

Any pointers on why this could be happening?

The nvme devices used are both identical intel datacenter ssd 400G.

Thanks

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 1496 bytes --]