[SPDK] Re: About SPDK I/OAT Driver.

[SPDK] Re: About SPDK I/OAT Driver.

[qiang su]

[-- Attachment #1: Type: text/plain, Size: 56 bytes --]

Hi Jim,

Got it. Thanks a lot!

Best Regards,
Qiang

[SPDK] Re: About SPDK I/OAT Driver.

[qiang su]

[-- Attachment #1: Type: text/plain, Size: 888 bytes --]

Hi Jim,

Thanks for your reply.
Now, I am trying to use SPDK I/OAT driver to copy data.  
Copy data between SSD's CMB and host memory. 
The CMB type can be configured as prefetchable or non-prefetchable by upgrade SSD firmware.

The test application use "spdk_ioat_submit_copy" to copy data , 
Host memory is allocated by spdk_dma_zmalloc(...), CMB is allocated by spdk_nvme_ctrlr_map_cmb(...).

Test result found that the prefetchable CMB performance is poorer than non-prefetchable CMB.
Could you give me advice why prefetchable CMB is poorer than non-prefetchable？
Thanks a lot!

Test environment：

IOAT device info :
  Found matching device at 0000:80:04.7 vendor:0x8086 device:0x2021
attach ioat chan 0x171d010

CPU：
 Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz

Motherboard : Supermicro X11DPU

SPDK git version : 307b8c112ffd90a26d53dd15fad67bd9038ef526

[SPDK] Re: About SPDK I/OAT Driver.

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 1768 bytes --]

Great, thanks for the clarification!

From: Harris, James R <james.r.harris(a)intel.com>
Date: Tuesday, October 25, 2022 at 7:01 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Re: About SPDK I/OAT Driver.
Those DCA related defines were pulled from much older versions of the IOAT driver, when DCA had to be enabled on a per-driver/device basis.

-Jim


From: Luse, Paul E <paul.e.luse(a)intel.com>
Date: Tuesday, October 25, 2022 at 6:57 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Re: About SPDK I/OAT Driver.
Good question, Jim I would have expected this line in ioat_channel_start()

ioat->regs->chanctrl = SPDK_IOAT_CHANCTRL_ANY_ERR_ABORT_EN;

to read:

ioat->regs->chanctrl = SPDK_IOAT_CHANCTRL_ANY_ERR_ABORT_EN | SPDK_IOAT_CHANCTRL_COMPL_DCA_EN;

But I don’t have the spec in front of me to confirm whether this is really required or not, just looking at the code and the defines 😊

thx
Paul


From: qiang su <qsu(a)petaio.com>
Date: Tuesday, October 25, 2022 at 5:56 AM
To: spdk(a)lists.01.org <spdk(a)lists.01.org>
Subject: [SPDK] Re: About SPDK I/OAT Driver.
Hi Jim,

Thanks for your reply.
Is the DCA enabled by default in SPDK I/OAT driver?
I haven't found any parameter or API to config DCA.
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org

[SPDK] Re: About SPDK I/OAT Driver.

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 1373 bytes --]

Those DCA related defines were pulled from much older versions of the IOAT driver, when DCA had to be enabled on a per-driver/device basis.

-Jim


From: Luse, Paul E <paul.e.luse(a)intel.com>
Date: Tuesday, October 25, 2022 at 6:57 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Re: About SPDK I/OAT Driver.
Good question, Jim I would have expected this line in ioat_channel_start()

ioat->regs->chanctrl = SPDK_IOAT_CHANCTRL_ANY_ERR_ABORT_EN;

to read:

ioat->regs->chanctrl = SPDK_IOAT_CHANCTRL_ANY_ERR_ABORT_EN | SPDK_IOAT_CHANCTRL_COMPL_DCA_EN;

But I don’t have the spec in front of me to confirm whether this is really required or not, just looking at the code and the defines 😊

thx
Paul


From: qiang su <qsu(a)petaio.com>
Date: Tuesday, October 25, 2022 at 5:56 AM
To: spdk(a)lists.01.org <spdk(a)lists.01.org>
Subject: [SPDK] Re: About SPDK I/OAT Driver.
Hi Jim,

Thanks for your reply.
Is the DCA enabled by default in SPDK I/OAT driver?
I haven't found any parameter or API to config DCA.
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org

[SPDK] Re: About SPDK I/OAT Driver.

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 559 bytes --]

Hi,

I/OAT driver does not need to enable DCA/DDIO.  DDIO is a platform feature.

-Jim


From: qiang su <qsu(a)petaio.com>
Date: Tuesday, October 25, 2022 at 5:56 AM
To: spdk(a)lists.01.org <spdk(a)lists.01.org>
Subject: [SPDK] Re: About SPDK I/OAT Driver.
Hi Jim,

Thanks for your reply.
Is the DCA enabled by default in SPDK I/OAT driver?
I haven't found any parameter or API to config DCA.
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org

[SPDK] Re: About SPDK I/OAT Driver.

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 871 bytes --]

Good question, Jim I would have expected this line in ioat_channel_start()

ioat->regs->chanctrl = SPDK_IOAT_CHANCTRL_ANY_ERR_ABORT_EN;

to read:

ioat->regs->chanctrl = SPDK_IOAT_CHANCTRL_ANY_ERR_ABORT_EN | SPDK_IOAT_CHANCTRL_COMPL_DCA_EN;

But I don’t have the spec in front of me to confirm whether this is really required or not, just looking at the code and the defines 😊

thx
Paul


From: qiang su <qsu(a)petaio.com>
Date: Tuesday, October 25, 2022 at 5:56 AM
To: spdk(a)lists.01.org <spdk(a)lists.01.org>
Subject: [SPDK] Re: About SPDK I/OAT Driver.
Hi Jim,

Thanks for your reply.
Is the DCA enabled by default in SPDK I/OAT driver?
I haven't found any parameter or API to config DCA.
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org

[SPDK] Re: About SPDK I/OAT Driver.

[qiang su]

[-- Attachment #1: Type: text/plain, Size: 141 bytes --]

Hi Jim,

Thanks for your reply.
Is the DCA enabled by default in SPDK I/OAT driver?  
I haven't found any parameter or API to config DCA.

[SPDK] Re: About SPDK I/OAT Driver.

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 1118 bytes --]

Hi,

I/OAT uses DDIO to place data into LLC.  DDIO is the latest implementation of DCA on Intel platforms.

-Jim



From: qsu(a)petaio.com <qsu(a)petaio.com>
Date: Monday, October 24, 2022 at 4:12 AM
To: spdk(a)lists.01.org <spdk(a)lists.01.org>
Subject: [SPDK] About SPDK I/OAT Driver.
Hello,

I am studying SPDK I/OAT Driver, I don't understand if SPDK's IOAT DMA will access(via) CPU cache?
Can someone give me an advice?
Thanks a lot.

 SPDK says I/OAT driver is a DMA engine,
 I read some introductions about I/OAT on the intel website: (https://www.intel.com/content/www/us/en/wireless-network/accel-technology.html).
 It says I/OAT includes DCA, It looks like that DCA will access CPU cache.

"Direct Cache Access (DCA) allows a capable I/O device, such as a network controller, to place data directly into CPU cache, reducing cache misses and improving application response times."

Any help is appreciated.

Best Regards,
Qiang Su
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org