Re: [SPDK] How to handle not contiguous data in end-to-end data protection and use ISA-L in SPDK

Re: [SPDK] How to handle not contiguous data in end-to-end data protection and use ISA-L in SPDK

[@ 2018-09-11 4:44 UTC (permalink / raw)]

[-- Attachment #1: Type: text/plain, Size: 1759 bytes --]

Hi All,


To simplify the design, I will start my work by setting a requirment that length of each iovec must be multiple of extended LBAs, i.e., multiple of 520 if block size is 512 and metadata size is 8.

I hope I will submit some patches to Gerrit soon.


If you have any other feedback, please let me know. Otherwise, I'm happy if you just stay tune.


Thanks,

Shuhei


________________________________
差出人: 松本周平 / MATSUMOTO，SHUUHEI
送信日時: 2018年9月11日 8:08
宛先: spdk(a)lists.01.org
件名: How to handle not contiguous data in end-to-end data protection and use ISA-L in SPDK


Hi All,


I have started to investigate end-to-end data protection, i.e. DIF/DIX in SPDK.


I have a couple of things to get any feedback.


I'm focusing on not separated data and metadata but interleaved metadata, i.e. extended LBA first.

and I'm investigating to implement verification of DIF in the bdev layer.


SPDK support not contiguous data but it will be easier if SPDK can avoid not contiguous read of metadata, i.e. read two 4 bytes from different buffer and concatenate them.


Can we set any requirement to the alignment of the data? As long as I understand, NVMe don't set strict requirement to alignment/granularity.

Or do we have to support all pattern of splitted reads?

Or should we copy not contiguous data to contiguous dynamically allocated buffer and walk the latter?


Do you have any other idea?



Secondly, about CRC calculation, I observed that ISA-L (Intel Storage Application Library) has a hundred times better performance than current SPDK's function.

Should I use ISA-L by my own risk or can we integrate ISA-L into SPDK master?


Thanks,

Shuhei

[SPDK] How to handle not contiguous data in end-to-end data protection and use ISA-L in SPDK

[@ 2018-09-10 23:08 UTC (permalink / raw)]

[-- Attachment #1: Type: text/plain, Size: 1121 bytes --]

Hi All,


I have started to investigate end-to-end data protection, i.e. DIF/DIX in SPDK.


I have a couple of things to get any feedback.


I'm focusing on not separated data and metadata but interleaved metadata, i.e. extended LBA first.

and I'm investigating to implement verification of DIF in the bdev layer.


SPDK support not contiguous data but it will be easier if SPDK can avoid not contiguous read of metadata, i.e. read two 4 bytes from different buffer and concatenate them.


Can we set any requirement to the alignment of the data? As long as I understand, NVMe don't set strict requirement to alignment/granularity.

Or do we have to support all pattern of splitted reads?

Or should we copy not contiguous data to contiguous dynamically allocated buffer and walk the latter?


Do you have any other idea?



Secondly, about CRC calculation, I observed that ISA-L (Intel Storage Application Library) has a hundred times better performance than current SPDK's function.

Should I use ISA-L by my own risk or can we integrate ISA-L into SPDK master?


Thanks,

Shuhei