[SPDK] SPDK pooled volume module in bdev

[SPDK] SPDK pooled volume module in bdev

[Sablok, Kunal]

[-- Attachment #1: Type: text/plain, Size: 839 bytes --]

Hi,
SPDK pooled volume (PVOL) module is a new bdev module which is responsible for striping various NVMe devices and expose the pooled volume to bdev layer which would enhance the performance and capacity. It can support theoretically 255 base devices which can be easily enhanced if required (currently it is being tested max upto 8 base devices). Multiple strip sizes like 32KB, 64KB, 128KB, 256KB, 512KB etc is supported. New RPC commands like "create pvol", "destroy pvol" and "get pvols" are introduced to configure pooled volume dynamically in a running SPDK system.
Please find attached more information in a ppt with system end to end testing done on this module in SPDK stack in SPDK multi-core environment.
Please find below gerrithub review details:
https://review.gerrithub.io/#/c/spdk/spdk/+/410484/


Regards,
Kunal

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 7659 bytes --]

[-- Attachment #3: SPDK_Pooled_Volume_Module.pptx --]
[-- Type: application/vnd.openxmlformats-officedocument.presentationml.presentation, Size: 635477 bytes --]

Re: [SPDK] SPDK pooled volume module in bdev

[Marushak, Nathan]

[-- Attachment #1: Type: text/plain, Size: 10943 bytes --]

Hi Senthil,

A virtual BDEV can provide varying types of functionality, it doesn't necessarily have to be for a logical or virtual volume. You could create a vbdev module that provides encryption support (as Paul Luse has been doing), you could also create a separate vbdev to provide compression if you'd like. You can then decide to stack vbdevs on top of one another to provide whatever features you desire for your application. You could layer these on top of the LVOL vbdev or even create your own individual vbdev that performs multiple storage features (e.g. encryption, compression, EC, dedup, etc.).

Hope this helps clarify.

Thanks,
Nate

> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Senthil Kumar
> Veluswamy
> Sent: Tuesday, June 19, 2018 7:25 PM
> To: Walker, Benjamin <benjamin.walker(a)intel.com>
> Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] SPDK pooled volume module in bdev
> 
> Thank you Ben and Maciek for the explanation.
> 
> Hi Ben,
>      If we can draw a parallel of VBDEV in SAN, can we say VBDEV are
> similar to "Conglomerate Logical unit/Virtual Volumes (vVOLs)"  in terms of
> its functionality/usage?
> 
> Regards,
> Senthil Kumar V.
> 
> 
> -----Original Message-----
> From: SPDK <spdk-bounces(a)lists.01.org> On Behalf Of Walker, Benjamin
> Sent: Tuesday, June 19, 2018 10:22 PM
> To: spdk(a)lists.01.org; Sablok, Kunal <kunal.sablok(a)intel.com>; Luse, Paul E
> <paul.e.luse(a)intel.com>
> Subject: Re: [SPDK] SPDK pooled volume module in bdev
> 
> Hi all,
> 
> SPDK's virtual block devices (vbdevs) are a framework for creating an I/O
> processing pipeline by stacking block devices (bdevs) on top of one
> another.
> Each block device has a type and has internal logic implemented by a bdev
> module. For example, there is an NVMe bdev, which routes I/O to NVMe
> devices, and there is also a logical volume (lvol) bdev, which carves up an
> underlying bdev into many smaller bdevs dynamically.
> 
> The (v)bdev layer does not contain any actual I/O processing logic in and
> of itself - it's just the framework. It delegates to bdev modules to
> implement the actual logic. So the bdev layer today does not implement
> concatenation or anything of that nature.
> 
> What Kunal is implementing here is a bdev module that implements RAID0. It
> stripes the RAID volume across N underlying bdevs and exposes a single bdev
> on top.
> 
> Thanks,
> Ben
> 
> On Tue, 2018-06-19 at 03:23 +0000, Senthil Kumar Veluswamy wrote:
> > Hi Kunal,
> >      >> I am assuming here you are asking about vbdev concatenation vs
> > PVOL concatenation only. I don’t think vbdev is supporting concatenation.
> >
> > I’m going thru the Paul’s presentation from last SPDK US submit on
> > “Virtual
> > Bdevs: The secret to customizing SPDK” and found below slides.
> >
> >
> >
> > As you can see, from VBDEV explanation it looks very similar to PVOL .
> > This is where the confusion is. If I understand it correctly, even
> > VBDEV can concatenate more than one BDEVs.
> >
> > May be Paul can help here from VBDEV perspective..
> >
> > Hi Paul,
> >      Can you please help here to understand the use-cases for VBDEV and
> PVOLs?
> > & How they are architecturally different and how they can co-exists?
> >
> > Thank you.
> >
> > Regards,
> > Senthil Kumar V.
> >
> > From: SPDK <spdk-bounces(a)lists.01.org> On Behalf Of Sablok, Kunal
> > Sent: Monday, June 18, 2018 10:12 PM
> > To: Senthil Kumar Veluswamy <Senthil.Kumar.Veluswamy(a)wdc.com>
> > Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: Re: [SPDK] SPDK pooled volume module in bdev
> >
> > Hi Senthil,
> > Please find inline.
> >
> > Regards,
> > Kunal
> >
> > From: Senthil Kumar Veluswamy [mailto:Senthil.Kumar.Veluswamy(a)wdc.com]
> > Sent: Sunday, June 17, 2018 10:07 AM
> > To: Sablok, Kunal <kunal.sablok(a)intel.com>
> > Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: RE: SPDK pooled volume module in bdev
> >
> > Hi Kunal,
> >      Thank you for the explanation. I’ve few more questions. My
> > response inline tagged with “[Senthil-1]”.
> >
> > Regards,
> > Senthil Kumar V.
> >
> > From: Sablok, Kunal <kunal.sablok(a)intel.com>
> > Sent: Friday, June 15, 2018 10:40 PM
> > To: Senthil Kumar Veluswamy <Senthil.Kumar.Veluswamy(a)wdc.com>
> > Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: RE: SPDK pooled volume module in bdev
> >
> > Hi Senthil,
> > Please find inline.
> >
> > Regards,
> > Kunal
> >
> > From: Senthil Kumar Veluswamy [mailto:Senthil.Kumar.Veluswamy(a)wdc.com]
> > Sent: Thursday, June 14, 2018 12:20 PM
> > To: Sablok, Kunal <kunal.sablok(a)intel.com>
> > Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: RE: SPDK pooled volume module in bdev
> >
> > Hi Kunal,
> >      I’m trying understand the pVol architecture and have few questions.
> >
> > 1.      How this pvol model co-exists with vbdev that is present today?
> > [Kunal] In current SPDK, when any bdev is getting unregistered,
> > spdk_bdev_unregister() removes all the underlying base bdevs from
> > vbdev but for PVOL scenario (since it is similar to Raid0 device),
> > only PVOL bdev gets unregistered and it still needs to keep the
> > information of current base devices which are still claimed. PVOL
> > needs to keep many more information apart from base_bdev reference per
> > base_bdev level like base_bdev's descriptor, some flags for hotplug
> > scenarios etc[will upload in hotplug review] which it is managing itself
> in pvol module.
> >
> > During hotplug scenarios, when one of the base bdev of PVOL is removed
> > and PVOL gets its notification, it un-registers the PVOL from bdev
> > layer since all underlying base_bdevs are not present (but bdev layer
> > implementation also removes all the base bdevs in unregister call) but
> > in this case also it still needs to keep the base bdevs and other
> > information of the base bdevs which are still present and claimed by
> > PVOL. When the removed base_bdev is added again, PVOL gets the
> > notification via examine() and it will claim only that base_bdev now,
> > all other base_bdevs were already present and so PVOL gets registered
> again with bdev layer.
> >
> > Base bdevs and its information can only be removed from PVOL bdev if
> > below layer like NVMe notifies the PVOL explicitly of base device
> > removal. So, idea is to hold the base_bdevs in PVOL which are still
> > present, claimed and not removed for various reasons, so as of now
> > PVOL is using spdk_bdev_register() to register itself with bdev layer,
> > and other modules like LVS/LVOL can create logical volumes over PVOL
> [physical volume].
> >
> > 2.      If I understand correctly, vbdev also allows to concatenate more
> than
> > one base physical bdevs and used for creating logical volume. So shall
> > we say pvol is an extension of vbdev?
> > [Kunal] We can say it is an extension of vbdev but it maintains more
> > information than just base_bdevs in its own module itself to have
> > better control as explained above.
> >
> > [Senthil-1] Since Pvol maintains more info of base_bdevs than vbdev,
> > can we say that PVol is a superset of vbdev? If so, can you give an
> > example of their data structure mappings?
> > [Kunal] PVOL maintains the base bdev information in its module itself
> > and registers with bdev layer via spdk_bdev_register(). So there is no
> > mapping exists between PVOL and vbdev. However once PVOL is created
> > with 1 or more NVMe bdevs, LVS/LVOl can be created over PVOL itself.
> >
> > [Senthil-1] Would it possible for you to explain the diff b/w PVol &
> > vbdev by their real-time use-cases? Like when PVol will be used vs
> > vbdev. I assuming both vbdev and PVol are going to coexist.
> > [Kunal] I am assuming here you are asking about vbdev concatenation vs
> > PVOL concatenation only. I don’t think vbdev is supporting
> > concatenation. For that once code is pushed, PVOL will be used for
> > striping across drives which will have IO split logic to split IOs and
> > submit to underlying nvme bdevs. To create PVOL across 1 or more base
> > nvme bdevs, either RPC command can be used or pvol config file can be
> > updated with the details to create PVOL during bootup. Once PVOL bdev
> > is created, other modules get examine calls for pvol bdev also. LVS
> > and LVOL can be create over PVOL and exposed to initiator via subsystems.
> >
> > 3.      It would be good if you could modify your “Pooled volume” PPT
> slides
> > to show how vbdev fits in there.
> >
> > Regards,
> > Senthil Kumar V.
> >
> > From: SPDK <spdk-bounces(a)lists.01.org> On Behalf Of Luse, Paul E
> > Sent: Friday, May 18, 2018 12:15 AM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: Re: [SPDK] SPDK pooled volume module in bdev
> >
> > Excellent, thanks Kunal!  There’s a lot to look at here and with
> > Fenggang’s aggregation patch as well.  I encourage everyone to take a
> > look at both (and I promise I will soon too after post-summit catch-up
> > stuff)
> >
> > J
> >
> >
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sablok,
> > Kunal
> > Sent: Sunday, May 13, 2018 9:51 AM
> > To: spdk(a)lists.01.org
> > Subject: [SPDK] SPDK pooled volume module in bdev
> >
> > Hi,
> > SPDK pooled volume (PVOL) module is a new bdev module which is
> > responsible for striping various NVMe devices and expose the pooled
> > volume to bdev layer which would enhance the performance and capacity.
> > It can support theoretically 255 base devices which can be easily
> > enhanced if required (currently it is being tested max upto 8 base
> > devices). Multiple strip sizes like 32KB, 64KB, 128KB, 256KB, 512KB
> > etc is supported. New RPC commands like "create pvol", "destroy pvol"
> > and "get pvols" are introduced to configure pooled volume dynamically in
> a running SPDK system.
> > Please find attached more information in a ppt with system end to end
> > testing done on this module in SPDK stack in SPDK multi-core environment.
> > Please find below gerrithub review details:
> > https://review.gerrithub.io/#/c/spdk/spdk/+/410484/
> >
> >
> > Regards,
> > Kunal
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] SPDK pooled volume module in bdev

[Senthil Kumar Veluswamy]

[-- Attachment #1: Type: text/plain, Size: 9463 bytes --]

Thank you Ben and Maciek for the explanation.

Hi Ben,
     If we can draw a parallel of VBDEV in SAN, can we say VBDEV are similar to "Conglomerate Logical unit/Virtual Volumes (vVOLs)"  in terms of its functionality/usage? 

Regards,
Senthil Kumar V.


-----Original Message-----
From: SPDK <spdk-bounces(a)lists.01.org> On Behalf Of Walker, Benjamin
Sent: Tuesday, June 19, 2018 10:22 PM
To: spdk(a)lists.01.org; Sablok, Kunal <kunal.sablok(a)intel.com>; Luse, Paul E <paul.e.luse(a)intel.com>
Subject: Re: [SPDK] SPDK pooled volume module in bdev

Hi all,

SPDK's virtual block devices (vbdevs) are a framework for creating an I/O processing pipeline by stacking block devices (bdevs) on top of one another.
Each block device has a type and has internal logic implemented by a bdev module. For example, there is an NVMe bdev, which routes I/O to NVMe devices, and there is also a logical volume (lvol) bdev, which carves up an underlying bdev into many smaller bdevs dynamically.

The (v)bdev layer does not contain any actual I/O processing logic in and of itself - it's just the framework. It delegates to bdev modules to implement the actual logic. So the bdev layer today does not implement concatenation or anything of that nature.

What Kunal is implementing here is a bdev module that implements RAID0. It stripes the RAID volume across N underlying bdevs and exposes a single bdev on top.

Thanks,
Ben

On Tue, 2018-06-19 at 03:23 +0000, Senthil Kumar Veluswamy wrote:
> Hi Kunal,
>      >> I am assuming here you are asking about vbdev concatenation vs 
> PVOL concatenation only. I don’t think vbdev is supporting concatenation.
>  
> I’m going thru the Paul’s presentation from last SPDK US submit on 
> “Virtual
> Bdevs: The secret to customizing SPDK” and found below slides.
>  
>      
>  
> As you can see, from VBDEV explanation it looks very similar to PVOL . 
> This is where the confusion is. If I understand it correctly, even 
> VBDEV can concatenate more than one BDEVs.
>  
> May be Paul can help here from VBDEV perspective..
>  
> Hi Paul,
>      Can you please help here to understand the use-cases for VBDEV and PVOLs?
> & How they are architecturally different and how they can co-exists?
>  
> Thank you.
>  
> Regards,
> Senthil Kumar V.
>  
> From: SPDK <spdk-bounces(a)lists.01.org> On Behalf Of Sablok, Kunal
> Sent: Monday, June 18, 2018 10:12 PM
> To: Senthil Kumar Veluswamy <Senthil.Kumar.Veluswamy(a)wdc.com>
> Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] SPDK pooled volume module in bdev
>  
> Hi Senthil,
> Please find inline.
>  
> Regards,
> Kunal
>  
> From: Senthil Kumar Veluswamy [mailto:Senthil.Kumar.Veluswamy(a)wdc.com]
> Sent: Sunday, June 17, 2018 10:07 AM
> To: Sablok, Kunal <kunal.sablok(a)intel.com>
> Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: RE: SPDK pooled volume module in bdev
>  
> Hi Kunal,
>      Thank you for the explanation. I’ve few more questions. My 
> response inline tagged with “[Senthil-1]”.
>  
> Regards,
> Senthil Kumar V.
>  
> From: Sablok, Kunal <kunal.sablok(a)intel.com>
> Sent: Friday, June 15, 2018 10:40 PM
> To: Senthil Kumar Veluswamy <Senthil.Kumar.Veluswamy(a)wdc.com>
> Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: RE: SPDK pooled volume module in bdev
>  
> Hi Senthil,
> Please find inline.
>  
> Regards,
> Kunal
>  
> From: Senthil Kumar Veluswamy [mailto:Senthil.Kumar.Veluswamy(a)wdc.com]
> Sent: Thursday, June 14, 2018 12:20 PM
> To: Sablok, Kunal <kunal.sablok(a)intel.com>
> Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: RE: SPDK pooled volume module in bdev
>  
> Hi Kunal,
>      I’m trying understand the pVol architecture and have few questions.
>  
> 1.      How this pvol model co-exists with vbdev that is present today?
> [Kunal] In current SPDK, when any bdev is getting unregistered,
> spdk_bdev_unregister() removes all the underlying base bdevs from 
> vbdev but for PVOL scenario (since it is similar to Raid0 device), 
> only PVOL bdev gets unregistered and it still needs to keep the 
> information of current base devices which are still claimed. PVOL 
> needs to keep many more information apart from base_bdev reference per 
> base_bdev level like base_bdev's descriptor, some flags for hotplug 
> scenarios etc[will upload in hotplug review] which it is managing itself in pvol module.
>  
> During hotplug scenarios, when one of the base bdev of PVOL is removed 
> and PVOL gets its notification, it un-registers the PVOL from bdev 
> layer since all underlying base_bdevs are not present (but bdev layer 
> implementation also removes all the base bdevs in unregister call) but 
> in this case also it still needs to keep the base bdevs and other 
> information of the base bdevs which are still present and claimed by 
> PVOL. When the removed base_bdev is added again, PVOL gets the 
> notification via examine() and it will claim only that base_bdev now, 
> all other base_bdevs were already present and so PVOL gets registered again with bdev layer.
>  
> Base bdevs and its information can only be removed from PVOL bdev if 
> below layer like NVMe notifies the PVOL explicitly of base device 
> removal. So, idea is to hold the base_bdevs in PVOL which are still 
> present, claimed and not removed for various reasons, so as of now 
> PVOL is using spdk_bdev_register() to register itself with bdev layer, 
> and other modules like LVS/LVOL can create logical volumes over PVOL [physical volume].
>  
> 2.      If I understand correctly, vbdev also allows to concatenate more than
> one base physical bdevs and used for creating logical volume. So shall 
> we say pvol is an extension of vbdev?
> [Kunal] We can say it is an extension of vbdev but it maintains more 
> information than just base_bdevs in its own module itself to have 
> better control as explained above.
>  
> [Senthil-1] Since Pvol maintains more info of base_bdevs than vbdev, 
> can we say that PVol is a superset of vbdev? If so, can you give an 
> example of their data structure mappings?
> [Kunal] PVOL maintains the base bdev information in its module itself 
> and registers with bdev layer via spdk_bdev_register(). So there is no 
> mapping exists between PVOL and vbdev. However once PVOL is created 
> with 1 or more NVMe bdevs, LVS/LVOl can be created over PVOL itself.
>  
> [Senthil-1] Would it possible for you to explain the diff b/w PVol & 
> vbdev by their real-time use-cases? Like when PVol will be used vs 
> vbdev. I assuming both vbdev and PVol are going to coexist.
> [Kunal] I am assuming here you are asking about vbdev concatenation vs 
> PVOL concatenation only. I don’t think vbdev is supporting 
> concatenation. For that once code is pushed, PVOL will be used for 
> striping across drives which will have IO split logic to split IOs and 
> submit to underlying nvme bdevs. To create PVOL across 1 or more base 
> nvme bdevs, either RPC command can be used or pvol config file can be 
> updated with the details to create PVOL during bootup. Once PVOL bdev 
> is created, other modules get examine calls for pvol bdev also. LVS 
> and LVOL can be create over PVOL and exposed to initiator via subsystems.
>  
> 3.      It would be good if you could modify your “Pooled volume” PPT slides
> to show how vbdev fits in there.
>  
> Regards,
> Senthil Kumar V.
>  
> From: SPDK <spdk-bounces(a)lists.01.org> On Behalf Of Luse, Paul E
> Sent: Friday, May 18, 2018 12:15 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] SPDK pooled volume module in bdev
>  
> Excellent, thanks Kunal!  There’s a lot to look at here and with 
> Fenggang’s aggregation patch as well.  I encourage everyone to take a 
> look at both (and I promise I will soon too after post-summit catch-up 
> stuff)
>  
> J
>  
>  
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sablok, 
> Kunal
> Sent: Sunday, May 13, 2018 9:51 AM
> To: spdk(a)lists.01.org
> Subject: [SPDK] SPDK pooled volume module in bdev
>  
> Hi,
> SPDK pooled volume (PVOL) module is a new bdev module which is 
> responsible for striping various NVMe devices and expose the pooled 
> volume to bdev layer which would enhance the performance and capacity. 
> It can support theoretically 255 base devices which can be easily 
> enhanced if required (currently it is being tested max upto 8 base 
> devices). Multiple strip sizes like 32KB, 64KB, 128KB, 256KB, 512KB 
> etc is supported. New RPC commands like "create pvol", "destroy pvol" 
> and "get pvols" are introduced to configure pooled volume dynamically in a running SPDK system.
> Please find attached more information in a ppt with system end to end 
> testing done on this module in SPDK stack in SPDK multi-core environment.
> Please find below gerrithub review details:
> https://review.gerrithub.io/#/c/spdk/spdk/+/410484/
>  
>  
> Regards,
> Kunal
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] SPDK pooled volume module in bdev

[Senthil Kumar Veluswamy]

[-- Attachment #1: Type: text/plain, Size: 9463 bytes --]

Thank you Ben and Maciek for the explanation.

Hi Ben,
     If we can draw a parallel of VBDEV in SAN, can we say VBDEV are similar to "Conglomerate Logical unit/Virtual Volumes (vVOLs)"  in terms of its functionality/usage? 

Regards,
Senthil Kumar V.


-----Original Message-----
From: SPDK <spdk-bounces(a)lists.01.org> On Behalf Of Walker, Benjamin
Sent: Tuesday, June 19, 2018 10:22 PM
To: spdk(a)lists.01.org; Sablok, Kunal <kunal.sablok(a)intel.com>; Luse, Paul E <paul.e.luse(a)intel.com>
Subject: Re: [SPDK] SPDK pooled volume module in bdev

Hi all,

SPDK's virtual block devices (vbdevs) are a framework for creating an I/O processing pipeline by stacking block devices (bdevs) on top of one another.
Each block device has a type and has internal logic implemented by a bdev module. For example, there is an NVMe bdev, which routes I/O to NVMe devices, and there is also a logical volume (lvol) bdev, which carves up an underlying bdev into many smaller bdevs dynamically.

The (v)bdev layer does not contain any actual I/O processing logic in and of itself - it's just the framework. It delegates to bdev modules to implement the actual logic. So the bdev layer today does not implement concatenation or anything of that nature.

What Kunal is implementing here is a bdev module that implements RAID0. It stripes the RAID volume across N underlying bdevs and exposes a single bdev on top.

Thanks,
Ben

On Tue, 2018-06-19 at 03:23 +0000, Senthil Kumar Veluswamy wrote:
> Hi Kunal,
>      >> I am assuming here you are asking about vbdev concatenation vs 
> PVOL concatenation only. I don’t think vbdev is supporting concatenation.
>  
> I’m going thru the Paul’s presentation from last SPDK US submit on 
> “Virtual
> Bdevs: The secret to customizing SPDK” and found below slides.
>  
>      
>  
> As you can see, from VBDEV explanation it looks very similar to PVOL . 
> This is where the confusion is. If I understand it correctly, even 
> VBDEV can concatenate more than one BDEVs.
>  
> May be Paul can help here from VBDEV perspective..
>  
> Hi Paul,
>      Can you please help here to understand the use-cases for VBDEV and PVOLs?
> & How they are architecturally different and how they can co-exists?
>  
> Thank you.
>  
> Regards,
> Senthil Kumar V.
>  
> From: SPDK <spdk-bounces(a)lists.01.org> On Behalf Of Sablok, Kunal
> Sent: Monday, June 18, 2018 10:12 PM
> To: Senthil Kumar Veluswamy <Senthil.Kumar.Veluswamy(a)wdc.com>
> Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] SPDK pooled volume module in bdev
>  
> Hi Senthil,
> Please find inline.
>  
> Regards,
> Kunal
>  
> From: Senthil Kumar Veluswamy [mailto:Senthil.Kumar.Veluswamy(a)wdc.com]
> Sent: Sunday, June 17, 2018 10:07 AM
> To: Sablok, Kunal <kunal.sablok(a)intel.com>
> Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: RE: SPDK pooled volume module in bdev
>  
> Hi Kunal,
>      Thank you for the explanation. I’ve few more questions. My 
> response inline tagged with “[Senthil-1]”.
>  
> Regards,
> Senthil Kumar V.
>  
> From: Sablok, Kunal <kunal.sablok(a)intel.com>
> Sent: Friday, June 15, 2018 10:40 PM
> To: Senthil Kumar Veluswamy <Senthil.Kumar.Veluswamy(a)wdc.com>
> Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: RE: SPDK pooled volume module in bdev
>  
> Hi Senthil,
> Please find inline.
>  
> Regards,
> Kunal
>  
> From: Senthil Kumar Veluswamy [mailto:Senthil.Kumar.Veluswamy(a)wdc.com]
> Sent: Thursday, June 14, 2018 12:20 PM
> To: Sablok, Kunal <kunal.sablok(a)intel.com>
> Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: RE: SPDK pooled volume module in bdev
>  
> Hi Kunal,
>      I’m trying understand the pVol architecture and have few questions.
>  
> 1.      How this pvol model co-exists with vbdev that is present today?
> [Kunal] In current SPDK, when any bdev is getting unregistered,
> spdk_bdev_unregister() removes all the underlying base bdevs from 
> vbdev but for PVOL scenario (since it is similar to Raid0 device), 
> only PVOL bdev gets unregistered and it still needs to keep the 
> information of current base devices which are still claimed. PVOL 
> needs to keep many more information apart from base_bdev reference per 
> base_bdev level like base_bdev's descriptor, some flags for hotplug 
> scenarios etc[will upload in hotplug review] which it is managing itself in pvol module.
>  
> During hotplug scenarios, when one of the base bdev of PVOL is removed 
> and PVOL gets its notification, it un-registers the PVOL from bdev 
> layer since all underlying base_bdevs are not present (but bdev layer 
> implementation also removes all the base bdevs in unregister call) but 
> in this case also it still needs to keep the base bdevs and other 
> information of the base bdevs which are still present and claimed by 
> PVOL. When the removed base_bdev is added again, PVOL gets the 
> notification via examine() and it will claim only that base_bdev now, 
> all other base_bdevs were already present and so PVOL gets registered again with bdev layer.
>  
> Base bdevs and its information can only be removed from PVOL bdev if 
> below layer like NVMe notifies the PVOL explicitly of base device 
> removal. So, idea is to hold the base_bdevs in PVOL which are still 
> present, claimed and not removed for various reasons, so as of now 
> PVOL is using spdk_bdev_register() to register itself with bdev layer, 
> and other modules like LVS/LVOL can create logical volumes over PVOL [physical volume].
>  
> 2.      If I understand correctly, vbdev also allows to concatenate more than
> one base physical bdevs and used for creating logical volume. So shall 
> we say pvol is an extension of vbdev?
> [Kunal] We can say it is an extension of vbdev but it maintains more 
> information than just base_bdevs in its own module itself to have 
> better control as explained above.
>  
> [Senthil-1] Since Pvol maintains more info of base_bdevs than vbdev, 
> can we say that PVol is a superset of vbdev? If so, can you give an 
> example of their data structure mappings?
> [Kunal] PVOL maintains the base bdev information in its module itself 
> and registers with bdev layer via spdk_bdev_register(). So there is no 
> mapping exists between PVOL and vbdev. However once PVOL is created 
> with 1 or more NVMe bdevs, LVS/LVOl can be created over PVOL itself.
>  
> [Senthil-1] Would it possible for you to explain the diff b/w PVol & 
> vbdev by their real-time use-cases? Like when PVol will be used vs 
> vbdev. I assuming both vbdev and PVol are going to coexist.
> [Kunal] I am assuming here you are asking about vbdev concatenation vs 
> PVOL concatenation only. I don’t think vbdev is supporting 
> concatenation. For that once code is pushed, PVOL will be used for 
> striping across drives which will have IO split logic to split IOs and 
> submit to underlying nvme bdevs. To create PVOL across 1 or more base 
> nvme bdevs, either RPC command can be used or pvol config file can be 
> updated with the details to create PVOL during bootup. Once PVOL bdev 
> is created, other modules get examine calls for pvol bdev also. LVS 
> and LVOL can be create over PVOL and exposed to initiator via subsystems.
>  
> 3.      It would be good if you could modify your “Pooled volume” PPT slides
> to show how vbdev fits in there.
>  
> Regards,
> Senthil Kumar V.
>  
> From: SPDK <spdk-bounces(a)lists.01.org> On Behalf Of Luse, Paul E
> Sent: Friday, May 18, 2018 12:15 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] SPDK pooled volume module in bdev
>  
> Excellent, thanks Kunal!  There’s a lot to look at here and with 
> Fenggang’s aggregation patch as well.  I encourage everyone to take a 
> look at both (and I promise I will soon too after post-summit catch-up 
> stuff)
>  
> J
>  
>  
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sablok, 
> Kunal
> Sent: Sunday, May 13, 2018 9:51 AM
> To: spdk(a)lists.01.org
> Subject: [SPDK] SPDK pooled volume module in bdev
>  
> Hi,
> SPDK pooled volume (PVOL) module is a new bdev module which is 
> responsible for striping various NVMe devices and expose the pooled 
> volume to bdev layer which would enhance the performance and capacity. 
> It can support theoretically 255 base devices which can be easily 
> enhanced if required (currently it is being tested max upto 8 base 
> devices). Multiple strip sizes like 32KB, 64KB, 128KB, 256KB, 512KB 
> etc is supported. New RPC commands like "create pvol", "destroy pvol" 
> and "get pvols" are introduced to configure pooled volume dynamically in a running SPDK system.
> Please find attached more information in a ppt with system end to end 
> testing done on this module in SPDK stack in SPDK multi-core environment.
> Please find below gerrithub review details:
> https://review.gerrithub.io/#/c/spdk/spdk/+/410484/
>  
>  
> Regards,
> Kunal
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] SPDK pooled volume module in bdev

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 8712 bytes --]

Hi all,

SPDK's virtual block devices (vbdevs) are a framework for creating an I/O
processing pipeline by stacking block devices (bdevs) on top of one another.
Each block device has a type and has internal logic implemented by a bdev
module. For example, there is an NVMe bdev, which routes I/O to NVMe devices,
and there is also a logical volume (lvol) bdev, which carves up an underlying
bdev into many smaller bdevs dynamically.

The (v)bdev layer does not contain any actual I/O processing logic in and of
itself - it's just the framework. It delegates to bdev modules to implement the
actual logic. So the bdev layer today does not implement concatenation or
anything of that nature.

What Kunal is implementing here is a bdev module that implements RAID0. It
stripes the RAID volume across N underlying bdevs and exposes a single bdev on
top.

Thanks,
Ben

On Tue, 2018-06-19 at 03:23 +0000, Senthil Kumar Veluswamy wrote:
> Hi Kunal,
>      >> I am assuming here you are asking about vbdev concatenation vs PVOL
> concatenation only. I don’t think vbdev is supporting concatenation.
>  
> I’m going thru the Paul’s presentation from last SPDK US submit on “Virtual
> Bdevs: The secret to customizing SPDK” and found below slides.
>  
>      
>  
> As you can see, from VBDEV explanation it looks very similar to PVOL . This is
> where the confusion is. If I understand it correctly, even VBDEV can
> concatenate more than one BDEVs.
>  
> May be Paul can help here from VBDEV perspective..
>  
> Hi Paul,
>      Can you please help here to understand the use-cases for VBDEV and PVOLs?
> & How they are architecturally different and how they can co-exists?
>  
> Thank you.
>  
> Regards,
> Senthil Kumar V.
>  
> From: SPDK <spdk-bounces(a)lists.01.org> On Behalf Of Sablok, Kunal
> Sent: Monday, June 18, 2018 10:12 PM
> To: Senthil Kumar Veluswamy <Senthil.Kumar.Veluswamy(a)wdc.com>
> Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] SPDK pooled volume module in bdev
>  
> Hi Senthil,
> Please find inline.
>  
> Regards,
> Kunal
>  
> From: Senthil Kumar Veluswamy [mailto:Senthil.Kumar.Veluswamy(a)wdc.com] 
> Sent: Sunday, June 17, 2018 10:07 AM
> To: Sablok, Kunal <kunal.sablok(a)intel.com>
> Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: RE: SPDK pooled volume module in bdev
>  
> Hi Kunal,
>      Thank you for the explanation. I’ve few more questions. My response
> inline tagged with “[Senthil-1]”.
>  
> Regards,
> Senthil Kumar V.
>  
> From: Sablok, Kunal <kunal.sablok(a)intel.com> 
> Sent: Friday, June 15, 2018 10:40 PM
> To: Senthil Kumar Veluswamy <Senthil.Kumar.Veluswamy(a)wdc.com>
> Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: RE: SPDK pooled volume module in bdev
>  
> Hi Senthil,
> Please find inline.
>  
> Regards,
> Kunal
>  
> From: Senthil Kumar Veluswamy [mailto:Senthil.Kumar.Veluswamy(a)wdc.com] 
> Sent: Thursday, June 14, 2018 12:20 PM
> To: Sablok, Kunal <kunal.sablok(a)intel.com>
> Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: RE: SPDK pooled volume module in bdev
>  
> Hi Kunal,
>      I’m trying understand the pVol architecture and have few questions.
>  
> 1.      How this pvol model co-exists with vbdev that is present today?
> [Kunal] In current SPDK, when any bdev is getting unregistered,
> spdk_bdev_unregister() removes all the underlying base bdevs from vbdev but
> for PVOL scenario (since it is similar to Raid0 device), only PVOL bdev gets
> unregistered and it still needs to keep the information of current base
> devices which are still claimed. PVOL needs to keep many more information
> apart from base_bdev reference per base_bdev level like base_bdev's
> descriptor, some flags for hotplug scenarios etc[will upload in hotplug
> review] which it is managing itself in pvol module.
>  
> During hotplug scenarios, when one of the base bdev of PVOL is removed and
> PVOL gets its notification, it un-registers the PVOL from bdev layer since all
> underlying base_bdevs are not present (but bdev layer implementation also
> removes all the base bdevs in unregister call) but in this case also it still
> needs to keep the base bdevs and other information of the base bdevs which are
> still present and claimed by PVOL. When the removed base_bdev is added again,
> PVOL gets the notification via examine() and it will claim only that base_bdev
> now, all other base_bdevs were already present and so PVOL gets registered
> again with bdev layer.
>  
> Base bdevs and its information can only be removed from PVOL bdev if below
> layer like NVMe notifies the PVOL explicitly of base device removal. So, idea
> is to hold the base_bdevs in PVOL which are still present, claimed and not
> removed for various reasons, so as of now PVOL is using spdk_bdev_register()
> to register itself with bdev layer, and other modules like LVS/LVOL can create
> logical volumes over PVOL [physical volume].
>  
> 2.      If I understand correctly, vbdev also allows to concatenate more than
> one base physical bdevs and used for creating logical volume. So shall we say
> pvol is an extension of vbdev?
> [Kunal] We can say it is an extension of vbdev but it maintains more
> information than just base_bdevs in its own module itself to have better
> control as explained above.
>  
> [Senthil-1] Since Pvol maintains more info of base_bdevs than vbdev, can we
> say that PVol is a superset of vbdev? If so, can you give an example of their
> data structure mappings?
> [Kunal] PVOL maintains the base bdev information in its module itself and
> registers with bdev layer via spdk_bdev_register(). So there is no mapping
> exists between PVOL and vbdev. However once PVOL is created with 1 or more
> NVMe bdevs, LVS/LVOl can be created over PVOL itself.
>  
> [Senthil-1] Would it possible for you to explain the diff b/w PVol & vbdev by
> their real-time use-cases? Like when PVol will be used vs vbdev. I assuming
> both vbdev and PVol are going to coexist.
> [Kunal] I am assuming here you are asking about vbdev concatenation vs PVOL
> concatenation only. I don’t think vbdev is supporting concatenation. For that
> once code is pushed, PVOL will be used for striping across drives which will
> have IO split logic to split IOs and submit to underlying nvme bdevs. To
> create PVOL across 1 or more base nvme bdevs, either RPC command can be used
> or pvol config file can be updated with the details to create PVOL during
> bootup. Once PVOL bdev is created, other modules get examine calls for pvol
> bdev also. LVS and LVOL can be create over PVOL and exposed to initiator via
> subsystems.
>  
> 3.      It would be good if you could modify your “Pooled volume” PPT slides
> to show how vbdev fits in there.
>  
> Regards,
> Senthil Kumar V.
>  
> From: SPDK <spdk-bounces(a)lists.01.org> On Behalf Of Luse, Paul E
> Sent: Friday, May 18, 2018 12:15 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] SPDK pooled volume module in bdev
>  
> Excellent, thanks Kunal!  There’s a lot to look at here and with Fenggang’s
> aggregation patch as well.  I encourage everyone to take a look at both (and I
> promise I will soon too after post-summit catch-up stuff)
>  
> J
>  
>  
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sablok, Kunal
> Sent: Sunday, May 13, 2018 9:51 AM
> To: spdk(a)lists.01.org
> Subject: [SPDK] SPDK pooled volume module in bdev
>  
> Hi,
> SPDK pooled volume (PVOL) module is a new bdev module which is responsible for
> striping various NVMe devices and expose the pooled volume to bdev layer which
> would enhance the performance and capacity. It can support theoretically 255
> base devices which can be easily enhanced if required (currently it is being
> tested max upto 8 base devices). Multiple strip sizes like 32KB, 64KB, 128KB,
> 256KB, 512KB etc is supported. New RPC commands like "create pvol", "destroy
> pvol" and "get pvols" are introduced to configure pooled volume dynamically in
> a running SPDK system.
> Please find attached more information in a ppt with system end to end testing
> done on this module in SPDK stack in SPDK multi-core environment.
> Please find below gerrithub review details:
> https://review.gerrithub.io/#/c/spdk/spdk/+/410484/
>  
>  
> Regards,
> Kunal
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] SPDK pooled volume module in bdev

[Szwed, Maciej]

[-- Attachment #1: Type: text/plain, Size: 8214 bytes --]

Hi Senthil,
Until now vbdevs usually were based on single device / memory entity or split them to smaller vbdevs like split module or lvols. PVOls are merging many bdevs/vbdevs to one.

Regards,
Maciek

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Senthil Kumar Veluswamy
Sent: Tuesday, June 19, 2018 5:23 AM
To: Sablok, Kunal <kunal.sablok(a)intel.com>; Luse, Paul E <paul.e.luse(a)intel.com>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] SPDK pooled volume module in bdev

Hi Kunal,
     >> I am assuming here you are asking about vbdev concatenation vs PVOL concatenation only. I don't think vbdev is supporting concatenation.

I'm going thru the Paul's presentation from last SPDK US submit on "Virtual Bdevs: The secret to customizing SPDK" and found below slides.

[cid:image001.jpg(a)01D407C0.9B7B2080]     [cid:image002.jpg(a)01D407C0.9B7B2080]

As you can see, from VBDEV explanation it looks very similar to PVOL . This is where the confusion is. If I understand it correctly, even VBDEV can concatenate more than one BDEVs.

May be Paul can help here from VBDEV perspective..

Hi Paul,
     Can you please help here to understand the use-cases for VBDEV and PVOLs? & How they are architecturally different and how they can co-exists?

Thank you.

Regards,
Senthil Kumar V.

From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> On Behalf Of Sablok, Kunal
Sent: Monday, June 18, 2018 10:12 PM
To: Senthil Kumar Veluswamy <Senthil.Kumar.Veluswamy(a)wdc.com<mailto:Senthil.Kumar.Veluswamy(a)wdc.com>>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] SPDK pooled volume module in bdev

Hi Senthil,
Please find inline.

Regards,
Kunal

From: Senthil Kumar Veluswamy [mailto:Senthil.Kumar.Veluswamy(a)wdc.com]
Sent: Sunday, June 17, 2018 10:07 AM
To: Sablok, Kunal <kunal.sablok(a)intel.com<mailto:kunal.sablok(a)intel.com>>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: RE: SPDK pooled volume module in bdev

Hi Kunal,
     Thank you for the explanation. I've few more questions. My response inline tagged with "[Senthil-1]".

Regards,
Senthil Kumar V.

From: Sablok, Kunal <kunal.sablok(a)intel.com<mailto:kunal.sablok(a)intel.com>>
Sent: Friday, June 15, 2018 10:40 PM
To: Senthil Kumar Veluswamy <Senthil.Kumar.Veluswamy(a)wdc.com<mailto:Senthil.Kumar.Veluswamy(a)wdc.com>>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: RE: SPDK pooled volume module in bdev

Hi Senthil,
Please find inline.

Regards,
Kunal

From: Senthil Kumar Veluswamy [mailto:Senthil.Kumar.Veluswamy(a)wdc.com]
Sent: Thursday, June 14, 2018 12:20 PM
To: Sablok, Kunal <kunal.sablok(a)intel.com<mailto:kunal.sablok(a)intel.com>>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: RE: SPDK pooled volume module in bdev

Hi Kunal,
     I'm trying understand the pVol architecture and have few questions.


1.      How this pvol model co-exists with vbdev that is present today?
[Kunal] In current SPDK, when any bdev is getting unregistered, spdk_bdev_unregister() removes all the underlying base bdevs from vbdev but for PVOL scenario (since it is similar to Raid0 device), only PVOL bdev gets unregistered and it still needs to keep the information of current base devices which are still claimed. PVOL needs to keep many more information apart from base_bdev reference per base_bdev level like base_bdev's descriptor, some flags for hotplug scenarios etc[will upload in hotplug review] which it is managing itself in pvol module.

During hotplug scenarios, when one of the base bdev of PVOL is removed and PVOL gets its notification, it un-registers the PVOL from bdev layer since all underlying base_bdevs are not present (but bdev layer implementation also removes all the base bdevs in unregister call) but in this case also it still needs to keep the base bdevs and other information of the base bdevs which are still present and claimed by PVOL. When the removed base_bdev is added again, PVOL gets the notification via examine() and it will claim only that base_bdev now, all other base_bdevs were already present and so PVOL gets registered again with bdev layer.

Base bdevs and its information can only be removed from PVOL bdev if below layer like NVMe notifies the PVOL explicitly of base device removal. So, idea is to hold the base_bdevs in PVOL which are still present, claimed and not removed for various reasons, so as of now PVOL is using spdk_bdev_register() to register itself with bdev layer, and other modules like LVS/LVOL can create logical volumes over PVOL [physical volume].


2.      If I understand correctly, vbdev also allows to concatenate more than one base physical bdevs and used for creating logical volume. So shall we say pvol is an extension of vbdev?
[Kunal] We can say it is an extension of vbdev but it maintains more information than just base_bdevs in its own module itself to have better control as explained above.

[Senthil-1] Since Pvol maintains more info of base_bdevs than vbdev, can we say that PVol is a superset of vbdev? If so, can you give an example of their data structure mappings?
[Kunal] PVOL maintains the base bdev information in its module itself and registers with bdev layer via spdk_bdev_register(). So there is no mapping exists between PVOL and vbdev. However once PVOL is created with 1 or more NVMe bdevs, LVS/LVOl can be created over PVOL itself.

[Senthil-1] Would it possible for you to explain the diff b/w PVol & vbdev by their real-time use-cases? Like when PVol will be used vs vbdev. I assuming both vbdev and PVol are going to coexist.
[Kunal] I am assuming here you are asking about vbdev concatenation vs PVOL concatenation only. I don't think vbdev is supporting concatenation. For that once code is pushed, PVOL will be used for striping across drives which will have IO split logic to split IOs and submit to underlying nvme bdevs. To create PVOL across 1 or more base nvme bdevs, either RPC command can be used or pvol config file can be updated with the details to create PVOL during bootup. Once PVOL bdev is created, other modules get examine calls for pvol bdev also. LVS and LVOL can be create over PVOL and exposed to initiator via subsystems.


3.      It would be good if you could modify your "Pooled volume" PPT slides to show how vbdev fits in there.

Regards,
Senthil Kumar V.

From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> On Behalf Of Luse, Paul E
Sent: Friday, May 18, 2018 12:15 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] SPDK pooled volume module in bdev

Excellent, thanks Kunal!  There's a lot to look at here and with Fenggang's aggregation patch as well.  I encourage everyone to take a look at both (and I promise I will soon too after post-summit catch-up stuff)

:)


From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sablok, Kunal
Sent: Sunday, May 13, 2018 9:51 AM
To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
Subject: [SPDK] SPDK pooled volume module in bdev

Hi,
SPDK pooled volume (PVOL) module is a new bdev module which is responsible for striping various NVMe devices and expose the pooled volume to bdev layer which would enhance the performance and capacity. It can support theoretically 255 base devices which can be easily enhanced if required (currently it is being tested max upto 8 base devices). Multiple strip sizes like 32KB, 64KB, 128KB, 256KB, 512KB etc is supported. New RPC commands like "create pvol", "destroy pvol" and "get pvols" are introduced to configure pooled volume dynamically in a running SPDK system.
Please find attached more information in a ppt with system end to end testing done on this module in SPDK stack in SPDK multi-core environment.
Please find below gerrithub review details:
https://review.gerrithub.io/#/c/spdk/spdk/+/410484/


Regards,
Kunal

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 23539 bytes --]

[-- Attachment #3: image001.jpg --]
[-- Type: image/jpeg, Size: 17220 bytes --]

[-- Attachment #4: image002.jpg --]
[-- Type: image/jpeg, Size: 24625 bytes --]

Re: [SPDK] SPDK pooled volume module in bdev

[Senthil Kumar Veluswamy]

[-- Attachment #1: Type: text/plain, Size: 7569 bytes --]

Hi Kunal,
     >> I am assuming here you are asking about vbdev concatenation vs PVOL concatenation only. I don't think vbdev is supporting concatenation.

I'm going thru the Paul's presentation from last SPDK US submit on "Virtual Bdevs: The secret to customizing SPDK" and found below slides.

[cid:image003.jpg(a)01D407AA.F6C44680]     [cid:image005.jpg(a)01D407AA.F6C44680]

As you can see, from VBDEV explanation it looks very similar to PVOL . This is where the confusion is. If I understand it correctly, even VBDEV can concatenate more than one BDEVs.

May be Paul can help here from VBDEV perspective..

Hi Paul,
     Can you please help here to understand the use-cases for VBDEV and PVOLs? & How they are architecturally different and how they can co-exists?

Thank you.

Regards,
Senthil Kumar V.

From: SPDK <spdk-bounces(a)lists.01.org> On Behalf Of Sablok, Kunal
Sent: Monday, June 18, 2018 10:12 PM
To: Senthil Kumar Veluswamy <Senthil.Kumar.Veluswamy(a)wdc.com>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] SPDK pooled volume module in bdev

Hi Senthil,
Please find inline.

Regards,
Kunal

From: Senthil Kumar Veluswamy [mailto:Senthil.Kumar.Veluswamy(a)wdc.com]
Sent: Sunday, June 17, 2018 10:07 AM
To: Sablok, Kunal <kunal.sablok(a)intel.com<mailto:kunal.sablok(a)intel.com>>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: RE: SPDK pooled volume module in bdev

Hi Kunal,
     Thank you for the explanation. I've few more questions. My response inline tagged with "[Senthil-1]".

Regards,
Senthil Kumar V.

From: Sablok, Kunal <kunal.sablok(a)intel.com<mailto:kunal.sablok(a)intel.com>>
Sent: Friday, June 15, 2018 10:40 PM
To: Senthil Kumar Veluswamy <Senthil.Kumar.Veluswamy(a)wdc.com<mailto:Senthil.Kumar.Veluswamy(a)wdc.com>>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: RE: SPDK pooled volume module in bdev

Hi Senthil,
Please find inline.

Regards,
Kunal

From: Senthil Kumar Veluswamy [mailto:Senthil.Kumar.Veluswamy(a)wdc.com]
Sent: Thursday, June 14, 2018 12:20 PM
To: Sablok, Kunal <kunal.sablok(a)intel.com<mailto:kunal.sablok(a)intel.com>>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: RE: SPDK pooled volume module in bdev

Hi Kunal,
     I'm trying understand the pVol architecture and have few questions.


1.      How this pvol model co-exists with vbdev that is present today?
[Kunal] In current SPDK, when any bdev is getting unregistered, spdk_bdev_unregister() removes all the underlying base bdevs from vbdev but for PVOL scenario (since it is similar to Raid0 device), only PVOL bdev gets unregistered and it still needs to keep the information of current base devices which are still claimed. PVOL needs to keep many more information apart from base_bdev reference per base_bdev level like base_bdev's descriptor, some flags for hotplug scenarios etc[will upload in hotplug review] which it is managing itself in pvol module.

During hotplug scenarios, when one of the base bdev of PVOL is removed and PVOL gets its notification, it un-registers the PVOL from bdev layer since all underlying base_bdevs are not present (but bdev layer implementation also removes all the base bdevs in unregister call) but in this case also it still needs to keep the base bdevs and other information of the base bdevs which are still present and claimed by PVOL. When the removed base_bdev is added again, PVOL gets the notification via examine() and it will claim only that base_bdev now, all other base_bdevs were already present and so PVOL gets registered again with bdev layer.

Base bdevs and its information can only be removed from PVOL bdev if below layer like NVMe notifies the PVOL explicitly of base device removal. So, idea is to hold the base_bdevs in PVOL which are still present, claimed and not removed for various reasons, so as of now PVOL is using spdk_bdev_register() to register itself with bdev layer, and other modules like LVS/LVOL can create logical volumes over PVOL [physical volume].


2.      If I understand correctly, vbdev also allows to concatenate more than one base physical bdevs and used for creating logical volume. So shall we say pvol is an extension of vbdev?
[Kunal] We can say it is an extension of vbdev but it maintains more information than just base_bdevs in its own module itself to have better control as explained above.

[Senthil-1] Since Pvol maintains more info of base_bdevs than vbdev, can we say that PVol is a superset of vbdev? If so, can you give an example of their data structure mappings?
[Kunal] PVOL maintains the base bdev information in its module itself and registers with bdev layer via spdk_bdev_register(). So there is no mapping exists between PVOL and vbdev. However once PVOL is created with 1 or more NVMe bdevs, LVS/LVOl can be created over PVOL itself.

[Senthil-1] Would it possible for you to explain the diff b/w PVol & vbdev by their real-time use-cases? Like when PVol will be used vs vbdev. I assuming both vbdev and PVol are going to coexist.
[Kunal] I am assuming here you are asking about vbdev concatenation vs PVOL concatenation only. I don't think vbdev is supporting concatenation. For that once code is pushed, PVOL will be used for striping across drives which will have IO split logic to split IOs and submit to underlying nvme bdevs. To create PVOL across 1 or more base nvme bdevs, either RPC command can be used or pvol config file can be updated with the details to create PVOL during bootup. Once PVOL bdev is created, other modules get examine calls for pvol bdev also. LVS and LVOL can be create over PVOL and exposed to initiator via subsystems.


3.      It would be good if you could modify your "Pooled volume" PPT slides to show how vbdev fits in there.

Regards,
Senthil Kumar V.

From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> On Behalf Of Luse, Paul E
Sent: Friday, May 18, 2018 12:15 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] SPDK pooled volume module in bdev

Excellent, thanks Kunal!  There's a lot to look at here and with Fenggang's aggregation patch as well.  I encourage everyone to take a look at both (and I promise I will soon too after post-summit catch-up stuff)

:)


From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sablok, Kunal
Sent: Sunday, May 13, 2018 9:51 AM
To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
Subject: [SPDK] SPDK pooled volume module in bdev

Hi,
SPDK pooled volume (PVOL) module is a new bdev module which is responsible for striping various NVMe devices and expose the pooled volume to bdev layer which would enhance the performance and capacity. It can support theoretically 255 base devices which can be easily enhanced if required (currently it is being tested max upto 8 base devices). Multiple strip sizes like 32KB, 64KB, 128KB, 256KB, 512KB etc is supported. New RPC commands like "create pvol", "destroy pvol" and "get pvols" are introduced to configure pooled volume dynamically in a running SPDK system.
Please find attached more information in a ppt with system end to end testing done on this module in SPDK stack in SPDK multi-core environment.
Please find below gerrithub review details:
https://review.gerrithub.io/#/c/spdk/spdk/+/410484/


Regards,
Kunal

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 20311 bytes --]

[-- Attachment #3: image003.jpg --]
[-- Type: image/jpeg, Size: 17220 bytes --]

[-- Attachment #4: image005.jpg --]
[-- Type: image/jpeg, Size: 24625 bytes --]

Re: [SPDK] SPDK pooled volume module in bdev

[Sablok, Kunal]

[-- Attachment #1: Type: text/plain, Size: 6384 bytes --]

Hi Senthil,
Please find inline.

Regards,
Kunal

From: Senthil Kumar Veluswamy [mailto:Senthil.Kumar.Veluswamy(a)wdc.com]
Sent: Sunday, June 17, 2018 10:07 AM
To: Sablok, Kunal <kunal.sablok(a)intel.com>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: RE: SPDK pooled volume module in bdev

Hi Kunal,
     Thank you for the explanation. I've few more questions. My response inline tagged with "[Senthil-1]".

Regards,
Senthil Kumar V.

From: Sablok, Kunal <kunal.sablok(a)intel.com<mailto:kunal.sablok(a)intel.com>>
Sent: Friday, June 15, 2018 10:40 PM
To: Senthil Kumar Veluswamy <Senthil.Kumar.Veluswamy(a)wdc.com<mailto:Senthil.Kumar.Veluswamy(a)wdc.com>>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: RE: SPDK pooled volume module in bdev

Hi Senthil,
Please find inline.

Regards,
Kunal

From: Senthil Kumar Veluswamy [mailto:Senthil.Kumar.Veluswamy(a)wdc.com]
Sent: Thursday, June 14, 2018 12:20 PM
To: Sablok, Kunal <kunal.sablok(a)intel.com<mailto:kunal.sablok(a)intel.com>>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: RE: SPDK pooled volume module in bdev

Hi Kunal,
     I'm trying understand the pVol architecture and have few questions.


1.     How this pvol model co-exists with vbdev that is present today?
[Kunal] In current SPDK, when any bdev is getting unregistered, spdk_bdev_unregister() removes all the underlying base bdevs from vbdev but for PVOL scenario (since it is similar to Raid0 device), only PVOL bdev gets unregistered and it still needs to keep the information of current base devices which are still claimed. PVOL needs to keep many more information apart from base_bdev reference per base_bdev level like base_bdev's descriptor, some flags for hotplug scenarios etc[will upload in hotplug review] which it is managing itself in pvol module.

During hotplug scenarios, when one of the base bdev of PVOL is removed and PVOL gets its notification, it un-registers the PVOL from bdev layer since all underlying base_bdevs are not present (but bdev layer implementation also removes all the base bdevs in unregister call) but in this case also it still needs to keep the base bdevs and other information of the base bdevs which are still present and claimed by PVOL. When the removed base_bdev is added again, PVOL gets the notification via examine() and it will claim only that base_bdev now, all other base_bdevs were already present and so PVOL gets registered again with bdev layer.

Base bdevs and its information can only be removed from PVOL bdev if below layer like NVMe notifies the PVOL explicitly of base device removal. So, idea is to hold the base_bdevs in PVOL which are still present, claimed and not removed for various reasons, so as of now PVOL is using spdk_bdev_register() to register itself with bdev layer, and other modules like LVS/LVOL can create logical volumes over PVOL [physical volume].


2.     If I understand correctly, vbdev also allows to concatenate more than one base physical bdevs and used for creating logical volume. So shall we say pvol is an extension of vbdev?
[Kunal] We can say it is an extension of vbdev but it maintains more information than just base_bdevs in its own module itself to have better control as explained above.

[Senthil-1] Since Pvol maintains more info of base_bdevs than vbdev, can we say that PVol is a superset of vbdev? If so, can you give an example of their data structure mappings?
[Kunal] PVOL maintains the base bdev information in its module itself and registers with bdev layer via spdk_bdev_register(). So there is no mapping exists between PVOL and vbdev. However once PVOL is created with 1 or more NVMe bdevs, LVS/LVOl can be created over PVOL itself.

[Senthil-1] Would it possible for you to explain the diff b/w PVol & vbdev by their real-time use-cases? Like when PVol will be used vs vbdev. I assuming both vbdev and PVol are going to coexist.
[Kunal] I am assuming here you are asking about vbdev concatenation vs PVOL concatenation only. I don't think vbdev is supporting concatenation. For that once code is pushed, PVOL will be used for striping across drives which will have IO split logic to split IOs and submit to underlying nvme bdevs. To create PVOL across 1 or more base nvme bdevs, either RPC command can be used or pvol config file can be updated with the details to create PVOL during bootup. Once PVOL bdev is created, other modules get examine calls for pvol bdev also. LVS and LVOL can be create over PVOL and exposed to initiator via subsystems.


3.     It would be good if you could modify your "Pooled volume" PPT slides to show how vbdev fits in there.

Regards,
Senthil Kumar V.

From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> On Behalf Of Luse, Paul E
Sent: Friday, May 18, 2018 12:15 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] SPDK pooled volume module in bdev

Excellent, thanks Kunal!  There's a lot to look at here and with Fenggang's aggregation patch as well.  I encourage everyone to take a look at both (and I promise I will soon too after post-summit catch-up stuff)

:)


From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sablok, Kunal
Sent: Sunday, May 13, 2018 9:51 AM
To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
Subject: [SPDK] SPDK pooled volume module in bdev

Hi,
SPDK pooled volume (PVOL) module is a new bdev module which is responsible for striping various NVMe devices and expose the pooled volume to bdev layer which would enhance the performance and capacity. It can support theoretically 255 base devices which can be easily enhanced if required (currently it is being tested max upto 8 base devices). Multiple strip sizes like 32KB, 64KB, 128KB, 256KB, 512KB etc is supported. New RPC commands like "create pvol", "destroy pvol" and "get pvols" are introduced to configure pooled volume dynamically in a running SPDK system.
Please find attached more information in a ppt with system end to end testing done on this module in SPDK stack in SPDK multi-core environment.
Please find below gerrithub review details:
https://review.gerrithub.io/#/c/spdk/spdk/+/410484/


Regards,
Kunal

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 16746 bytes --]

Re: [SPDK] SPDK pooled volume module in bdev

[Senthil Kumar Veluswamy]

[-- Attachment #1: Type: text/plain, Size: 5161 bytes --]

Hi Kunal,
     Thank you for the explanation. I've few more questions. My response inline tagged with "[Senthil-1]".

Regards,
Senthil Kumar V.

From: Sablok, Kunal <kunal.sablok(a)intel.com<mailto:kunal.sablok(a)intel.com>>
Sent: Friday, June 15, 2018 10:40 PM
To: Senthil Kumar Veluswamy <Senthil.Kumar.Veluswamy(a)wdc.com<mailto:Senthil.Kumar.Veluswamy(a)wdc.com>>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: RE: SPDK pooled volume module in bdev

Hi Senthil,
Please find inline.

Regards,
Kunal

From: Senthil Kumar Veluswamy [mailto:Senthil.Kumar.Veluswamy(a)wdc.com]
Sent: Thursday, June 14, 2018 12:20 PM
To: Sablok, Kunal <kunal.sablok(a)intel.com<mailto:kunal.sablok(a)intel.com>>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: RE: SPDK pooled volume module in bdev

Hi Kunal,
     I'm trying understand the pVol architecture and have few questions.


1.      How this pvol model co-exists with vbdev that is present today?
[Kunal] In current SPDK, when any bdev is getting unregistered, spdk_bdev_unregister() removes all the underlying base bdevs from vbdev but for PVOL scenario (since it is similar to Raid0 device), only PVOL bdev gets unregistered and it still needs to keep the information of current base devices which are still claimed. PVOL needs to keep many more information apart from base_bdev reference per base_bdev level like base_bdev's descriptor, some flags for hotplug scenarios etc[will upload in hotplug review] which it is managing itself in pvol module.

During hotplug scenarios, when one of the base bdev of PVOL is removed and PVOL gets its notification, it un-registers the PVOL from bdev layer since all underlying base_bdevs are not present (but bdev layer implementation also removes all the base bdevs in unregister call) but in this case also it still needs to keep the base bdevs and other information of the base bdevs which are still present and claimed by PVOL. When the removed base_bdev is added again, PVOL gets the notification via examine() and it will claim only that base_bdev now, all other base_bdevs were already present and so PVOL gets registered again with bdev layer.

Base bdevs and its information can only be removed from PVOL bdev if below layer like NVMe notifies the PVOL explicitly of base device removal. So, idea is to hold the base_bdevs in PVOL which are still present, claimed and not removed for various reasons, so as of now PVOL is using spdk_bdev_register() to register itself with bdev layer, and other modules like LVS/LVOL can create logical volumes over PVOL [physical volume].


2.      If I understand correctly, vbdev also allows to concatenate more than one base physical bdevs and used for creating logical volume. So shall we say pvol is an extension of vbdev?
[Kunal] We can say it is an extension of vbdev but it maintains more information than just base_bdevs in its own module itself to have better control as explained above.

[Senthil-1] Since Pvol maintains more info of base_bdevs than vbdev, can we say that PVol is a superset of vbdev? If so, can you give an example of their data structure mappings?

[Senthil-1] Would it possible for you to explain the diff b/w PVol & vbdev by their real-time use-cases? Like when PVol will be used vs vbdev. I assuming both vbdev and PVol are going to coexist.


3.      It would be good if you could modify your "Pooled volume" PPT slides to show how vbdev fits in there.

Regards,
Senthil Kumar V.

From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> On Behalf Of Luse, Paul E
Sent: Friday, May 18, 2018 12:15 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] SPDK pooled volume module in bdev

Excellent, thanks Kunal!  There's a lot to look at here and with Fenggang's aggregation patch as well.  I encourage everyone to take a look at both (and I promise I will soon too after post-summit catch-up stuff)

:)


From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sablok, Kunal
Sent: Sunday, May 13, 2018 9:51 AM
To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
Subject: [SPDK] SPDK pooled volume module in bdev

Hi,
SPDK pooled volume (PVOL) module is a new bdev module which is responsible for striping various NVMe devices and expose the pooled volume to bdev layer which would enhance the performance and capacity. It can support theoretically 255 base devices which can be easily enhanced if required (currently it is being tested max upto 8 base devices). Multiple strip sizes like 32KB, 64KB, 128KB, 256KB, 512KB etc is supported. New RPC commands like "create pvol", "destroy pvol" and "get pvols" are introduced to configure pooled volume dynamically in a running SPDK system.
Please find attached more information in a ppt with system end to end testing done on this module in SPDK stack in SPDK multi-core environment.
Please find below gerrithub review details:
https://review.gerrithub.io/#/c/spdk/spdk/+/410484/


Regards,
Kunal

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 14662 bytes --]

Re: [SPDK] SPDK pooled volume module in bdev

[Sablok, Kunal]

[-- Attachment #1: Type: text/plain, Size: 4198 bytes --]

Hi Senthil,
Please find inline.

Regards,
Kunal

From: Senthil Kumar Veluswamy [mailto:Senthil.Kumar.Veluswamy(a)wdc.com]
Sent: Thursday, June 14, 2018 12:20 PM
To: Sablok, Kunal <kunal.sablok(a)intel.com>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: RE: SPDK pooled volume module in bdev

Hi Kunal,
     I'm trying understand the pVol architecture and have few questions.


1.     How this pvol model co-exists with vbdev that is present today?
[Kunal] In current SPDK, when any bdev is getting unregistered, spdk_bdev_unregister() removes all the underlying base bdevs from vbdev but for PVOL scenario (since it is similar to Raid0 device), only PVOL bdev gets unregistered and it still needs to keep the information of current base devices which are still claimed. PVOL needs to keep many more information apart from base_bdev reference per base_bdev level like base_bdev's descriptor, some flags for hotplug scenarios etc[will upload in hotplug review] which it is managing itself in pvol module.

During hotplug scenarios, when one of the base bdev of PVOL is removed and PVOL gets its notification, it un-registers the PVOL from bdev layer since all underlying base_bdevs are not present (but bdev layer implementation also removes all the base bdevs in unregister call) but in this case also it still needs to keep the base bdevs and other information of the base bdevs which are still present and claimed by PVOL. When the removed base_bdev is added again, PVOL gets the notification via examine() and it will claim only that base_bdev now, all other base_bdevs were already present and so PVOL gets registered again with bdev layer.

Base bdevs and its information can only be removed from PVOL bdev if below layer like NVMe notifies the PVOL explicitly of base device removal. So, idea is to hold the base_bdevs in PVOL which are still present, claimed and not removed for various reasons, so as of now PVOL is using spdk_bdev_register() to register itself with bdev layer, and other modules like LVS/LVOL can create logical volumes over PVOL [physical volume].


2.     If I understand correctly, vbdev also allows to concatenate more than one base physical bdevs and used for creating logical volume. So shall we say pvol is an extension of vbdev?
[Kunal] We can say it is an extension of vbdev but it maintains more information than just base_bdevs in its own module itself to have better control as explained above.


3.     It would be good if you could modify your "Pooled volume" PPT slides to show how vbdev fits in there.

Regards,
Senthil Kumar V.

From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> On Behalf Of Luse, Paul E
Sent: Friday, May 18, 2018 12:15 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] SPDK pooled volume module in bdev

Excellent, thanks Kunal!  There's a lot to look at here and with Fenggang's aggregation patch as well.  I encourage everyone to take a look at both (and I promise I will soon too after post-summit catch-up stuff)

:)


From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sablok, Kunal
Sent: Sunday, May 13, 2018 9:51 AM
To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
Subject: [SPDK] SPDK pooled volume module in bdev

Hi,
SPDK pooled volume (PVOL) module is a new bdev module which is responsible for striping various NVMe devices and expose the pooled volume to bdev layer which would enhance the performance and capacity. It can support theoretically 255 base devices which can be easily enhanced if required (currently it is being tested max upto 8 base devices). Multiple strip sizes like 32KB, 64KB, 128KB, 256KB, 512KB etc is supported. New RPC commands like "create pvol", "destroy pvol" and "get pvols" are introduced to configure pooled volume dynamically in a running SPDK system.
Please find attached more information in a ppt with system end to end testing done on this module in SPDK stack in SPDK multi-core environment.
Please find below gerrithub review details:
https://review.gerrithub.io/#/c/spdk/spdk/+/410484/


Regards,
Kunal

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 12424 bytes --]

Re: [SPDK] SPDK pooled volume module in bdev

[Senthil Kumar Veluswamy]

[-- Attachment #1: Type: text/plain, Size: 2003 bytes --]

Hi Kunal,
     I'm trying understand the pVol architecture and have few questions.


1.      How this pvol model co-exists with vbdev that is present today?

2.      If I understand correctly, vbdev also allows to concatenate more than one base physical bdevs and used for creating logical volume. So shall we say pvol is an extension of vbdev?

3.      It would be good if you could modify your "Pooled volume" PPT slides to show how vbdev fits in there.

Regards,
Senthil Kumar V.

From: SPDK <spdk-bounces(a)lists.01.org> On Behalf Of Luse, Paul E
Sent: Friday, May 18, 2018 12:15 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] SPDK pooled volume module in bdev

Excellent, thanks Kunal!  There's a lot to look at here and with Fenggang's aggregation patch as well.  I encourage everyone to take a look at both (and I promise I will soon too after post-summit catch-up stuff)

:)


From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sablok, Kunal
Sent: Sunday, May 13, 2018 9:51 AM
To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
Subject: [SPDK] SPDK pooled volume module in bdev

Hi,
SPDK pooled volume (PVOL) module is a new bdev module which is responsible for striping various NVMe devices and expose the pooled volume to bdev layer which would enhance the performance and capacity. It can support theoretically 255 base devices which can be easily enhanced if required (currently it is being tested max upto 8 base devices). Multiple strip sizes like 32KB, 64KB, 128KB, 256KB, 512KB etc is supported. New RPC commands like "create pvol", "destroy pvol" and "get pvols" are introduced to configure pooled volume dynamically in a running SPDK system.
Please find attached more information in a ppt with system end to end testing done on this module in SPDK stack in SPDK multi-core environment.
Please find below gerrithub review details:
https://review.gerrithub.io/#/c/spdk/spdk/+/410484/


Regards,
Kunal

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 8885 bytes --]

Re: [SPDK] SPDK pooled volume module in bdev

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 1253 bytes --]

Excellent, thanks Kunal!  There's a lot to look at here and with Fenggang's aggregation patch as well.  I encourage everyone to take a look at both (and I promise I will soon too after post-summit catch-up stuff)

:)


From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sablok, Kunal
Sent: Sunday, May 13, 2018 9:51 AM
To: spdk(a)lists.01.org
Subject: [SPDK] SPDK pooled volume module in bdev

Hi,
SPDK pooled volume (PVOL) module is a new bdev module which is responsible for striping various NVMe devices and expose the pooled volume to bdev layer which would enhance the performance and capacity. It can support theoretically 255 base devices which can be easily enhanced if required (currently it is being tested max upto 8 base devices). Multiple strip sizes like 32KB, 64KB, 128KB, 256KB, 512KB etc is supported. New RPC commands like "create pvol", "destroy pvol" and "get pvols" are introduced to configure pooled volume dynamically in a running SPDK system.
Please find attached more information in a ppt with system end to end testing done on this module in SPDK stack in SPDK multi-core environment.
Please find below gerrithub review details:
https://review.gerrithub.io/#/c/spdk/spdk/+/410484/


Regards,
Kunal

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 4370 bytes --]