Re: [SPDK] About per client QoS (IOPS rate limiting)

Re: [SPDK] About per client QoS (IOPS rate limiting)

[Cao, Gang]

[-- Attachment #1: Type: text/plain, Size: 10083 bytes --]

Thanks Paul on your comments.

Maybe here we are talking about to rename or having a new vbdev module is to make it  “generic vbdev shared code”. If not, current three Passthru vbdev patches are already working now.

Would like to hear other voice?

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Sunday, April 15, 2018 4:44 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Thanks Gang. Yeah, that’s what I thought.  Personally I think these are fine for the passthru example, they don’t’ directly tie to QoS and especially the first two were things that were going to need to go into the example at some point anywhere  If we duplicate this and rename things we’ve just got 2 vbdev modules to maintain.  If we wanted a QoS ‘do nothing’ module either because (a) we think it’s going to grow w/QoS specific stuff anyway or because (b) we want it to use more common code like the other part* modules that’s a different story.  However, I’d consider renaming if not slightly refactoring stuff in part.c to see if we can make it much more “generic vbdev shared code” as opposed to “shared code between vbdev partition related modules plus a few others that have nothing to do with part stuff” ☺

I’m happy to address that last suggestion w/a patch if desired...

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Friday, April 13, 2018 6:51 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

For the outstanding three (passthru) vbdev related patches, the first two is for the RPC method. The very first one is to have a common function to support the later RPC call. The last one may have some relationship to QoS through the support of vbdev on shared bdev (1 read/write vbdev and N-1 read-only vbdev).

https://review.gerrithub.io/#/c/406888/     vbdev: update and have a create_passthru_disk function
https://review.gerrithub.io/#/c/406891/     vbdev: add the construct_passthru_bdev RPC method
https://review.gerrithub.io/#/c/406977/     vbdev: make the passthru vbdev share the same underlying bdev

As mentioned, the QoS is implemented in the common way and on the common bdev, so there is no direct changes to have QoS stuff on the vbdev here.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, April 13, 2018 10:49 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Yeah I was about to add… from the patches I looked at so far there was no QoS in them, they just add RPC support which I think is very good for the example.  I haven’t looked in the last two days, but what specific functionality is required in the vbdev module for QoS, from the answer to my question below it didn’t sound like there was anything.

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Friday, April 13, 2018 1:40 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Sounds good to me.

Another question is that do we consider to add the RPC support to the “example” module as the example on using RPC to construct the vbdev?

Or just leave it in the current state.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Friday, April 13, 2018 3:52 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Had an OOB chat with Nate on this – what if we rename Paul’s module to “example”, and reappropriate the “passthru” name for this new module that we are proposing?

-Jim


From: James Harris <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>>
Date: Thursday, April 12, 2018 at 11:11 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

I agree that this should probably go into a new vbdev module.  We do want to keep the existing pass-through module as the example which is as simple as possible.  Then if we find we need other changes to support Gang’s per-client use case, we don’t have to worry about further complicating the example.

The question is then what to name it.  I don’t think “qos” is the best name – since this module isn’t actually doing any QoS work – it just enables per-client QoS.  All of the QoS work is done in the common bdev layer – not a bdev module.  Maybe something along the lines of “shared_passthru”?  I don’t particularly care for that name either – if anyone has another idea, please advise.

This new module can also utilize the spdk_bdev_part API to significantly simplify the code.  Paul purposely did not use spdk_bdev_part in the passthru example because he wanted to explicitly show how everything should be implemented in the example bdev module itself.  But for this module, since it’s not an example module, we don’t have that same concern.

-Jim


From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of "Kariuki, John K" <john.k.kariuki(a)intel.com<mailto:john.k.kariuki(a)intel.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Thursday, April 12, 2018 at 10:01 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Gang/Paul
Does this functionality belong in the passthru bdev or should it be implemented in another vbdev called something like “qos”? My understanding of the passthru bdev is that it’s a template that anyone looking to build a vbdev can use as a starting point. It has all the functions needed to create an SPDK module stub out so that someone can pick it up, make a copy and a few modifications and then they are off to the races implementing their new virtual bdev

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Tuesday, April 10, 2018 6:05 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Yes. It’s a good extension to couple with the vbdev and bdev to meet the client’s QoS goal. For a client, from my understanding, a “device” will always be linked through its own specific “connection”. The per client (or the per connection) QoS can be also achieved by the per “device”.

The other comment is that to move this QoS rate limiting from the client side to the target side. Client can has its own QoS for the connection before submitting I/Os and with this work, it can also let the target side to do the similar thing and may not need a QoS at client side.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Tuesday, April 10, 2018 11:56 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Hi Gang,

This is pretty cool.  So essentially the PT vbdev is just a per-client hook for QoS?

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Monday, April 9, 2018 9:03 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] About per client QoS (IOPS rate limiting)

Hi all,

There has been some work done to have SPDK support the QoS per bdev which means that each bdev has its assigned IOPS rate limiting. This QoS functionality is below the storage protocols like NVMe-oF, iSCSI and so on.

At the time of last PRC SPDK summit, there are some questions on the per client QoS (i.e., the NVMe-oF iniator / host) besides the per bdev QoS. Each client like NVMe-oF initiator can see its own IOPS rate limiting not matter whether the connected device is exclusive or shared at the target side.

Based on Paul’s great stuff of the Passthru vbdev, I’ve updated through below three patches to achieve the per client QoS. Each client sees its own passthru vbdev and QoS assigned on this vbdev. These vbdev shares the same bdev. Currently it only supports shareable and read only vbdev on same bdev.

https://review.gerrithub.io/#/c/406888/     vbdev: update and have a create_passthru_disk function
https://review.gerrithub.io/#/c/406891/     vbdev: add the construct_passthru_bdev RPC method
https://review.gerrithub.io/#/c/406977/     vbdev: make the passthru vbdev share the same underlying bdev

The possible usage as following:
[Take NVMe-oF target as example]

1.  In the nvmf.conf file,
# Configure the Passthru vbdev on same bdev like Malloc0
[Passthru]
PT Malloc0 PT0
PT Malloc0 PT1

# The subsystem is configured with shareable bdev by Passthru vbdevs
[Subsystem2]
  NQN nqn.2016-06.io.spdk:cnode2
  Listen RDMA 192.168.2.21:4420
  AllowAnyHost No
  Host nqn.2016-06.io.spdk:init
  SN SPDK00000000000002
  Namespace PT0   >>>> This can be connected by one client
  Namespace PT1   >>>> This can be connected by another client

# Assign different QoS IOPS limiting on Passthru vbdevs
[QoS]
Limit_IOPS PT0 20000
Limit_IOPS PT1 30000

2. Use RPC method to add the Passthru vbdev at runtime
a. python ./scripts/rpc.py construct_passthru_bdev Malloc0 PT3
b. python ./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2 PT3
c. python ./scripts/rpc.py enable_bdev_qos PT3 20000

If there is any concern or comment, please feel free to let me know.

Thanks,
Gang

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 30956 bytes --]

Re: [SPDK] About per client QoS (IOPS rate limiting)

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 12189 bytes --]

The patch listed below isn’t necessarily QoS-specific, but it may start to complicate the passthrough example module in ways that make it less clear what a new module actually needs to implement:

https://review.gerrithub.io/#/c/406977/     vbdev: make the passthru vbdev share the same underlying bdev

I’ve posted some other comments on this patch which would need to be addressed if we decide to just use the example module for this per-client QoS use case.

Paul – you may want to take a look at the comments I made.  If you’re OK with just making changes to the existing example module rather than creating a new module, then I’m OK with it too.

-Jim


From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Paul E Luse <paul.e.luse(a)intel.com>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Monday, April 16, 2018 at 7:07 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Gang,

Let me try and simplify the couple of points I’ve been trying to make and then you guys can collectively decide how to proceed ☺


·         I think it’s fine that the passthru module be used by the per client QoS use case provided there are no QoS specific changes required. I believe all 3 patches are good generic supplements to the passthru example module

·         If for whatever reason we move forward with a new vbdev for the per client Qos module and leverage common code in part.c, as Jim suggested in one of the email chains, I would suggest that we rename appropriate elements in part.c (including the filename) to reflect that fact that it would no longer contain common code used only by partition related vbdevs as the new QoS vbdev would, of course, not be partition related.

Hope that makes sense.

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Monday, April 16, 2018 2:40 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Thanks Paul on your comments.

Maybe here we are talking about to rename or having a new vbdev module is to make it  “generic vbdev shared code”. If not, current three Passthru vbdev patches are already working now.

Would like to hear other voice?

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Sunday, April 15, 2018 4:44 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Thanks Gang. Yeah, that’s what I thought.  Personally I think these are fine for the passthru example, they don’t’ directly tie to QoS and especially the first two were things that were going to need to go into the example at some point anywhere  If we duplicate this and rename things we’ve just got 2 vbdev modules to maintain.  If we wanted a QoS ‘do nothing’ module either because (a) we think it’s going to grow w/QoS specific stuff anyway or because (b) we want it to use more common code like the other part* modules that’s a different story.  However, I’d consider renaming if not slightly refactoring stuff in part.c to see if we can make it much more “generic vbdev shared code” as opposed to “shared code between vbdev partition related modules plus a few others that have nothing to do with part stuff” ☺

I’m happy to address that last suggestion w/a patch if desired...

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Friday, April 13, 2018 6:51 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

For the outstanding three (passthru) vbdev related patches, the first two is for the RPC method. The very first one is to have a common function to support the later RPC call. The last one may have some relationship to QoS through the support of vbdev on shared bdev (1 read/write vbdev and N-1 read-only vbdev).

https://review.gerrithub.io/#/c/406888/     vbdev: update and have a create_passthru_disk function
https://review.gerrithub.io/#/c/406891/     vbdev: add the construct_passthru_bdev RPC method
https://review.gerrithub.io/#/c/406977/     vbdev: make the passthru vbdev share the same underlying bdev

As mentioned, the QoS is implemented in the common way and on the common bdev, so there is no direct changes to have QoS stuff on the vbdev here.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, April 13, 2018 10:49 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Yeah I was about to add… from the patches I looked at so far there was no QoS in them, they just add RPC support which I think is very good for the example.  I haven’t looked in the last two days, but what specific functionality is required in the vbdev module for QoS, from the answer to my question below it didn’t sound like there was anything.

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Friday, April 13, 2018 1:40 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Sounds good to me.

Another question is that do we consider to add the RPC support to the “example” module as the example on using RPC to construct the vbdev?

Or just leave it in the current state.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Friday, April 13, 2018 3:52 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Had an OOB chat with Nate on this – what if we rename Paul’s module to “example”, and reappropriate the “passthru” name for this new module that we are proposing?

-Jim


From: James Harris <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>>
Date: Thursday, April 12, 2018 at 11:11 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

I agree that this should probably go into a new vbdev module.  We do want to keep the existing pass-through module as the example which is as simple as possible.  Then if we find we need other changes to support Gang’s per-client use case, we don’t have to worry about further complicating the example.

The question is then what to name it.  I don’t think “qos” is the best name – since this module isn’t actually doing any QoS work – it just enables per-client QoS.  All of the QoS work is done in the common bdev layer – not a bdev module.  Maybe something along the lines of “shared_passthru”?  I don’t particularly care for that name either – if anyone has another idea, please advise.

This new module can also utilize the spdk_bdev_part API to significantly simplify the code.  Paul purposely did not use spdk_bdev_part in the passthru example because he wanted to explicitly show how everything should be implemented in the example bdev module itself.  But for this module, since it’s not an example module, we don’t have that same concern.

-Jim


From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of "Kariuki, John K" <john.k.kariuki(a)intel.com<mailto:john.k.kariuki(a)intel.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Thursday, April 12, 2018 at 10:01 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Gang/Paul
Does this functionality belong in the passthru bdev or should it be implemented in another vbdev called something like “qos”? My understanding of the passthru bdev is that it’s a template that anyone looking to build a vbdev can use as a starting point. It has all the functions needed to create an SPDK module stub out so that someone can pick it up, make a copy and a few modifications and then they are off to the races implementing their new virtual bdev

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Tuesday, April 10, 2018 6:05 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Yes. It’s a good extension to couple with the vbdev and bdev to meet the client’s QoS goal. For a client, from my understanding, a “device” will always be linked through its own specific “connection”. The per client (or the per connection) QoS can be also achieved by the per “device”.

The other comment is that to move this QoS rate limiting from the client side to the target side. Client can has its own QoS for the connection before submitting I/Os and with this work, it can also let the target side to do the similar thing and may not need a QoS at client side.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Tuesday, April 10, 2018 11:56 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Hi Gang,

This is pretty cool.  So essentially the PT vbdev is just a per-client hook for QoS?

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Monday, April 9, 2018 9:03 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] About per client QoS (IOPS rate limiting)

Hi all,

There has been some work done to have SPDK support the QoS per bdev which means that each bdev has its assigned IOPS rate limiting. This QoS functionality is below the storage protocols like NVMe-oF, iSCSI and so on.

At the time of last PRC SPDK summit, there are some questions on the per client QoS (i.e., the NVMe-oF iniator / host) besides the per bdev QoS. Each client like NVMe-oF initiator can see its own IOPS rate limiting not matter whether the connected device is exclusive or shared at the target side.

Based on Paul’s great stuff of the Passthru vbdev, I’ve updated through below three patches to achieve the per client QoS. Each client sees its own passthru vbdev and QoS assigned on this vbdev. These vbdev shares the same bdev. Currently it only supports shareable and read only vbdev on same bdev.

https://review.gerrithub.io/#/c/406888/     vbdev: update and have a create_passthru_disk function
https://review.gerrithub.io/#/c/406891/     vbdev: add the construct_passthru_bdev RPC method
https://review.gerrithub.io/#/c/406977/     vbdev: make the passthru vbdev share the same underlying bdev

The possible usage as following:
[Take NVMe-oF target as example]

1.  In the nvmf.conf file,
# Configure the Passthru vbdev on same bdev like Malloc0
[Passthru]
PT Malloc0 PT0
PT Malloc0 PT1

# The subsystem is configured with shareable bdev by Passthru vbdevs
[Subsystem2]
  NQN nqn.2016-06.io.spdk:cnode2
  Listen RDMA 192.168.2.21:4420
  AllowAnyHost No
  Host nqn.2016-06.io.spdk:init
  SN SPDK00000000000002
  Namespace PT0   >>>> This can be connected by one client
  Namespace PT1   >>>> This can be connected by another client

# Assign different QoS IOPS limiting on Passthru vbdevs
[QoS]
Limit_IOPS PT0 20000
Limit_IOPS PT1 30000

2. Use RPC method to add the Passthru vbdev at runtime
a. python ./scripts/rpc.py construct_passthru_bdev Malloc0 PT3
b. python ./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2 PT3
c. python ./scripts/rpc.py enable_bdev_qos PT3 20000

If there is any concern or comment, please feel free to let me know.

Thanks,
Gang

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 36622 bytes --]

Re: [SPDK] About per client QoS (IOPS rate limiting)

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 11176 bytes --]

Gang,

Let me try and simplify the couple of points I’ve been trying to make and then you guys can collectively decide how to proceed ☺


·        I think it’s fine that the passthru module be used by the per client QoS use case provided there are no QoS specific changes required. I believe all 3 patches are good generic supplements to the passthru example module

·        If for whatever reason we move forward with a new vbdev for the per client Qos module and leverage common code in part.c, as Jim suggested in one of the email chains, I would suggest that we rename appropriate elements in part.c (including the filename) to reflect that fact that it would no longer contain common code used only by partition related vbdevs as the new QoS vbdev would, of course, not be partition related.

Hope that makes sense.

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Monday, April 16, 2018 2:40 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Thanks Paul on your comments.

Maybe here we are talking about to rename or having a new vbdev module is to make it  “generic vbdev shared code”. If not, current three Passthru vbdev patches are already working now.

Would like to hear other voice?

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Sunday, April 15, 2018 4:44 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Thanks Gang. Yeah, that’s what I thought.  Personally I think these are fine for the passthru example, they don’t’ directly tie to QoS and especially the first two were things that were going to need to go into the example at some point anywhere  If we duplicate this and rename things we’ve just got 2 vbdev modules to maintain.  If we wanted a QoS ‘do nothing’ module either because (a) we think it’s going to grow w/QoS specific stuff anyway or because (b) we want it to use more common code like the other part* modules that’s a different story.  However, I’d consider renaming if not slightly refactoring stuff in part.c to see if we can make it much more “generic vbdev shared code” as opposed to “shared code between vbdev partition related modules plus a few others that have nothing to do with part stuff” ☺

I’m happy to address that last suggestion w/a patch if desired...

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Friday, April 13, 2018 6:51 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

For the outstanding three (passthru) vbdev related patches, the first two is for the RPC method. The very first one is to have a common function to support the later RPC call. The last one may have some relationship to QoS through the support of vbdev on shared bdev (1 read/write vbdev and N-1 read-only vbdev).

https://review.gerrithub.io/#/c/406888/     vbdev: update and have a create_passthru_disk function
https://review.gerrithub.io/#/c/406891/     vbdev: add the construct_passthru_bdev RPC method
https://review.gerrithub.io/#/c/406977/     vbdev: make the passthru vbdev share the same underlying bdev

As mentioned, the QoS is implemented in the common way and on the common bdev, so there is no direct changes to have QoS stuff on the vbdev here.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, April 13, 2018 10:49 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Yeah I was about to add… from the patches I looked at so far there was no QoS in them, they just add RPC support which I think is very good for the example.  I haven’t looked in the last two days, but what specific functionality is required in the vbdev module for QoS, from the answer to my question below it didn’t sound like there was anything.

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Friday, April 13, 2018 1:40 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Sounds good to me.

Another question is that do we consider to add the RPC support to the “example” module as the example on using RPC to construct the vbdev?

Or just leave it in the current state.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Friday, April 13, 2018 3:52 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Had an OOB chat with Nate on this – what if we rename Paul’s module to “example”, and reappropriate the “passthru” name for this new module that we are proposing?

-Jim


From: James Harris <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>>
Date: Thursday, April 12, 2018 at 11:11 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

I agree that this should probably go into a new vbdev module.  We do want to keep the existing pass-through module as the example which is as simple as possible.  Then if we find we need other changes to support Gang’s per-client use case, we don’t have to worry about further complicating the example.

The question is then what to name it.  I don’t think “qos” is the best name – since this module isn’t actually doing any QoS work – it just enables per-client QoS.  All of the QoS work is done in the common bdev layer – not a bdev module.  Maybe something along the lines of “shared_passthru”?  I don’t particularly care for that name either – if anyone has another idea, please advise.

This new module can also utilize the spdk_bdev_part API to significantly simplify the code.  Paul purposely did not use spdk_bdev_part in the passthru example because he wanted to explicitly show how everything should be implemented in the example bdev module itself.  But for this module, since it’s not an example module, we don’t have that same concern.

-Jim


From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of "Kariuki, John K" <john.k.kariuki(a)intel.com<mailto:john.k.kariuki(a)intel.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Thursday, April 12, 2018 at 10:01 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Gang/Paul
Does this functionality belong in the passthru bdev or should it be implemented in another vbdev called something like “qos”? My understanding of the passthru bdev is that it’s a template that anyone looking to build a vbdev can use as a starting point. It has all the functions needed to create an SPDK module stub out so that someone can pick it up, make a copy and a few modifications and then they are off to the races implementing their new virtual bdev

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Tuesday, April 10, 2018 6:05 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Yes. It’s a good extension to couple with the vbdev and bdev to meet the client’s QoS goal. For a client, from my understanding, a “device” will always be linked through its own specific “connection”. The per client (or the per connection) QoS can be also achieved by the per “device”.

The other comment is that to move this QoS rate limiting from the client side to the target side. Client can has its own QoS for the connection before submitting I/Os and with this work, it can also let the target side to do the similar thing and may not need a QoS at client side.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Tuesday, April 10, 2018 11:56 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Hi Gang,

This is pretty cool.  So essentially the PT vbdev is just a per-client hook for QoS?

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Monday, April 9, 2018 9:03 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] About per client QoS (IOPS rate limiting)

Hi all,

There has been some work done to have SPDK support the QoS per bdev which means that each bdev has its assigned IOPS rate limiting. This QoS functionality is below the storage protocols like NVMe-oF, iSCSI and so on.

At the time of last PRC SPDK summit, there are some questions on the per client QoS (i.e., the NVMe-oF iniator / host) besides the per bdev QoS. Each client like NVMe-oF initiator can see its own IOPS rate limiting not matter whether the connected device is exclusive or shared at the target side.

Based on Paul’s great stuff of the Passthru vbdev, I’ve updated through below three patches to achieve the per client QoS. Each client sees its own passthru vbdev and QoS assigned on this vbdev. These vbdev shares the same bdev. Currently it only supports shareable and read only vbdev on same bdev.

https://review.gerrithub.io/#/c/406888/     vbdev: update and have a create_passthru_disk function
https://review.gerrithub.io/#/c/406891/     vbdev: add the construct_passthru_bdev RPC method
https://review.gerrithub.io/#/c/406977/     vbdev: make the passthru vbdev share the same underlying bdev

The possible usage as following:
[Take NVMe-oF target as example]

1.  In the nvmf.conf file,
# Configure the Passthru vbdev on same bdev like Malloc0
[Passthru]
PT Malloc0 PT0
PT Malloc0 PT1

# The subsystem is configured with shareable bdev by Passthru vbdevs
[Subsystem2]
  NQN nqn.2016-06.io.spdk:cnode2
  Listen RDMA 192.168.2.21:4420
  AllowAnyHost No
  Host nqn.2016-06.io.spdk:init
  SN SPDK00000000000002
  Namespace PT0   >>>> This can be connected by one client
  Namespace PT1   >>>> This can be connected by another client

# Assign different QoS IOPS limiting on Passthru vbdevs
[QoS]
Limit_IOPS PT0 20000
Limit_IOPS PT1 30000

2. Use RPC method to add the Passthru vbdev at runtime
a. python ./scripts/rpc.py construct_passthru_bdev Malloc0 PT3
b. python ./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2 PT3
c. python ./scripts/rpc.py enable_bdev_qos PT3 20000

If there is any concern or comment, please feel free to let me know.

Thanks,
Gang

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 39486 bytes --]

Re: [SPDK] About per client QoS (IOPS rate limiting)

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 7844 bytes --]

That’s all that the first 2 patches on this thread do – add RPC.  The third just adds a generic capability for the vbdev to have multiple underlying bdevs with one R/W and the rest read only. That too seems like a reasonable thing to show in the example as well.

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Friday, April 13, 2018 12:52 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

I think adding an RPC to the example passthrough module is a good idea.  A separate patch obviously from everything else we’re talking about in this thread though.

-Jim


From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of "Cao, Gang" <gang.cao(a)intel.com<mailto:gang.cao(a)intel.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Friday, April 13, 2018 at 1:40 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Sounds good to me.

Another question is that do we consider to add the RPC support to the “example” module as the example on using RPC to construct the vbdev?

Or just leave it in the current state.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Friday, April 13, 2018 3:52 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Had an OOB chat with Nate on this – what if we rename Paul’s module to “example”, and reappropriate the “passthru” name for this new module that we are proposing?

-Jim


From: James Harris <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>>
Date: Thursday, April 12, 2018 at 11:11 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

I agree that this should probably go into a new vbdev module.  We do want to keep the existing pass-through module as the example which is as simple as possible.  Then if we find we need other changes to support Gang’s per-client use case, we don’t have to worry about further complicating the example.

The question is then what to name it.  I don’t think “qos” is the best name – since this module isn’t actually doing any QoS work – it just enables per-client QoS.  All of the QoS work is done in the common bdev layer – not a bdev module.  Maybe something along the lines of “shared_passthru”?  I don’t particularly care for that name either – if anyone has another idea, please advise.

This new module can also utilize the spdk_bdev_part API to significantly simplify the code.  Paul purposely did not use spdk_bdev_part in the passthru example because he wanted to explicitly show how everything should be implemented in the example bdev module itself.  But for this module, since it’s not an example module, we don’t have that same concern.

-Jim


From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of "Kariuki, John K" <john.k.kariuki(a)intel.com<mailto:john.k.kariuki(a)intel.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Thursday, April 12, 2018 at 10:01 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Gang/Paul
Does this functionality belong in the passthru bdev or should it be implemented in another vbdev called something like “qos”? My understanding of the passthru bdev is that it’s a template that anyone looking to build a vbdev can use as a starting point. It has all the functions needed to create an SPDK module stub out so that someone can pick it up, make a copy and a few modifications and then they are off to the races implementing their new virtual bdev

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Tuesday, April 10, 2018 6:05 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Yes. It’s a good extension to couple with the vbdev and bdev to meet the client’s QoS goal. For a client, from my understanding, a “device” will always be linked through its own specific “connection”. The per client (or the per connection) QoS can be also achieved by the per “device”.

The other comment is that to move this QoS rate limiting from the client side to the target side. Client can has its own QoS for the connection before submitting I/Os and with this work, it can also let the target side to do the similar thing and may not need a QoS at client side.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Tuesday, April 10, 2018 11:56 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Hi Gang,

This is pretty cool.  So essentially the PT vbdev is just a per-client hook for QoS?

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Monday, April 9, 2018 9:03 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] About per client QoS (IOPS rate limiting)

Hi all,

There has been some work done to have SPDK support the QoS per bdev which means that each bdev has its assigned IOPS rate limiting. This QoS functionality is below the storage protocols like NVMe-oF, iSCSI and so on.

At the time of last PRC SPDK summit, there are some questions on the per client QoS (i.e., the NVMe-oF iniator / host) besides the per bdev QoS. Each client like NVMe-oF initiator can see its own IOPS rate limiting not matter whether the connected device is exclusive or shared at the target side.

Based on Paul’s great stuff of the Passthru vbdev, I’ve updated through below three patches to achieve the per client QoS. Each client sees its own passthru vbdev and QoS assigned on this vbdev. These vbdev shares the same bdev. Currently it only supports shareable and read only vbdev on same bdev.

https://review.gerrithub.io/#/c/406888/     vbdev: update and have a create_passthru_disk function
https://review.gerrithub.io/#/c/406891/     vbdev: add the construct_passthru_bdev RPC method
https://review.gerrithub.io/#/c/406977/     vbdev: make the passthru vbdev share the same underlying bdev

The possible usage as following:
[Take NVMe-oF target as example]

1.  In the nvmf.conf file,
# Configure the Passthru vbdev on same bdev like Malloc0
[Passthru]
PT Malloc0 PT0
PT Malloc0 PT1

# The subsystem is configured with shareable bdev by Passthru vbdevs
[Subsystem2]
  NQN nqn.2016-06.io.spdk:cnode2
  Listen RDMA 192.168.2.21:4420
  AllowAnyHost No
  Host nqn.2016-06.io.spdk:init
  SN SPDK00000000000002
  Namespace PT0   >>>> This can be connected by one client
  Namespace PT1   >>>> This can be connected by another client

# Assign different QoS IOPS limiting on Passthru vbdevs
[QoS]
Limit_IOPS PT0 20000
Limit_IOPS PT1 30000

2. Use RPC method to add the Passthru vbdev at runtime
a. python ./scripts/rpc.py construct_passthru_bdev Malloc0 PT3
b. python ./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2 PT3
c. python ./scripts/rpc.py enable_bdev_qos PT3 20000

If there is any concern or comment, please feel free to let me know.

Thanks,
Gang

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 22403 bytes --]

Re: [SPDK] About per client QoS (IOPS rate limiting)

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 9537 bytes --]

Thanks Gang. Yeah, that’s what I thought.  Personally I think these are fine for the passthru example, they don’t’ directly tie to QoS and especially the first two were things that were going to need to go into the example at some point anywhere  If we duplicate this and rename things we’ve just got 2 vbdev modules to maintain.  If we wanted a QoS ‘do nothing’ module either because (a) we think it’s going to grow w/QoS specific stuff anyway or because (b) we want it to use more common code like the other part* modules that’s a different story.  However, I’d consider renaming if not slightly refactoring stuff in part.c to see if we can make it much more “generic vbdev shared code” as opposed to “shared code between vbdev partition related modules plus a few others that have nothing to do with part stuff” ☺

I’m happy to address that last suggestion w/a patch if desired...

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Friday, April 13, 2018 6:51 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

For the outstanding three (passthru) vbdev related patches, the first two is for the RPC method. The very first one is to have a common function to support the later RPC call. The last one may have some relationship to QoS through the support of vbdev on shared bdev (1 read/write vbdev and N-1 read-only vbdev).

https://review.gerrithub.io/#/c/406888/     vbdev: update and have a create_passthru_disk function
https://review.gerrithub.io/#/c/406891/     vbdev: add the construct_passthru_bdev RPC method
https://review.gerrithub.io/#/c/406977/     vbdev: make the passthru vbdev share the same underlying bdev

As mentioned, the QoS is implemented in the common way and on the common bdev, so there is no direct changes to have QoS stuff on the vbdev here.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, April 13, 2018 10:49 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Yeah I was about to add… from the patches I looked at so far there was no QoS in them, they just add RPC support which I think is very good for the example.  I haven’t looked in the last two days, but what specific functionality is required in the vbdev module for QoS, from the answer to my question below it didn’t sound like there was anything.

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Friday, April 13, 2018 1:40 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Sounds good to me.

Another question is that do we consider to add the RPC support to the “example” module as the example on using RPC to construct the vbdev?

Or just leave it in the current state.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Friday, April 13, 2018 3:52 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Had an OOB chat with Nate on this – what if we rename Paul’s module to “example”, and reappropriate the “passthru” name for this new module that we are proposing?

-Jim


From: James Harris <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>>
Date: Thursday, April 12, 2018 at 11:11 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

I agree that this should probably go into a new vbdev module.  We do want to keep the existing pass-through module as the example which is as simple as possible.  Then if we find we need other changes to support Gang’s per-client use case, we don’t have to worry about further complicating the example.

The question is then what to name it.  I don’t think “qos” is the best name – since this module isn’t actually doing any QoS work – it just enables per-client QoS.  All of the QoS work is done in the common bdev layer – not a bdev module.  Maybe something along the lines of “shared_passthru”?  I don’t particularly care for that name either – if anyone has another idea, please advise.

This new module can also utilize the spdk_bdev_part API to significantly simplify the code.  Paul purposely did not use spdk_bdev_part in the passthru example because he wanted to explicitly show how everything should be implemented in the example bdev module itself.  But for this module, since it’s not an example module, we don’t have that same concern.

-Jim


From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of "Kariuki, John K" <john.k.kariuki(a)intel.com<mailto:john.k.kariuki(a)intel.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Thursday, April 12, 2018 at 10:01 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Gang/Paul
Does this functionality belong in the passthru bdev or should it be implemented in another vbdev called something like “qos”? My understanding of the passthru bdev is that it’s a template that anyone looking to build a vbdev can use as a starting point. It has all the functions needed to create an SPDK module stub out so that someone can pick it up, make a copy and a few modifications and then they are off to the races implementing their new virtual bdev

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Tuesday, April 10, 2018 6:05 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Yes. It’s a good extension to couple with the vbdev and bdev to meet the client’s QoS goal. For a client, from my understanding, a “device” will always be linked through its own specific “connection”. The per client (or the per connection) QoS can be also achieved by the per “device”.

The other comment is that to move this QoS rate limiting from the client side to the target side. Client can has its own QoS for the connection before submitting I/Os and with this work, it can also let the target side to do the similar thing and may not need a QoS at client side.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Tuesday, April 10, 2018 11:56 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Hi Gang,

This is pretty cool.  So essentially the PT vbdev is just a per-client hook for QoS?

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Monday, April 9, 2018 9:03 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] About per client QoS (IOPS rate limiting)

Hi all,

There has been some work done to have SPDK support the QoS per bdev which means that each bdev has its assigned IOPS rate limiting. This QoS functionality is below the storage protocols like NVMe-oF, iSCSI and so on.

At the time of last PRC SPDK summit, there are some questions on the per client QoS (i.e., the NVMe-oF iniator / host) besides the per bdev QoS. Each client like NVMe-oF initiator can see its own IOPS rate limiting not matter whether the connected device is exclusive or shared at the target side.

Based on Paul’s great stuff of the Passthru vbdev, I’ve updated through below three patches to achieve the per client QoS. Each client sees its own passthru vbdev and QoS assigned on this vbdev. These vbdev shares the same bdev. Currently it only supports shareable and read only vbdev on same bdev.

https://review.gerrithub.io/#/c/406888/     vbdev: update and have a create_passthru_disk function
https://review.gerrithub.io/#/c/406891/     vbdev: add the construct_passthru_bdev RPC method
https://review.gerrithub.io/#/c/406977/     vbdev: make the passthru vbdev share the same underlying bdev

The possible usage as following:
[Take NVMe-oF target as example]

1.  In the nvmf.conf file,
# Configure the Passthru vbdev on same bdev like Malloc0
[Passthru]
PT Malloc0 PT0
PT Malloc0 PT1

# The subsystem is configured with shareable bdev by Passthru vbdevs
[Subsystem2]
  NQN nqn.2016-06.io.spdk:cnode2
  Listen RDMA 192.168.2.21:4420
  AllowAnyHost No
  Host nqn.2016-06.io.spdk:init
  SN SPDK00000000000002
  Namespace PT0   >>>> This can be connected by one client
  Namespace PT1   >>>> This can be connected by another client

# Assign different QoS IOPS limiting on Passthru vbdevs
[QoS]
Limit_IOPS PT0 20000
Limit_IOPS PT1 30000

2. Use RPC method to add the Passthru vbdev at runtime
a. python ./scripts/rpc.py construct_passthru_bdev Malloc0 PT3
b. python ./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2 PT3
c. python ./scripts/rpc.py enable_bdev_qos PT3 20000

If there is any concern or comment, please feel free to let me know.

Thanks,
Gang

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 31422 bytes --]

Re: [SPDK] About per client QoS (IOPS rate limiting)

[Cao, Gang]

[-- Attachment #1: Type: text/plain, Size: 8352 bytes --]

For the outstanding three (passthru) vbdev related patches, the first two is for the RPC method. The very first one is to have a common function to support the later RPC call. The last one may have some relationship to QoS through the support of vbdev on shared bdev (1 read/write vbdev and N-1 read-only vbdev).

https://review.gerrithub.io/#/c/406888/     vbdev: update and have a create_passthru_disk function
https://review.gerrithub.io/#/c/406891/     vbdev: add the construct_passthru_bdev RPC method
https://review.gerrithub.io/#/c/406977/     vbdev: make the passthru vbdev share the same underlying bdev

As mentioned, the QoS is implemented in the common way and on the common bdev, so there is no direct changes to have QoS stuff on the vbdev here.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, April 13, 2018 10:49 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Yeah I was about to add… from the patches I looked at so far there was no QoS in them, they just add RPC support which I think is very good for the example.  I haven’t looked in the last two days, but what specific functionality is required in the vbdev module for QoS, from the answer to my question below it didn’t sound like there was anything.

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Friday, April 13, 2018 1:40 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Sounds good to me.

Another question is that do we consider to add the RPC support to the “example” module as the example on using RPC to construct the vbdev?

Or just leave it in the current state.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Friday, April 13, 2018 3:52 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Had an OOB chat with Nate on this – what if we rename Paul’s module to “example”, and reappropriate the “passthru” name for this new module that we are proposing?

-Jim


From: James Harris <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>>
Date: Thursday, April 12, 2018 at 11:11 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

I agree that this should probably go into a new vbdev module.  We do want to keep the existing pass-through module as the example which is as simple as possible.  Then if we find we need other changes to support Gang’s per-client use case, we don’t have to worry about further complicating the example.

The question is then what to name it.  I don’t think “qos” is the best name – since this module isn’t actually doing any QoS work – it just enables per-client QoS.  All of the QoS work is done in the common bdev layer – not a bdev module.  Maybe something along the lines of “shared_passthru”?  I don’t particularly care for that name either – if anyone has another idea, please advise.

This new module can also utilize the spdk_bdev_part API to significantly simplify the code.  Paul purposely did not use spdk_bdev_part in the passthru example because he wanted to explicitly show how everything should be implemented in the example bdev module itself.  But for this module, since it’s not an example module, we don’t have that same concern.

-Jim


From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of "Kariuki, John K" <john.k.kariuki(a)intel.com<mailto:john.k.kariuki(a)intel.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Thursday, April 12, 2018 at 10:01 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Gang/Paul
Does this functionality belong in the passthru bdev or should it be implemented in another vbdev called something like “qos”? My understanding of the passthru bdev is that it’s a template that anyone looking to build a vbdev can use as a starting point. It has all the functions needed to create an SPDK module stub out so that someone can pick it up, make a copy and a few modifications and then they are off to the races implementing their new virtual bdev

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Tuesday, April 10, 2018 6:05 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Yes. It’s a good extension to couple with the vbdev and bdev to meet the client’s QoS goal. For a client, from my understanding, a “device” will always be linked through its own specific “connection”. The per client (or the per connection) QoS can be also achieved by the per “device”.

The other comment is that to move this QoS rate limiting from the client side to the target side. Client can has its own QoS for the connection before submitting I/Os and with this work, it can also let the target side to do the similar thing and may not need a QoS at client side.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Tuesday, April 10, 2018 11:56 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Hi Gang,

This is pretty cool.  So essentially the PT vbdev is just a per-client hook for QoS?

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Monday, April 9, 2018 9:03 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] About per client QoS (IOPS rate limiting)

Hi all,

There has been some work done to have SPDK support the QoS per bdev which means that each bdev has its assigned IOPS rate limiting. This QoS functionality is below the storage protocols like NVMe-oF, iSCSI and so on.

At the time of last PRC SPDK summit, there are some questions on the per client QoS (i.e., the NVMe-oF iniator / host) besides the per bdev QoS. Each client like NVMe-oF initiator can see its own IOPS rate limiting not matter whether the connected device is exclusive or shared at the target side.

Based on Paul’s great stuff of the Passthru vbdev, I’ve updated through below three patches to achieve the per client QoS. Each client sees its own passthru vbdev and QoS assigned on this vbdev. These vbdev shares the same bdev. Currently it only supports shareable and read only vbdev on same bdev.

https://review.gerrithub.io/#/c/406888/     vbdev: update and have a create_passthru_disk function
https://review.gerrithub.io/#/c/406891/     vbdev: add the construct_passthru_bdev RPC method
https://review.gerrithub.io/#/c/406977/     vbdev: make the passthru vbdev share the same underlying bdev

The possible usage as following:
[Take NVMe-oF target as example]

1.  In the nvmf.conf file,
# Configure the Passthru vbdev on same bdev like Malloc0
[Passthru]
PT Malloc0 PT0
PT Malloc0 PT1

# The subsystem is configured with shareable bdev by Passthru vbdevs
[Subsystem2]
  NQN nqn.2016-06.io.spdk:cnode2
  Listen RDMA 192.168.2.21:4420
  AllowAnyHost No
  Host nqn.2016-06.io.spdk:init
  SN SPDK00000000000002
  Namespace PT0   >>>> This can be connected by one client
  Namespace PT1   >>>> This can be connected by another client

# Assign different QoS IOPS limiting on Passthru vbdevs
[QoS]
Limit_IOPS PT0 20000
Limit_IOPS PT1 30000

2. Use RPC method to add the Passthru vbdev at runtime
a. python ./scripts/rpc.py construct_passthru_bdev Malloc0 PT3
b. python ./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2 PT3
c. python ./scripts/rpc.py enable_bdev_qos PT3 20000

If there is any concern or comment, please feel free to let me know.

Thanks,
Gang

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 26493 bytes --]

Re: [SPDK] About per client QoS (IOPS rate limiting)

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 7167 bytes --]

I think adding an RPC to the example passthrough module is a good idea.  A separate patch obviously from everything else we’re talking about in this thread though.

-Jim


From: SPDK <spdk-bounces(a)lists.01.org> on behalf of "Cao, Gang" <gang.cao(a)intel.com>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Friday, April 13, 2018 at 1:40 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Sounds good to me.

Another question is that do we consider to add the RPC support to the “example” module as the example on using RPC to construct the vbdev?

Or just leave it in the current state.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Friday, April 13, 2018 3:52 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Had an OOB chat with Nate on this – what if we rename Paul’s module to “example”, and reappropriate the “passthru” name for this new module that we are proposing?

-Jim


From: James Harris <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>>
Date: Thursday, April 12, 2018 at 11:11 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

I agree that this should probably go into a new vbdev module.  We do want to keep the existing pass-through module as the example which is as simple as possible.  Then if we find we need other changes to support Gang’s per-client use case, we don’t have to worry about further complicating the example.

The question is then what to name it.  I don’t think “qos” is the best name – since this module isn’t actually doing any QoS work – it just enables per-client QoS.  All of the QoS work is done in the common bdev layer – not a bdev module.  Maybe something along the lines of “shared_passthru”?  I don’t particularly care for that name either – if anyone has another idea, please advise.

This new module can also utilize the spdk_bdev_part API to significantly simplify the code.  Paul purposely did not use spdk_bdev_part in the passthru example because he wanted to explicitly show how everything should be implemented in the example bdev module itself.  But for this module, since it’s not an example module, we don’t have that same concern.

-Jim


From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of "Kariuki, John K" <john.k.kariuki(a)intel.com<mailto:john.k.kariuki(a)intel.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Thursday, April 12, 2018 at 10:01 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Gang/Paul
Does this functionality belong in the passthru bdev or should it be implemented in another vbdev called something like “qos”? My understanding of the passthru bdev is that it’s a template that anyone looking to build a vbdev can use as a starting point. It has all the functions needed to create an SPDK module stub out so that someone can pick it up, make a copy and a few modifications and then they are off to the races implementing their new virtual bdev

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Tuesday, April 10, 2018 6:05 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Yes. It’s a good extension to couple with the vbdev and bdev to meet the client’s QoS goal. For a client, from my understanding, a “device” will always be linked through its own specific “connection”. The per client (or the per connection) QoS can be also achieved by the per “device”.

The other comment is that to move this QoS rate limiting from the client side to the target side. Client can has its own QoS for the connection before submitting I/Os and with this work, it can also let the target side to do the similar thing and may not need a QoS at client side.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Tuesday, April 10, 2018 11:56 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Hi Gang,

This is pretty cool.  So essentially the PT vbdev is just a per-client hook for QoS?

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Monday, April 9, 2018 9:03 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] About per client QoS (IOPS rate limiting)

Hi all,

There has been some work done to have SPDK support the QoS per bdev which means that each bdev has its assigned IOPS rate limiting. This QoS functionality is below the storage protocols like NVMe-oF, iSCSI and so on.

At the time of last PRC SPDK summit, there are some questions on the per client QoS (i.e., the NVMe-oF iniator / host) besides the per bdev QoS. Each client like NVMe-oF initiator can see its own IOPS rate limiting not matter whether the connected device is exclusive or shared at the target side.

Based on Paul’s great stuff of the Passthru vbdev, I’ve updated through below three patches to achieve the per client QoS. Each client sees its own passthru vbdev and QoS assigned on this vbdev. These vbdev shares the same bdev. Currently it only supports shareable and read only vbdev on same bdev.

https://review.gerrithub.io/#/c/406888/     vbdev: update and have a create_passthru_disk function
https://review.gerrithub.io/#/c/406891/     vbdev: add the construct_passthru_bdev RPC method
https://review.gerrithub.io/#/c/406977/     vbdev: make the passthru vbdev share the same underlying bdev

The possible usage as following:
[Take NVMe-oF target as example]

1.  In the nvmf.conf file,
# Configure the Passthru vbdev on same bdev like Malloc0
[Passthru]
PT Malloc0 PT0
PT Malloc0 PT1

# The subsystem is configured with shareable bdev by Passthru vbdevs
[Subsystem2]
  NQN nqn.2016-06.io.spdk:cnode2
  Listen RDMA 192.168.2.21:4420
  AllowAnyHost No
  Host nqn.2016-06.io.spdk:init
  SN SPDK00000000000002
  Namespace PT0   >>>> This can be connected by one client
  Namespace PT1   >>>> This can be connected by another client

# Assign different QoS IOPS limiting on Passthru vbdevs
[QoS]
Limit_IOPS PT0 20000
Limit_IOPS PT1 30000

2. Use RPC method to add the Passthru vbdev at runtime
a. python ./scripts/rpc.py construct_passthru_bdev Malloc0 PT3
b. python ./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2 PT3
c. python ./scripts/rpc.py enable_bdev_qos PT3 20000

If there is any concern or comment, please feel free to let me know.

Thanks,
Gang

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 20532 bytes --]

Re: [SPDK] About per client QoS (IOPS rate limiting)

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 7296 bytes --]

Yeah I was about to add… from the patches I looked at so far there was no QoS in them, they just add RPC support which I think is very good for the example.  I haven’t looked in the last two days, but what specific functionality is required in the vbdev module for QoS, from the answer to my question below it didn’t sound like there was anything.

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Friday, April 13, 2018 1:40 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Sounds good to me.

Another question is that do we consider to add the RPC support to the “example” module as the example on using RPC to construct the vbdev?

Or just leave it in the current state.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Friday, April 13, 2018 3:52 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Had an OOB chat with Nate on this – what if we rename Paul’s module to “example”, and reappropriate the “passthru” name for this new module that we are proposing?

-Jim


From: James Harris <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>>
Date: Thursday, April 12, 2018 at 11:11 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

I agree that this should probably go into a new vbdev module.  We do want to keep the existing pass-through module as the example which is as simple as possible.  Then if we find we need other changes to support Gang’s per-client use case, we don’t have to worry about further complicating the example.

The question is then what to name it.  I don’t think “qos” is the best name – since this module isn’t actually doing any QoS work – it just enables per-client QoS.  All of the QoS work is done in the common bdev layer – not a bdev module.  Maybe something along the lines of “shared_passthru”?  I don’t particularly care for that name either – if anyone has another idea, please advise.

This new module can also utilize the spdk_bdev_part API to significantly simplify the code.  Paul purposely did not use spdk_bdev_part in the passthru example because he wanted to explicitly show how everything should be implemented in the example bdev module itself.  But for this module, since it’s not an example module, we don’t have that same concern.

-Jim


From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of "Kariuki, John K" <john.k.kariuki(a)intel.com<mailto:john.k.kariuki(a)intel.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Thursday, April 12, 2018 at 10:01 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Gang/Paul
Does this functionality belong in the passthru bdev or should it be implemented in another vbdev called something like “qos”? My understanding of the passthru bdev is that it’s a template that anyone looking to build a vbdev can use as a starting point. It has all the functions needed to create an SPDK module stub out so that someone can pick it up, make a copy and a few modifications and then they are off to the races implementing their new virtual bdev

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Tuesday, April 10, 2018 6:05 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Yes. It’s a good extension to couple with the vbdev and bdev to meet the client’s QoS goal. For a client, from my understanding, a “device” will always be linked through its own specific “connection”. The per client (or the per connection) QoS can be also achieved by the per “device”.

The other comment is that to move this QoS rate limiting from the client side to the target side. Client can has its own QoS for the connection before submitting I/Os and with this work, it can also let the target side to do the similar thing and may not need a QoS at client side.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Tuesday, April 10, 2018 11:56 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Hi Gang,

This is pretty cool.  So essentially the PT vbdev is just a per-client hook for QoS?

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Monday, April 9, 2018 9:03 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] About per client QoS (IOPS rate limiting)

Hi all,

There has been some work done to have SPDK support the QoS per bdev which means that each bdev has its assigned IOPS rate limiting. This QoS functionality is below the storage protocols like NVMe-oF, iSCSI and so on.

At the time of last PRC SPDK summit, there are some questions on the per client QoS (i.e., the NVMe-oF iniator / host) besides the per bdev QoS. Each client like NVMe-oF initiator can see its own IOPS rate limiting not matter whether the connected device is exclusive or shared at the target side.

Based on Paul’s great stuff of the Passthru vbdev, I’ve updated through below three patches to achieve the per client QoS. Each client sees its own passthru vbdev and QoS assigned on this vbdev. These vbdev shares the same bdev. Currently it only supports shareable and read only vbdev on same bdev.

https://review.gerrithub.io/#/c/406888/     vbdev: update and have a create_passthru_disk function
https://review.gerrithub.io/#/c/406891/     vbdev: add the construct_passthru_bdev RPC method
https://review.gerrithub.io/#/c/406977/     vbdev: make the passthru vbdev share the same underlying bdev

The possible usage as following:
[Take NVMe-oF target as example]

1.  In the nvmf.conf file,
# Configure the Passthru vbdev on same bdev like Malloc0
[Passthru]
PT Malloc0 PT0
PT Malloc0 PT1

# The subsystem is configured with shareable bdev by Passthru vbdevs
[Subsystem2]
  NQN nqn.2016-06.io.spdk:cnode2
  Listen RDMA 192.168.2.21:4420
  AllowAnyHost No
  Host nqn.2016-06.io.spdk:init
  SN SPDK00000000000002
  Namespace PT0   >>>> This can be connected by one client
  Namespace PT1   >>>> This can be connected by another client

# Assign different QoS IOPS limiting on Passthru vbdevs
[QoS]
Limit_IOPS PT0 20000
Limit_IOPS PT1 30000

2. Use RPC method to add the Passthru vbdev at runtime
a. python ./scripts/rpc.py construct_passthru_bdev Malloc0 PT3
b. python ./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2 PT3
c. python ./scripts/rpc.py enable_bdev_qos PT3 20000

If there is any concern or comment, please feel free to let me know.

Thanks,
Gang

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 25798 bytes --]

Re: [SPDK] About per client QoS (IOPS rate limiting)

[Cao, Gang]

[-- Attachment #1: Type: text/plain, Size: 6660 bytes --]

Sounds good to me.

Another question is that do we consider to add the RPC support to the “example” module as the example on using RPC to construct the vbdev?

Or just leave it in the current state.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Friday, April 13, 2018 3:52 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Had an OOB chat with Nate on this – what if we rename Paul’s module to “example”, and reappropriate the “passthru” name for this new module that we are proposing?

-Jim


From: James Harris <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>>
Date: Thursday, April 12, 2018 at 11:11 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

I agree that this should probably go into a new vbdev module.  We do want to keep the existing pass-through module as the example which is as simple as possible.  Then if we find we need other changes to support Gang’s per-client use case, we don’t have to worry about further complicating the example.

The question is then what to name it.  I don’t think “qos” is the best name – since this module isn’t actually doing any QoS work – it just enables per-client QoS.  All of the QoS work is done in the common bdev layer – not a bdev module.  Maybe something along the lines of “shared_passthru”?  I don’t particularly care for that name either – if anyone has another idea, please advise.

This new module can also utilize the spdk_bdev_part API to significantly simplify the code.  Paul purposely did not use spdk_bdev_part in the passthru example because he wanted to explicitly show how everything should be implemented in the example bdev module itself.  But for this module, since it’s not an example module, we don’t have that same concern.

-Jim


From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of "Kariuki, John K" <john.k.kariuki(a)intel.com<mailto:john.k.kariuki(a)intel.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Thursday, April 12, 2018 at 10:01 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Gang/Paul
Does this functionality belong in the passthru bdev or should it be implemented in another vbdev called something like “qos”? My understanding of the passthru bdev is that it’s a template that anyone looking to build a vbdev can use as a starting point. It has all the functions needed to create an SPDK module stub out so that someone can pick it up, make a copy and a few modifications and then they are off to the races implementing their new virtual bdev

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Tuesday, April 10, 2018 6:05 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Yes. It’s a good extension to couple with the vbdev and bdev to meet the client’s QoS goal. For a client, from my understanding, a “device” will always be linked through its own specific “connection”. The per client (or the per connection) QoS can be also achieved by the per “device”.

The other comment is that to move this QoS rate limiting from the client side to the target side. Client can has its own QoS for the connection before submitting I/Os and with this work, it can also let the target side to do the similar thing and may not need a QoS at client side.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Tuesday, April 10, 2018 11:56 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Hi Gang,

This is pretty cool.  So essentially the PT vbdev is just a per-client hook for QoS?

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Monday, April 9, 2018 9:03 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] About per client QoS (IOPS rate limiting)

Hi all,

There has been some work done to have SPDK support the QoS per bdev which means that each bdev has its assigned IOPS rate limiting. This QoS functionality is below the storage protocols like NVMe-oF, iSCSI and so on.

At the time of last PRC SPDK summit, there are some questions on the per client QoS (i.e., the NVMe-oF iniator / host) besides the per bdev QoS. Each client like NVMe-oF initiator can see its own IOPS rate limiting not matter whether the connected device is exclusive or shared at the target side.

Based on Paul’s great stuff of the Passthru vbdev, I’ve updated through below three patches to achieve the per client QoS. Each client sees its own passthru vbdev and QoS assigned on this vbdev. These vbdev shares the same bdev. Currently it only supports shareable and read only vbdev on same bdev.

https://review.gerrithub.io/#/c/406888/     vbdev: update and have a create_passthru_disk function
https://review.gerrithub.io/#/c/406891/     vbdev: add the construct_passthru_bdev RPC method
https://review.gerrithub.io/#/c/406977/     vbdev: make the passthru vbdev share the same underlying bdev

The possible usage as following:
[Take NVMe-oF target as example]

1.  In the nvmf.conf file,
# Configure the Passthru vbdev on same bdev like Malloc0
[Passthru]
PT Malloc0 PT0
PT Malloc0 PT1

# The subsystem is configured with shareable bdev by Passthru vbdevs
[Subsystem2]
  NQN nqn.2016-06.io.spdk:cnode2
  Listen RDMA 192.168.2.21:4420
  AllowAnyHost No
  Host nqn.2016-06.io.spdk:init
  SN SPDK00000000000002
  Namespace PT0   >>>> This can be connected by one client
  Namespace PT1   >>>> This can be connected by another client

# Assign different QoS IOPS limiting on Passthru vbdevs
[QoS]
Limit_IOPS PT0 20000
Limit_IOPS PT1 30000

2. Use RPC method to add the Passthru vbdev at runtime
a. python ./scripts/rpc.py construct_passthru_bdev Malloc0 PT3
b. python ./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2 PT3
c. python ./scripts/rpc.py enable_bdev_qos PT3 20000

If there is any concern or comment, please feel free to let me know.

Thanks,
Gang

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 21898 bytes --]

Re: [SPDK] About per client QoS (IOPS rate limiting)

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 5971 bytes --]

Had an OOB chat with Nate on this – what if we rename Paul’s module to “example”, and reappropriate the “passthru” name for this new module that we are proposing?

-Jim


From: James Harris <james.r.harris(a)intel.com>
Date: Thursday, April 12, 2018 at 11:11 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

I agree that this should probably go into a new vbdev module.  We do want to keep the existing pass-through module as the example which is as simple as possible.  Then if we find we need other changes to support Gang’s per-client use case, we don’t have to worry about further complicating the example.

The question is then what to name it.  I don’t think “qos” is the best name – since this module isn’t actually doing any QoS work – it just enables per-client QoS.  All of the QoS work is done in the common bdev layer – not a bdev module.  Maybe something along the lines of “shared_passthru”?  I don’t particularly care for that name either – if anyone has another idea, please advise.

This new module can also utilize the spdk_bdev_part API to significantly simplify the code.  Paul purposely did not use spdk_bdev_part in the passthru example because he wanted to explicitly show how everything should be implemented in the example bdev module itself.  But for this module, since it’s not an example module, we don’t have that same concern.

-Jim


From: SPDK <spdk-bounces(a)lists.01.org> on behalf of "Kariuki, John K" <john.k.kariuki(a)intel.com>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Thursday, April 12, 2018 at 10:01 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Gang/Paul
Does this functionality belong in the passthru bdev or should it be implemented in another vbdev called something like “qos”? My understanding of the passthru bdev is that it’s a template that anyone looking to build a vbdev can use as a starting point. It has all the functions needed to create an SPDK module stub out so that someone can pick it up, make a copy and a few modifications and then they are off to the races implementing their new virtual bdev

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Tuesday, April 10, 2018 6:05 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Yes. It’s a good extension to couple with the vbdev and bdev to meet the client’s QoS goal. For a client, from my understanding, a “device” will always be linked through its own specific “connection”. The per client (or the per connection) QoS can be also achieved by the per “device”.

The other comment is that to move this QoS rate limiting from the client side to the target side. Client can has its own QoS for the connection before submitting I/Os and with this work, it can also let the target side to do the similar thing and may not need a QoS at client side.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Tuesday, April 10, 2018 11:56 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Hi Gang,

This is pretty cool.  So essentially the PT vbdev is just a per-client hook for QoS?

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Monday, April 9, 2018 9:03 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] About per client QoS (IOPS rate limiting)

Hi all,

There has been some work done to have SPDK support the QoS per bdev which means that each bdev has its assigned IOPS rate limiting. This QoS functionality is below the storage protocols like NVMe-oF, iSCSI and so on.

At the time of last PRC SPDK summit, there are some questions on the per client QoS (i.e., the NVMe-oF iniator / host) besides the per bdev QoS. Each client like NVMe-oF initiator can see its own IOPS rate limiting not matter whether the connected device is exclusive or shared at the target side.

Based on Paul’s great stuff of the Passthru vbdev, I’ve updated through below three patches to achieve the per client QoS. Each client sees its own passthru vbdev and QoS assigned on this vbdev. These vbdev shares the same bdev. Currently it only supports shareable and read only vbdev on same bdev.

https://review.gerrithub.io/#/c/406888/     vbdev: update and have a create_passthru_disk function
https://review.gerrithub.io/#/c/406891/     vbdev: add the construct_passthru_bdev RPC method
https://review.gerrithub.io/#/c/406977/     vbdev: make the passthru vbdev share the same underlying bdev

The possible usage as following:
[Take NVMe-oF target as example]

1.  In the nvmf.conf file,
# Configure the Passthru vbdev on same bdev like Malloc0
[Passthru]
PT Malloc0 PT0
PT Malloc0 PT1

# The subsystem is configured with shareable bdev by Passthru vbdevs
[Subsystem2]
  NQN nqn.2016-06.io.spdk:cnode2
  Listen RDMA 192.168.2.21:4420
  AllowAnyHost No
  Host nqn.2016-06.io.spdk:init
  SN SPDK00000000000002
  Namespace PT0   >>>> This can be connected by one client
  Namespace PT1   >>>> This can be connected by another client

# Assign different QoS IOPS limiting on Passthru vbdevs
[QoS]
Limit_IOPS PT0 20000
Limit_IOPS PT1 30000

2. Use RPC method to add the Passthru vbdev at runtime
a. python ./scripts/rpc.py construct_passthru_bdev Malloc0 PT3
b. python ./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2 PT3
c. python ./scripts/rpc.py enable_bdev_qos PT3 20000

If there is any concern or comment, please feel free to let me know.

Thanks,
Gang

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 16757 bytes --]

Re: [SPDK] About per client QoS (IOPS rate limiting)

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 5562 bytes --]

I agree that this should probably go into a new vbdev module.  We do want to keep the existing pass-through module as the example which is as simple as possible.  Then if we find we need other changes to support Gang’s per-client use case, we don’t have to worry about further complicating the example.

The question is then what to name it.  I don’t think “qos” is the best name – since this module isn’t actually doing any QoS work – it just enables per-client QoS.  All of the QoS work is done in the common bdev layer – not a bdev module.  Maybe something along the lines of “shared_passthru”?  I don’t particularly care for that name either – if anyone has another idea, please advise.

This new module can also utilize the spdk_bdev_part API to significantly simplify the code.  Paul purposely did not use spdk_bdev_part in the passthru example because he wanted to explicitly show how everything should be implemented in the example bdev module itself.  But for this module, since it’s not an example module, we don’t have that same concern.

-Jim


From: SPDK <spdk-bounces(a)lists.01.org> on behalf of "Kariuki, John K" <john.k.kariuki(a)intel.com>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Thursday, April 12, 2018 at 10:01 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Gang/Paul
Does this functionality belong in the passthru bdev or should it be implemented in another vbdev called something like “qos”? My understanding of the passthru bdev is that it’s a template that anyone looking to build a vbdev can use as a starting point. It has all the functions needed to create an SPDK module stub out so that someone can pick it up, make a copy and a few modifications and then they are off to the races implementing their new virtual bdev

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Tuesday, April 10, 2018 6:05 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Yes. It’s a good extension to couple with the vbdev and bdev to meet the client’s QoS goal. For a client, from my understanding, a “device” will always be linked through its own specific “connection”. The per client (or the per connection) QoS can be also achieved by the per “device”.

The other comment is that to move this QoS rate limiting from the client side to the target side. Client can has its own QoS for the connection before submitting I/Os and with this work, it can also let the target side to do the similar thing and may not need a QoS at client side.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Tuesday, April 10, 2018 11:56 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Hi Gang,

This is pretty cool.  So essentially the PT vbdev is just a per-client hook for QoS?

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Monday, April 9, 2018 9:03 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] About per client QoS (IOPS rate limiting)

Hi all,

There has been some work done to have SPDK support the QoS per bdev which means that each bdev has its assigned IOPS rate limiting. This QoS functionality is below the storage protocols like NVMe-oF, iSCSI and so on.

At the time of last PRC SPDK summit, there are some questions on the per client QoS (i.e., the NVMe-oF iniator / host) besides the per bdev QoS. Each client like NVMe-oF initiator can see its own IOPS rate limiting not matter whether the connected device is exclusive or shared at the target side.

Based on Paul’s great stuff of the Passthru vbdev, I’ve updated through below three patches to achieve the per client QoS. Each client sees its own passthru vbdev and QoS assigned on this vbdev. These vbdev shares the same bdev. Currently it only supports shareable and read only vbdev on same bdev.

https://review.gerrithub.io/#/c/406888/     vbdev: update and have a create_passthru_disk function
https://review.gerrithub.io/#/c/406891/     vbdev: add the construct_passthru_bdev RPC method
https://review.gerrithub.io/#/c/406977/     vbdev: make the passthru vbdev share the same underlying bdev

The possible usage as following:
[Take NVMe-oF target as example]

1.  In the nvmf.conf file,
# Configure the Passthru vbdev on same bdev like Malloc0
[Passthru]
PT Malloc0 PT0
PT Malloc0 PT1

# The subsystem is configured with shareable bdev by Passthru vbdevs
[Subsystem2]
  NQN nqn.2016-06.io.spdk:cnode2
  Listen RDMA 192.168.2.21:4420
  AllowAnyHost No
  Host nqn.2016-06.io.spdk:init
  SN SPDK00000000000002
  Namespace PT0   >>>> This can be connected by one client
  Namespace PT1   >>>> This can be connected by another client

# Assign different QoS IOPS limiting on Passthru vbdevs
[QoS]
Limit_IOPS PT0 20000
Limit_IOPS PT1 30000

2. Use RPC method to add the Passthru vbdev at runtime
a. python ./scripts/rpc.py construct_passthru_bdev Malloc0 PT3
b. python ./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2 PT3
c. python ./scripts/rpc.py enable_bdev_qos PT3 20000

If there is any concern or comment, please feel free to let me know.

Thanks,
Gang

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 15163 bytes --]

Re: [SPDK] About per client QoS (IOPS rate limiting)

[Kariuki, John K]

[-- Attachment #1: Type: text/plain, Size: 4097 bytes --]

Gang/Paul
Does this functionality belong in the passthru bdev or should it be implemented in another vbdev called something like "qos"? My understanding of the passthru bdev is that it's a template that anyone looking to build a vbdev can use as a starting point. It has all the functions needed to create an SPDK module stub out so that someone can pick it up, make a copy and a few modifications and then they are off to the races implementing their new virtual bdev

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Tuesday, April 10, 2018 6:05 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Yes. It's a good extension to couple with the vbdev and bdev to meet the client's QoS goal. For a client, from my understanding, a "device" will always be linked through its own specific "connection". The per client (or the per connection) QoS can be also achieved by the per "device".

The other comment is that to move this QoS rate limiting from the client side to the target side. Client can has its own QoS for the connection before submitting I/Os and with this work, it can also let the target side to do the similar thing and may not need a QoS at client side.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Tuesday, April 10, 2018 11:56 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Hi Gang,

This is pretty cool.  So essentially the PT vbdev is just a per-client hook for QoS?

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Monday, April 9, 2018 9:03 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] About per client QoS (IOPS rate limiting)

Hi all,

There has been some work done to have SPDK support the QoS per bdev which means that each bdev has its assigned IOPS rate limiting. This QoS functionality is below the storage protocols like NVMe-oF, iSCSI and so on.

At the time of last PRC SPDK summit, there are some questions on the per client QoS (i.e., the NVMe-oF iniator / host) besides the per bdev QoS. Each client like NVMe-oF initiator can see its own IOPS rate limiting not matter whether the connected device is exclusive or shared at the target side.

Based on Paul's great stuff of the Passthru vbdev, I've updated through below three patches to achieve the per client QoS. Each client sees its own passthru vbdev and QoS assigned on this vbdev. These vbdev shares the same bdev. Currently it only supports shareable and read only vbdev on same bdev.

https://review.gerrithub.io/#/c/406888/     vbdev: update and have a create_passthru_disk function
https://review.gerrithub.io/#/c/406891/     vbdev: add the construct_passthru_bdev RPC method
https://review.gerrithub.io/#/c/406977/     vbdev: make the passthru vbdev share the same underlying bdev

The possible usage as following:
[Take NVMe-oF target as example]

1.  In the nvmf.conf file,
# Configure the Passthru vbdev on same bdev like Malloc0
[Passthru]
PT Malloc0 PT0
PT Malloc0 PT1

# The subsystem is configured with shareable bdev by Passthru vbdevs
[Subsystem2]
  NQN nqn.2016-06.io.spdk:cnode2
  Listen RDMA 192.168.2.21:4420
  AllowAnyHost No
  Host nqn.2016-06.io.spdk:init
  SN SPDK00000000000002
  Namespace PT0   >>>> This can be connected by one client
  Namespace PT1   >>>> This can be connected by another client

# Assign different QoS IOPS limiting on Passthru vbdevs
[QoS]
Limit_IOPS PT0 20000
Limit_IOPS PT1 30000

2. Use RPC method to add the Passthru vbdev at runtime
a. python ./scripts/rpc.py construct_passthru_bdev Malloc0 PT3
b. python ./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2 PT3
c. python ./scripts/rpc.py enable_bdev_qos PT3 20000

If there is any concern or comment, please feel free to let me know.

Thanks,
Gang

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 13482 bytes --]

Re: [SPDK] About per client QoS (IOPS rate limiting)

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 3868 bytes --]

Thanks, that makes sense.  Funny how the "do nothing" vbdev found a real reason to exist, besides being an example, so quickly.  Gotta love this stuff!  I'm out for a few days but will continue reviewing the patches for sure.

Thx
Paul


From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Tuesday, April 10, 2018 6:05 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Yes. It's a good extension to couple with the vbdev and bdev to meet the client's QoS goal. For a client, from my understanding, a "device" will always be linked through its own specific "connection". The per client (or the per connection) QoS can be also achieved by the per "device".

The other comment is that to move this QoS rate limiting from the client side to the target side. Client can has its own QoS for the connection before submitting I/Os and with this work, it can also let the target side to do the similar thing and may not need a QoS at client side.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Tuesday, April 10, 2018 11:56 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Hi Gang,

This is pretty cool.  So essentially the PT vbdev is just a per-client hook for QoS?

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Monday, April 9, 2018 9:03 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] About per client QoS (IOPS rate limiting)

Hi all,

There has been some work done to have SPDK support the QoS per bdev which means that each bdev has its assigned IOPS rate limiting. This QoS functionality is below the storage protocols like NVMe-oF, iSCSI and so on.

At the time of last PRC SPDK summit, there are some questions on the per client QoS (i.e., the NVMe-oF iniator / host) besides the per bdev QoS. Each client like NVMe-oF initiator can see its own IOPS rate limiting not matter whether the connected device is exclusive or shared at the target side.

Based on Paul's great stuff of the Passthru vbdev, I've updated through below three patches to achieve the per client QoS. Each client sees its own passthru vbdev and QoS assigned on this vbdev. These vbdev shares the same bdev. Currently it only supports shareable and read only vbdev on same bdev.

https://review.gerrithub.io/#/c/406888/     vbdev: update and have a create_passthru_disk function
https://review.gerrithub.io/#/c/406891/     vbdev: add the construct_passthru_bdev RPC method
https://review.gerrithub.io/#/c/406977/     vbdev: make the passthru vbdev share the same underlying bdev

The possible usage as following:
[Take NVMe-oF target as example]

1.  In the nvmf.conf file,
# Configure the Passthru vbdev on same bdev like Malloc0
[Passthru]
PT Malloc0 PT0
PT Malloc0 PT1

# The subsystem is configured with shareable bdev by Passthru vbdevs
[Subsystem2]
  NQN nqn.2016-06.io.spdk:cnode2
  Listen RDMA 192.168.2.21:4420
  AllowAnyHost No
  Host nqn.2016-06.io.spdk:init
  SN SPDK00000000000002
  Namespace PT0   >>>> This can be connected by one client
  Namespace PT1   >>>> This can be connected by another client

# Assign different QoS IOPS limiting on Passthru vbdevs
[QoS]
Limit_IOPS PT0 20000
Limit_IOPS PT1 30000

2. Use RPC method to add the Passthru vbdev at runtime
a. python ./scripts/rpc.py construct_passthru_bdev Malloc0 PT3
b. python ./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2 PT3
c. python ./scripts/rpc.py enable_bdev_qos PT3 20000

If there is any concern or comment, please feel free to let me know.

Thanks,
Gang

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 13540 bytes --]

Re: [SPDK] About per client QoS (IOPS rate limiting)

[Cao, Gang]

[-- Attachment #1: Type: text/plain, Size: 3357 bytes --]

Yes. It's a good extension to couple with the vbdev and bdev to meet the client's QoS goal. For a client, from my understanding, a "device" will always be linked through its own specific "connection". The per client (or the per connection) QoS can be also achieved by the per "device".

The other comment is that to move this QoS rate limiting from the client side to the target side. Client can has its own QoS for the connection before submitting I/Os and with this work, it can also let the target side to do the similar thing and may not need a QoS at client side.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Tuesday, April 10, 2018 11:56 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] About per client QoS (IOPS rate limiting)

Hi Gang,

This is pretty cool.  So essentially the PT vbdev is just a per-client hook for QoS?

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Monday, April 9, 2018 9:03 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] About per client QoS (IOPS rate limiting)

Hi all,

There has been some work done to have SPDK support the QoS per bdev which means that each bdev has its assigned IOPS rate limiting. This QoS functionality is below the storage protocols like NVMe-oF, iSCSI and so on.

At the time of last PRC SPDK summit, there are some questions on the per client QoS (i.e., the NVMe-oF iniator / host) besides the per bdev QoS. Each client like NVMe-oF initiator can see its own IOPS rate limiting not matter whether the connected device is exclusive or shared at the target side.

Based on Paul's great stuff of the Passthru vbdev, I've updated through below three patches to achieve the per client QoS. Each client sees its own passthru vbdev and QoS assigned on this vbdev. These vbdev shares the same bdev. Currently it only supports shareable and read only vbdev on same bdev.

https://review.gerrithub.io/#/c/406888/     vbdev: update and have a create_passthru_disk function
https://review.gerrithub.io/#/c/406891/     vbdev: add the construct_passthru_bdev RPC method
https://review.gerrithub.io/#/c/406977/     vbdev: make the passthru vbdev share the same underlying bdev

The possible usage as following:
[Take NVMe-oF target as example]

1.  In the nvmf.conf file,
# Configure the Passthru vbdev on same bdev like Malloc0
[Passthru]
PT Malloc0 PT0
PT Malloc0 PT1

# The subsystem is configured with shareable bdev by Passthru vbdevs
[Subsystem2]
  NQN nqn.2016-06.io.spdk:cnode2
  Listen RDMA 192.168.2.21:4420
  AllowAnyHost No
  Host nqn.2016-06.io.spdk:init
  SN SPDK00000000000002
  Namespace PT0   >>>> This can be connected by one client
  Namespace PT1   >>>> This can be connected by another client

# Assign different QoS IOPS limiting on Passthru vbdevs
[QoS]
Limit_IOPS PT0 20000
Limit_IOPS PT1 30000

2. Use RPC method to add the Passthru vbdev at runtime
a. python ./scripts/rpc.py construct_passthru_bdev Malloc0 PT3
b. python ./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2 PT3
c. python ./scripts/rpc.py enable_bdev_qos PT3 20000

If there is any concern or comment, please feel free to let me know.

Thanks,
Gang

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 10804 bytes --]

Re: [SPDK] About per client QoS (IOPS rate limiting)

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 2495 bytes --]

Hi Gang,

This is pretty cool.  So essentially the PT vbdev is just a per-client hook for QoS?

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Cao, Gang
Sent: Monday, April 9, 2018 9:03 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] About per client QoS (IOPS rate limiting)

Hi all,

There has been some work done to have SPDK support the QoS per bdev which means that each bdev has its assigned IOPS rate limiting. This QoS functionality is below the storage protocols like NVMe-oF, iSCSI and so on.

At the time of last PRC SPDK summit, there are some questions on the per client QoS (i.e., the NVMe-oF iniator / host) besides the per bdev QoS. Each client like NVMe-oF initiator can see its own IOPS rate limiting not matter whether the connected device is exclusive or shared at the target side.

Based on Paul's great stuff of the Passthru vbdev, I've updated through below three patches to achieve the per client QoS. Each client sees its own passthru vbdev and QoS assigned on this vbdev. These vbdev shares the same bdev. Currently it only supports shareable and read only vbdev on same bdev.

https://review.gerrithub.io/#/c/406888/     vbdev: update and have a create_passthru_disk function
https://review.gerrithub.io/#/c/406891/     vbdev: add the construct_passthru_bdev RPC method
https://review.gerrithub.io/#/c/406977/     vbdev: make the passthru vbdev share the same underlying bdev

The possible usage as following:
[Take NVMe-oF target as example]

1.  In the nvmf.conf file,
# Configure the Passthru vbdev on same bdev like Malloc0
[Passthru]
PT Malloc0 PT0
PT Malloc0 PT1

# The subsystem is configured with shareable bdev by Passthru vbdevs
[Subsystem2]
  NQN nqn.2016-06.io.spdk:cnode2
  Listen RDMA 192.168.2.21:4420
  AllowAnyHost No
  Host nqn.2016-06.io.spdk:init
  SN SPDK00000000000002
  Namespace PT0   >>>> This can be connected by one client
  Namespace PT1   >>>> This can be connected by another client

# Assign different QoS IOPS limiting on Passthru vbdevs
[QoS]
Limit_IOPS PT0 20000
Limit_IOPS PT1 30000

2. Use RPC method to add the Passthru vbdev at runtime
a. python ./scripts/rpc.py construct_passthru_bdev Malloc0 PT3
b. python ./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2 PT3
c. python ./scripts/rpc.py enable_bdev_qos PT3 20000

If there is any concern or comment, please feel free to let me know.

Thanks,
Gang

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 9393 bytes --]

[SPDK] About per client QoS (IOPS rate limiting)

[Cao, Gang]

[-- Attachment #1: Type: text/plain, Size: 2149 bytes --]

Hi all,

There has been some work done to have SPDK support the QoS per bdev which means that each bdev has its assigned IOPS rate limiting. This QoS functionality is below the storage protocols like NVMe-oF, iSCSI and so on.

At the time of last PRC SPDK summit, there are some questions on the per client QoS (i.e., the NVMe-oF iniator / host) besides the per bdev QoS. Each client like NVMe-oF initiator can see its own IOPS rate limiting not matter whether the connected device is exclusive or shared at the target side.

Based on Paul's great stuff of the Passthru vbdev, I've updated through below three patches to achieve the per client QoS. Each client sees its own passthru vbdev and QoS assigned on this vbdev. These vbdev shares the same bdev. Currently it only supports shareable and read only vbdev on same bdev.

https://review.gerrithub.io/#/c/406888/     vbdev: update and have a create_passthru_disk function
https://review.gerrithub.io/#/c/406891/     vbdev: add the construct_passthru_bdev RPC method
https://review.gerrithub.io/#/c/406977/     vbdev: make the passthru vbdev share the same underlying bdev

The possible usage as following:
[Take NVMe-oF target as example]

1.  In the nvmf.conf file,
# Configure the Passthru vbdev on same bdev like Malloc0
[Passthru]
PT Malloc0 PT0
PT Malloc0 PT1

# The subsystem is configured with shareable bdev by Passthru vbdevs
[Subsystem2]
  NQN nqn.2016-06.io.spdk:cnode2
  Listen RDMA 192.168.2.21:4420
  AllowAnyHost No
  Host nqn.2016-06.io.spdk:init
  SN SPDK00000000000002
  Namespace PT0   >>>> This can be connected by one client
  Namespace PT1   >>>> This can be connected by another client

# Assign different QoS IOPS limiting on Passthru vbdevs
[QoS]
Limit_IOPS PT0 20000
Limit_IOPS PT1 30000

2. Use RPC method to add the Passthru vbdev at runtime
a. python ./scripts/rpc.py construct_passthru_bdev Malloc0 PT3
b. python ./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2 PT3
c. python ./scripts/rpc.py enable_bdev_qos PT3 20000

If there is any concern or comment, please feel free to let me know.

Thanks,
Gang

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 7095 bytes --]