[PATCH v3 1/1] virtio-blk: add num_request_queues module parameter

[PATCH v3 1/1] virtio-blk: add num_request_queues module parameter

[Max Gurtovoy]

Sometimes a user would like to control the amount of request queues to
be created for a block device. For example, for limiting the memory
footprint of virtio-blk devices.

Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
---

changes from v2:
 - renamed num_io_queues to num_request_queues (from Stefan)
 - added Reviewed-by signatures (from Stefan and Christoph)

changes from v1:
 - use param_set_uint_minmax (from Christoph)
 - added "Should > 0" to module description

Note: This commit apply on top of Jens's branch for-5.15/drivers

---
 drivers/block/virtio_blk.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
index 4b49df2dfd23..aaa2833a4734 100644
--- a/drivers/block/virtio_blk.c
+++ b/drivers/block/virtio_blk.c
@@ -24,6 +24,23 @@
 /* The maximum number of sg elements that fit into a virtqueue */
 #define VIRTIO_BLK_MAX_SG_ELEMS 32768
 
+static int virtblk_queue_count_set(const char *val,
+		const struct kernel_param *kp)
+{
+	return param_set_uint_minmax(val, kp, 1, nr_cpu_ids);
+}
+
+static const struct kernel_param_ops queue_count_ops = {
+	.set = virtblk_queue_count_set,
+	.get = param_get_uint,
+};
+
+static unsigned int num_request_queues;
+module_param_cb(num_request_queues, &queue_count_ops, &num_request_queues,
+		0644);
+MODULE_PARM_DESC(num_request_queues,
+		 "Number of request queues to use for blk device. Should > 0");
+
 static int major;
 static DEFINE_IDA(vd_index_ida);
 
@@ -501,7 +518,9 @@ static int init_vq(struct virtio_blk *vblk)
 	if (err)
 		num_vqs = 1;
 
-	num_vqs = min_t(unsigned int, nr_cpu_ids, num_vqs);
+	num_vqs = min_t(unsigned int,
+			min_not_zero(num_request_queues, nr_cpu_ids),
+			num_vqs);
 
 	vblk->vqs = kmalloc_array(num_vqs, sizeof(*vblk->vqs), GFP_KERNEL);
 	if (!vblk->vqs)
-- 
2.18.1

Re: [PATCH v3 1/1] virtio-blk: add num_request_queues module parameter

[Pankaj Gupta]

> Sometimes a user would like to control the amount of request queues to
> be created for a block device. For example, for limiting the memory
> footprint of virtio-blk devices.
>
> Reviewed-by: Christoph Hellwig <hch@lst.de>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> ---
>
> changes from v2:
>  - renamed num_io_queues to num_request_queues (from Stefan)
>  - added Reviewed-by signatures (from Stefan and Christoph)
>
> changes from v1:
>  - use param_set_uint_minmax (from Christoph)
>  - added "Should > 0" to module description
>
> Note: This commit apply on top of Jens's branch for-5.15/drivers
>
> ---
>  drivers/block/virtio_blk.c | 21 ++++++++++++++++++++-
>  1 file changed, 20 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
> index 4b49df2dfd23..aaa2833a4734 100644
> --- a/drivers/block/virtio_blk.c
> +++ b/drivers/block/virtio_blk.c
> @@ -24,6 +24,23 @@
>  /* The maximum number of sg elements that fit into a virtqueue */
>  #define VIRTIO_BLK_MAX_SG_ELEMS 32768
>
> +static int virtblk_queue_count_set(const char *val,
> +               const struct kernel_param *kp)
> +{
> +       return param_set_uint_minmax(val, kp, 1, nr_cpu_ids);
> +}
> +
> +static const struct kernel_param_ops queue_count_ops = {
> +       .set = virtblk_queue_count_set,
> +       .get = param_get_uint,
> +};
> +
> +static unsigned int num_request_queues;
> +module_param_cb(num_request_queues, &queue_count_ops, &num_request_queues,
> +               0644);
> +MODULE_PARM_DESC(num_request_queues,
> +                "Number of request queues to use for blk device. Should > 0");
> +
>  static int major;
>  static DEFINE_IDA(vd_index_ida);
>
> @@ -501,7 +518,9 @@ static int init_vq(struct virtio_blk *vblk)
>         if (err)
>                 num_vqs = 1;
>
> -       num_vqs = min_t(unsigned int, nr_cpu_ids, num_vqs);
> +       num_vqs = min_t(unsigned int,
> +                       min_not_zero(num_request_queues, nr_cpu_ids),
> +                       num_vqs);
>
>         vblk->vqs = kmalloc_array(num_vqs, sizeof(*vblk->vqs), GFP_KERNEL);
>         if (!vblk->vqs)
> --

Reviewed-by: Pankaj Gupta <pankaj.gupta@ionos.com>

Re: [PATCH v3 1/1] virtio-blk: add num_request_queues module parameter

[Leon Romanovsky]

On Thu, Sep 02, 2021 at 11:46:22PM +0300, Max Gurtovoy wrote:
> Sometimes a user would like to control the amount of request queues to
> be created for a block device. For example, for limiting the memory
> footprint of virtio-blk devices.
> 
> Reviewed-by: Christoph Hellwig <hch@lst.de>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> ---
> 
> changes from v2:
>  - renamed num_io_queues to num_request_queues (from Stefan)
>  - added Reviewed-by signatures (from Stefan and Christoph)
> 
> changes from v1:
>  - use param_set_uint_minmax (from Christoph)
>  - added "Should > 0" to module description
> 
> Note: This commit apply on top of Jens's branch for-5.15/drivers
> 
> ---
>  drivers/block/virtio_blk.c | 21 ++++++++++++++++++++-
>  1 file changed, 20 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
> index 4b49df2dfd23..aaa2833a4734 100644
> --- a/drivers/block/virtio_blk.c
> +++ b/drivers/block/virtio_blk.c
> @@ -24,6 +24,23 @@
>  /* The maximum number of sg elements that fit into a virtqueue */
>  #define VIRTIO_BLK_MAX_SG_ELEMS 32768
>  
> +static int virtblk_queue_count_set(const char *val,
> +		const struct kernel_param *kp)
> +{
> +	return param_set_uint_minmax(val, kp, 1, nr_cpu_ids);
> +}
> +
> +static const struct kernel_param_ops queue_count_ops = {
> +	.set = virtblk_queue_count_set,
> +	.get = param_get_uint,
> +};
> +
> +static unsigned int num_request_queues;
> +module_param_cb(num_request_queues, &queue_count_ops, &num_request_queues,
> +		0644);
> +MODULE_PARM_DESC(num_request_queues,
> +		 "Number of request queues to use for blk device. Should > 0");
> +

Won't it limit all virtio block devices to the same limit?

It is very common to see multiple virtio-blk devices on the same system
and they probably need different limits.

Thanks

Re: [PATCH v3 1/1] virtio-blk: add num_request_queues module parameter

[Leon Romanovsky]

On Thu, Sep 02, 2021 at 11:46:22PM +0300, Max Gurtovoy wrote:
> Sometimes a user would like to control the amount of request queues to
> be created for a block device. For example, for limiting the memory
> footprint of virtio-blk devices.
> 
> Reviewed-by: Christoph Hellwig <hch@lst.de>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> ---
> 
> changes from v2:
>  - renamed num_io_queues to num_request_queues (from Stefan)
>  - added Reviewed-by signatures (from Stefan and Christoph)
> 
> changes from v1:
>  - use param_set_uint_minmax (from Christoph)
>  - added "Should > 0" to module description
> 
> Note: This commit apply on top of Jens's branch for-5.15/drivers
> 
> ---
>  drivers/block/virtio_blk.c | 21 ++++++++++++++++++++-
>  1 file changed, 20 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
> index 4b49df2dfd23..aaa2833a4734 100644
> --- a/drivers/block/virtio_blk.c
> +++ b/drivers/block/virtio_blk.c
> @@ -24,6 +24,23 @@
>  /* The maximum number of sg elements that fit into a virtqueue */
>  #define VIRTIO_BLK_MAX_SG_ELEMS 32768
>  
> +static int virtblk_queue_count_set(const char *val,
> +		const struct kernel_param *kp)
> +{
> +	return param_set_uint_minmax(val, kp, 1, nr_cpu_ids);
> +}
> +
> +static const struct kernel_param_ops queue_count_ops = {
> +	.set = virtblk_queue_count_set,
> +	.get = param_get_uint,
> +};
> +
> +static unsigned int num_request_queues;
> +module_param_cb(num_request_queues, &queue_count_ops, &num_request_queues,
> +		0644);
> +MODULE_PARM_DESC(num_request_queues,
> +		 "Number of request queues to use for blk device. Should > 0");
> +

Won't it limit all virtio block devices to the same limit?

It is very common to see multiple virtio-blk devices on the same system
and they probably need different limits.

Thanks
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH v3 1/1] virtio-blk: add num_request_queues module parameter

[Chaitanya Kulkarni]

On 9/5/2021 12:46 AM, Leon Romanovsky wrote:
>> +static unsigned int num_request_queues;
>> +module_param_cb(num_request_queues, &queue_count_ops, &num_request_queues,
>> +		0644);
>> +MODULE_PARM_DESC(num_request_queues,
>> +		 "Number of request queues to use for blk device. Should > 0");
>> +
> Won't it limit all virtio block devices to the same limit?
>
> It is very common to see multiple virtio-blk devices on the same system
> and they probably need different limits.
>
> Thanks


Without looking into the code, that can be done adding a configfs

interface and overriding a global value (module param) when it is set from

configfs.

Re: [PATCH v3 1/1] virtio-blk: add num_request_queues module parameter

[Max Gurtovoy]

On 9/5/2021 11:49 AM, Chaitanya Kulkarni wrote:
>
> On 9/5/2021 12:46 AM, Leon Romanovsky wrote:
>>> +static unsigned int num_request_queues;
>>> +module_param_cb(num_request_queues, &queue_count_ops, 
>>> &num_request_queues,
>>> +        0644);
>>> +MODULE_PARM_DESC(num_request_queues,
>>> +         "Number of request queues to use for blk device. Should > 
>>> 0");
>>> +
>> Won't it limit all virtio block devices to the same limit?
>>
>> It is very common to see multiple virtio-blk devices on the same system
>> and they probably need different limits.
>>
>> Thanks
>
>
> Without looking into the code, that can be done adding a configfs
>
> interface and overriding a global value (module param) when it is set 
> from
>
> configfs.
>
>
You have many ways to overcome this issue.

For example:

# ls -l /sys/block/vda/mq/
drwxr-xr-x 18 root root 0 Sep  5 12:14 0
drwxr-xr-x 18 root root 0 Sep  5 12:14 1
drwxr-xr-x 18 root root 0 Sep  5 12:14 2
drwxr-xr-x 18 root root 0 Sep  5 12:14 3

# echo virtio0 > /sys/bus/virtio/drivers/virtio_blk/unbind

# echo 8 > /sys/module/virtio_blk/parameters/num_request_queues

# echo virtio0 > /sys/bus/virtio/drivers/virtio_blk/bind

# ls -l /sys/block/vda/mq/
drwxr-xr-x 10 root root 0 Sep  5 12:17 0
drwxr-xr-x 10 root root 0 Sep  5 12:17 1
drwxr-xr-x 10 root root 0 Sep  5 12:17 2
drwxr-xr-x 10 root root 0 Sep  5 12:17 3
drwxr-xr-x 10 root root 0 Sep  5 12:17 4
drwxr-xr-x 10 root root 0 Sep  5 12:17 5
drwxr-xr-x 10 root root 0 Sep  5 12:17 6
drwxr-xr-x 10 root root 0 Sep  5 12:17 7

-Max.

Re: [PATCH v3 1/1] virtio-blk: add num_request_queues module parameter

[Leon Romanovsky]

On Sun, Sep 05, 2021 at 12:19:09PM +0300, Max Gurtovoy wrote:
> 
> On 9/5/2021 11:49 AM, Chaitanya Kulkarni wrote:
> > 
> > On 9/5/2021 12:46 AM, Leon Romanovsky wrote:
> > > > +static unsigned int num_request_queues;
> > > > +module_param_cb(num_request_queues, &queue_count_ops,
> > > > &num_request_queues,
> > > > +        0644);
> > > > +MODULE_PARM_DESC(num_request_queues,
> > > > +         "Number of request queues to use for blk device.
> > > > Should > 0");
> > > > +
> > > Won't it limit all virtio block devices to the same limit?
> > > 
> > > It is very common to see multiple virtio-blk devices on the same system
> > > and they probably need different limits.
> > > 
> > > Thanks
> > 
> > 
> > Without looking into the code, that can be done adding a configfs
> > 
> > interface and overriding a global value (module param) when it is set
> > from
> > 
> > configfs.
> > 
> > 
> You have many ways to overcome this issue.
> 
> For example:
> 
> # ls -l /sys/block/vda/mq/
> drwxr-xr-x 18 root root 0 Sep  5 12:14 0
> drwxr-xr-x 18 root root 0 Sep  5 12:14 1
> drwxr-xr-x 18 root root 0 Sep  5 12:14 2
> drwxr-xr-x 18 root root 0 Sep  5 12:14 3
> 
> # echo virtio0 > /sys/bus/virtio/drivers/virtio_blk/unbind
> 
> # echo 8 > /sys/module/virtio_blk/parameters/num_request_queues

This is global to all virtio-blk devices.

> 
> # echo virtio0 > /sys/bus/virtio/drivers/virtio_blk/bind
> 
> # ls -l /sys/block/vda/mq/
> drwxr-xr-x 10 root root 0 Sep  5 12:17 0
> drwxr-xr-x 10 root root 0 Sep  5 12:17 1
> drwxr-xr-x 10 root root 0 Sep  5 12:17 2
> drwxr-xr-x 10 root root 0 Sep  5 12:17 3
> drwxr-xr-x 10 root root 0 Sep  5 12:17 4
> drwxr-xr-x 10 root root 0 Sep  5 12:17 5
> drwxr-xr-x 10 root root 0 Sep  5 12:17 6
> drwxr-xr-x 10 root root 0 Sep  5 12:17 7
> 
> -Max.
> 
> 

Re: [PATCH v3 1/1] virtio-blk: add num_request_queues module parameter

[Leon Romanovsky]

On Sun, Sep 05, 2021 at 12:19:09PM +0300, Max Gurtovoy wrote:
> 
> On 9/5/2021 11:49 AM, Chaitanya Kulkarni wrote:
> > 
> > On 9/5/2021 12:46 AM, Leon Romanovsky wrote:
> > > > +static unsigned int num_request_queues;
> > > > +module_param_cb(num_request_queues, &queue_count_ops,
> > > > &num_request_queues,
> > > > +        0644);
> > > > +MODULE_PARM_DESC(num_request_queues,
> > > > +         "Number of request queues to use for blk device.
> > > > Should > 0");
> > > > +
> > > Won't it limit all virtio block devices to the same limit?
> > > 
> > > It is very common to see multiple virtio-blk devices on the same system
> > > and they probably need different limits.
> > > 
> > > Thanks
> > 
> > 
> > Without looking into the code, that can be done adding a configfs
> > 
> > interface and overriding a global value (module param) when it is set
> > from
> > 
> > configfs.
> > 
> > 
> You have many ways to overcome this issue.
> 
> For example:
> 
> # ls -l /sys/block/vda/mq/
> drwxr-xr-x 18 root root 0 Sep  5 12:14 0
> drwxr-xr-x 18 root root 0 Sep  5 12:14 1
> drwxr-xr-x 18 root root 0 Sep  5 12:14 2
> drwxr-xr-x 18 root root 0 Sep  5 12:14 3
> 
> # echo virtio0 > /sys/bus/virtio/drivers/virtio_blk/unbind
> 
> # echo 8 > /sys/module/virtio_blk/parameters/num_request_queues

This is global to all virtio-blk devices.

> 
> # echo virtio0 > /sys/bus/virtio/drivers/virtio_blk/bind
> 
> # ls -l /sys/block/vda/mq/
> drwxr-xr-x 10 root root 0 Sep  5 12:17 0
> drwxr-xr-x 10 root root 0 Sep  5 12:17 1
> drwxr-xr-x 10 root root 0 Sep  5 12:17 2
> drwxr-xr-x 10 root root 0 Sep  5 12:17 3
> drwxr-xr-x 10 root root 0 Sep  5 12:17 4
> drwxr-xr-x 10 root root 0 Sep  5 12:17 5
> drwxr-xr-x 10 root root 0 Sep  5 12:17 6
> drwxr-xr-x 10 root root 0 Sep  5 12:17 7
> 
> -Max.
> 
> 
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH v3 1/1] virtio-blk: add num_request_queues module parameter

[Leon Romanovsky]

On Sun, Sep 05, 2021 at 01:49:46AM -0700, Chaitanya Kulkarni wrote:
> 
> On 9/5/2021 12:46 AM, Leon Romanovsky wrote:
> > > +static unsigned int num_request_queues;
> > > +module_param_cb(num_request_queues, &queue_count_ops, &num_request_queues,
> > > +		0644);
> > > +MODULE_PARM_DESC(num_request_queues,
> > > +		 "Number of request queues to use for blk device. Should > 0");
> > > +
> > Won't it limit all virtio block devices to the same limit?
> > 
> > It is very common to see multiple virtio-blk devices on the same system
> > and they probably need different limits.
> > 
> > Thanks
> 
> 
> Without looking into the code, that can be done adding a configfs
> 
> interface and overriding a global value (module param) when it is set from
> 
> configfs.

So why should we do double work instead of providing one working
interface from the beginning?

Thanks

> 
> 

Re: [PATCH v3 1/1] virtio-blk: add num_request_queues module parameter

[Leon Romanovsky]

On Sun, Sep 05, 2021 at 01:49:46AM -0700, Chaitanya Kulkarni wrote:
> 
> On 9/5/2021 12:46 AM, Leon Romanovsky wrote:
> > > +static unsigned int num_request_queues;
> > > +module_param_cb(num_request_queues, &queue_count_ops, &num_request_queues,
> > > +		0644);
> > > +MODULE_PARM_DESC(num_request_queues,
> > > +		 "Number of request queues to use for blk device. Should > 0");
> > > +
> > Won't it limit all virtio block devices to the same limit?
> > 
> > It is very common to see multiple virtio-blk devices on the same system
> > and they probably need different limits.
> > 
> > Thanks
> 
> 
> Without looking into the code, that can be done adding a configfs
> 
> interface and overriding a global value (module param) when it is set from
> 
> configfs.

So why should we do double work instead of providing one working
interface from the beginning?

Thanks

> 
> 
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH v3 1/1] virtio-blk: add num_request_queues module parameter

[Max Gurtovoy]

On 9/5/2021 1:19 PM, Leon Romanovsky wrote:
> On Sun, Sep 05, 2021 at 12:19:09PM +0300, Max Gurtovoy wrote:
>> On 9/5/2021 11:49 AM, Chaitanya Kulkarni wrote:
>>> On 9/5/2021 12:46 AM, Leon Romanovsky wrote:
>>>>> +static unsigned int num_request_queues;
>>>>> +module_param_cb(num_request_queues, &queue_count_ops,
>>>>> &num_request_queues,
>>>>> +        0644);
>>>>> +MODULE_PARM_DESC(num_request_queues,
>>>>> +         "Number of request queues to use for blk device.
>>>>> Should > 0");
>>>>> +
>>>> Won't it limit all virtio block devices to the same limit?
>>>>
>>>> It is very common to see multiple virtio-blk devices on the same system
>>>> and they probably need different limits.
>>>>
>>>> Thanks
>>>
>>> Without looking into the code, that can be done adding a configfs
>>>
>>> interface and overriding a global value (module param) when it is set
>>> from
>>>
>>> configfs.
>>>
>>>
>> You have many ways to overcome this issue.
>>
>> For example:
>>
>> # ls -l /sys/block/vda/mq/
>> drwxr-xr-x 18 root root 0 Sep  5 12:14 0
>> drwxr-xr-x 18 root root 0 Sep  5 12:14 1
>> drwxr-xr-x 18 root root 0 Sep  5 12:14 2
>> drwxr-xr-x 18 root root 0 Sep  5 12:14 3
>>
>> # echo virtio0 > /sys/bus/virtio/drivers/virtio_blk/unbind
>>
>> # echo 8 > /sys/module/virtio_blk/parameters/num_request_queues
> This is global to all virtio-blk devices.

You define a global module param but you bind/unbind a specific device.

Do you have a better way to control it ?

if the device is already probed, it's too late to change the queue_num.


>
>> # echo virtio0 > /sys/bus/virtio/drivers/virtio_blk/bind
>>
>> # ls -l /sys/block/vda/mq/
>> drwxr-xr-x 10 root root 0 Sep  5 12:17 0
>> drwxr-xr-x 10 root root 0 Sep  5 12:17 1
>> drwxr-xr-x 10 root root 0 Sep  5 12:17 2
>> drwxr-xr-x 10 root root 0 Sep  5 12:17 3
>> drwxr-xr-x 10 root root 0 Sep  5 12:17 4
>> drwxr-xr-x 10 root root 0 Sep  5 12:17 5
>> drwxr-xr-x 10 root root 0 Sep  5 12:17 6
>> drwxr-xr-x 10 root root 0 Sep  5 12:17 7
>>
>> -Max.
>>
>>

Re: [PATCH v3 1/1] virtio-blk: add num_request_queues module parameter

[Leon Romanovsky]

On Sun, Sep 05, 2021 at 02:16:58PM +0300, Max Gurtovoy wrote:
> 
> On 9/5/2021 1:19 PM, Leon Romanovsky wrote:
> > On Sun, Sep 05, 2021 at 12:19:09PM +0300, Max Gurtovoy wrote:
> > > On 9/5/2021 11:49 AM, Chaitanya Kulkarni wrote:
> > > > On 9/5/2021 12:46 AM, Leon Romanovsky wrote:
> > > > > > +static unsigned int num_request_queues;
> > > > > > +module_param_cb(num_request_queues, &queue_count_ops,
> > > > > > &num_request_queues,
> > > > > > +        0644);
> > > > > > +MODULE_PARM_DESC(num_request_queues,
> > > > > > +         "Number of request queues to use for blk device.
> > > > > > Should > 0");
> > > > > > +
> > > > > Won't it limit all virtio block devices to the same limit?
> > > > > 
> > > > > It is very common to see multiple virtio-blk devices on the same system
> > > > > and they probably need different limits.
> > > > > 
> > > > > Thanks
> > > > 
> > > > Without looking into the code, that can be done adding a configfs
> > > > 
> > > > interface and overriding a global value (module param) when it is set
> > > > from
> > > > 
> > > > configfs.
> > > > 
> > > > 
> > > You have many ways to overcome this issue.
> > > 
> > > For example:
> > > 
> > > # ls -l /sys/block/vda/mq/
> > > drwxr-xr-x 18 root root 0 Sep  5 12:14 0
> > > drwxr-xr-x 18 root root 0 Sep  5 12:14 1
> > > drwxr-xr-x 18 root root 0 Sep  5 12:14 2
> > > drwxr-xr-x 18 root root 0 Sep  5 12:14 3
> > > 
> > > # echo virtio0 > /sys/bus/virtio/drivers/virtio_blk/unbind
> > > 
> > > # echo 8 > /sys/module/virtio_blk/parameters/num_request_queues
> > This is global to all virtio-blk devices.
> 
> You define a global module param but you bind/unbind a specific device.
> 
> Do you have a better way to control it ?

One of the possible solutions will be to extend virtio bus to allow
setting of such pre-probe parameters. However I don't know if it is
really worth doing it,

> 
> if the device is already probed, it's too late to change the queue_num.
> 
> 
> > 
> > > # echo virtio0 > /sys/bus/virtio/drivers/virtio_blk/bind
> > > 
> > > # ls -l /sys/block/vda/mq/
> > > drwxr-xr-x 10 root root 0 Sep  5 12:17 0
> > > drwxr-xr-x 10 root root 0 Sep  5 12:17 1
> > > drwxr-xr-x 10 root root 0 Sep  5 12:17 2
> > > drwxr-xr-x 10 root root 0 Sep  5 12:17 3
> > > drwxr-xr-x 10 root root 0 Sep  5 12:17 4
> > > drwxr-xr-x 10 root root 0 Sep  5 12:17 5
> > > drwxr-xr-x 10 root root 0 Sep  5 12:17 6
> > > drwxr-xr-x 10 root root 0 Sep  5 12:17 7
> > > 
> > > -Max.
> > > 
> > > 

Re: [PATCH v3 1/1] virtio-blk: add num_request_queues module parameter

[Leon Romanovsky]

On Sun, Sep 05, 2021 at 02:16:58PM +0300, Max Gurtovoy wrote:
> 
> On 9/5/2021 1:19 PM, Leon Romanovsky wrote:
> > On Sun, Sep 05, 2021 at 12:19:09PM +0300, Max Gurtovoy wrote:
> > > On 9/5/2021 11:49 AM, Chaitanya Kulkarni wrote:
> > > > On 9/5/2021 12:46 AM, Leon Romanovsky wrote:
> > > > > > +static unsigned int num_request_queues;
> > > > > > +module_param_cb(num_request_queues, &queue_count_ops,
> > > > > > &num_request_queues,
> > > > > > +        0644);
> > > > > > +MODULE_PARM_DESC(num_request_queues,
> > > > > > +         "Number of request queues to use for blk device.
> > > > > > Should > 0");
> > > > > > +
> > > > > Won't it limit all virtio block devices to the same limit?
> > > > > 
> > > > > It is very common to see multiple virtio-blk devices on the same system
> > > > > and they probably need different limits.
> > > > > 
> > > > > Thanks
> > > > 
> > > > Without looking into the code, that can be done adding a configfs
> > > > 
> > > > interface and overriding a global value (module param) when it is set
> > > > from
> > > > 
> > > > configfs.
> > > > 
> > > > 
> > > You have many ways to overcome this issue.
> > > 
> > > For example:
> > > 
> > > # ls -l /sys/block/vda/mq/
> > > drwxr-xr-x 18 root root 0 Sep  5 12:14 0
> > > drwxr-xr-x 18 root root 0 Sep  5 12:14 1
> > > drwxr-xr-x 18 root root 0 Sep  5 12:14 2
> > > drwxr-xr-x 18 root root 0 Sep  5 12:14 3
> > > 
> > > # echo virtio0 > /sys/bus/virtio/drivers/virtio_blk/unbind
> > > 
> > > # echo 8 > /sys/module/virtio_blk/parameters/num_request_queues
> > This is global to all virtio-blk devices.
> 
> You define a global module param but you bind/unbind a specific device.
> 
> Do you have a better way to control it ?

One of the possible solutions will be to extend virtio bus to allow
setting of such pre-probe parameters. However I don't know if it is
really worth doing it,

> 
> if the device is already probed, it's too late to change the queue_num.
> 
> 
> > 
> > > # echo virtio0 > /sys/bus/virtio/drivers/virtio_blk/bind
> > > 
> > > # ls -l /sys/block/vda/mq/
> > > drwxr-xr-x 10 root root 0 Sep  5 12:17 0
> > > drwxr-xr-x 10 root root 0 Sep  5 12:17 1
> > > drwxr-xr-x 10 root root 0 Sep  5 12:17 2
> > > drwxr-xr-x 10 root root 0 Sep  5 12:17 3
> > > drwxr-xr-x 10 root root 0 Sep  5 12:17 4
> > > drwxr-xr-x 10 root root 0 Sep  5 12:17 5
> > > drwxr-xr-x 10 root root 0 Sep  5 12:17 6
> > > drwxr-xr-x 10 root root 0 Sep  5 12:17 7
> > > 
> > > -Max.
> > > 
> > > 
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH v3 1/1] virtio-blk: add num_request_queues module parameter

[Max Gurtovoy]

On 9/5/2021 4:10 PM, Leon Romanovsky wrote:
> On Sun, Sep 05, 2021 at 02:16:58PM +0300, Max Gurtovoy wrote:
>> On 9/5/2021 1:19 PM, Leon Romanovsky wrote:
>>> On Sun, Sep 05, 2021 at 12:19:09PM +0300, Max Gurtovoy wrote:
>>>> On 9/5/2021 11:49 AM, Chaitanya Kulkarni wrote:
>>>>> On 9/5/2021 12:46 AM, Leon Romanovsky wrote:
>>>>>>> +static unsigned int num_request_queues;
>>>>>>> +module_param_cb(num_request_queues, &queue_count_ops,
>>>>>>> &num_request_queues,
>>>>>>> +        0644);
>>>>>>> +MODULE_PARM_DESC(num_request_queues,
>>>>>>> +         "Number of request queues to use for blk device.
>>>>>>> Should > 0");
>>>>>>> +
>>>>>> Won't it limit all virtio block devices to the same limit?
>>>>>>
>>>>>> It is very common to see multiple virtio-blk devices on the same system
>>>>>> and they probably need different limits.
>>>>>>
>>>>>> Thanks
>>>>> Without looking into the code, that can be done adding a configfs
>>>>>
>>>>> interface and overriding a global value (module param) when it is set
>>>>> from
>>>>>
>>>>> configfs.
>>>>>
>>>>>
>>>> You have many ways to overcome this issue.
>>>>
>>>> For example:
>>>>
>>>> # ls -l /sys/block/vda/mq/
>>>> drwxr-xr-x 18 root root 0 Sep  5 12:14 0
>>>> drwxr-xr-x 18 root root 0 Sep  5 12:14 1
>>>> drwxr-xr-x 18 root root 0 Sep  5 12:14 2
>>>> drwxr-xr-x 18 root root 0 Sep  5 12:14 3
>>>>
>>>> # echo virtio0 > /sys/bus/virtio/drivers/virtio_blk/unbind
>>>>
>>>> # echo 8 > /sys/module/virtio_blk/parameters/num_request_queues
>>> This is global to all virtio-blk devices.
>> You define a global module param but you bind/unbind a specific device.
>>
>> Do you have a better way to control it ?
> One of the possible solutions will be to extend virtio bus to allow
> setting of such pre-probe parameters. However I don't know if it is
> really worth doing it,

So lets keep it as it is now.

Thanks.

>> if the device is already probed, it's too late to change the queue_num.
>>
>>
>>>> # echo virtio0 > /sys/bus/virtio/drivers/virtio_blk/bind
>>>>
>>>> # ls -l /sys/block/vda/mq/
>>>> drwxr-xr-x 10 root root 0 Sep  5 12:17 0
>>>> drwxr-xr-x 10 root root 0 Sep  5 12:17 1
>>>> drwxr-xr-x 10 root root 0 Sep  5 12:17 2
>>>> drwxr-xr-x 10 root root 0 Sep  5 12:17 3
>>>> drwxr-xr-x 10 root root 0 Sep  5 12:17 4
>>>> drwxr-xr-x 10 root root 0 Sep  5 12:17 5
>>>> drwxr-xr-x 10 root root 0 Sep  5 12:17 6
>>>> drwxr-xr-x 10 root root 0 Sep  5 12:17 7
>>>>
>>>> -Max.
>>>>
>>>>

Re: [PATCH v3 1/1] virtio-blk: add num_request_queues module parameter

[Michael S. Tsirkin]

On Sun, Sep 05, 2021 at 01:20:24PM +0300, Leon Romanovsky wrote:
> On Sun, Sep 05, 2021 at 01:49:46AM -0700, Chaitanya Kulkarni wrote:
> > 
> > On 9/5/2021 12:46 AM, Leon Romanovsky wrote:
> > > > +static unsigned int num_request_queues;
> > > > +module_param_cb(num_request_queues, &queue_count_ops, &num_request_queues,
> > > > +		0644);
> > > > +MODULE_PARM_DESC(num_request_queues,
> > > > +		 "Number of request queues to use for blk device. Should > 0");
> > > > +
> > > Won't it limit all virtio block devices to the same limit?
> > > 
> > > It is very common to see multiple virtio-blk devices on the same system
> > > and they probably need different limits.
> > > 
> > > Thanks
> > 
> > 
> > Without looking into the code, that can be done adding a configfs
> > 
> > interface and overriding a global value (module param) when it is set from
> > 
> > configfs.
> 
> So why should we do double work instead of providing one working
> interface from the beginning?
> 
> Thanks
> 
> > 
> > 

The main way to do it is really from the hypervisor. This one
is a pretty blunt instrument, Max here says it's useful to reduce
memory usage of the driver. If that's the usecase then a global limit
seems sufficient.

-- 
MST

Re: [PATCH v3 1/1] virtio-blk: add num_request_queues module parameter

[Michael S. Tsirkin]

On Sun, Sep 05, 2021 at 01:20:24PM +0300, Leon Romanovsky wrote:
> On Sun, Sep 05, 2021 at 01:49:46AM -0700, Chaitanya Kulkarni wrote:
> > 
> > On 9/5/2021 12:46 AM, Leon Romanovsky wrote:
> > > > +static unsigned int num_request_queues;
> > > > +module_param_cb(num_request_queues, &queue_count_ops, &num_request_queues,
> > > > +		0644);
> > > > +MODULE_PARM_DESC(num_request_queues,
> > > > +		 "Number of request queues to use for blk device. Should > 0");
> > > > +
> > > Won't it limit all virtio block devices to the same limit?
> > > 
> > > It is very common to see multiple virtio-blk devices on the same system
> > > and they probably need different limits.
> > > 
> > > Thanks
> > 
> > 
> > Without looking into the code, that can be done adding a configfs
> > 
> > interface and overriding a global value (module param) when it is set from
> > 
> > configfs.
> 
> So why should we do double work instead of providing one working
> interface from the beginning?
> 
> Thanks
> 
> > 
> > 

The main way to do it is really from the hypervisor. This one
is a pretty blunt instrument, Max here says it's useful to reduce
memory usage of the driver. If that's the usecase then a global limit
seems sufficient.

-- 
MST

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH v3 1/1] virtio-blk: add num_request_queues module parameter

[Leon Romanovsky]

On Sun, Sep 05, 2021 at 11:15:16AM -0400, Michael S. Tsirkin wrote:
> On Sun, Sep 05, 2021 at 01:20:24PM +0300, Leon Romanovsky wrote:
> > On Sun, Sep 05, 2021 at 01:49:46AM -0700, Chaitanya Kulkarni wrote:
> > > 
> > > On 9/5/2021 12:46 AM, Leon Romanovsky wrote:
> > > > > +static unsigned int num_request_queues;
> > > > > +module_param_cb(num_request_queues, &queue_count_ops, &num_request_queues,
> > > > > +		0644);
> > > > > +MODULE_PARM_DESC(num_request_queues,
> > > > > +		 "Number of request queues to use for blk device. Should > 0");
> > > > > +
> > > > Won't it limit all virtio block devices to the same limit?
> > > > 
> > > > It is very common to see multiple virtio-blk devices on the same system
> > > > and they probably need different limits.
> > > > 
> > > > Thanks
> > > 
> > > 
> > > Without looking into the code, that can be done adding a configfs
> > > 
> > > interface and overriding a global value (module param) when it is set from
> > > 
> > > configfs.
> > 
> > So why should we do double work instead of providing one working
> > interface from the beginning?
> > 
> > Thanks
> > 
> > > 
> > > 
> 
> The main way to do it is really from the hypervisor. This one
> is a pretty blunt instrument, Max here says it's useful to reduce
> memory usage of the driver. If that's the usecase then a global limit
> seems sufficient.

How memory will you reduce? It is worth to write it in the commit message.

Thanks

> 
> -- 
> MST
> 

Re: [PATCH v3 1/1] virtio-blk: add num_request_queues module parameter

[Leon Romanovsky]

On Sun, Sep 05, 2021 at 11:15:16AM -0400, Michael S. Tsirkin wrote:
> On Sun, Sep 05, 2021 at 01:20:24PM +0300, Leon Romanovsky wrote:
> > On Sun, Sep 05, 2021 at 01:49:46AM -0700, Chaitanya Kulkarni wrote:
> > > 
> > > On 9/5/2021 12:46 AM, Leon Romanovsky wrote:
> > > > > +static unsigned int num_request_queues;
> > > > > +module_param_cb(num_request_queues, &queue_count_ops, &num_request_queues,
> > > > > +		0644);
> > > > > +MODULE_PARM_DESC(num_request_queues,
> > > > > +		 "Number of request queues to use for blk device. Should > 0");
> > > > > +
> > > > Won't it limit all virtio block devices to the same limit?
> > > > 
> > > > It is very common to see multiple virtio-blk devices on the same system
> > > > and they probably need different limits.
> > > > 
> > > > Thanks
> > > 
> > > 
> > > Without looking into the code, that can be done adding a configfs
> > > 
> > > interface and overriding a global value (module param) when it is set from
> > > 
> > > configfs.
> > 
> > So why should we do double work instead of providing one working
> > interface from the beginning?
> > 
> > Thanks
> > 
> > > 
> > > 
> 
> The main way to do it is really from the hypervisor. This one
> is a pretty blunt instrument, Max here says it's useful to reduce
> memory usage of the driver. If that's the usecase then a global limit
> seems sufficient.

How memory will you reduce? It is worth to write it in the commit message.

Thanks

> 
> -- 
> MST
> 
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH v3 1/1] virtio-blk: add num_request_queues module parameter

[Michael S. Tsirkin]

On Thu, Sep 02, 2021 at 11:46:22PM +0300, Max Gurtovoy wrote:
> Sometimes a user would like to control the amount of request queues to
> be created for a block device. For example, for limiting the memory
> footprint of virtio-blk devices.
> 
> Reviewed-by: Christoph Hellwig <hch@lst.de>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> ---
> 
> changes from v2:
>  - renamed num_io_queues to num_request_queues (from Stefan)
>  - added Reviewed-by signatures (from Stefan and Christoph)
> 
> changes from v1:
>  - use param_set_uint_minmax (from Christoph)
>  - added "Should > 0" to module description
> 
> Note: This commit apply on top of Jens's branch for-5.15/drivers
> 
> ---
>  drivers/block/virtio_blk.c | 21 ++++++++++++++++++++-
>  1 file changed, 20 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
> index 4b49df2dfd23..aaa2833a4734 100644
> --- a/drivers/block/virtio_blk.c
> +++ b/drivers/block/virtio_blk.c
> @@ -24,6 +24,23 @@
>  /* The maximum number of sg elements that fit into a virtqueue */
>  #define VIRTIO_BLK_MAX_SG_ELEMS 32768
>  
> +static int virtblk_queue_count_set(const char *val,
> +		const struct kernel_param *kp)
> +{
> +	return param_set_uint_minmax(val, kp, 1, nr_cpu_ids);
> +}
> +
> +static const struct kernel_param_ops queue_count_ops = {
> +	.set = virtblk_queue_count_set,
> +	.get = param_get_uint,
> +};
> +
> +static unsigned int num_request_queues;
> +module_param_cb(num_request_queues, &queue_count_ops, &num_request_queues,
> +		0644);
> +MODULE_PARM_DESC(num_request_queues,
> +		 "Number of request queues to use for blk device. Should > 0");
> +
>  static int major;
>  static DEFINE_IDA(vd_index_ida);
>  

I wasn't happy with the message here so I tweaked it.

Please look at it in linux-next and confirm. Thanks!


> @@ -501,7 +518,9 @@ static int init_vq(struct virtio_blk *vblk)
>  	if (err)
>  		num_vqs = 1;
>  
> -	num_vqs = min_t(unsigned int, nr_cpu_ids, num_vqs);
> +	num_vqs = min_t(unsigned int,
> +			min_not_zero(num_request_queues, nr_cpu_ids),
> +			num_vqs);
>  
>  	vblk->vqs = kmalloc_array(num_vqs, sizeof(*vblk->vqs), GFP_KERNEL);
>  	if (!vblk->vqs)
> -- 
> 2.18.1

Re: [PATCH v3 1/1] virtio-blk: add num_request_queues module parameter

[Michael S. Tsirkin]

On Thu, Sep 02, 2021 at 11:46:22PM +0300, Max Gurtovoy wrote:
> Sometimes a user would like to control the amount of request queues to
> be created for a block device. For example, for limiting the memory
> footprint of virtio-blk devices.
> 
> Reviewed-by: Christoph Hellwig <hch@lst.de>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> ---
> 
> changes from v2:
>  - renamed num_io_queues to num_request_queues (from Stefan)
>  - added Reviewed-by signatures (from Stefan and Christoph)
> 
> changes from v1:
>  - use param_set_uint_minmax (from Christoph)
>  - added "Should > 0" to module description
> 
> Note: This commit apply on top of Jens's branch for-5.15/drivers
> 
> ---
>  drivers/block/virtio_blk.c | 21 ++++++++++++++++++++-
>  1 file changed, 20 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
> index 4b49df2dfd23..aaa2833a4734 100644
> --- a/drivers/block/virtio_blk.c
> +++ b/drivers/block/virtio_blk.c
> @@ -24,6 +24,23 @@
>  /* The maximum number of sg elements that fit into a virtqueue */
>  #define VIRTIO_BLK_MAX_SG_ELEMS 32768
>  
> +static int virtblk_queue_count_set(const char *val,
> +		const struct kernel_param *kp)
> +{
> +	return param_set_uint_minmax(val, kp, 1, nr_cpu_ids);
> +}
> +
> +static const struct kernel_param_ops queue_count_ops = {
> +	.set = virtblk_queue_count_set,
> +	.get = param_get_uint,
> +};
> +
> +static unsigned int num_request_queues;
> +module_param_cb(num_request_queues, &queue_count_ops, &num_request_queues,
> +		0644);
> +MODULE_PARM_DESC(num_request_queues,
> +		 "Number of request queues to use for blk device. Should > 0");
> +
>  static int major;
>  static DEFINE_IDA(vd_index_ida);
>  

I wasn't happy with the message here so I tweaked it.

Please look at it in linux-next and confirm. Thanks!


> @@ -501,7 +518,9 @@ static int init_vq(struct virtio_blk *vblk)
>  	if (err)
>  		num_vqs = 1;
>  
> -	num_vqs = min_t(unsigned int, nr_cpu_ids, num_vqs);
> +	num_vqs = min_t(unsigned int,
> +			min_not_zero(num_request_queues, nr_cpu_ids),
> +			num_vqs);
>  
>  	vblk->vqs = kmalloc_array(num_vqs, sizeof(*vblk->vqs), GFP_KERNEL);
>  	if (!vblk->vqs)
> -- 
> 2.18.1

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH v3 1/1] virtio-blk: add num_request_queues module parameter

[Max Gurtovoy]

On 10/22/2021 12:30 PM, Michael S. Tsirkin wrote:
> On Thu, Sep 02, 2021 at 11:46:22PM +0300, Max Gurtovoy wrote:
>> Sometimes a user would like to control the amount of request queues to
>> be created for a block device. For example, for limiting the memory
>> footprint of virtio-blk devices.
>>
>> Reviewed-by: Christoph Hellwig <hch@lst.de>
>> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
>> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>> ---
>>
>> changes from v2:
>>   - renamed num_io_queues to num_request_queues (from Stefan)
>>   - added Reviewed-by signatures (from Stefan and Christoph)
>>
>> changes from v1:
>>   - use param_set_uint_minmax (from Christoph)
>>   - added "Should > 0" to module description
>>
>> Note: This commit apply on top of Jens's branch for-5.15/drivers
>>
>> ---
>>   drivers/block/virtio_blk.c | 21 ++++++++++++++++++++-
>>   1 file changed, 20 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
>> index 4b49df2dfd23..aaa2833a4734 100644
>> --- a/drivers/block/virtio_blk.c
>> +++ b/drivers/block/virtio_blk.c
>> @@ -24,6 +24,23 @@
>>   /* The maximum number of sg elements that fit into a virtqueue */
>>   #define VIRTIO_BLK_MAX_SG_ELEMS 32768
>>   
>> +static int virtblk_queue_count_set(const char *val,
>> +		const struct kernel_param *kp)
>> +{
>> +	return param_set_uint_minmax(val, kp, 1, nr_cpu_ids);
>> +}
>> +
>> +static const struct kernel_param_ops queue_count_ops = {
>> +	.set = virtblk_queue_count_set,
>> +	.get = param_get_uint,
>> +};
>> +
>> +static unsigned int num_request_queues;
>> +module_param_cb(num_request_queues, &queue_count_ops, &num_request_queues,
>> +		0644);
>> +MODULE_PARM_DESC(num_request_queues,
>> +		 "Number of request queues to use for blk device. Should > 0");
>> +
>>   static int major;
>>   static DEFINE_IDA(vd_index_ida);
>>   
> I wasn't happy with the message here so I tweaked it.
>
> Please look at it in linux-next and confirm. Thanks!

Looks good.


>
>
>> @@ -501,7 +518,9 @@ static int init_vq(struct virtio_blk *vblk)
>>   	if (err)
>>   		num_vqs = 1;
>>   
>> -	num_vqs = min_t(unsigned int, nr_cpu_ids, num_vqs);
>> +	num_vqs = min_t(unsigned int,
>> +			min_not_zero(num_request_queues, nr_cpu_ids),
>> +			num_vqs);
>>   
>>   	vblk->vqs = kmalloc_array(num_vqs, sizeof(*vblk->vqs), GFP_KERNEL);
>>   	if (!vblk->vqs)
>> -- 
>> 2.18.1

Re: [PATCH v3 1/1] virtio-blk: add num_request_queues module parameter

[Max Gurtovoy]

On 10/24/2021 10:19 AM, Max Gurtovoy wrote:
>
> On 10/22/2021 12:30 PM, Michael S. Tsirkin wrote:
>> On Thu, Sep 02, 2021 at 11:46:22PM +0300, Max Gurtovoy wrote:
>>> Sometimes a user would like to control the amount of request queues to
>>> be created for a block device. For example, for limiting the memory
>>> footprint of virtio-blk devices.
>>>
>>> Reviewed-by: Christoph Hellwig <hch@lst.de>
>>> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
>>> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>>> ---
>>>
>>> changes from v2:
>>>   - renamed num_io_queues to num_request_queues (from Stefan)
>>>   - added Reviewed-by signatures (from Stefan and Christoph)
>>>
>>> changes from v1:
>>>   - use param_set_uint_minmax (from Christoph)
>>>   - added "Should > 0" to module description
>>>
>>> Note: This commit apply on top of Jens's branch for-5.15/drivers
>>>
>>> ---
>>>   drivers/block/virtio_blk.c | 21 ++++++++++++++++++++-
>>>   1 file changed, 20 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
>>> index 4b49df2dfd23..aaa2833a4734 100644
>>> --- a/drivers/block/virtio_blk.c
>>> +++ b/drivers/block/virtio_blk.c
>>> @@ -24,6 +24,23 @@
>>>   /* The maximum number of sg elements that fit into a virtqueue */
>>>   #define VIRTIO_BLK_MAX_SG_ELEMS 32768
>>>   +static int virtblk_queue_count_set(const char *val,
>>> +        const struct kernel_param *kp)
>>> +{
>>> +    return param_set_uint_minmax(val, kp, 1, nr_cpu_ids);
>>> +}
>>> +

BTW, I've noticed in your new message you allow setting 0 so you might 
want to change the code to

param_set_uint_minmax(val, kp, 0, nr_cpu_ids);

to a case a user will load the module with num_request_queues=0.

>>> +static const struct kernel_param_ops queue_count_ops = {
>>> +    .set = virtblk_queue_count_set,
>>> +    .get = param_get_uint,
>>> +};
>>> +
>>> +static unsigned int num_request_queues;
>>> +module_param_cb(num_request_queues, &queue_count_ops, 
>>> &num_request_queues,
>>> +        0644);
>>> +MODULE_PARM_DESC(num_request_queues,
>>> +         "Number of request queues to use for blk device. Should > 
>>> 0");
>>> +
>>>   static int major;
>>>   static DEFINE_IDA(vd_index_ida);
>> I wasn't happy with the message here so I tweaked it.
>>
>> Please look at it in linux-next and confirm. Thanks!
>
> Looks good.
>
>
>>
>>
>>> @@ -501,7 +518,9 @@ static int init_vq(struct virtio_blk *vblk)
>>>       if (err)
>>>           num_vqs = 1;
>>>   -    num_vqs = min_t(unsigned int, nr_cpu_ids, num_vqs);
>>> +    num_vqs = min_t(unsigned int,
>>> +            min_not_zero(num_request_queues, nr_cpu_ids),
>>> +            num_vqs);
>>>         vblk->vqs = kmalloc_array(num_vqs, sizeof(*vblk->vqs), 
>>> GFP_KERNEL);
>>>       if (!vblk->vqs)
>>> -- 
>>> 2.18.1

Re: [PATCH v3 1/1] virtio-blk: add num_request_queues module parameter

[Michael S. Tsirkin]

On Sun, Oct 24, 2021 at 11:12:26AM +0300, Max Gurtovoy wrote:
> 
> On 10/24/2021 10:19 AM, Max Gurtovoy wrote:
> > 
> > On 10/22/2021 12:30 PM, Michael S. Tsirkin wrote:
> > > On Thu, Sep 02, 2021 at 11:46:22PM +0300, Max Gurtovoy wrote:
> > > > Sometimes a user would like to control the amount of request queues to
> > > > be created for a block device. For example, for limiting the memory
> > > > footprint of virtio-blk devices.
> > > > 
> > > > Reviewed-by: Christoph Hellwig <hch@lst.de>
> > > > Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> > > > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> > > > ---
> > > > 
> > > > changes from v2:
> > > >   - renamed num_io_queues to num_request_queues (from Stefan)
> > > >   - added Reviewed-by signatures (from Stefan and Christoph)
> > > > 
> > > > changes from v1:
> > > >   - use param_set_uint_minmax (from Christoph)
> > > >   - added "Should > 0" to module description
> > > > 
> > > > Note: This commit apply on top of Jens's branch for-5.15/drivers
> > > > 
> > > > ---
> > > >   drivers/block/virtio_blk.c | 21 ++++++++++++++++++++-
> > > >   1 file changed, 20 insertions(+), 1 deletion(-)
> > > > 
> > > > diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
> > > > index 4b49df2dfd23..aaa2833a4734 100644
> > > > --- a/drivers/block/virtio_blk.c
> > > > +++ b/drivers/block/virtio_blk.c
> > > > @@ -24,6 +24,23 @@
> > > >   /* The maximum number of sg elements that fit into a virtqueue */
> > > >   #define VIRTIO_BLK_MAX_SG_ELEMS 32768
> > > >   +static int virtblk_queue_count_set(const char *val,
> > > > +        const struct kernel_param *kp)
> > > > +{
> > > > +    return param_set_uint_minmax(val, kp, 1, nr_cpu_ids);
> > > > +}
> > > > +
> 
> BTW, I've noticed in your new message you allow setting 0 so you might want
> to change the code to
> 
> param_set_uint_minmax(val, kp, 0, nr_cpu_ids);
> 
> to a case a user will load the module with num_request_queues=0.

Oh. So as written the default forces 1 queue?
Send a patch please!

> > > > +static const struct kernel_param_ops queue_count_ops = {
> > > > +    .set = virtblk_queue_count_set,
> > > > +    .get = param_get_uint,
> > > > +};
> > > > +
> > > > +static unsigned int num_request_queues;
> > > > +module_param_cb(num_request_queues, &queue_count_ops,
> > > > &num_request_queues,
> > > > +        0644);
> > > > +MODULE_PARM_DESC(num_request_queues,
> > > > +         "Number of request queues to use for blk device.
> > > > Should > 0");
> > > > +
> > > >   static int major;
> > > >   static DEFINE_IDA(vd_index_ida);
> > > I wasn't happy with the message here so I tweaked it.
> > > 
> > > Please look at it in linux-next and confirm. Thanks!
> > 
> > Looks good.
> > 
> > 
> > > 
> > > 
> > > > @@ -501,7 +518,9 @@ static int init_vq(struct virtio_blk *vblk)
> > > >       if (err)
> > > >           num_vqs = 1;
> > > >   -    num_vqs = min_t(unsigned int, nr_cpu_ids, num_vqs);
> > > > +    num_vqs = min_t(unsigned int,
> > > > +            min_not_zero(num_request_queues, nr_cpu_ids),
> > > > +            num_vqs);
> > > >         vblk->vqs = kmalloc_array(num_vqs, sizeof(*vblk->vqs),
> > > > GFP_KERNEL);
> > > >       if (!vblk->vqs)
> > > > -- 
> > > > 2.18.1

Re: [PATCH v3 1/1] virtio-blk: add num_request_queues module parameter

[Michael S. Tsirkin]

On Sun, Oct 24, 2021 at 11:12:26AM +0300, Max Gurtovoy wrote:
> 
> On 10/24/2021 10:19 AM, Max Gurtovoy wrote:
> > 
> > On 10/22/2021 12:30 PM, Michael S. Tsirkin wrote:
> > > On Thu, Sep 02, 2021 at 11:46:22PM +0300, Max Gurtovoy wrote:
> > > > Sometimes a user would like to control the amount of request queues to
> > > > be created for a block device. For example, for limiting the memory
> > > > footprint of virtio-blk devices.
> > > > 
> > > > Reviewed-by: Christoph Hellwig <hch@lst.de>
> > > > Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> > > > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> > > > ---
> > > > 
> > > > changes from v2:
> > > >   - renamed num_io_queues to num_request_queues (from Stefan)
> > > >   - added Reviewed-by signatures (from Stefan and Christoph)
> > > > 
> > > > changes from v1:
> > > >   - use param_set_uint_minmax (from Christoph)
> > > >   - added "Should > 0" to module description
> > > > 
> > > > Note: This commit apply on top of Jens's branch for-5.15/drivers
> > > > 
> > > > ---
> > > >   drivers/block/virtio_blk.c | 21 ++++++++++++++++++++-
> > > >   1 file changed, 20 insertions(+), 1 deletion(-)
> > > > 
> > > > diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
> > > > index 4b49df2dfd23..aaa2833a4734 100644
> > > > --- a/drivers/block/virtio_blk.c
> > > > +++ b/drivers/block/virtio_blk.c
> > > > @@ -24,6 +24,23 @@
> > > >   /* The maximum number of sg elements that fit into a virtqueue */
> > > >   #define VIRTIO_BLK_MAX_SG_ELEMS 32768
> > > >   +static int virtblk_queue_count_set(const char *val,
> > > > +        const struct kernel_param *kp)
> > > > +{
> > > > +    return param_set_uint_minmax(val, kp, 1, nr_cpu_ids);
> > > > +}
> > > > +
> 
> BTW, I've noticed in your new message you allow setting 0 so you might want
> to change the code to
> 
> param_set_uint_minmax(val, kp, 0, nr_cpu_ids);
> 
> to a case a user will load the module with num_request_queues=0.

Oh. So as written the default forces 1 queue?
Send a patch please!

> > > > +static const struct kernel_param_ops queue_count_ops = {
> > > > +    .set = virtblk_queue_count_set,
> > > > +    .get = param_get_uint,
> > > > +};
> > > > +
> > > > +static unsigned int num_request_queues;
> > > > +module_param_cb(num_request_queues, &queue_count_ops,
> > > > &num_request_queues,
> > > > +        0644);
> > > > +MODULE_PARM_DESC(num_request_queues,
> > > > +         "Number of request queues to use for blk device.
> > > > Should > 0");
> > > > +
> > > >   static int major;
> > > >   static DEFINE_IDA(vd_index_ida);
> > > I wasn't happy with the message here so I tweaked it.
> > > 
> > > Please look at it in linux-next and confirm. Thanks!
> > 
> > Looks good.
> > 
> > 
> > > 
> > > 
> > > > @@ -501,7 +518,9 @@ static int init_vq(struct virtio_blk *vblk)
> > > >       if (err)
> > > >           num_vqs = 1;
> > > >   -    num_vqs = min_t(unsigned int, nr_cpu_ids, num_vqs);
> > > > +    num_vqs = min_t(unsigned int,
> > > > +            min_not_zero(num_request_queues, nr_cpu_ids),
> > > > +            num_vqs);
> > > >         vblk->vqs = kmalloc_array(num_vqs, sizeof(*vblk->vqs),
> > > > GFP_KERNEL);
> > > >       if (!vblk->vqs)
> > > > -- 
> > > > 2.18.1

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH v3 1/1] virtio-blk: add num_request_queues module parameter

[Max Gurtovoy]

On 10/24/2021 11:48 AM, Michael S. Tsirkin wrote:
> On Sun, Oct 24, 2021 at 11:12:26AM +0300, Max Gurtovoy wrote:
>> On 10/24/2021 10:19 AM, Max Gurtovoy wrote:
>>> On 10/22/2021 12:30 PM, Michael S. Tsirkin wrote:
>>>> On Thu, Sep 02, 2021 at 11:46:22PM +0300, Max Gurtovoy wrote:
>>>>> Sometimes a user would like to control the amount of request queues to
>>>>> be created for a block device. For example, for limiting the memory
>>>>> footprint of virtio-blk devices.
>>>>>
>>>>> Reviewed-by: Christoph Hellwig <hch@lst.de>
>>>>> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
>>>>> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>>>>> ---
>>>>>
>>>>> changes from v2:
>>>>>    - renamed num_io_queues to num_request_queues (from Stefan)
>>>>>    - added Reviewed-by signatures (from Stefan and Christoph)
>>>>>
>>>>> changes from v1:
>>>>>    - use param_set_uint_minmax (from Christoph)
>>>>>    - added "Should > 0" to module description
>>>>>
>>>>> Note: This commit apply on top of Jens's branch for-5.15/drivers
>>>>>
>>>>> ---
>>>>>    drivers/block/virtio_blk.c | 21 ++++++++++++++++++++-
>>>>>    1 file changed, 20 insertions(+), 1 deletion(-)
>>>>>
>>>>> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
>>>>> index 4b49df2dfd23..aaa2833a4734 100644
>>>>> --- a/drivers/block/virtio_blk.c
>>>>> +++ b/drivers/block/virtio_blk.c
>>>>> @@ -24,6 +24,23 @@
>>>>>    /* The maximum number of sg elements that fit into a virtqueue */
>>>>>    #define VIRTIO_BLK_MAX_SG_ELEMS 32768
>>>>>    +static int virtblk_queue_count_set(const char *val,
>>>>> +        const struct kernel_param *kp)
>>>>> +{
>>>>> +    return param_set_uint_minmax(val, kp, 1, nr_cpu_ids);
>>>>> +}
>>>>> +
>> BTW, I've noticed in your new message you allow setting 0 so you might want
>> to change the code to
>>
>> param_set_uint_minmax(val, kp, 0, nr_cpu_ids);
>>
>> to a case a user will load the module with num_request_queues=0.
> Oh. So as written the default forces 1 queue?
> Send a patch please!

No. The default (if nobody touch this param) is 0 and everything works 
as today. but the user can't do "modprobe virtio_blk num_request_queues=0".

My comment was right (should be set > 0).

You changed and wrote "0 for no limit." but the user can't set to 0. So 
if you want the user to change to 0, please change the above code as I 
mentioned.


>
>>>>> +static const struct kernel_param_ops queue_count_ops = {
>>>>> +    .set = virtblk_queue_count_set,
>>>>> +    .get = param_get_uint,
>>>>> +};
>>>>> +
>>>>> +static unsigned int num_request_queues;
>>>>> +module_param_cb(num_request_queues, &queue_count_ops,
>>>>> &num_request_queues,
>>>>> +        0644);
>>>>> +MODULE_PARM_DESC(num_request_queues,
>>>>> +         "Number of request queues to use for blk device.
>>>>> Should > 0");
>>>>> +
>>>>>    static int major;
>>>>>    static DEFINE_IDA(vd_index_ida);
>>>> I wasn't happy with the message here so I tweaked it.
>>>>
>>>> Please look at it in linux-next and confirm. Thanks!
>>> Looks good.
>>>
>>>
>>>>
>>>>> @@ -501,7 +518,9 @@ static int init_vq(struct virtio_blk *vblk)
>>>>>        if (err)
>>>>>            num_vqs = 1;
>>>>>    -    num_vqs = min_t(unsigned int, nr_cpu_ids, num_vqs);
>>>>> +    num_vqs = min_t(unsigned int,
>>>>> +            min_not_zero(num_request_queues, nr_cpu_ids),
>>>>> +            num_vqs);
>>>>>          vblk->vqs = kmalloc_array(num_vqs, sizeof(*vblk->vqs),
>>>>> GFP_KERNEL);
>>>>>        if (!vblk->vqs)
>>>>> -- 
>>>>> 2.18.1