Re: [PATCH 1/1] virtio-blk: add num_io_queues module parameter

[Max Gurtovoy <mgurtovoy@nvidia.com>]

On 8/31/2021 12:48 AM, Michael S. Tsirkin wrote:
> On Mon, Aug 30, 2021 at 03:00:23PM +0300, Max Gurtovoy wrote:
>> Sometimes a user would like to control the amount of IO queues to be
>> created for a block device. For example, for limiting the memory
>> footprint of virtio-blk devices.
>>
>> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>
> Hmm. It's already limited by # of CPUs... Why not just limit
> from the hypervisor side? What's the actual use-case here?

Limiting and minimizing resource allocation is a real use case.

# of CPUs today might be 64 or 128. HW virtio-blk device might have this 
amount of queues (or at least 32).

But it's a waste to use all the queues since the device may reach to max 
IOPs with less queues. Multiply this by 16 or 32 devices we get a lot of 
memory wasted without a real need.

It's a common configuration we do in NVMf connect command and it can 
also be seen in other drivers in some variation (null_blk.submit_queues, 
ib_srp.ch_count and more).

Another use case is to add some flexibility for QOS.

Also if we can set the queue depth, it's a good idea to control the 
queue count as well.

If no objections, I'll take the comment from Christoph and send v2.

>
>> ---
>>   drivers/block/virtio_blk.c | 26 +++++++++++++++++++++++++-
>>   1 file changed, 25 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
>> index e574fbf5e6df..77e8468e8593 100644
>> --- a/drivers/block/virtio_blk.c
>> +++ b/drivers/block/virtio_blk.c
>> @@ -24,6 +24,28 @@
>>   /* The maximum number of sg elements that fit into a virtqueue */
>>   #define VIRTIO_BLK_MAX_SG_ELEMS 32768
>>   
>> +static int virtblk_queue_count_set(const char *val,
>> +		const struct kernel_param *kp)
>> +{
>> +	unsigned int n;
>> +	int ret;
>> +
>> +	ret = kstrtouint(val, 10, &n);
>> +	if (ret != 0 || n > nr_cpu_ids)
>> +		return -EINVAL;
>> +	return param_set_uint(val, kp);
>> +}
>> +
>> +static const struct kernel_param_ops queue_count_ops = {
>> +	.set = virtblk_queue_count_set,
>> +	.get = param_get_uint,
>> +};
>> +
>> +static unsigned int num_io_queues;
>> +module_param_cb(num_io_queues, &queue_count_ops, &num_io_queues, 0644);
>> +MODULE_PARM_DESC(num_io_queues,
>> +		 "Number of IO virt queues to use for blk device.");
>> +
>>   static int major;
>>   static DEFINE_IDA(vd_index_ida);
>>   
>> @@ -501,7 +523,9 @@ static int init_vq(struct virtio_blk *vblk)
>>   	if (err)
>>   		num_vqs = 1;
>>   
>> -	num_vqs = min_t(unsigned int, nr_cpu_ids, num_vqs);
>> +	num_vqs = min_t(unsigned int,
>> +			min_not_zero(num_io_queues, nr_cpu_ids),
>> +			num_vqs);
>>   
>>   	vblk->vqs = kmalloc_array(num_vqs, sizeof(*vblk->vqs), GFP_KERNEL);
>>   	if (!vblk->vqs)
>> -- 
>> 2.18.1