From: "Michael S. Tsirkin" <mst@redhat.com>
To: Dongli Zhang <dongli.zhang@oracle.com>
Cc: virtualization@lists.linux-foundation.org,
linux-block@vger.kernel.org, axboe@kernel.dk,
jasowang@redhat.com, linux-kernel@vger.kernel.org
Subject: Re: virtio-blk: should num_vqs be limited by num_possible_cpus()?
Date: Thu, 14 Mar 2019 08:32:58 -0400 [thread overview]
Message-ID: <20190314082926-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <e4afe4c5-0262-4500-aeec-60f30734b4fc@default>
On Tue, Mar 12, 2019 at 10:22:46AM -0700, Dongli Zhang wrote:
> I observed that there is one msix vector for config and one shared vector
> for all queues in below qemu cmdline, when the num-queues for virtio-blk
> is more than the number of possible cpus:
>
> qemu: "-smp 4" while "-device virtio-blk-pci,drive=drive-0,id=virtblk0,num-queues=6"
So why do this?
> # cat /proc/interrupts
> CPU0 CPU1 CPU2 CPU3
> ... ...
> 24: 0 0 0 0 PCI-MSI 65536-edge virtio0-config
> 25: 0 0 0 59 PCI-MSI 65537-edge virtio0-virtqueues
> ... ...
>
>
> However, when num-queues is the same as number of possible cpus:
>
> qemu: "-smp 4" while "-device virtio-blk-pci,drive=drive-0,id=virtblk0,num-queues=4"
>
> # cat /proc/interrupts
> CPU0 CPU1 CPU2 CPU3
> ... ...
> 24: 0 0 0 0 PCI-MSI 65536-edge virtio0-config
> 25: 2 0 0 0 PCI-MSI 65537-edge virtio0-req.0
> 26: 0 35 0 0 PCI-MSI 65538-edge virtio0-req.1
> 27: 0 0 32 0 PCI-MSI 65539-edge virtio0-req.2
> 28: 0 0 0 0 PCI-MSI 65540-edge virtio0-req.3
> ... ...
>
> In above case, there is one msix vector per queue.
>
>
> This is because the max number of queues is not limited by the number of
> possible cpus.
>
> By default, nvme (regardless about write_queues and poll_queues) and
> xen-blkfront limit the number of queues with num_possible_cpus().
>
>
> Is this by design on purpose, or can we fix with below?
>
>
> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
> index 4bc083b..df95ce3 100644
> --- a/drivers/block/virtio_blk.c
> +++ b/drivers/block/virtio_blk.c
> @@ -513,6 +513,8 @@ static int init_vq(struct virtio_blk *vblk)
> if (err)
> num_vqs = 1;
>
> + num_vqs = min(num_possible_cpus(), num_vqs);
> +
> vblk->vqs = kmalloc_array(num_vqs, sizeof(*vblk->vqs), GFP_KERNEL);
> if (!vblk->vqs)
> return -ENOMEM;
> --
>
>
> PS: The same issue is applicable to virtio-scsi as well.
>
> Thank you very much!
>
> Dongli Zhang
I don't think this will address the issue if there's vcpu hotplug though.
Because it's not about num_possible_cpus it's about the # of active VCPUs,
right? Does block hangle CPU hotplug generally?
We could maybe address that by switching vq to msi vector mapping in
a cpu hotplug notifier...
--
MST
next prev parent reply other threads:[~2019-03-14 12:33 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-03-12 17:22 virtio-blk: should num_vqs be limited by num_possible_cpus()? Dongli Zhang
2019-03-12 17:33 ` Cornelia Huck
2019-03-13 3:26 ` Dongli Zhang
2019-03-13 9:39 ` Cornelia Huck
2019-03-14 6:12 ` Dongli Zhang
2019-03-14 12:13 ` Cornelia Huck
2019-03-14 16:08 ` Dongli Zhang
2019-03-15 4:50 ` Jason Wang
2019-03-15 12:41 ` Cornelia Huck
2019-03-18 7:47 ` Jason Wang
2019-03-19 2:22 ` Dongli Zhang
2019-03-20 12:53 ` Jason Wang
2019-03-21 2:14 ` Dongli Zhang
2019-03-21 15:57 ` Stefan Hajnoczi
2019-03-14 12:32 ` Michael S. Tsirkin [this message]
2019-03-14 15:36 ` Dongli Zhang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190314082926-mutt-send-email-mst@kernel.org \
--to=mst@redhat.com \
--cc=axboe@kernel.dk \
--cc=dongli.zhang@oracle.com \
--cc=jasowang@redhat.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=virtualization@lists.linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).