Inconsistent virtio-net-pci MSI vector count

Inconsistent virtio-net-pci MSI vector count

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 931 bytes --]

Hi Jason,
I stumbled across something strange with virtio-net multi-queue today.
It doesn't seem to be a bug in practice, just an inconsistency. Here are
the details in case you think something needs to be changed:

libvirt uses the vectors = 2 * N + 2 formula from
https://www.linux-kvm.org/page/Multiqueue to calculate the number of PCI
MSI vectors, where N is the number of rx/tx queue pairs.

QEMU's virtio-net-pci device has 3 MSI vectors by default. This is
inconsistent with the formula that libvirt uses (should be 4 instead of
3).

Luckily, the Linux virtio_net.ko driver does not configure a callback
function for the control virtqueue. Therefore it can still use MSI with
only 3 vectors (config, rx, tx) instead of 4 (config, rx, tx, ctrl).

But other driver implementations might need the ctrl vq vector and would
not have enough MSI vectors.

Perhaps new QEMU machine types should set vectors to 4 by default?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Inconsistent virtio-net-pci MSI vector count

[Jason Wang]

On 2021/3/3 6:53 下午, Stefan Hajnoczi wrote:
> Hi Jason,
> I stumbled across something strange with virtio-net multi-queue today.
> It doesn't seem to be a bug in practice, just an inconsistency. Here are
> the details in case you think something needs to be changed:
>
> libvirt uses the vectors = 2 * N + 2 formula from
> https://www.linux-kvm.org/page/Multiqueue to calculate the number of PCI
> MSI vectors, where N is the number of rx/tx queue pairs.
>
> QEMU's virtio-net-pci device has 3 MSI vectors by default. This is
> inconsistent with the formula that libvirt uses (should be 4 instead of
> 3).


Yes.


>
> Luckily, the Linux virtio_net.ko driver does not configure a callback
> function for the control virtqueue. Therefore it can still use MSI with
> only 3 vectors (config, rx, tx) instead of 4 (config, rx, tx, ctrl).
>
> But other driver implementations might need the ctrl vq vector and would
> not have enough MSI vectors.
>
> Perhaps new QEMU machine types should set vectors to 4 by default?


Or it's time to accept this:

https://lists.gnu.org/archive/html/qemu-devel/2014-03/msg01330.html

Thanks


>
> Stefan

Re: Inconsistent virtio-net-pci MSI vector count

[Michael S. Tsirkin]

On Wed, Mar 03, 2021 at 10:53:14AM +0000, Stefan Hajnoczi wrote:
> Hi Jason,
> I stumbled across something strange with virtio-net multi-queue today.
> It doesn't seem to be a bug in practice, just an inconsistency. Here are
> the details in case you think something needs to be changed:
> 
> libvirt uses the vectors = 2 * N + 2 formula from
> https://www.linux-kvm.org/page/Multiqueue to calculate the number of PCI
> MSI vectors, where N is the number of rx/tx queue pairs.
> 
> QEMU's virtio-net-pci device has 3 MSI vectors by default. This is
> inconsistent with the formula that libvirt uses (should be 4 instead of
> 3).
> 
> Luckily, the Linux virtio_net.ko driver does not configure a callback
> function for the control virtqueue. Therefore it can still use MSI with
> only 3 vectors (config, rx, tx) instead of 4 (config, rx, tx, ctrl).
> 
> But other driver implementations might need the ctrl vq vector and would
> not have enough MSI vectors.

At a guess control vq can share a vector with config.
Linux is not clever enough to do it yet but it's possible.

> Perhaps new QEMU machine types should set vectors to 4 by default?
> 
> Stefan

Re: Inconsistent virtio-net-pci MSI vector count

[Michael S. Tsirkin]

On Wed, Mar 03, 2021 at 09:08:27PM +0800, Jason Wang wrote:
> 
> On 2021/3/3 6:53 下午, Stefan Hajnoczi wrote:
> > Hi Jason,
> > I stumbled across something strange with virtio-net multi-queue today.
> > It doesn't seem to be a bug in practice, just an inconsistency. Here are
> > the details in case you think something needs to be changed:
> > 
> > libvirt uses the vectors = 2 * N + 2 formula from
> > https://www.linux-kvm.org/page/Multiqueue to calculate the number of PCI
> > MSI vectors, where N is the number of rx/tx queue pairs.
> > 
> > QEMU's virtio-net-pci device has 3 MSI vectors by default. This is
> > inconsistent with the formula that libvirt uses (should be 4 instead of
> > 3).
> 
> 
> Yes.
> 
> 
> > 
> > Luckily, the Linux virtio_net.ko driver does not configure a callback
> > function for the control virtqueue. Therefore it can still use MSI with
> > only 3 vectors (config, rx, tx) instead of 4 (config, rx, tx, ctrl).
> > 
> > But other driver implementations might need the ctrl vq vector and would
> > not have enough MSI vectors.
> > 
> > Perhaps new QEMU machine types should set vectors to 4 by default?
> 
> 
> Or it's time to accept this:
> 
> https://lists.gnu.org/archive/html/qemu-devel/2014-03/msg01330.html
> 
> Thanks

okay, please repost. If possible please add code comments
explaining the math.

Thanks!

> 
> > 
> > Stefan

Re: Inconsistent virtio-net-pci MSI vector count

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1546 bytes --]

On Wed, Mar 03, 2021 at 11:12:51AM -0500, Michael S. Tsirkin wrote:
> On Wed, Mar 03, 2021 at 09:08:27PM +0800, Jason Wang wrote:
> > 
> > On 2021/3/3 6:53 下午, Stefan Hajnoczi wrote:
> > > Hi Jason,
> > > I stumbled across something strange with virtio-net multi-queue today.
> > > It doesn't seem to be a bug in practice, just an inconsistency. Here are
> > > the details in case you think something needs to be changed:
> > > 
> > > libvirt uses the vectors = 2 * N + 2 formula from
> > > https://www.linux-kvm.org/page/Multiqueue to calculate the number of PCI
> > > MSI vectors, where N is the number of rx/tx queue pairs.
> > > 
> > > QEMU's virtio-net-pci device has 3 MSI vectors by default. This is
> > > inconsistent with the formula that libvirt uses (should be 4 instead of
> > > 3).
> > 
> > 
> > Yes.
> > 
> > 
> > > 
> > > Luckily, the Linux virtio_net.ko driver does not configure a callback
> > > function for the control virtqueue. Therefore it can still use MSI with
> > > only 3 vectors (config, rx, tx) instead of 4 (config, rx, tx, ctrl).
> > > 
> > > But other driver implementations might need the ctrl vq vector and would
> > > not have enough MSI vectors.
> > > 
> > > Perhaps new QEMU machine types should set vectors to 4 by default?
> > 
> > 
> > Or it's time to accept this:
> > 
> > https://lists.gnu.org/archive/html/qemu-devel/2014-03/msg01330.html
> > 
> > Thanks
> 
> okay, please repost. If possible please add code comments
> explaining the math.

Nice!

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]