Re: [PATCH v3 2/3] uio_pci_generic: add MSI/MSI-X support

From: Avi Kivity <avi@scylladb.com>
To: Greg KH <gregkh@linuxfoundation.org>
Cc: Vlad Zolotarov <vladz@cloudius-systems.com>,
	linux-kernel@vger.kernel.org, mst@redhat.com, hjk@hansjkoch.de,
	corbet@lwn.net, bruce.richardson@intel.com,
	avi@cloudius-systems.com, gleb@cloudius-systems.com,
	stephen@networkplumber.org, alexander.duyck@gmail.com
Subject: Re: [PATCH v3 2/3] uio_pci_generic: add MSI/MSI-X support
Date: Mon, 5 Oct 2015 13:20:11 +0300	[thread overview]
Message-ID: <56124EDB.3070701@scylladb.com> (raw)
In-Reply-To: <20151005094932.GA5236@kroah.com>

On 10/05/2015 12:49 PM, Greg KH wrote:
> On Mon, Oct 05, 2015 at 11:28:03AM +0300, Avi Kivity wrote:
>> Of course it has to be documented, but this just follows vfio.
>>
>> Eventfd is a natural enough representation of an interrupt; both kvm and
>> vfio use it, and are also able to share the eventfd, allowing a vfio
>> interrupt to generate a kvm interrupt, without userspace intervention, and
>> one day without even kernel intervention.
> That's nice and wonderful, but it's not how UIO works today, so this is
> now going to be a mix and match type interface, with no justification so
> far as to why to create this new api and exactly how this is all going
> to be used from userspace.

The intended user is dpdk (http://dpdk.org), which is a family of 
userspace networking drivers for high performance networking applications.

The natural device driver for dpdk is vfio, which both provides memory 
protection and exposes msi/msix interrupts.  However, in many cases vfio 
cannot be used, either due to the lack of an iommu (for example, in 
virtualized environments) or out of a desire to avoid the iommus 
performance impact.

The challenge in exposing msix interrupts to user space is that there 
are many of them, so you can't simply poll the device fd.  If you do, 
how do you know which interrupt was triggered?  The solution that vfio 
adopted was to associate each interrupt with an eventfd, allowing it to 
be individually polled.  Since you can pass an eventfd with SCM_RIGHTS, 
and since kvm can trigger guest interrupts using an eventfd, the 
solution is very flexible.

> Example code would be even better...
>
>

This is the vfio dpdk interface code:

http://dpdk.org/browse/dpdk/tree/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c

basically, the equivalent uio msix code would be very similar if uio 
adopts a similar interface:

http://dpdk.org/browse/dpdk/tree/lib/librte_eal/linuxapp/eal/eal_pci_uio.c

(current code lacks msi/msix support, of course).