Using VFIO vs. developing a kernel module

Using VFIO vs. developing a kernel module

[sainath grandhi]

Hello,
Our project has an FPGA connected to BMC as a PCIe endpoint. This
endpoint provides a set of registers via MMIO and an interrupt for
notifying completion of work. This endpoint also implements AER
capability.

We have two options to enable this endpoint.
1) Write a new kernel module with a character device interface for
user-space interaction.
2) Use VFIO infrastructure provided by Linux and write an user-space
application.

I am reaching out to the community to check if there is any
recommended option, using VFIO vs. implementing a new kernel module,
or any previous experiences weighing in one option over the other.

A quick grep on https://github.com/openbmc/openbmc with CONFIG_VFIO
does not return any default configs using it.

Please advise.

Re: Using VFIO vs. developing a kernel module

[Andrew Jeffery]

On Thu, 27 May 2021, at 22:53, sainath grandhi wrote:
> Hello,
> Our project has an FPGA connected to BMC as a PCIe endpoint. This
> endpoint provides a set of registers via MMIO and an interrupt for
> notifying completion of work. This endpoint also implements AER
> capability.
> 
> We have two options to enable this endpoint.
> 1) Write a new kernel module with a character device interface for
> user-space interaction.
> 2) Use VFIO infrastructure provided by Linux and write an user-space
> application.
> 
> I am reaching out to the community to check if there is any
> recommended option, using VFIO vs. implementing a new kernel module,
> or any previous experiences weighing in one option over the other.

I don't have any experience with VFIO, so take this with a grain of salt.

Generally you should write an in-kernel driver for it. The reason you 
might not want to do so is if the device's register interface changes 
frequently, as it's more pain to update the kernel than some userspace 
application, which slows iteration. But handling DMAs and interrupts 
make userspace more painful, so unless VFIO helps there (I assume it 
does), then that would push the implementation back towards the kernel.

Andrew

Re: Using VFIO vs. developing a kernel module

[Dhananjay Phadke]

On Fri, 28 May 2021, Andrew Jeffery wrote:
>
> On Thu, 27 May 2021, at 22:53, sainath grandhi wrote:
>> Hello,
>> Our project has an FPGA connected to BMC as a PCIe endpoint. This
>> endpoint provides a set of registers via MMIO and an interrupt for
>> notifying completion of work. This endpoint also implements AER
>> capability.
>>
>> We have two options to enable this endpoint.
>> 1) Write a new kernel module with a character device interface for
>> user-space interaction.
>> 2) Use VFIO infrastructure provided by Linux and write an user-space
>> application.
>>
>> I am reaching out to the community to check if there is any
>> recommended option, using VFIO vs. implementing a new kernel module,
>> or any previous experiences weighing in one option over the other.
>
> I don't have any experience with VFIO, so take this with a grain of salt.
>
> Generally you should write an in-kernel driver for it. The reason you
> might not want to do so is if the device's register interface changes
> frequently, as it's more pain to update the kernel than some userspace
> application, which slows iteration. But handling DMAs and interrupts
> make userspace more painful, so unless VFIO helps there (I assume it
> does), then that would push the implementation back towards the kernel.

VFIO requires IOMMU to protect kernel memory corruption by device DMA,
which is programmed by (untrusted) userspace app. Unless the BMC SoC
implements IOMMU (SMMU) for I/O virtualization (I/O page tables), it
would not be possible/safe to use VFIO for DMA/interrupts.

See https://www.kernel.org/doc/Documentation/vfio.txt

If you just want to program MMIO registers, you could mmap() PCIe sysfs
resource files that represent MMIO bars and let application write and
read registers, including polling for completion.

See https://www.kernel.org/doc/Documentation/PCI/sysfs-pci.rst

If none of these options are viable, then a custom kernel driver would
be necessary.

Regards,
Dhananjay

Re: Using VFIO vs. developing a kernel module

[sainath grandhi]

[-- Attachment #1: Type: text/plain, Size: 2497 bytes --]

On Fri, May 28, 2021, 6:12 PM Dhananjay Phadke <dphadke@linux.microsoft.com>
wrote:

> On Fri, 28 May 2021, Andrew Jeffery wrote:
> >
> > On Thu, 27 May 2021, at 22:53, sainath grandhi wrote:
> >> Hello,
> >> Our project has an FPGA connected to BMC as a PCIe endpoint. This
> >> endpoint provides a set of registers via MMIO and an interrupt for
> >> notifying completion of work. This endpoint also implements AER
> >> capability.
> >>
> >> We have two options to enable this endpoint.
> >> 1) Write a new kernel module with a character device interface for
> >> user-space interaction.
> >> 2) Use VFIO infrastructure provided by Linux and write an user-space
> >> application.
> >>
> >> I am reaching out to the community to check if there is any
> >> recommended option, using VFIO vs. implementing a new kernel module,
> >> or any previous experiences weighing in one option over the other.
> >
> > I don't have any experience with VFIO, so take this with a grain of salt.
> >
> > Generally you should write an in-kernel driver for it. The reason you
> > might not want to do so is if the device's register interface changes
> > frequently, as it's more pain to update the kernel than some userspace
> > application, which slows iteration. But handling DMAs and interrupts
> > make userspace more painful, so unless VFIO helps there (I assume it
> > does), then that would push the implementation back towards the kernel.
>
> VFIO requires IOMMU to protect kernel memory corruption by device DMA,
> which is programmed by (untrusted) userspace app. Unless the BMC SoC
> implements IOMMU (SMMU) for I/O virtualization (I/O page tables), it
> would not be possible/safe to use VFIO for DMA/interrupts.
>
> See https://www.kernel.org/doc/Documentation/vfio.txt
>
> Yeah. In our case the endpoint does not have DMA.  And uses INTx for
> interrupt to avoid memory corruption issues.


Our BMC does not support SMMU, so I am using VFIO with no IOMMU
> configuration. Linux kernel introduced VFIO without IOMMU for such
> platforms so that userspace applications can still take advantage of the
> VFIO facilities.
>

If you just want to program MMIO registers, you could mmap() PCIe sysfs
> resource files that represent MMIO bars and let application write and
> read registers, including polling for completion.
>
> See https://www.kernel.org/doc/Documentation/PCI/sysfs-pci.rst
>
> If none of these options are viable, then a custom kernel driver would
> be necessary.
>
> Regards,
> Dhananjay
>

[-- Attachment #2: Type: text/html, Size: 3683 bytes --]