* Using VFIO vs. developing a kernel module @ 2021-05-27 13:23 sainath grandhi 2021-05-27 23:16 ` Andrew Jeffery 0 siblings, 1 reply; 4+ messages in thread From: sainath grandhi @ 2021-05-27 13:23 UTC (permalink / raw) To: openbmc Hello, Our project has an FPGA connected to BMC as a PCIe endpoint. This endpoint provides a set of registers via MMIO and an interrupt for notifying completion of work. This endpoint also implements AER capability. We have two options to enable this endpoint. 1) Write a new kernel module with a character device interface for user-space interaction. 2) Use VFIO infrastructure provided by Linux and write an user-space application. I am reaching out to the community to check if there is any recommended option, using VFIO vs. implementing a new kernel module, or any previous experiences weighing in one option over the other. A quick grep on https://github.com/openbmc/openbmc with CONFIG_VFIO does not return any default configs using it. Please advise. ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Using VFIO vs. developing a kernel module 2021-05-27 13:23 Using VFIO vs. developing a kernel module sainath grandhi @ 2021-05-27 23:16 ` Andrew Jeffery 2021-05-28 23:12 ` Dhananjay Phadke 0 siblings, 1 reply; 4+ messages in thread From: Andrew Jeffery @ 2021-05-27 23:16 UTC (permalink / raw) To: sainath grandhi, openbmc On Thu, 27 May 2021, at 22:53, sainath grandhi wrote: > Hello, > Our project has an FPGA connected to BMC as a PCIe endpoint. This > endpoint provides a set of registers via MMIO and an interrupt for > notifying completion of work. This endpoint also implements AER > capability. > > We have two options to enable this endpoint. > 1) Write a new kernel module with a character device interface for > user-space interaction. > 2) Use VFIO infrastructure provided by Linux and write an user-space > application. > > I am reaching out to the community to check if there is any > recommended option, using VFIO vs. implementing a new kernel module, > or any previous experiences weighing in one option over the other. I don't have any experience with VFIO, so take this with a grain of salt. Generally you should write an in-kernel driver for it. The reason you might not want to do so is if the device's register interface changes frequently, as it's more pain to update the kernel than some userspace application, which slows iteration. But handling DMAs and interrupts make userspace more painful, so unless VFIO helps there (I assume it does), then that would push the implementation back towards the kernel. Andrew ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Using VFIO vs. developing a kernel module 2021-05-27 23:16 ` Andrew Jeffery @ 2021-05-28 23:12 ` Dhananjay Phadke 2021-05-29 15:16 ` sainath grandhi 0 siblings, 1 reply; 4+ messages in thread From: Dhananjay Phadke @ 2021-05-28 23:12 UTC (permalink / raw) To: Andrew Jeffery, sainath grandhi, openbmc On Fri, 28 May 2021, Andrew Jeffery wrote: > > On Thu, 27 May 2021, at 22:53, sainath grandhi wrote: >> Hello, >> Our project has an FPGA connected to BMC as a PCIe endpoint. This >> endpoint provides a set of registers via MMIO and an interrupt for >> notifying completion of work. This endpoint also implements AER >> capability. >> >> We have two options to enable this endpoint. >> 1) Write a new kernel module with a character device interface for >> user-space interaction. >> 2) Use VFIO infrastructure provided by Linux and write an user-space >> application. >> >> I am reaching out to the community to check if there is any >> recommended option, using VFIO vs. implementing a new kernel module, >> or any previous experiences weighing in one option over the other. > > I don't have any experience with VFIO, so take this with a grain of salt. > > Generally you should write an in-kernel driver for it. The reason you > might not want to do so is if the device's register interface changes > frequently, as it's more pain to update the kernel than some userspace > application, which slows iteration. But handling DMAs and interrupts > make userspace more painful, so unless VFIO helps there (I assume it > does), then that would push the implementation back towards the kernel. VFIO requires IOMMU to protect kernel memory corruption by device DMA, which is programmed by (untrusted) userspace app. Unless the BMC SoC implements IOMMU (SMMU) for I/O virtualization (I/O page tables), it would not be possible/safe to use VFIO for DMA/interrupts. See https://www.kernel.org/doc/Documentation/vfio.txt If you just want to program MMIO registers, you could mmap() PCIe sysfs resource files that represent MMIO bars and let application write and read registers, including polling for completion. See https://www.kernel.org/doc/Documentation/PCI/sysfs-pci.rst If none of these options are viable, then a custom kernel driver would be necessary. Regards, Dhananjay ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Using VFIO vs. developing a kernel module 2021-05-28 23:12 ` Dhananjay Phadke @ 2021-05-29 15:16 ` sainath grandhi 0 siblings, 0 replies; 4+ messages in thread From: sainath grandhi @ 2021-05-29 15:16 UTC (permalink / raw) To: Dhananjay Phadke; +Cc: Andrew Jeffery, openbmc [-- Attachment #1: Type: text/plain, Size: 2497 bytes --] On Fri, May 28, 2021, 6:12 PM Dhananjay Phadke <dphadke@linux.microsoft.com> wrote: > On Fri, 28 May 2021, Andrew Jeffery wrote: > > > > On Thu, 27 May 2021, at 22:53, sainath grandhi wrote: > >> Hello, > >> Our project has an FPGA connected to BMC as a PCIe endpoint. This > >> endpoint provides a set of registers via MMIO and an interrupt for > >> notifying completion of work. This endpoint also implements AER > >> capability. > >> > >> We have two options to enable this endpoint. > >> 1) Write a new kernel module with a character device interface for > >> user-space interaction. > >> 2) Use VFIO infrastructure provided by Linux and write an user-space > >> application. > >> > >> I am reaching out to the community to check if there is any > >> recommended option, using VFIO vs. implementing a new kernel module, > >> or any previous experiences weighing in one option over the other. > > > > I don't have any experience with VFIO, so take this with a grain of salt. > > > > Generally you should write an in-kernel driver for it. The reason you > > might not want to do so is if the device's register interface changes > > frequently, as it's more pain to update the kernel than some userspace > > application, which slows iteration. But handling DMAs and interrupts > > make userspace more painful, so unless VFIO helps there (I assume it > > does), then that would push the implementation back towards the kernel. > > VFIO requires IOMMU to protect kernel memory corruption by device DMA, > which is programmed by (untrusted) userspace app. Unless the BMC SoC > implements IOMMU (SMMU) for I/O virtualization (I/O page tables), it > would not be possible/safe to use VFIO for DMA/interrupts. > > See https://www.kernel.org/doc/Documentation/vfio.txt > > Yeah. In our case the endpoint does not have DMA. And uses INTx for > interrupt to avoid memory corruption issues. Our BMC does not support SMMU, so I am using VFIO with no IOMMU > configuration. Linux kernel introduced VFIO without IOMMU for such > platforms so that userspace applications can still take advantage of the > VFIO facilities. > If you just want to program MMIO registers, you could mmap() PCIe sysfs > resource files that represent MMIO bars and let application write and > read registers, including polling for completion. > > See https://www.kernel.org/doc/Documentation/PCI/sysfs-pci.rst > > If none of these options are viable, then a custom kernel driver would > be necessary. > > Regards, > Dhananjay > [-- Attachment #2: Type: text/html, Size: 3683 bytes --] ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2021-05-29 15:17 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-05-27 13:23 Using VFIO vs. developing a kernel module sainath grandhi 2021-05-27 23:16 ` Andrew Jeffery 2021-05-28 23:12 ` Dhananjay Phadke 2021-05-29 15:16 ` sainath grandhi
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).