vfio/pci - uAPI for WC

vfio/pci - uAPI for WC

[Arinzon, David]

Hi,

There's currently no mechanism for vfio that exposes WC-related operations (check if memory is WC capable, ask to WC memory) to user space module entities, such as DPDK, for example.
This topic has been previously discussed in [1], [2] and [3], but there was no follow-up.
This capability is very useful for DPDK, specifically to the DPDK ENA driver that uses vfio-pci, which requires memory to be WC on the TX path. Without WC, higher CPU utilization and performance degradation are observed.
In the above mentioned discussions, three options were suggested: sysfs, ioctl, mmap extension (extra attributes).

Was there any progress on this area? Is there someone who's looking into this?
We're leaning towards the ioctl option, if there are no objections, we'd come up with an RFC.

[1]: https://patchwork.kernel.org/project/kvm/patch/20171009025000.39435-1-aik@ozlabs.ru/
[2]: https://lore.kernel.org/linux-pci/2b539df4c9ec703458e46da2fc879ee3b310b31c.camel@kernel.crashing.org/
[3]: https://lore.kernel.org/lkml/20210429162906.32742-2-sdonthineni@nvidia.com/

Thanks,
David

Re: vfio/pci - uAPI for WC

[Bjorn Helgaas]

[+cc Alex, Cornelia, kvm, lkml (from "get_maintainer.pl drivers/vfio")
and rewrapped for plain-text readability]
On Thu, Aug 04, 2022 at 09:47:36AM +0000, Arinzon, David wrote:
> Hi,
> 
> There's currently no mechanism for vfio that exposes WC-related
> operations (check if memory is WC capable, ask to WC memory) to user
> space module entities, such as DPDK, for example.
>
> This topic has been previously discussed in [1], [2] and [3], but
> there was no follow-up.
>
> This capability is very useful for DPDK, specifically to the DPDK
> ENA driver that uses vfio-pci, which requires memory to be WC on the
> TX path. Without WC, higher CPU utilization and performance
> degradation are observed.
>
> In the above mentioned discussions, three options were suggested:
> sysfs, ioctl, mmap extension (extra attributes).
> 
> Was there any progress on this area? Is there someone who's looking
> into this?
>
> We're leaning towards the ioctl option, if there are no objections,
> we'd come up with an RFC.

> [1]: https://patchwork.kernel.org/project/kvm/patch/20171009025000.39435-1-aik@ozlabs.ru/
> [2]: https://lore.kernel.org/linux-pci/2b539df4c9ec703458e46da2fc879ee3b310b31c.camel@kernel.crashing.org/
> [3]: https://lore.kernel.org/lkml/20210429162906.32742-2-sdonthineni@nvidia.com/
> 
> Thanks,
> David

Re: vfio/pci - uAPI for WC

[Alex Williamson]

On Fri, 5 Aug 2022 11:05:45 -0500
Bjorn Helgaas <helgaas@kernel.org> wrote:

> [+cc Alex, Cornelia, kvm, lkml (from "get_maintainer.pl drivers/vfio")
> and rewrapped for plain-text readability]
> On Thu, Aug 04, 2022 at 09:47:36AM +0000, Arinzon, David wrote:
> > Hi,
> > 
> > There's currently no mechanism for vfio that exposes WC-related
> > operations (check if memory is WC capable, ask to WC memory) to user
> > space module entities, such as DPDK, for example.
> >
> > This topic has been previously discussed in [1], [2] and [3], but
> > there was no follow-up.
> >
> > This capability is very useful for DPDK, specifically to the DPDK
> > ENA driver that uses vfio-pci, which requires memory to be WC on the
> > TX path. Without WC, higher CPU utilization and performance
> > degradation are observed.
> >
> > In the above mentioned discussions, three options were suggested:
> > sysfs, ioctl, mmap extension (extra attributes).
> > 
> > Was there any progress on this area? Is there someone who's looking
> > into this?

IIRC, much of the discussion was related to VM use cases on ARM and
they may have found alternate ways to do things more like x86.  I'm not
aware of any current development towards a uAPI change to enable this.
Thanks,

Alex