Re: Running DPDK as an unprivileged user

[Alejandro Lucero <alejandro.lucero@netronome.com>]

On Mon, Nov 27, 2017 at 5:58 PM, Walker, Benjamin <benjamin.walker@intel.com
> wrote:

> On Sun, 2017-11-05 at 01:17 +0100, Thomas Monjalon wrote:
> > Hi, restarting an old topic,
> >
> > 05/01/2017 16:52, Tan, Jianfeng:
> > > On 1/5/2017 5:34 AM, Walker, Benjamin wrote:
> > > > > > Note that this
> > > > > > probably means that using uio on recent kernels is subtly
> > > > > > broken and cannot be supported going forward because there
> > > > > > is no uio mechanism to pin the memory.
> > > > > >
> > > > > > The first open question I have is whether DPDK should allow
> > > > > > uio at all on recent (4.x) kernels. My current understanding
> > > > > > is that there is no way to pin memory and hugepages can now
> > > > > > be moved around, so uio would be unsafe. What does the
> > > > > > community think here?
> > >
> > > Back to this question, removing uio support in DPDK seems a little
> > > overkill to me. Can we just document it down? Like, firstly warn users
> > > do not invoke migrate_pages() or move_pages() to a DPDK process; as for
> > > the kcompactd daemon and some more cases (like compaction could be
> > > triggered by alloc_pages()), could we just recommend to disable
> > > CONFIG_COMPACTION?
> >
> > We really need to better document the limitations of UIO.
> > May we have some suggestions here?
> >
> > > Another side, how does vfio pin those memory? Through memlock (from
> code
> > > in vfio_pin_pages())? So why not just mlock those hugepages?
> >
> > Good question. Why not mlock the hugepages?
>
> mlock just guarantees that a virtual page is always backed by *some*
> physical
> page of memory. It does not guarantee that over the lifetime of the
> process a
> virtual page is mapped to the *same* physical page. The kernel is free to
> transparently move memory around, compress it, dedupe it, etc.
>
> vfio is not pinning the memory, but instead is using the IOMMU (a piece of
> hardware) to participate in the memory management on the platform. If a
> device
> begins a DMA transfer to an I/O virtual address, the IOMMU will coordinate
> with
> the main MMU to make sure that the data ends up in the correct location,
> even as
> the virtual to physical mappings are being modified.


This last comment confused me because you said VFIO did the page pinning in
your first email.
I have been looking at the kernel code and the VFIO driver does pin the
pages, at least the iommu type 1.

I can see a problem adding support to UIO for doing the same, because that
implies there is a device
doing DMAs and programmed from user space, which is something the UIO
maintainer is against. But because
vfio-noiommu mode was implemented just for this, I guess that could be
added to the VFIO driver. This does not
solve the problem of software not using vfio though.

Apart from improving the UIO documentation when used with DPDK, maybe some
sort of check could be done
and DPDK requiring a explicit parameter for making the user aware of the
potential risk when UIO is used and the
kernel page migration is enabled. Not sure if this last thing could be
easily known from user space.

On another side, we suffered a similar problem when VMs were using SRIOV
and memory balloning. The IOMMU was
removing the mapping for the memory removed, but the kernel inside the VM
did not get any event and the device
ended up doing some wrong DMA operation.