Re: [RFC v4 PATCH 00/49] Initial support of multi-process qemu - status update

[Felipe Franciosi <felipe@nutanix.com>]

> On Dec 20, 2019, at 9:50 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> 
> On 20/12/19 10:47, Stefan Hajnoczi wrote:
>>> If it doesn't create too large of a burden to support both, then I think
>>> it is very desirable. IIUC, this is saying a kernel based solution as the
>>> optimized/optimal solution, and userspace UNIX socket based option as the
>>> generic "works everywhere" fallback solution.
>> I'm slightly in favor of the kernel implementation because it keeps us
>> better aligned with VFIO.  That means solving problems in one place only
>> and less reinventing the wheel.
> 
> I think there are anyway going to be some differences with VFIO.
> 
> For example, currently VFIO requires pinning user memory.  Is that a
> limitation for muser too?  If so, that would be a big disadvantage; if
> not, however, management tools need to learn that muser devices unlike
> other VFIO devices do not prevent overcommit.

More or less. We pin them today, but I think we don't really have to.
I created an issue to look into it:
https://github.com/nutanix/muser/issues/28

In any case, if Qemu is ballooning and calls UNMAP_DMA for memory that
has been ballooned out, then we would release it.

The reason we keep it pinned is to support libmuser restarts. IIRC,
VFIO doesn't need to pin pages for mdev devices (that's the job of the
mdev driver on the other end via vfio_pin_pages()). It only keeps the
DMA entries in a RB tree.

If my understanding is right, then we can probably just keep the map
Qemu registered (without holding the pages) and call vfio_pin_pages()
on demand when libmuser restarts.

For context, this is how the DMA memory registration works today:

1) Qemu calls ioctl(vfio_fd, IOMMU_MAP_DMA, &vm_map);

2) The iommu driver notifies muser.ko

3) Muser.ko pins the pages (in get_dma_map(), called from below)
(https://github.com/nutanix/muser/blob/master/kmod/muser.c#L711)

4) Muser.ko notifies libmuser about the memory registration
(The iommu driver context goes to sleep, hence the pinning)

5) Libmuser wakes up and calls mmap() on muser.ko

6) Muser.ko inserts the VM memory in libmuser's context
(https://github.com/nutanix/muser/blob/master/kmod/muser.c#L543)

7) Libmuser tells muser.ko that it's done

8) Muser.ko iommu callback context that was sleeping wakes up

9) Muser.ko places the memory in a "dma_list" for the mudev and returns.

We could potentially modify the last step to unpin and keep only what
we need for a future call to vfio_pin_pages(), but I need to check if
that works.

Cheers,
Felipe

> 
> Paolo
>