RFC: New device for zero-copy VM memory access

RFC: New device for zero-copy VM memory access

[geoff]

Hi All,

Over the past week, I have been working to come up with a solution to 
the
memory transfer performance issues that hinder the Looking Glass 
Project.

Currently Looking Glass works by using the IVSHMEM shared memory device 
which
is fed by an application that captures the guest's video output. While 
this
works it is sub-optimal because we first have to perform a CPU copy of 
the
captured frame into shared RAM, and then back out again for display. 
Because
the destination buffers are allocated by closed proprietary code 
(DirectX, or
NVidia NvFBC) there is no way to have the frame placed directly into the
IVSHMEM shared ram.

This new device, currently named `introspection` (which needs a more 
suitable
name, porthole perhaps?), provides a means of translating guest physical
addresses to host virtual addresses, and finally to the host offsets in 
RAM for
file-backed memory guests. It does this by means of a simple protocol 
over a
unix socket (chardev) which is supplied the appropriate fd for the VM's 
system
RAM. The guest (in this case, Windows), when presented with the address 
of a
userspace buffer and size, will mlock the appropriate pages into RAM and 
pass
guest physical addresses to the virtual device.

This device and the windows driver have been designed in such a way that 
it's a
utility device for any project and/or application that could make use of 
it.
The PCI subsystem vendor and device ID are used to provide a means of 
device
identification in cases where multiple devices may be in use for 
differing
applications. This also allows one common driver to be used for any 
other
projects wishing to build on this device.

My ultimate goal is to get this to a state where it could be accepted 
upstream
into Qemu at which point Looking Glass would be modified to use it 
instead of
the IVSHMEM device.

My git repository with the new device can be found at:
https://github.com/gnif/qemu

The new device is:
https://github.com/gnif/qemu/blob/master/hw/misc/introspection.c

Looking Glass:
https://looking-glass.hostfission.com/

The windows driver, while working, needs some cleanup before the source 
is
published. I intend to maintain both this device and the windows driver
including producing a signed Windows 10 driver if Redhat are unwilling 
or
unable.

Kind Regards,
Geoffrey McRae

HostFission
https://hostfission.com

Re: RFC: New device for zero-copy VM memory access

[geoff]

Just to follow this up, here is a sample client application for this 
device

https://gist.github.com/gnif/77e7fb54604b42a1a98ecb8bf3d2cf46

-Geoff

On 2019-10-30 01:31, geoff@hostfission.com wrote:
> Hi All,
> 
> Over the past week, I have been working to come up with a solution to 
> the
> memory transfer performance issues that hinder the Looking Glass 
> Project.
> 
> Currently Looking Glass works by using the IVSHMEM shared memory device 
> which
> is fed by an application that captures the guest's video output. While 
> this
> works it is sub-optimal because we first have to perform a CPU copy of 
> the
> captured frame into shared RAM, and then back out again for display. 
> Because
> the destination buffers are allocated by closed proprietary code 
> (DirectX, or
> NVidia NvFBC) there is no way to have the frame placed directly into 
> the
> IVSHMEM shared ram.
> 
> This new device, currently named `introspection` (which needs a more 
> suitable
> name, porthole perhaps?), provides a means of translating guest 
> physical
> addresses to host virtual addresses, and finally to the host offsets in 
> RAM for
> file-backed memory guests. It does this by means of a simple protocol 
> over a
> unix socket (chardev) which is supplied the appropriate fd for the VM's 
> system
> RAM. The guest (in this case, Windows), when presented with the address 
> of a
> userspace buffer and size, will mlock the appropriate pages into RAM 
> and pass
> guest physical addresses to the virtual device.
> 
> This device and the windows driver have been designed in such a way 
> that it's a
> utility device for any project and/or application that could make use 
> of it.
> The PCI subsystem vendor and device ID are used to provide a means of 
> device
> identification in cases where multiple devices may be in use for 
> differing
> applications. This also allows one common driver to be used for any 
> other
> projects wishing to build on this device.
> 
> My ultimate goal is to get this to a state where it could be accepted 
> upstream
> into Qemu at which point Looking Glass would be modified to use it 
> instead of
> the IVSHMEM device.
> 
> My git repository with the new device can be found at:
> https://github.com/gnif/qemu
> 
> The new device is:
> https://github.com/gnif/qemu/blob/master/hw/misc/introspection.c
> 
> Looking Glass:
> https://looking-glass.hostfission.com/
> 
> The windows driver, while working, needs some cleanup before the source 
> is
> published. I intend to maintain both this device and the windows driver
> including producing a signed Windows 10 driver if Redhat are unwilling 
> or
> unable.
> 
> Kind Regards,
> Geoffrey McRae
> 
> HostFission
> https://hostfission.com

Re: RFC: New device for zero-copy VM memory access

[geoff]

The windows driver source is now available also.

https://github.com/gnif/Porthole-Guest-Driver

I have also opted to rename the device to 'porthole', hopefully this 
name is
acceptable.

On 2019-10-30 09:53, geoff@hostfission.com wrote:
> Just to follow this up, here is a sample client application for this 
> device
> 
> https://gist.github.com/gnif/77e7fb54604b42a1a98ecb8bf3d2cf46
> 
> -Geoff
> 
> On 2019-10-30 01:31, geoff@hostfission.com wrote:
>> Hi All,
>> 
>> Over the past week, I have been working to come up with a solution to 
>> the
>> memory transfer performance issues that hinder the Looking Glass 
>> Project.
>> 
>> Currently Looking Glass works by using the IVSHMEM shared memory 
>> device which
>> is fed by an application that captures the guest's video output. While 
>> this
>> works it is sub-optimal because we first have to perform a CPU copy of 
>> the
>> captured frame into shared RAM, and then back out again for display. 
>> Because
>> the destination buffers are allocated by closed proprietary code 
>> (DirectX, or
>> NVidia NvFBC) there is no way to have the frame placed directly into 
>> the
>> IVSHMEM shared ram.
>> 
>> This new device, currently named `introspection` (which needs a more 
>> suitable
>> name, porthole perhaps?), provides a means of translating guest 
>> physical
>> addresses to host virtual addresses, and finally to the host offsets 
>> in RAM for
>> file-backed memory guests. It does this by means of a simple protocol 
>> over a
>> unix socket (chardev) which is supplied the appropriate fd for the 
>> VM's system
>> RAM. The guest (in this case, Windows), when presented with the 
>> address of a
>> userspace buffer and size, will mlock the appropriate pages into RAM 
>> and pass
>> guest physical addresses to the virtual device.
>> 
>> This device and the windows driver have been designed in such a way 
>> that it's a
>> utility device for any project and/or application that could make use 
>> of it.
>> The PCI subsystem vendor and device ID are used to provide a means of 
>> device
>> identification in cases where multiple devices may be in use for 
>> differing
>> applications. This also allows one common driver to be used for any 
>> other
>> projects wishing to build on this device.
>> 
>> My ultimate goal is to get this to a state where it could be accepted 
>> upstream
>> into Qemu at which point Looking Glass would be modified to use it 
>> instead of
>> the IVSHMEM device.
>> 
>> My git repository with the new device can be found at:
>> https://github.com/gnif/qemu
>> 
>> The new device is:
>> https://github.com/gnif/qemu/blob/master/hw/misc/introspection.c
>> 
>> Looking Glass:
>> https://looking-glass.hostfission.com/
>> 
>> The windows driver, while working, needs some cleanup before the 
>> source is
>> published. I intend to maintain both this device and the windows 
>> driver
>> including producing a signed Windows 10 driver if Redhat are unwilling 
>> or
>> unable.
>> 
>> Kind Regards,
>> Geoffrey McRae
>> 
>> HostFission
>> https://hostfission.com

Re: RFC: New device for zero-copy VM memory access

[Dr. David Alan Gilbert]

* geoff@hostfission.com (geoff@hostfission.com) wrote:
> Hi All,
> 
> Over the past week, I have been working to come up with a solution to the
> memory transfer performance issues that hinder the Looking Glass Project.
> 
> Currently Looking Glass works by using the IVSHMEM shared memory device
> which
> is fed by an application that captures the guest's video output. While this
> works it is sub-optimal because we first have to perform a CPU copy of the
> captured frame into shared RAM, and then back out again for display. Because
> the destination buffers are allocated by closed proprietary code (DirectX,
> or
> NVidia NvFBC) there is no way to have the frame placed directly into the
> IVSHMEM shared ram.
> 
> This new device, currently named `introspection` (which needs a more
> suitable
> name, porthole perhaps?), provides a means of translating guest physical
> addresses to host virtual addresses, and finally to the host offsets in RAM
> for
> file-backed memory guests. It does this by means of a simple protocol over a
> unix socket (chardev) which is supplied the appropriate fd for the VM's
> system
> RAM. The guest (in this case, Windows), when presented with the address of a
> userspace buffer and size, will mlock the appropriate pages into RAM and
> pass
> guest physical addresses to the virtual device.

Hi Geroggrey,
  I wonder if the same thing can be done by using the existing vhost-user
mechanism.

  vhost-user is intended for implementing a virtio device outside of the
qemu process; so it has a character device that qemu passes commands down
to the other process, where qemu mostly passes commands via the virtio
queues.   To be able to read the virtio queues, the external process
mmap's the same memory as the guest - it gets passed a 'set mem table'
command by qemu that includes fd's for the RAM, and includes base/offset
pairs saying that a particular chunk of RAM is mapped at a particular
guest physical address.

  Whether or not you make use of virtio queues, I think the mechanism
for the device to tell the external process the mappings might be what
you're after.

Dave

> This device and the windows driver have been designed in such a way that
> it's a
> utility device for any project and/or application that could make use of it.
> The PCI subsystem vendor and device ID are used to provide a means of device
> identification in cases where multiple devices may be in use for differing
> applications. This also allows one common driver to be used for any other
> projects wishing to build on this device.
> 
> My ultimate goal is to get this to a state where it could be accepted
> upstream
> into Qemu at which point Looking Glass would be modified to use it instead
> of
> the IVSHMEM device.
> 
> My git repository with the new device can be found at:
> https://github.com/gnif/qemu
> 
> The new device is:
> https://github.com/gnif/qemu/blob/master/hw/misc/introspection.c
> 
> Looking Glass:
> https://looking-glass.hostfission.com/
> 
> The windows driver, while working, needs some cleanup before the source is
> published. I intend to maintain both this device and the windows driver
> including producing a signed Windows 10 driver if Redhat are unwilling or
> unable.
> 
> Kind Regards,
> Geoffrey McRae
> 
> HostFission
> https://hostfission.com
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: RFC: New device for zero-copy VM memory access

[geoff]

Hi Dave,

On 2019-10-31 05:52, Dr. David Alan Gilbert wrote:
> * geoff@hostfission.com (geoff@hostfission.com) wrote:
>> Hi All,
>> 
>> Over the past week, I have been working to come up with a solution to 
>> the
>> memory transfer performance issues that hinder the Looking Glass 
>> Project.
>> 
>> Currently Looking Glass works by using the IVSHMEM shared memory 
>> device
>> which
>> is fed by an application that captures the guest's video output. While 
>> this
>> works it is sub-optimal because we first have to perform a CPU copy of 
>> the
>> captured frame into shared RAM, and then back out again for display. 
>> Because
>> the destination buffers are allocated by closed proprietary code 
>> (DirectX,
>> or
>> NVidia NvFBC) there is no way to have the frame placed directly into 
>> the
>> IVSHMEM shared ram.
>> 
>> This new device, currently named `introspection` (which needs a more
>> suitable
>> name, porthole perhaps?), provides a means of translating guest 
>> physical
>> addresses to host virtual addresses, and finally to the host offsets 
>> in RAM
>> for
>> file-backed memory guests. It does this by means of a simple protocol 
>> over a
>> unix socket (chardev) which is supplied the appropriate fd for the 
>> VM's
>> system
>> RAM. The guest (in this case, Windows), when presented with the 
>> address of a
>> userspace buffer and size, will mlock the appropriate pages into RAM 
>> and
>> pass
>> guest physical addresses to the virtual device.
> 
> Hi Geroggrey,
>   I wonder if the same thing can be done by using the existing 
> vhost-user
> mechanism.
> 
>   vhost-user is intended for implementing a virtio device outside of 
> the
> qemu process; so it has a character device that qemu passes commands 
> down
> to the other process, where qemu mostly passes commands via the virtio
> queues.   To be able to read the virtio queues, the external process
> mmap's the same memory as the guest - it gets passed a 'set mem table'
> command by qemu that includes fd's for the RAM, and includes 
> base/offset
> pairs saying that a particular chunk of RAM is mapped at a particular
> guest physical address.
> 
>   Whether or not you make use of virtio queues, I think the mechanism
> for the device to tell the external process the mappings might be what
> you're after.
> 
> Dave
> 

While normally I would be all for re-using such code, the vhost-user 
while
being very feature-complete from what I understand is overkill for our
requirements. It will still allocate a communication ring and an events 
system
that we will not be using. The goal of this device is to provide a dumb 
&
simple method of sharing system ram, both for this project and for 
others that
work on a simple polling mechanism, it is not intended to be an 
end-to-end
solution like vhost-user is.

If you still believe that vhost-user should be used, I will do what I 
can to
implement it, but for such a simple device I honestly believe it is 
overkill.

-Geoff

>> This device and the windows driver have been designed in such a way 
>> that
>> it's a
>> utility device for any project and/or application that could make use 
>> of it.
>> The PCI subsystem vendor and device ID are used to provide a means of 
>> device
>> identification in cases where multiple devices may be in use for 
>> differing
>> applications. This also allows one common driver to be used for any 
>> other
>> projects wishing to build on this device.
>> 
>> My ultimate goal is to get this to a state where it could be accepted
>> upstream
>> into Qemu at which point Looking Glass would be modified to use it 
>> instead
>> of
>> the IVSHMEM device.
>> 
>> My git repository with the new device can be found at:
>> https://github.com/gnif/qemu
>> 
>> The new device is:
>> https://github.com/gnif/qemu/blob/master/hw/misc/introspection.c
>> 
>> Looking Glass:
>> https://looking-glass.hostfission.com/
>> 
>> The windows driver, while working, needs some cleanup before the 
>> source is
>> published. I intend to maintain both this device and the windows 
>> driver
>> including producing a signed Windows 10 driver if Redhat are unwilling 
>> or
>> unable.
>> 
>> Kind Regards,
>> Geoffrey McRae
>> 
>> HostFission
>> https://hostfission.com
>> 
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: RFC: New device for zero-copy VM memory access

[geoff]

Another update to this that adds support for unmap notification. The 
device
has also been renamed to `porthole` and now resides here:

https://github.com/gnif/qemu/blob/master/hw/misc/porthole.c

And here is the updated Linux test client.

https://gist.github.com/gnif/77e7fb54604b42a1a98ecb8bf3d2cf46

-Geoff

On 2019-10-31 13:55, geoff@hostfission.com wrote:
> Hi Dave,
> 
> On 2019-10-31 05:52, Dr. David Alan Gilbert wrote:
>> * geoff@hostfission.com (geoff@hostfission.com) wrote:
>>> Hi All,
>>> 
>>> Over the past week, I have been working to come up with a solution to 
>>> the
>>> memory transfer performance issues that hinder the Looking Glass 
>>> Project.
>>> 
>>> Currently Looking Glass works by using the IVSHMEM shared memory 
>>> device
>>> which
>>> is fed by an application that captures the guest's video output. 
>>> While this
>>> works it is sub-optimal because we first have to perform a CPU copy 
>>> of the
>>> captured frame into shared RAM, and then back out again for display. 
>>> Because
>>> the destination buffers are allocated by closed proprietary code 
>>> (DirectX,
>>> or
>>> NVidia NvFBC) there is no way to have the frame placed directly into 
>>> the
>>> IVSHMEM shared ram.
>>> 
>>> This new device, currently named `introspection` (which needs a more
>>> suitable
>>> name, porthole perhaps?), provides a means of translating guest 
>>> physical
>>> addresses to host virtual addresses, and finally to the host offsets 
>>> in RAM
>>> for
>>> file-backed memory guests. It does this by means of a simple protocol 
>>> over a
>>> unix socket (chardev) which is supplied the appropriate fd for the 
>>> VM's
>>> system
>>> RAM. The guest (in this case, Windows), when presented with the 
>>> address of a
>>> userspace buffer and size, will mlock the appropriate pages into RAM 
>>> and
>>> pass
>>> guest physical addresses to the virtual device.
>> 
>> Hi Geroggrey,
>>   I wonder if the same thing can be done by using the existing 
>> vhost-user
>> mechanism.
>> 
>>   vhost-user is intended for implementing a virtio device outside of 
>> the
>> qemu process; so it has a character device that qemu passes commands 
>> down
>> to the other process, where qemu mostly passes commands via the virtio
>> queues.   To be able to read the virtio queues, the external process
>> mmap's the same memory as the guest - it gets passed a 'set mem table'
>> command by qemu that includes fd's for the RAM, and includes 
>> base/offset
>> pairs saying that a particular chunk of RAM is mapped at a particular
>> guest physical address.
>> 
>>   Whether or not you make use of virtio queues, I think the mechanism
>> for the device to tell the external process the mappings might be what
>> you're after.
>> 
>> Dave
>> 
> 
> While normally I would be all for re-using such code, the vhost-user 
> while
> being very feature-complete from what I understand is overkill for our
> requirements. It will still allocate a communication ring and an events 
> system
> that we will not be using. The goal of this device is to provide a dumb 
> &
> simple method of sharing system ram, both for this project and for 
> others that
> work on a simple polling mechanism, it is not intended to be an 
> end-to-end
> solution like vhost-user is.
> 
> If you still believe that vhost-user should be used, I will do what I 
> can to
> implement it, but for such a simple device I honestly believe it is 
> overkill.
> 
> -Geoff
> 
>>> This device and the windows driver have been designed in such a way 
>>> that
>>> it's a
>>> utility device for any project and/or application that could make use 
>>> of it.
>>> The PCI subsystem vendor and device ID are used to provide a means of 
>>> device
>>> identification in cases where multiple devices may be in use for 
>>> differing
>>> applications. This also allows one common driver to be used for any 
>>> other
>>> projects wishing to build on this device.
>>> 
>>> My ultimate goal is to get this to a state where it could be accepted
>>> upstream
>>> into Qemu at which point Looking Glass would be modified to use it 
>>> instead
>>> of
>>> the IVSHMEM device.
>>> 
>>> My git repository with the new device can be found at:
>>> https://github.com/gnif/qemu
>>> 
>>> The new device is:
>>> https://github.com/gnif/qemu/blob/master/hw/misc/introspection.c
>>> 
>>> Looking Glass:
>>> https://looking-glass.hostfission.com/
>>> 
>>> The windows driver, while working, needs some cleanup before the 
>>> source is
>>> published. I intend to maintain both this device and the windows 
>>> driver
>>> including producing a signed Windows 10 driver if Redhat are 
>>> unwilling or
>>> unable.
>>> 
>>> Kind Regards,
>>> Geoffrey McRae
>>> 
>>> HostFission
>>> https://hostfission.com
>>> 
>> --
>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: RFC: New device for zero-copy VM memory access

[Peter Maydell]

On Thu, 31 Oct 2019 at 02:56, <geoff@hostfission.com> wrote:
> If you still believe that vhost-user should be used, I will do what I
> can to
> implement it, but for such a simple device I honestly believe it is
> overkill.

While not making a statement on whether vhost-user is technically
suitable in this case, I would note that we have a strong bias
towards "use facilities we already have rather than adding new ones",
because having multiple ways to do something in QEMU is more
maintenance effort and also confusing. It doesn't seem to me to
matter if vhost-user provides extra features your specific use case
doesn't need - just don't use them...

thanks
-- PMM

Re: RFC: New device for zero-copy VM memory access

[Dr. David Alan Gilbert]

* geoff@hostfission.com (geoff@hostfission.com) wrote:
> Hi Dave,
> 
> On 2019-10-31 05:52, Dr. David Alan Gilbert wrote:
> > * geoff@hostfission.com (geoff@hostfission.com) wrote:
> > > Hi All,
> > > 
> > > Over the past week, I have been working to come up with a solution
> > > to the
> > > memory transfer performance issues that hinder the Looking Glass
> > > Project.
> > > 
> > > Currently Looking Glass works by using the IVSHMEM shared memory
> > > device
> > > which
> > > is fed by an application that captures the guest's video output.
> > > While this
> > > works it is sub-optimal because we first have to perform a CPU copy
> > > of the
> > > captured frame into shared RAM, and then back out again for display.
> > > Because
> > > the destination buffers are allocated by closed proprietary code
> > > (DirectX,
> > > or
> > > NVidia NvFBC) there is no way to have the frame placed directly into
> > > the
> > > IVSHMEM shared ram.
> > > 
> > > This new device, currently named `introspection` (which needs a more
> > > suitable
> > > name, porthole perhaps?), provides a means of translating guest
> > > physical
> > > addresses to host virtual addresses, and finally to the host offsets
> > > in RAM
> > > for
> > > file-backed memory guests. It does this by means of a simple
> > > protocol over a
> > > unix socket (chardev) which is supplied the appropriate fd for the
> > > VM's
> > > system
> > > RAM. The guest (in this case, Windows), when presented with the
> > > address of a
> > > userspace buffer and size, will mlock the appropriate pages into RAM
> > > and
> > > pass
> > > guest physical addresses to the virtual device.
> > 
> > Hi Geroggrey,
> >   I wonder if the same thing can be done by using the existing
> > vhost-user
> > mechanism.
> > 
> >   vhost-user is intended for implementing a virtio device outside of the
> > qemu process; so it has a character device that qemu passes commands
> > down
> > to the other process, where qemu mostly passes commands via the virtio
> > queues.   To be able to read the virtio queues, the external process
> > mmap's the same memory as the guest - it gets passed a 'set mem table'
> > command by qemu that includes fd's for the RAM, and includes base/offset
> > pairs saying that a particular chunk of RAM is mapped at a particular
> > guest physical address.
> > 
> >   Whether or not you make use of virtio queues, I think the mechanism
> > for the device to tell the external process the mappings might be what
> > you're after.
> > 
> > Dave
> > 
> 
> While normally I would be all for re-using such code, the vhost-user while
> being very feature-complete from what I understand is overkill for our
> requirements. It will still allocate a communication ring and an events
> system
> that we will not be using. The goal of this device is to provide a dumb &
> simple method of sharing system ram, both for this project and for others
> that
> work on a simple polling mechanism, it is not intended to be an end-to-end
> solution like vhost-user is.
> 
> If you still believe that vhost-user should be used, I will do what I can to
> implement it, but for such a simple device I honestly believe it is
> overkill.

It's certainly worth having a look at vhost-user even if you don't use
most of it;  you can configure it down to 1 (maybe 0?) queues if you're
really desperate - and you might find it comes in useful!  The actual
setup is pretty easy.

The process of synchronising with (potentially changing) host memory
mapping is a bit hairy; so if we can share it with vhost it's probably
worth it.

Dave

> -Geoff
> 
> > > This device and the windows driver have been designed in such a way
> > > that
> > > it's a
> > > utility device for any project and/or application that could make
> > > use of it.
> > > The PCI subsystem vendor and device ID are used to provide a means
> > > of device
> > > identification in cases where multiple devices may be in use for
> > > differing
> > > applications. This also allows one common driver to be used for any
> > > other
> > > projects wishing to build on this device.
> > > 
> > > My ultimate goal is to get this to a state where it could be accepted
> > > upstream
> > > into Qemu at which point Looking Glass would be modified to use it
> > > instead
> > > of
> > > the IVSHMEM device.
> > > 
> > > My git repository with the new device can be found at:
> > > https://github.com/gnif/qemu
> > > 
> > > The new device is:
> > > https://github.com/gnif/qemu/blob/master/hw/misc/introspection.c
> > > 
> > > Looking Glass:
> > > https://looking-glass.hostfission.com/
> > > 
> > > The windows driver, while working, needs some cleanup before the
> > > source is
> > > published. I intend to maintain both this device and the windows
> > > driver
> > > including producing a signed Windows 10 driver if Redhat are
> > > unwilling or
> > > unable.
> > > 
> > > Kind Regards,
> > > Geoffrey McRae
> > > 
> > > HostFission
> > > https://hostfission.com
> > > 
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: RFC: New device for zero-copy VM memory access

[geoff]

On 2019-11-01 00:24, Dr. David Alan Gilbert wrote:
> * geoff@hostfission.com (geoff@hostfission.com) wrote:
>> Hi Dave,
>> 
>> On 2019-10-31 05:52, Dr. David Alan Gilbert wrote:
>> > * geoff@hostfission.com (geoff@hostfission.com) wrote:
>> > > Hi All,
>> > >
>> > > Over the past week, I have been working to come up with a solution
>> > > to the
>> > > memory transfer performance issues that hinder the Looking Glass
>> > > Project.
>> > >
>> > > Currently Looking Glass works by using the IVSHMEM shared memory
>> > > device
>> > > which
>> > > is fed by an application that captures the guest's video output.
>> > > While this
>> > > works it is sub-optimal because we first have to perform a CPU copy
>> > > of the
>> > > captured frame into shared RAM, and then back out again for display.
>> > > Because
>> > > the destination buffers are allocated by closed proprietary code
>> > > (DirectX,
>> > > or
>> > > NVidia NvFBC) there is no way to have the frame placed directly into
>> > > the
>> > > IVSHMEM shared ram.
>> > >
>> > > This new device, currently named `introspection` (which needs a more
>> > > suitable
>> > > name, porthole perhaps?), provides a means of translating guest
>> > > physical
>> > > addresses to host virtual addresses, and finally to the host offsets
>> > > in RAM
>> > > for
>> > > file-backed memory guests. It does this by means of a simple
>> > > protocol over a
>> > > unix socket (chardev) which is supplied the appropriate fd for the
>> > > VM's
>> > > system
>> > > RAM. The guest (in this case, Windows), when presented with the
>> > > address of a
>> > > userspace buffer and size, will mlock the appropriate pages into RAM
>> > > and
>> > > pass
>> > > guest physical addresses to the virtual device.
>> >
>> > Hi Geroggrey,
>> >   I wonder if the same thing can be done by using the existing
>> > vhost-user
>> > mechanism.
>> >
>> >   vhost-user is intended for implementing a virtio device outside of the
>> > qemu process; so it has a character device that qemu passes commands
>> > down
>> > to the other process, where qemu mostly passes commands via the virtio
>> > queues.   To be able to read the virtio queues, the external process
>> > mmap's the same memory as the guest - it gets passed a 'set mem table'
>> > command by qemu that includes fd's for the RAM, and includes base/offset
>> > pairs saying that a particular chunk of RAM is mapped at a particular
>> > guest physical address.
>> >
>> >   Whether or not you make use of virtio queues, I think the mechanism
>> > for the device to tell the external process the mappings might be what
>> > you're after.
>> >
>> > Dave
>> >
>> 
>> While normally I would be all for re-using such code, the vhost-user 
>> while
>> being very feature-complete from what I understand is overkill for our
>> requirements. It will still allocate a communication ring and an 
>> events
>> system
>> that we will not be using. The goal of this device is to provide a 
>> dumb &
>> simple method of sharing system ram, both for this project and for 
>> others
>> that
>> work on a simple polling mechanism, it is not intended to be an 
>> end-to-end
>> solution like vhost-user is.
>> 
>> If you still believe that vhost-user should be used, I will do what I 
>> can to
>> implement it, but for such a simple device I honestly believe it is
>> overkill.
> 
> It's certainly worth having a look at vhost-user even if you don't use
> most of it;  you can configure it down to 1 (maybe 0?) queues if you're
> really desperate - and you might find it comes in useful!  The actual
> setup is pretty easy.
> 
> The process of synchronising with (potentially changing) host memory
> mapping is a bit hairy; so if we can share it with vhost it's probably
> worth it.

Thanks, I will have a deeper dive into it, however the issues with
changing host memory, migration, and all that extra is of no concern or
use to us.

The audience that will be using this interface is not interested in
such features as the primary reason for Looking Glass is to allow a for
high-performance windows workstation for gaming and proprietary windows
only software. In these scenarios features like ram ballooning are
avoided like the plague as it hampers performance for use cases that
require consistent low latency for competitive gameplay.

As the author of Looking Glass, I also have to consider the maintenance
and the complexity of implementing the vhost protocol into the project.
At this time a complete Porthole client can be implemented in 150 lines
of C without external dependencies, and most of that is boilerplate
socket code. This IMO is a major factor in deciding to avoid vhost-user.

I also have to weigh up the cost of developing and maintaining the
windows driver for this device. I am very green when it comes to Windows
driver programming, it took weeks to write the first IVSHMEM driver, and
several days to write the Porthole driver which is far simpler than the
IVSHMEM driver. I'd hate to think of the time investment in maintaining
the vhost integration also (yes, I am aware there is a library).

These drivers are not complex and I am sure an experienced windows
driver developer could have thrown them together in a few hours, but
since our requirements are so niche and of little commercial value those
in our community that are using this project do not have the time and/or
ability to assist with the drivers.

 From my point of view, avoiding vhost-use seems like a better path to
take as I am able (and willing) to maintain the Porthole device in QEMU,
the OS drivers, and client interface. The Porthole device also doesn't
have any special or complex features keeping it very simple to maintain
and keeping the client protocol very simple.

There is also an open-source audio driver for windows called SCREAM
that was initially designed for broadcasting audio over a network,
however, it's authors have also implemented transport via shared RAM.
While vhost-user would make much more sense here as vring buffers
would be very useful, the barrier to entry is too high and as such the
developers have instead opted to use the simple IVSHMEM device instead.

That said, I will still have a deeper look into vhost-user but I hope
the above shows the merits of this simple method of guest ram access.

-Geoff

> 
> Dave
> 
>> -Geoff
>> 
>> > > This device and the windows driver have been designed in such a way
>> > > that
>> > > it's a
>> > > utility device for any project and/or application that could make
>> > > use of it.
>> > > The PCI subsystem vendor and device ID are used to provide a means
>> > > of device
>> > > identification in cases where multiple devices may be in use for
>> > > differing
>> > > applications. This also allows one common driver to be used for any
>> > > other
>> > > projects wishing to build on this device.
>> > >
>> > > My ultimate goal is to get this to a state where it could be accepted
>> > > upstream
>> > > into Qemu at which point Looking Glass would be modified to use it
>> > > instead
>> > > of
>> > > the IVSHMEM device.
>> > >
>> > > My git repository with the new device can be found at:
>> > > https://github.com/gnif/qemu
>> > >
>> > > The new device is:
>> > > https://github.com/gnif/qemu/blob/master/hw/misc/introspection.c
>> > >
>> > > Looking Glass:
>> > > https://looking-glass.hostfission.com/
>> > >
>> > > The windows driver, while working, needs some cleanup before the
>> > > source is
>> > > published. I intend to maintain both this device and the windows
>> > > driver
>> > > including producing a signed Windows 10 driver if Redhat are
>> > > unwilling or
>> > > unable.
>> > >
>> > > Kind Regards,
>> > > Geoffrey McRae
>> > >
>> > > HostFission
>> > > https://hostfission.com
>> > >
>> > --
>> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: RFC: New device for zero-copy VM memory access

[Peter Maydell]

On Thu, 31 Oct 2019 at 14:26, <geoff@hostfission.com> wrote:
> As the author of Looking Glass, I also have to consider the maintenance
> and the complexity of implementing the vhost protocol into the project.
> At this time a complete Porthole client can be implemented in 150 lines
> of C without external dependencies, and most of that is boilerplate
> socket code. This IMO is a major factor in deciding to avoid vhost-user.

This is essentially a proposal that we should make our project and
code more complicated so that your project and code can be simpler.
I hope you can see why this isn't necessarily an argument that will hold
very much weight for us :-)

thanks
-- PMM

Re: RFC: New device for zero-copy VM memory access

[geoff]

On 2019-11-01 01:52, Peter Maydell wrote:
> On Thu, 31 Oct 2019 at 14:26, <geoff@hostfission.com> wrote:
>> As the author of Looking Glass, I also have to consider the 
>> maintenance
>> and the complexity of implementing the vhost protocol into the 
>> project.
>> At this time a complete Porthole client can be implemented in 150 
>> lines
>> of C without external dependencies, and most of that is boilerplate
>> socket code. This IMO is a major factor in deciding to avoid 
>> vhost-user.
> 
> This is essentially a proposal that we should make our project and
> code more complicated so that your project and code can be simpler.
> I hope you can see why this isn't necessarily an argument that will 
> hold
> very much weight for us :-)

Certainly, I do which is why I am still going to see about using vhost,
however, a device that uses vhost is likely more complex then the device
as it stands right now and as such more maintenance would be involved on
your end also. Or have I missed something in that vhost-user can be used
directly as a device?

> 
> thanks
> -- PMM

Re: RFC: New device for zero-copy VM memory access

[Dr. David Alan Gilbert]

* geoff@hostfission.com (geoff@hostfission.com) wrote:
> 
> 
> On 2019-11-01 01:52, Peter Maydell wrote:
> > On Thu, 31 Oct 2019 at 14:26, <geoff@hostfission.com> wrote:
> > > As the author of Looking Glass, I also have to consider the
> > > maintenance
> > > and the complexity of implementing the vhost protocol into the
> > > project.
> > > At this time a complete Porthole client can be implemented in 150
> > > lines
> > > of C without external dependencies, and most of that is boilerplate
> > > socket code. This IMO is a major factor in deciding to avoid
> > > vhost-user.
> > 
> > This is essentially a proposal that we should make our project and
> > code more complicated so that your project and code can be simpler.
> > I hope you can see why this isn't necessarily an argument that will hold
> > very much weight for us :-)
> 
> Certainly, I do which is why I am still going to see about using vhost,
> however, a device that uses vhost is likely more complex then the device
> as it stands right now and as such more maintenance would be involved on
> your end also. Or have I missed something in that vhost-user can be used
> directly as a device?

The basic vhost-user stuff isn't actually that hard;  if you aren't
actually shuffling commands over the queues you should find it pretty
simple - so I think your assumption about it being simpler if you avoid
it might be wrong.  It might be easier if you use it!

Dave

> > 
> > thanks
> > -- PMM
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: RFC: New device for zero-copy VM memory access

[geoff]

On 2019-11-01 02:52, Dr. David Alan Gilbert wrote:
> * geoff@hostfission.com (geoff@hostfission.com) wrote:
>> 
>> 
>> On 2019-11-01 01:52, Peter Maydell wrote:
>> > On Thu, 31 Oct 2019 at 14:26, <geoff@hostfission.com> wrote:
>> > > As the author of Looking Glass, I also have to consider the
>> > > maintenance
>> > > and the complexity of implementing the vhost protocol into the
>> > > project.
>> > > At this time a complete Porthole client can be implemented in 150
>> > > lines
>> > > of C without external dependencies, and most of that is boilerplate
>> > > socket code. This IMO is a major factor in deciding to avoid
>> > > vhost-user.
>> >
>> > This is essentially a proposal that we should make our project and
>> > code more complicated so that your project and code can be simpler.
>> > I hope you can see why this isn't necessarily an argument that will hold
>> > very much weight for us :-)
>> 
>> Certainly, I do which is why I am still going to see about using 
>> vhost,
>> however, a device that uses vhost is likely more complex then the 
>> device
>> as it stands right now and as such more maintenance would be involved 
>> on
>> your end also. Or have I missed something in that vhost-user can be 
>> used
>> directly as a device?
> 
> The basic vhost-user stuff isn't actually that hard;  if you aren't
> actually shuffling commands over the queues you should find it pretty
> simple - so I think your assumption about it being simpler if you avoid
> it might be wrong.  It might be easier if you use it!

I have been looking into this and I am yet to find some decent
documentation or a simple device example I can use to understand how to
create such a device. Do you know of any reading or examples I can 
obtain
on how to get an initial do nothing device up and running?

-Geoff

> 
> Dave
> 
>> >
>> > thanks
>> > -- PMM
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: RFC: New device for zero-copy VM memory access

[geoff]

On 2019-11-03 21:10, geoff@hostfission.com wrote:
> On 2019-11-01 02:52, Dr. David Alan Gilbert wrote:
>> * geoff@hostfission.com (geoff@hostfission.com) wrote:
>>> 
>>> 
>>> On 2019-11-01 01:52, Peter Maydell wrote:
>>> > On Thu, 31 Oct 2019 at 14:26, <geoff@hostfission.com> wrote:
>>> > > As the author of Looking Glass, I also have to consider the
>>> > > maintenance
>>> > > and the complexity of implementing the vhost protocol into the
>>> > > project.
>>> > > At this time a complete Porthole client can be implemented in 150
>>> > > lines
>>> > > of C without external dependencies, and most of that is boilerplate
>>> > > socket code. This IMO is a major factor in deciding to avoid
>>> > > vhost-user.
>>> >
>>> > This is essentially a proposal that we should make our project and
>>> > code more complicated so that your project and code can be simpler.
>>> > I hope you can see why this isn't necessarily an argument that will hold
>>> > very much weight for us :-)
>>> 
>>> Certainly, I do which is why I am still going to see about using 
>>> vhost,
>>> however, a device that uses vhost is likely more complex then the 
>>> device
>>> as it stands right now and as such more maintenance would be involved 
>>> on
>>> your end also. Or have I missed something in that vhost-user can be 
>>> used
>>> directly as a device?
>> 
>> The basic vhost-user stuff isn't actually that hard;  if you aren't
>> actually shuffling commands over the queues you should find it pretty
>> simple - so I think your assumption about it being simpler if you 
>> avoid
>> it might be wrong.  It might be easier if you use it!
> 
> I have been looking into this and I am yet to find some decent
> documentation or a simple device example I can use to understand how to
> create such a device. Do you know of any reading or examples I can 
> obtain
> on how to get an initial do nothing device up and running?
> 
> -Geoff

Scratch that, the design just solidified for me and I am now making
progress, however it seems that vhost-user can't do what we need here:

1) I dont see any way to recieve notification of socket disconnection, 
in
our use case the client app needs to be able to be (re)connected
dynamically. It might be possible to get this event by registering it on
the chardev manually but this seems like it would be a kludge.

2) I don't see any method of notifying the vhost-user client of the
removal of a shared memory mapping. Again, these may not be persistently
mapped in the guest as we have no control over the buffer allocation, 
and
as such, we need a method to notify the client that the mapping has 
become
invalid.

3) VHOST_USER_SET_MEM_TABLE is a one time request, again this breaks our
usage as we need to change this dynamically at runtime.

Unless there are viable solutions to these problems there is no way that
vhost-user can be used for this kind of a device.

-Geoff

> 
>> 
>> Dave
>> 
>>> >
>>> > thanks
>>> > -- PMM
>> --
>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: RFC: New device for zero-copy VM memory access

[Dr. David Alan Gilbert]

* geoff@hostfission.com (geoff@hostfission.com) wrote:
> 
> 
> On 2019-11-03 21:10, geoff@hostfission.com wrote:
> > On 2019-11-01 02:52, Dr. David Alan Gilbert wrote:
> > > * geoff@hostfission.com (geoff@hostfission.com) wrote:
> > > > 
> > > > 
> > > > On 2019-11-01 01:52, Peter Maydell wrote:
> > > > > On Thu, 31 Oct 2019 at 14:26, <geoff@hostfission.com> wrote:
> > > > > > As the author of Looking Glass, I also have to consider the
> > > > > > maintenance
> > > > > > and the complexity of implementing the vhost protocol into the
> > > > > > project.
> > > > > > At this time a complete Porthole client can be implemented in 150
> > > > > > lines
> > > > > > of C without external dependencies, and most of that is boilerplate
> > > > > > socket code. This IMO is a major factor in deciding to avoid
> > > > > > vhost-user.
> > > > >
> > > > > This is essentially a proposal that we should make our project and
> > > > > code more complicated so that your project and code can be simpler.
> > > > > I hope you can see why this isn't necessarily an argument that will hold
> > > > > very much weight for us :-)
> > > > 
> > > > Certainly, I do which is why I am still going to see about using
> > > > vhost,
> > > > however, a device that uses vhost is likely more complex then
> > > > the device
> > > > as it stands right now and as such more maintenance would be
> > > > involved on
> > > > your end also. Or have I missed something in that vhost-user can
> > > > be used
> > > > directly as a device?
> > > 
> > > The basic vhost-user stuff isn't actually that hard;  if you aren't
> > > actually shuffling commands over the queues you should find it pretty
> > > simple - so I think your assumption about it being simpler if you
> > > avoid
> > > it might be wrong.  It might be easier if you use it!
> > 
> > I have been looking into this and I am yet to find some decent
> > documentation or a simple device example I can use to understand how to
> > create such a device. Do you know of any reading or examples I can
> > obtain
> > on how to get an initial do nothing device up and running?
> > 
> > -Geoff
> 
> Scratch that, the design just solidified for me and I am now making
> progress, however it seems that vhost-user can't do what we need here:
> 
> 1) I dont see any way to recieve notification of socket disconnection, in
> our use case the client app needs to be able to be (re)connected
> dynamically. It might be possible to get this event by registering it on
> the chardev manually but this seems like it would be a kludge.

My understanding was that someone added support for reconnection of
vhost-user;  I'm not sure of the detail - cc'ing in Maxime and
Marc-Andre.

> 2) I don't see any method of notifying the vhost-user client of the
> removal of a shared memory mapping. Again, these may not be persistently
> mapped in the guest as we have no control over the buffer allocation, and
> as such, we need a method to notify the client that the mapping has become
> invalid.
> 
> 3) VHOST_USER_SET_MEM_TABLE is a one time request, again this breaks our
> usage as we need to change this dynamically at runtime.

I've seen (3) being sent multiple times (It's messy but it happens); so
I think that fixes (2) as well for you.

Dave

> Unless there are viable solutions to these problems there is no way that
> vhost-user can be used for this kind of a device.
> 
> -Geoff
> 
> > 
> > > 
> > > Dave
> > > 
> > > > >
> > > > > thanks
> > > > > -- PMM
> > > --
> > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: RFC: New device for zero-copy VM memory access

[geoff]

On 2019-11-04 22:55, Dr. David Alan Gilbert wrote:
> * geoff@hostfission.com (geoff@hostfission.com) wrote:
>> 
>> 
>> On 2019-11-03 21:10, geoff@hostfission.com wrote:
>> > On 2019-11-01 02:52, Dr. David Alan Gilbert wrote:
>> > > * geoff@hostfission.com (geoff@hostfission.com) wrote:
>> > > >
>> > > >
>> > > > On 2019-11-01 01:52, Peter Maydell wrote:
>> > > > > On Thu, 31 Oct 2019 at 14:26, <geoff@hostfission.com> wrote:
>> > > > > > As the author of Looking Glass, I also have to consider the
>> > > > > > maintenance
>> > > > > > and the complexity of implementing the vhost protocol into the
>> > > > > > project.
>> > > > > > At this time a complete Porthole client can be implemented in 150
>> > > > > > lines
>> > > > > > of C without external dependencies, and most of that is boilerplate
>> > > > > > socket code. This IMO is a major factor in deciding to avoid
>> > > > > > vhost-user.
>> > > > >
>> > > > > This is essentially a proposal that we should make our project and
>> > > > > code more complicated so that your project and code can be simpler.
>> > > > > I hope you can see why this isn't necessarily an argument that will hold
>> > > > > very much weight for us :-)
>> > > >
>> > > > Certainly, I do which is why I am still going to see about using
>> > > > vhost,
>> > > > however, a device that uses vhost is likely more complex then
>> > > > the device
>> > > > as it stands right now and as such more maintenance would be
>> > > > involved on
>> > > > your end also. Or have I missed something in that vhost-user can
>> > > > be used
>> > > > directly as a device?
>> > >
>> > > The basic vhost-user stuff isn't actually that hard;  if you aren't
>> > > actually shuffling commands over the queues you should find it pretty
>> > > simple - so I think your assumption about it being simpler if you
>> > > avoid
>> > > it might be wrong.  It might be easier if you use it!
>> >
>> > I have been looking into this and I am yet to find some decent
>> > documentation or a simple device example I can use to understand how to
>> > create such a device. Do you know of any reading or examples I can
>> > obtain
>> > on how to get an initial do nothing device up and running?
>> >
>> > -Geoff
>> 
>> Scratch that, the design just solidified for me and I am now making
>> progress, however it seems that vhost-user can't do what we need here:
>> 
>> 1) I dont see any way to recieve notification of socket disconnection, 
>> in
>> our use case the client app needs to be able to be (re)connected
>> dynamically. It might be possible to get this event by registering it 
>> on
>> the chardev manually but this seems like it would be a kludge.
> 
> My understanding was that someone added support for reconnection of
> vhost-user;  I'm not sure of the detail - cc'ing in Maxime and
> Marc-Andre.
> 
>> 2) I don't see any method of notifying the vhost-user client of the
>> removal of a shared memory mapping. Again, these may not be 
>> persistently
>> mapped in the guest as we have no control over the buffer allocation, 
>> and
>> as such, we need a method to notify the client that the mapping has 
>> become
>> invalid.
>> 
>> 3) VHOST_USER_SET_MEM_TABLE is a one time request, again this breaks 
>> our
>> usage as we need to change this dynamically at runtime.
> 
> I've seen (3) being sent multiple times (It's messy but it happens); so
> I think that fixes (2) as well for you.

Yes, but it's ignored.

     /*
      * For non-vring specific requests, like VHOST_USER_SET_MEM_TABLE,
      * we just need send it once in the first time. For later such
      * request, we just ignore it.
      */
     if (vhost_user_one_time_request(msg->hdr.request) && dev->vq_index 
!= 0) {
          msg->hdr.flags &= ~VHOST_USER_NEED_REPLY_MASK;
          return 0;
     }

> 
> Dave
> 
>> Unless there are viable solutions to these problems there is no way 
>> that
>> vhost-user can be used for this kind of a device.
>> 
>> -Geoff
>> 
>> >
>> > >
>> > > Dave
>> > >
>> > > > >
>> > > > > thanks
>> > > > > -- PMM
>> > > --
>> > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: RFC: New device for zero-copy VM memory access

[Dr. David Alan Gilbert]

* geoff@hostfission.com (geoff@hostfission.com) wrote:
> 
> 
> On 2019-11-04 22:55, Dr. David Alan Gilbert wrote:
> > * geoff@hostfission.com (geoff@hostfission.com) wrote:
> > > 
> > > 
> > > On 2019-11-03 21:10, geoff@hostfission.com wrote:
> > > > On 2019-11-01 02:52, Dr. David Alan Gilbert wrote:
> > > > > * geoff@hostfission.com (geoff@hostfission.com) wrote:
> > > > > >
> > > > > >
> > > > > > On 2019-11-01 01:52, Peter Maydell wrote:
> > > > > > > On Thu, 31 Oct 2019 at 14:26, <geoff@hostfission.com> wrote:
> > > > > > > > As the author of Looking Glass, I also have to consider the
> > > > > > > > maintenance
> > > > > > > > and the complexity of implementing the vhost protocol into the
> > > > > > > > project.
> > > > > > > > At this time a complete Porthole client can be implemented in 150
> > > > > > > > lines
> > > > > > > > of C without external dependencies, and most of that is boilerplate
> > > > > > > > socket code. This IMO is a major factor in deciding to avoid
> > > > > > > > vhost-user.
> > > > > > >
> > > > > > > This is essentially a proposal that we should make our project and
> > > > > > > code more complicated so that your project and code can be simpler.
> > > > > > > I hope you can see why this isn't necessarily an argument that will hold
> > > > > > > very much weight for us :-)
> > > > > >
> > > > > > Certainly, I do which is why I am still going to see about using
> > > > > > vhost,
> > > > > > however, a device that uses vhost is likely more complex then
> > > > > > the device
> > > > > > as it stands right now and as such more maintenance would be
> > > > > > involved on
> > > > > > your end also. Or have I missed something in that vhost-user can
> > > > > > be used
> > > > > > directly as a device?
> > > > >
> > > > > The basic vhost-user stuff isn't actually that hard;  if you aren't
> > > > > actually shuffling commands over the queues you should find it pretty
> > > > > simple - so I think your assumption about it being simpler if you
> > > > > avoid
> > > > > it might be wrong.  It might be easier if you use it!
> > > >
> > > > I have been looking into this and I am yet to find some decent
> > > > documentation or a simple device example I can use to understand how to
> > > > create such a device. Do you know of any reading or examples I can
> > > > obtain
> > > > on how to get an initial do nothing device up and running?
> > > >
> > > > -Geoff
> > > 
> > > Scratch that, the design just solidified for me and I am now making
> > > progress, however it seems that vhost-user can't do what we need here:
> > > 
> > > 1) I dont see any way to recieve notification of socket
> > > disconnection, in
> > > our use case the client app needs to be able to be (re)connected
> > > dynamically. It might be possible to get this event by registering
> > > it on
> > > the chardev manually but this seems like it would be a kludge.
> > 
> > My understanding was that someone added support for reconnection of
> > vhost-user;  I'm not sure of the detail - cc'ing in Maxime and
> > Marc-Andre.
> > 
> > > 2) I don't see any method of notifying the vhost-user client of the
> > > removal of a shared memory mapping. Again, these may not be
> > > persistently
> > > mapped in the guest as we have no control over the buffer
> > > allocation, and
> > > as such, we need a method to notify the client that the mapping has
> > > become
> > > invalid.
> > > 
> > > 3) VHOST_USER_SET_MEM_TABLE is a one time request, again this breaks
> > > our
> > > usage as we need to change this dynamically at runtime.
> > 
> > I've seen (3) being sent multiple times (It's messy but it happens); so
> > I think that fixes (2) as well for you.
> 
> Yes, but it's ignored.
> 
>     /*
>      * For non-vring specific requests, like VHOST_USER_SET_MEM_TABLE,
>      * we just need send it once in the first time. For later such
>      * request, we just ignore it.
>      */
>     if (vhost_user_one_time_request(msg->hdr.request) && dev->vq_index != 0)
> {
>          msg->hdr.flags &= ~VHOST_USER_NEED_REPLY_MASK;
>          return 0;
>     }

Curious.  I could swear I'd already dealt with multiple copies of this
message coming over vhost-user and having to deal with it when it did.
But now I'm confused, isn't vq_index a unique number per queue, so is
this really stopping it happening multiple times, or just making sure it
only happens for the first queue?

Dave

> > 
> > Dave
> > 
> > > Unless there are viable solutions to these problems there is no way
> > > that
> > > vhost-user can be used for this kind of a device.
> > > 
> > > -Geoff
> > > 
> > > >
> > > > >
> > > > > Dave
> > > > >
> > > > > > >
> > > > > > > thanks
> > > > > > > -- PMM
> > > > > --
> > > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: RFC: New device for zero-copy VM memory access

[Marc-André Lureau]

Hi

On Mon, Nov 4, 2019 at 4:05 PM <geoff@hostfission.com> wrote:
>
>
>
> On 2019-11-04 22:55, Dr. David Alan Gilbert wrote:
> > * geoff@hostfission.com (geoff@hostfission.com) wrote:
> >>
> >>
> >> On 2019-11-03 21:10, geoff@hostfission.com wrote:
> >> > On 2019-11-01 02:52, Dr. David Alan Gilbert wrote:
> >> > > * geoff@hostfission.com (geoff@hostfission.com) wrote:
> >> > > >
> >> > > >
> >> > > > On 2019-11-01 01:52, Peter Maydell wrote:
> >> > > > > On Thu, 31 Oct 2019 at 14:26, <geoff@hostfission.com> wrote:
> >> > > > > > As the author of Looking Glass, I also have to consider the
> >> > > > > > maintenance
> >> > > > > > and the complexity of implementing the vhost protocol into the
> >> > > > > > project.
> >> > > > > > At this time a complete Porthole client can be implemented in 150
> >> > > > > > lines
> >> > > > > > of C without external dependencies, and most of that is boilerplate
> >> > > > > > socket code. This IMO is a major factor in deciding to avoid
> >> > > > > > vhost-user.
> >> > > > >
> >> > > > > This is essentially a proposal that we should make our project and
> >> > > > > code more complicated so that your project and code can be simpler.
> >> > > > > I hope you can see why this isn't necessarily an argument that will hold
> >> > > > > very much weight for us :-)
> >> > > >
> >> > > > Certainly, I do which is why I am still going to see about using
> >> > > > vhost,
> >> > > > however, a device that uses vhost is likely more complex then
> >> > > > the device
> >> > > > as it stands right now and as such more maintenance would be
> >> > > > involved on
> >> > > > your end also. Or have I missed something in that vhost-user can
> >> > > > be used
> >> > > > directly as a device?
> >> > >
> >> > > The basic vhost-user stuff isn't actually that hard;  if you aren't
> >> > > actually shuffling commands over the queues you should find it pretty
> >> > > simple - so I think your assumption about it being simpler if you
> >> > > avoid
> >> > > it might be wrong.  It might be easier if you use it!
> >> >
> >> > I have been looking into this and I am yet to find some decent
> >> > documentation or a simple device example I can use to understand how to
> >> > create such a device. Do you know of any reading or examples I can
> >> > obtain
> >> > on how to get an initial do nothing device up and running?
> >> >
> >> > -Geoff
> >>
> >> Scratch that, the design just solidified for me and I am now making
> >> progress, however it seems that vhost-user can't do what we need here:
> >>
> >> 1) I dont see any way to recieve notification of socket disconnection,
> >> in
> >> our use case the client app needs to be able to be (re)connected
> >> dynamically. It might be possible to get this event by registering it
> >> on
> >> the chardev manually but this seems like it would be a kludge.
> >
> > My understanding was that someone added support for reconnection of
> > vhost-user;  I'm not sure of the detail - cc'ing in Maxime and
> > Marc-Andre.
> >
> >> 2) I don't see any method of notifying the vhost-user client of the
> >> removal of a shared memory mapping. Again, these may not be
> >> persistently
> >> mapped in the guest as we have no control over the buffer allocation,
> >> and
> >> as such, we need a method to notify the client that the mapping has
> >> become
> >> invalid.
> >>
> >> 3) VHOST_USER_SET_MEM_TABLE is a one time request, again this breaks
> >> our
> >> usage as we need to change this dynamically at runtime.
> >
> > I've seen (3) being sent multiple times (It's messy but it happens); so
> > I think that fixes (2) as well for you.
>
> Yes, but it's ignored.
>
>      /*
>       * For non-vring specific requests, like VHOST_USER_SET_MEM_TABLE,
>       * we just need send it once in the first time. For later such
>       * request, we just ignore it.
>       */
>      if (vhost_user_one_time_request(msg->hdr.request) && dev->vq_index
> != 0) {
>           msg->hdr.flags &= ~VHOST_USER_NEED_REPLY_MASK;
>           return 0;
>      }

This code was added to avoid sending the same mapping for each queue:
https://github.com/qemu/qemu/commit/b931bfbf042983f311b3b09894d8030b2755a638
(arguably, the filtering could have been done earlier)

But if you reconnect, it should still send it again at least once (for vq #0).

vhost-user-bridge reconnect used to work quite reliably, I haven't
tested recently.


>
> >
> > Dave
> >
> >> Unless there are viable solutions to these problems there is no way
> >> that
> >> vhost-user can be used for this kind of a device.
> >>
> >> -Geoff
> >>
> >> >
> >> > >
> >> > > Dave
> >> > >
> >> > > > >
> >> > > > > thanks
> >> > > > > -- PMM
> >> > > --
> >> > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: RFC: New device for zero-copy VM memory access

[Dr. David Alan Gilbert]

* Marc-André Lureau (marcandre.lureau@redhat.com) wrote:
> Hi
> 
> On Mon, Nov 4, 2019 at 4:05 PM <geoff@hostfission.com> wrote:
> >
> >
> >
> > On 2019-11-04 22:55, Dr. David Alan Gilbert wrote:
> > > * geoff@hostfission.com (geoff@hostfission.com) wrote:
> > >>
> > >>
> > >> On 2019-11-03 21:10, geoff@hostfission.com wrote:
> > >> > On 2019-11-01 02:52, Dr. David Alan Gilbert wrote:
> > >> > > * geoff@hostfission.com (geoff@hostfission.com) wrote:
> > >> > > >
> > >> > > >
> > >> > > > On 2019-11-01 01:52, Peter Maydell wrote:
> > >> > > > > On Thu, 31 Oct 2019 at 14:26, <geoff@hostfission.com> wrote:
> > >> > > > > > As the author of Looking Glass, I also have to consider the
> > >> > > > > > maintenance
> > >> > > > > > and the complexity of implementing the vhost protocol into the
> > >> > > > > > project.
> > >> > > > > > At this time a complete Porthole client can be implemented in 150
> > >> > > > > > lines
> > >> > > > > > of C without external dependencies, and most of that is boilerplate
> > >> > > > > > socket code. This IMO is a major factor in deciding to avoid
> > >> > > > > > vhost-user.
> > >> > > > >
> > >> > > > > This is essentially a proposal that we should make our project and
> > >> > > > > code more complicated so that your project and code can be simpler.
> > >> > > > > I hope you can see why this isn't necessarily an argument that will hold
> > >> > > > > very much weight for us :-)
> > >> > > >
> > >> > > > Certainly, I do which is why I am still going to see about using
> > >> > > > vhost,
> > >> > > > however, a device that uses vhost is likely more complex then
> > >> > > > the device
> > >> > > > as it stands right now and as such more maintenance would be
> > >> > > > involved on
> > >> > > > your end also. Or have I missed something in that vhost-user can
> > >> > > > be used
> > >> > > > directly as a device?
> > >> > >
> > >> > > The basic vhost-user stuff isn't actually that hard;  if you aren't
> > >> > > actually shuffling commands over the queues you should find it pretty
> > >> > > simple - so I think your assumption about it being simpler if you
> > >> > > avoid
> > >> > > it might be wrong.  It might be easier if you use it!
> > >> >
> > >> > I have been looking into this and I am yet to find some decent
> > >> > documentation or a simple device example I can use to understand how to
> > >> > create such a device. Do you know of any reading or examples I can
> > >> > obtain
> > >> > on how to get an initial do nothing device up and running?
> > >> >
> > >> > -Geoff
> > >>
> > >> Scratch that, the design just solidified for me and I am now making
> > >> progress, however it seems that vhost-user can't do what we need here:
> > >>
> > >> 1) I dont see any way to recieve notification of socket disconnection,
> > >> in
> > >> our use case the client app needs to be able to be (re)connected
> > >> dynamically. It might be possible to get this event by registering it
> > >> on
> > >> the chardev manually but this seems like it would be a kludge.
> > >
> > > My understanding was that someone added support for reconnection of
> > > vhost-user;  I'm not sure of the detail - cc'ing in Maxime and
> > > Marc-Andre.
> > >
> > >> 2) I don't see any method of notifying the vhost-user client of the
> > >> removal of a shared memory mapping. Again, these may not be
> > >> persistently
> > >> mapped in the guest as we have no control over the buffer allocation,
> > >> and
> > >> as such, we need a method to notify the client that the mapping has
> > >> become
> > >> invalid.
> > >>
> > >> 3) VHOST_USER_SET_MEM_TABLE is a one time request, again this breaks
> > >> our
> > >> usage as we need to change this dynamically at runtime.
> > >
> > > I've seen (3) being sent multiple times (It's messy but it happens); so
> > > I think that fixes (2) as well for you.
> >
> > Yes, but it's ignored.
> >
> >      /*
> >       * For non-vring specific requests, like VHOST_USER_SET_MEM_TABLE,
> >       * we just need send it once in the first time. For later such
> >       * request, we just ignore it.
> >       */
> >      if (vhost_user_one_time_request(msg->hdr.request) && dev->vq_index
> > != 0) {
> >           msg->hdr.flags &= ~VHOST_USER_NEED_REPLY_MASK;
> >           return 0;
> >      }
> 
> This code was added to avoid sending the same mapping for each queue:
> https://github.com/qemu/qemu/commit/b931bfbf042983f311b3b09894d8030b2755a638
> (arguably, the filtering could have been done earlier)
> 
> But if you reconnect, it should still send it again at least once (for vq #0).
> 
> vhost-user-bridge reconnect used to work quite reliably, I haven't
> tested recently.

Doesn't this also happen sometimes if the guest sees memory devices
change?

Dave

> 
> >
> > >
> > > Dave
> > >
> > >> Unless there are viable solutions to these problems there is no way
> > >> that
> > >> vhost-user can be used for this kind of a device.
> > >>
> > >> -Geoff
> > >>
> > >> >
> > >> > >
> > >> > > Dave
> > >> > >
> > >> > > > >
> > >> > > > > thanks
> > >> > > > > -- PMM
> > >> > > --
> > >> > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> > > --
> > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: RFC: New device for zero-copy VM memory access

[Gerd Hoffmann]

  Hi,

> This new device, currently named `introspection` (which needs a more
> suitable name, porthole perhaps?), provides a means of translating
> guest physical addresses to host virtual addresses, and finally to the
> host offsets in RAM for file-backed memory guests. It does this by
> means of a simple protocol over a unix socket (chardev) which is
> supplied the appropriate fd for the VM's system RAM. The guest (in
> this case, Windows), when presented with the address of a userspace
> buffer and size, will mlock the appropriate pages into RAM and pass
> guest physical addresses to the virtual device.

So, if I understand things correctly, the workflow looks like this:

  (1) guest allocates buffers, using guest ram.
  (2) guest uses these buffers as render target for the gpu (pci-assigned I guess?).
  (3) guest passes guest physical address to qemu (via porthole device).
  (4) qemu translates gpa into file offset and passes offsets to
      the client application.
  (5) client application maps all guest ram, then uses the offsets from
      qemu to find the buffers.  Then goes displaying these buffers I guess.

Correct?

Performance aside for now, is it an option for your use case to simply
use both an emulated display device and the assigned gpu, then configure
screen mirroring inside the guest to get the guest display scanned out
to the host?

cheers,
  Gerd

Re: RFC: New device for zero-copy VM memory access

[geoff]

On 2019-11-04 21:26, Gerd Hoffmann wrote:
> Hi,
> 
>> This new device, currently named `introspection` (which needs a more
>> suitable name, porthole perhaps?), provides a means of translating
>> guest physical addresses to host virtual addresses, and finally to the
>> host offsets in RAM for file-backed memory guests. It does this by
>> means of a simple protocol over a unix socket (chardev) which is
>> supplied the appropriate fd for the VM's system RAM. The guest (in
>> this case, Windows), when presented with the address of a userspace
>> buffer and size, will mlock the appropriate pages into RAM and pass
>> guest physical addresses to the virtual device.
> 
> So, if I understand things correctly, the workflow looks like this:
> 
>   (1) guest allocates buffers, using guest ram.
>   (2) guest uses these buffers as render target for the gpu
> (pci-assigned I guess?).
>   (3) guest passes guest physical address to qemu (via porthole 
> device).
>   (4) qemu translates gpa into file offset and passes offsets to
>       the client application.
>   (5) client application maps all guest ram, then uses the offsets from
>       qemu to find the buffers.  Then goes displaying these buffers I 
> guess.
> 
> Correct?

Correct, however step 5 might be a proxy to copy the buffers into 
another
porthole device in a second VM allowing VM->VM transfers.

> 
> Performance aside for now, is it an option for your use case to simply
> use both an emulated display device and the assigned gpu, then 
> configure
> screen mirroring inside the guest to get the guest display scanned out
> to the host?

Unfortunately no, NVidia and AMD devices do not support mirroring their
outputs to a separate GPU unless it's a professional-grade GPU such as a
Quadro or Firepro.

-Geoff

> 
> cheers,
>   Gerd

Re: RFC: New device for zero-copy VM memory access

[Gerd Hoffmann]

On Mon, Nov 04, 2019 at 09:31:47PM +1100, geoff@hostfission.com wrote:
> 
> On 2019-11-04 21:26, Gerd Hoffmann wrote:
> > Hi,
> > 
> > > This new device, currently named `introspection` (which needs a more
> > > suitable name, porthole perhaps?), provides a means of translating
> > > guest physical addresses to host virtual addresses, and finally to the
> > > host offsets in RAM for file-backed memory guests. It does this by
> > > means of a simple protocol over a unix socket (chardev) which is
> > > supplied the appropriate fd for the VM's system RAM. The guest (in
> > > this case, Windows), when presented with the address of a userspace
> > > buffer and size, will mlock the appropriate pages into RAM and pass
> > > guest physical addresses to the virtual device.
> > 
> > So, if I understand things correctly, the workflow looks like this:
> > 
> >   (1) guest allocates buffers, using guest ram.
> >   (2) guest uses these buffers as render target for the gpu
> > (pci-assigned I guess?).
> >   (3) guest passes guest physical address to qemu (via porthole device).
> >   (4) qemu translates gpa into file offset and passes offsets to
> >       the client application.
> >   (5) client application maps all guest ram, then uses the offsets from
> >       qemu to find the buffers.  Then goes displaying these buffers I
> > guess.
> > 
> > Correct?
> 
> Correct, however step 5 might be a proxy to copy the buffers into another
> porthole device in a second VM allowing VM->VM transfers.
> 
> > Performance aside for now, is it an option for your use case to simply
> > use both an emulated display device and the assigned gpu, then configure
> > screen mirroring inside the guest to get the guest display scanned out
> > to the host?
> 
> Unfortunately no, NVidia and AMD devices do not support mirroring their
> outputs to a separate GPU unless it's a professional-grade GPU such as a
> Quadro or Firepro.

Ok.

We had discussions about buffer sharing between host and guest before.


One possible approach would be to use virtio-gpu for that, because it
already has the buffer management bits (and alot of other stuff not
needed for this use case).  There is no support for shared buffers right
now (atm there are guest-side and host-side buffers and commands for
data transfers).  Shared buffer support is being worked on though, this
(and other changes) are here (look for udmabuf commits):
    https://git.kraxel.org/cgit/qemu/log/?h=sirius/virtio-gpu-memory-v2

Note: udmabuf is a linux driver which allows to create dma-bufs from
guest memory pages.  These dma-bufs can be passed to other applications
using unix file descriptor passing, that way we could pass the buffers
from qemu to the client application.  The client can map them, or even
pass them on to the (host) gpu driver for display.

Requiring a full-blown display device just for buffer sharing might be a
bit of overkill though.  Another obvious drawback for your specific use
case is that there are no virtio-gpu windows drivers yet.


Another approach would be to design a new virtio device just for buffer
sharing.  Would probably be pretty simple, with one guest -> host queue
for sending buffer management commands.  Each buffer would be a list of
pages or (guest physical) address ranges.  Adding some properties would
probably be very useful too, so you can attach some metadata to the
buffers (i.e. id=42, application=porthole, width=1024, height=768,
stride=4094, format=XR24, you get the idea ...).

On the host side we could again have qemu using the udmabuf driver to
create dma-bufs and hand them out to other applications so they can use
the buffers.  Alternatively use the vhost-user approach outlined
elsewhere in this thread.  Having qemu manage the buffers makes client
reconnects and multiple parallel applications alot easier though.

cheers,
  Gerd