[RFC] remoting KMS

[RFC] remoting KMS

[Thomas Hellstrom]

Hi!

I'm currently working on a remoting KMS backend, and now I thought it 
would be a good time to get some feedback on a very rough design:

The idea is that we want to be get enough information on the backend 
side of KMS to be able to remote the display system over VNC or 
something similar. Initially I'd plug in an open-source VNC server as 
the "reference" user-space.

The solution should be general enough to plug into any driver, including 
VKMS, either mirroring an existing crtc/connector or adding a virtual 
one. The driver would typically only have to implement an efficient 
readback command, optionally with a diff-map or damage rect that tracks 
changed content for the VNC server to be able to do its work efficiently.

This would all be forwarded to a user-space interface and protocol with 
a slightly more elaborate user-space event-mechanism than the one we 
have in drm now. In particular we want to be able to read partial 
events, and resume event reading after a partial event. We would also 
want to be able to send commands either through an ioctl() or using 
write(). Current test code is using write().

Since the VNC server in our application typically will be system-wide, 
we need to resurrect a control-node functionality. I was trying to use 
sysfs attributes for this, but while sysfs supports rudimentary poll 
functionality. It's a bit too rudimentary and tracking client lifetime 
becomes racy. Perhaps control nodes renamed to "remoting" nodes or 
something similar. It might be that masters want to attach a VNC server 
to their current display and in that case we probably would want to 
expand primary nodes with this functionality as well.

Now to the open-source  problem. As mentioned, the reference user-space 
will be a solution based on libvncserver. In fact there will be a small 
user-space interface library that libvncserver can interact with, 
similar to libdrm. However, I envision in the long run VMWare would be 
interested to use other remoting protocols with closed source protocol 
encoders, using the open source reference library. In theory, this 
solution would also allow for simple (open- or closed source) user-space 
display drivers, very similar to DisplayLink's evdi solution, even if 
that's not a use-case VMWare is currently interested in.

Thoughts and comments appreciated.

/Thomas



_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC] remoting KMS

[Daniel Vetter]

On Thu, May 17, 2018 at 8:23 PM, Thomas Hellstrom <thomas@shipmail.org> wrote:
> Hi!
>
> I'm currently working on a remoting KMS backend, and now I thought it would
> be a good time to get some feedback on a very rough design:
>
> The idea is that we want to be get enough information on the backend side of
> KMS to be able to remote the display system over VNC or something similar.
> Initially I'd plug in an open-source VNC server as the "reference"
> user-space.
>
> The solution should be general enough to plug into any driver, including
> VKMS, either mirroring an existing crtc/connector or adding a virtual one.
> The driver would typically only have to implement an efficient readback
> command, optionally with a diff-map or damage rect that tracks changed
> content for the VNC server to be able to do its work efficiently.
>
> This would all be forwarded to a user-space interface and protocol with a
> slightly more elaborate user-space event-mechanism than the one we have in
> drm now. In particular we want to be able to read partial events, and resume
> event reading after a partial event. We would also want to be able to send
> commands either through an ioctl() or using write(). Current test code is
> using write().
>
> Since the VNC server in our application typically will be system-wide, we
> need to resurrect a control-node functionality. I was trying to use sysfs
> attributes for this, but while sysfs supports rudimentary poll
> functionality. It's a bit too rudimentary and tracking client lifetime
> becomes racy. Perhaps control nodes renamed to "remoting" nodes or something
> similar. It might be that masters want to attach a VNC server to their
> current display and in that case we probably would want to expand primary
> nodes with this functionality as well.
>
> Now to the open-source  problem. As mentioned, the reference user-space will
> be a solution based on libvncserver. In fact there will be a small
> user-space interface library that libvncserver can interact with, similar to
> libdrm. However, I envision in the long run VMWare would be interested to
> use other remoting protocols with closed source protocol encoders, using the
> open source reference library. In theory, this solution would also allow for
> simple (open- or closed source) user-space display drivers, very similar to
> DisplayLink's evdi solution, even if that's not a use-case VMWare is
> currently interested in.
>
> Thoughts and comments appreciated.

Just two very quick comments:

- For the uapi - can't we just use v4l for the output side? That gives
you buffer handoff, syncing (v4l is gaining dma_fence support) and
pretty much everything you might want. And we've talked about building
kms->v4l pipelines in a bunch of other use-cases already (like feeding
into video encoders for widi and stuff like that). Would give you the
userspace for free, only thing we really need is to (maybe, not sure
you'd even need that) expose what kms output corresponds to what v4l
pipeline.

- You say "The solution should be general enough to plug into any
driver" - is the idea that any hw driver would automatically support
this? Given that prime-like setups are getting ever more common
(mobile is essentially all prime with display and offloaded rendering)
I think it'd be cleanest if we don't tie that virtual output to other
drivers, but bind things together using buffer sharing/dma-fences and
prime. Adding v4l output to vkms is actually something we've already
discussed a bit on irc, but with the use-case that you could just
watch what your desktop/window manager is doing under a specific
simulated setup (e.g. pll sharing constraints, or special plane
features, or whatever other specific thing you want to test your
compositor against but isn't supported on all physical hw/drivers).

Cheers, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC] remoting KMS

[Thomas Hellstrom]

On 05/17/2018 09:20 PM, Daniel Vetter wrote:
> On Thu, May 17, 2018 at 8:23 PM, Thomas Hellstrom <thomas@shipmail.org> wrote:
>> Hi!
>>
>> I'm currently working on a remoting KMS backend, and now I thought it would
>> be a good time to get some feedback on a very rough design:
>>
>> The idea is that we want to be get enough information on the backend side of
>> KMS to be able to remote the display system over VNC or something similar.
>> Initially I'd plug in an open-source VNC server as the "reference"
>> user-space.
>>
>> The solution should be general enough to plug into any driver, including
>> VKMS, either mirroring an existing crtc/connector or adding a virtual one.
>> The driver would typically only have to implement an efficient readback
>> command, optionally with a diff-map or damage rect that tracks changed
>> content for the VNC server to be able to do its work efficiently.
>>
>> This would all be forwarded to a user-space interface and protocol with a
>> slightly more elaborate user-space event-mechanism than the one we have in
>> drm now. In particular we want to be able to read partial events, and resume
>> event reading after a partial event. We would also want to be able to send
>> commands either through an ioctl() or using write(). Current test code is
>> using write().
>>
>> Since the VNC server in our application typically will be system-wide, we
>> need to resurrect a control-node functionality. I was trying to use sysfs
>> attributes for this, but while sysfs supports rudimentary poll
>> functionality. It's a bit too rudimentary and tracking client lifetime
>> becomes racy. Perhaps control nodes renamed to "remoting" nodes or something
>> similar. It might be that masters want to attach a VNC server to their
>> current display and in that case we probably would want to expand primary
>> nodes with this functionality as well.
>>
>> Now to the open-source  problem. As mentioned, the reference user-space will
>> be a solution based on libvncserver. In fact there will be a small
>> user-space interface library that libvncserver can interact with, similar to
>> libdrm. However, I envision in the long run VMWare would be interested to
>> use other remoting protocols with closed source protocol encoders, using the
>> open source reference library. In theory, this solution would also allow for
>> simple (open- or closed source) user-space display drivers, very similar to
>> DisplayLink's evdi solution, even if that's not a use-case VMWare is
>> currently interested in.
>>
>> Thoughts and comments appreciated.
> Just two very quick comments:
>
> - For the uapi - can't we just use v4l for the output side? That gives
> you buffer handoff, syncing (v4l is gaining dma_fence support) and
> pretty much everything you might want. And we've talked about building
> kms->v4l pipelines in a bunch of other use-cases already (like feeding
> into video encoders for widi and stuff like that). Would give you the
> userspace for free, only thing we really need is to (maybe, not sure
> you'd even need that) expose what kms output corresponds to what v4l
> pipeline.

Hi!
Thanks for the comments.

I'll take a look whether this is possible to accomplish with v4l. Hadn't 
thought of that. However, I doubt it will be possible to support all 
things we need to support. Initially we started looking at this in the 
context of readback connectors but buffer management and readbacks are 
only a subset of what we want to do. We also need to pipe a subset of 
the (atomic) state and its updates, the handling of which actually 
becomes the vast major part of the code.
We also need to feed data back, similar to the vmwgfx update_layout 
ioctl, that sets suggested x and y position, preferred mode, and, on 
readback, obtain damage information. Perhaps it's possible to construct 
some kind of side-channel.

>
> - You say "The solution should be general enough to plug into any
> driver" - is the idea that any hw driver would automatically support
> this?
No, the initial idea is that any hw driver that wants to could use 
helpers and a readback implementation to support this

> Given that prime-like setups are getting ever more common
> (mobile is essentially all prime with display and offloaded rendering)
> I think it'd be cleanest if we don't tie that virtual output to other
> drivers, but bind things together using buffer sharing/dma-fences and
> prime.

We do need a front end, though, I assume you're suggestion is that we 
add support to vkms only. That would work for the main VMware use-case, 
but not for a general use-case where a user would want to share his 
existing screen using VNC.

>   Adding v4l output to vkms is actually something we've already
> discussed a bit on irc, but with the use-case that you could just
> watch what your desktop/window manager is doing under a specific
> simulated setup (e.g. pll sharing constraints, or special plane
> features, or whatever other specific thing you want to test your
> compositor against but isn't supported on all physical hw/drivers).
>
> Cheers, Daniel

Thanks,

Thomas


_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC] remoting KMS

[Daniel Vetter]

On Thu, May 17, 2018 at 11:18 PM, Thomas Hellstrom <thomas@shipmail.org> wrote:
>
> On 05/17/2018 09:20 PM, Daniel Vetter wrote:
>>
>> On Thu, May 17, 2018 at 8:23 PM, Thomas Hellstrom <thomas@shipmail.org>
>> wrote:
>>>
>>> Hi!
>>>
>>> I'm currently working on a remoting KMS backend, and now I thought it
>>> would
>>> be a good time to get some feedback on a very rough design:
>>>
>>> The idea is that we want to be get enough information on the backend side
>>> of
>>> KMS to be able to remote the display system over VNC or something
>>> similar.
>>> Initially I'd plug in an open-source VNC server as the "reference"
>>> user-space.
>>>
>>> The solution should be general enough to plug into any driver, including
>>> VKMS, either mirroring an existing crtc/connector or adding a virtual
>>> one.
>>> The driver would typically only have to implement an efficient readback
>>> command, optionally with a diff-map or damage rect that tracks changed
>>> content for the VNC server to be able to do its work efficiently.
>>>
>>> This would all be forwarded to a user-space interface and protocol with a
>>> slightly more elaborate user-space event-mechanism than the one we have
>>> in
>>> drm now. In particular we want to be able to read partial events, and
>>> resume
>>> event reading after a partial event. We would also want to be able to
>>> send
>>> commands either through an ioctl() or using write(). Current test code is
>>> using write().
>>>
>>> Since the VNC server in our application typically will be system-wide, we
>>> need to resurrect a control-node functionality. I was trying to use sysfs
>>> attributes for this, but while sysfs supports rudimentary poll
>>> functionality. It's a bit too rudimentary and tracking client lifetime
>>> becomes racy. Perhaps control nodes renamed to "remoting" nodes or
>>> something
>>> similar. It might be that masters want to attach a VNC server to their
>>> current display and in that case we probably would want to expand primary
>>> nodes with this functionality as well.
>>>
>>> Now to the open-source  problem. As mentioned, the reference user-space
>>> will
>>> be a solution based on libvncserver. In fact there will be a small
>>> user-space interface library that libvncserver can interact with, similar
>>> to
>>> libdrm. However, I envision in the long run VMWare would be interested to
>>> use other remoting protocols with closed source protocol encoders, using
>>> the
>>> open source reference library. In theory, this solution would also allow
>>> for
>>> simple (open- or closed source) user-space display drivers, very similar
>>> to
>>> DisplayLink's evdi solution, even if that's not a use-case VMWare is
>>> currently interested in.
>>>
>>> Thoughts and comments appreciated.
>>
>> Just two very quick comments:
>>
>> - For the uapi - can't we just use v4l for the output side? That gives
>> you buffer handoff, syncing (v4l is gaining dma_fence support) and
>> pretty much everything you might want. And we've talked about building
>> kms->v4l pipelines in a bunch of other use-cases already (like feeding
>> into video encoders for widi and stuff like that). Would give you the
>> userspace for free, only thing we really need is to (maybe, not sure
>> you'd even need that) expose what kms output corresponds to what v4l
>> pipeline.
>
>
> Hi!
> Thanks for the comments.
>
> I'll take a look whether this is possible to accomplish with v4l. Hadn't
> thought of that. However, I doubt it will be possible to support all things
> we need to support. Initially we started looking at this in the context of
> readback connectors but buffer management and readbacks are only a subset of
> what we want to do. We also need to pipe a subset of the (atomic) state and
> its updates, the handling of which actually becomes the vast major part of
> the code.
> We also need to feed data back, similar to the vmwgfx update_layout ioctl,
> that sets suggested x and y position, preferred mode, and, on readback,
> obtain damage information. Perhaps it's possible to construct some kind of
> side-channel.

Ok, that's a slightly different use-case than what I thought off,
sounds like you want to keep the special cursor meaning for
virtualization. And that wouldn't go through v4l I think.

>> - You say "The solution should be general enough to plug into any
>> driver" - is the idea that any hw driver would automatically support
>> this?
>
> No, the initial idea is that any hw driver that wants to could use helpers
> and a readback implementation to support this

Is the helper idea just what you have in mind for vmwgfx, or does your
use-case require that you patch this to all kms drivers (kinda like
all kms drivers have fbcon)? +/- a bunch you'll probably not care
about. If you want this I guess you need to explain a bit more what's
the intended use-case, since it seems to be more than just piping any
random compositor than can run on kms into something like a vnc server
for forwarding.

>> Given that prime-like setups are getting ever more common
>> (mobile is essentially all prime with display and offloaded rendering)
>> I think it'd be cleanest if we don't tie that virtual output to other
>> drivers, but bind things together using buffer sharing/dma-fences and
>> prime.
>
>
> We do need a front end, though, I assume you're suggestion is that we add
> support to vkms only. That would work for the main VMware use-case, but not
> for a general use-case where a user would want to share his existing screen
> using VNC.

Fire up vkms, set up as a cloning output? Of course that needs minimal
cooperation of the compositor, so depending upon what you're trying to
do it's not good enough.
-Daniel

>>   Adding v4l output to vkms is actually something we've already
>> discussed a bit on irc, but with the use-case that you could just
>> watch what your desktop/window manager is doing under a specific
>> simulated setup (e.g. pll sharing constraints, or special plane
>> features, or whatever other specific thing you want to test your
>> compositor against but isn't supported on all physical hw/drivers).
>>
>> Cheers, Daniel
>
>
> Thanks,
>
> Thomas
>
>



-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC] remoting KMS

[Thomas Hellstrom]

On 05/17/2018 11:40 PM, Daniel Vetter wrote:
> On Thu, May 17, 2018 at 11:18 PM, Thomas Hellstrom <thomas@shipmail.org> wrote:
>> On 05/17/2018 09:20 PM, Daniel Vetter wrote:
>>> On Thu, May 17, 2018 at 8:23 PM, Thomas Hellstrom <thomas@shipmail.org>
>>> wrote:
>>>> Hi!
>>>>
>>>> I'm currently working on a remoting KMS backend, and now I thought it
>>>> would
>>>> be a good time to get some feedback on a very rough design:
>>>>
>>>> The idea is that we want to be get enough information on the backend side
>>>> of
>>>> KMS to be able to remote the display system over VNC or something
>>>> similar.
>>>> Initially I'd plug in an open-source VNC server as the "reference"
>>>> user-space.
>>>>
>>>> The solution should be general enough to plug into any driver, including
>>>> VKMS, either mirroring an existing crtc/connector or adding a virtual
>>>> one.
>>>> The driver would typically only have to implement an efficient readback
>>>> command, optionally with a diff-map or damage rect that tracks changed
>>>> content for the VNC server to be able to do its work efficiently.
>>>>
>>>> This would all be forwarded to a user-space interface and protocol with a
>>>> slightly more elaborate user-space event-mechanism than the one we have
>>>> in
>>>> drm now. In particular we want to be able to read partial events, and
>>>> resume
>>>> event reading after a partial event. We would also want to be able to
>>>> send
>>>> commands either through an ioctl() or using write(). Current test code is
>>>> using write().
>>>>
>>>> Since the VNC server in our application typically will be system-wide, we
>>>> need to resurrect a control-node functionality. I was trying to use sysfs
>>>> attributes for this, but while sysfs supports rudimentary poll
>>>> functionality. It's a bit too rudimentary and tracking client lifetime
>>>> becomes racy. Perhaps control nodes renamed to "remoting" nodes or
>>>> something
>>>> similar. It might be that masters want to attach a VNC server to their
>>>> current display and in that case we probably would want to expand primary
>>>> nodes with this functionality as well.
>>>>
>>>> Now to the open-source  problem. As mentioned, the reference user-space
>>>> will
>>>> be a solution based on libvncserver. In fact there will be a small
>>>> user-space interface library that libvncserver can interact with, similar
>>>> to
>>>> libdrm. However, I envision in the long run VMWare would be interested to
>>>> use other remoting protocols with closed source protocol encoders, using
>>>> the
>>>> open source reference library. In theory, this solution would also allow
>>>> for
>>>> simple (open- or closed source) user-space display drivers, very similar
>>>> to
>>>> DisplayLink's evdi solution, even if that's not a use-case VMWare is
>>>> currently interested in.
>>>>
>>>> Thoughts and comments appreciated.
>>> Just two very quick comments:
>>>
>>> - For the uapi - can't we just use v4l for the output side? That gives
>>> you buffer handoff, syncing (v4l is gaining dma_fence support) and
>>> pretty much everything you might want. And we've talked about building
>>> kms->v4l pipelines in a bunch of other use-cases already (like feeding
>>> into video encoders for widi and stuff like that). Would give you the
>>> userspace for free, only thing we really need is to (maybe, not sure
>>> you'd even need that) expose what kms output corresponds to what v4l
>>> pipeline.
>>
>> Hi!
>> Thanks for the comments.
>>
>> I'll take a look whether this is possible to accomplish with v4l. Hadn't
>> thought of that. However, I doubt it will be possible to support all things
>> we need to support. Initially we started looking at this in the context of
>> readback connectors but buffer management and readbacks are only a subset of
>> what we want to do. We also need to pipe a subset of the (atomic) state and
>> its updates, the handling of which actually becomes the vast major part of
>> the code.
>> We also need to feed data back, similar to the vmwgfx update_layout ioctl,
>> that sets suggested x and y position, preferred mode, and, on readback,
>> obtain damage information. Perhaps it's possible to construct some kind of
>> side-channel.
> Ok, that's a slightly different use-case than what I thought off,
> sounds like you want to keep the special cursor meaning for
> virtualization. And that wouldn't go through v4l I think.
>
>>> - You say "The solution should be general enough to plug into any
>>> driver" - is the idea that any hw driver would automatically support
>>> this?
>> No, the initial idea is that any hw driver that wants to could use helpers
>> and a readback implementation to support this
> Is the helper idea just what you have in mind for vmwgfx, or does your
> use-case require that you patch this to all kms drivers (kinda like
> all kms drivers have fbcon)? +/- a bunch you'll probably not care
> about. If you want this I guess you need to explain a bit more what's
> the intended use-case, since it seems to be more than just piping any
> random compositor than can run on kms into something like a vnc server
> for forwarding.

Basically the helper idea is for vmwgfx. Currently I've hooked it up in 
atomic_flush(). It takes the atomic state, updates the remoting state 
(which is a subset) and sends changes to user-space. It also has a 
callback for readback which we need to implement in vmwgfx.


VMWare's use-cases are

1) Desktop remoting. The remote site forwards layout changes 
(resolution, multi-monitor position), keyboard clicks and mouse movement 
and gets back data encoded using VNC, h.264 or whatever protocol is 
suitable. There are numerous competing virtualization products that 
accomplish this today, VMWare's solution talks to the X server. The X 
server is disappearing. This is the primary need, and one would 
anticipate advanced use cases  with multiple desktops like 4x4K on a 
single server. Here I think a vkms/prime solution would work fine, as 
long as the fast readback problem with damage region can be solved. 
Encoding and sending redundant data from, for example, a full page-flip 
is a no-go. Ideally we'd need to know the damaged region even before 
reading back data over the PCI bus, since a competent server with many 
VMs and abundant processor power but GPUs sitting on a single PCI bus 
would rather easily saturate that PCI bus with readbacks...

2) Automated testing. With this solution one can easily verify the 
desktop content and layout and make sure that nothing was broken on its 
way through the driver stack.

And finally for my personal use I'd love to be able to open a VNC 
connection to the RPi on my boat to see what it is currently displaying, 
and what's currently displaying would not be vkms but the native display 
driver, which would require that the native display driver is 
remoting-enabled using the helpers + the readback solution.

In terms of the X solutions available today, vkms would correspond to 
Xvnc with prime availability (a dedicated X server) The RPi use-case 
would correspond to x11vnc (an X client that connects to an existing 
server).

Thanks,
Thomas

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel