Atomic KMS API lacks the ability to set cursor hot-spot coordinates

Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Hans de Goede]

Hi All,

I'm not sure if $subject was a conscious design decision, or an oversight,
but that does not really matter.

ATM the Atomic KMS API lacks the ability to set cursor hot-spot
coordinates. Mutter (and Weston) have tried to emulate this by shifting
the coordinates for where to draw the cursor by the hotspot-coordinates
and always using 0x0 for the hotspot.

But this breaks the so called "seamless mouse mode" for virtual-machines
and there really is no way to fix this but to allow passing the proper
hotspot coordinates to the virtual gfx-card.

Seamless-mode consists of 2 parts:

1) Letting the VM-viewer window-system draw the cursor as it normally
would draw it.

2) Giving absolute coordinates of the mouse to the VM by e.g. emulating
an USB drawing tablet. These coordinates come from translating the
coordinates where the VM-viewer window-system is drawing the cursor
to an absolute position using the top left corner of the view as 0x0
and the bottom right corner as max-abs-x,max-abs-y.

2) Means that any coordinates the window-system inside the VM passes to
the VM's gfx-card for where to draw the cursor are basically totally
ignored to avoid lag / flicker (and to not have to grab the cursor /
confine it to the VM-viewer window and to not have to warp the
pointer).

This means that the offset added to the coordinates by e.g. mutter to
emulate the hotspot are ignored. For Seamless mouse mode to keep working
properly the window-system inside the VM need to pass the VM's gfx-card
the correct hotspot when setting the cursor. Which currently is not
possible when restricting oneself to the atomic APIs.

Also see: https://gitlab.gnome.org/GNOME/mutter/issues/1094
Where this is currently being tracked from the mutter side. Mutter
internally has both atomic and legacy paths. The plan for now is to
push the hotspot-emulation by shifting coordinates thing into the
atomic path, fixing seamless mouse mode when running in legacy mode,
combined with blacklisting vboxvideo, vmwgfx, qxl and cirrus from
using atomic mode.

This is of course a workaround, eventually we would like to see
the atomic API extended to allow passing the cursor hot spot.

I'm not really familiar enough with the atomic API to come up with
an API design for this, but if there are suggestions on how this
should look like from the uAPI side then I can take a shot at
implementing this (and hooking it up in mutter's atomic code
paths to test it).

Regards,

Hans



p.s.

Before people start discussing how the VM / VM-viewer is broken here and
the VM needs to be fixed. Seamless mouse mode exists for at least a
decade and has worked fine during this entire decade. It also works
fine when using the legacy (non atomic) DRM_IOCTL_MODE_CURSOR2 ioctl;

Also this problem reproduces with 2 completely independent VM code-bases,
it has been seen on both qemu-kvm VMs and on VirtualBox VMs and I would
not be surprised if other hypervisors are also affected.

And on the API consumer side this problem has been triggered by both
mutter and Weston.

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Simon Ser]

Hi,

> 1) Letting the VM-viewer window-system draw the cursor as it normally
> would draw it.

Why is this important? Can't the VM viewer hide the cursor and use a
sub-surface to manually draw the cursor plane configured by the guest?

This would also allow the compositor running inside the VM to correctly
have control over the cursor position, which is necessary for pointer
constraints.

Thanks,

Simon
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Hans de Goede]

Hi,

On 3/18/20 3:28 PM, Hans de Goede wrote:
> Hi All,
> 
> I'm not sure if $subject was a conscious design decision, or an oversight,
> but that does not really matter.
> 
> ATM the Atomic KMS API lacks the ability to set cursor hot-spot
> coordinates. Mutter (and Weston) have tried to emulate this by shifting
> the coordinates for where to draw the cursor by the hotspot-coordinates
> and always using 0x0 for the hotspot.
> 
> But this breaks the so called "seamless mouse mode" for virtual-machines
> and there really is no way to fix this but to allow passing the proper
> hotspot coordinates to the virtual gfx-card.
> 
> Seamless-mode consists of 2 parts:
> 
> 1) Letting the VM-viewer window-system draw the cursor as it normally
> would draw it.
> 
> 2) Giving absolute coordinates of the mouse to the VM by e.g. emulating
> an USB drawing tablet. These coordinates come from translating the
> coordinates where the VM-viewer window-system is drawing the cursor
> to an absolute position using the top left corner of the view as 0x0
> and the bottom right corner as max-abs-x,max-abs-y.
> 
> 2) Means that any coordinates the window-system inside the VM passes to

This should be: "1) Means that ..."

> the VM's gfx-card for where to draw the cursor are basically totally
> ignored to avoid lag / flicker (and to not have to grab the cursor /
> confine it to the VM-viewer window and to not have to warp the
> pointer).
> 
> This means that the offset added to the coordinates by e.g. mutter to
> emulate the hotspot are ignored. For Seamless mouse mode to keep working
> properly the window-system inside the VM need to pass the VM's gfx-card
> the correct hotspot when setting the cursor. Which currently is not
> possible when restricting oneself to the atomic APIs.
> 
> Also see: https://gitlab.gnome.org/GNOME/mutter/issues/1094
> Where this is currently being tracked from the mutter side. Mutter
> internally has both atomic and legacy paths. The plan for now is to
> push the hotspot-emulation by shifting coordinates thing into the
> atomic path, fixing seamless mouse mode when running in legacy mode,
> combined with blacklisting vboxvideo, vmwgfx, qxl and cirrus from
> using atomic mode.
> 
> This is of course a workaround, eventually we would like to see
> the atomic API extended to allow passing the cursor hot spot.
> 
> I'm not really familiar enough with the atomic API to come up with
> an API design for this, but if there are suggestions on how this
> should look like from the uAPI side then I can take a shot at
> implementing this (and hooking it up in mutter's atomic code
> paths to test it).

Correction, I misunderstood the mutter devs that mutter already
has support for atomic, this support is in the works, but not yet
there yet. The internal mutter API has been reworked to closer
resemble the atomic APIs and that introduced the hotspot emulation
and broke seamless mouse mode in VMs.

Mutter does eventually want to switch to using the atomic APIs
and then this will become a real problem. This does mean that
testing any UAPI extension we come up with for this will be
harder then I thought.

Regards,

Hans


> p.s.
> 
> Before people start discussing how the VM / VM-viewer is broken here and
> the VM needs to be fixed. Seamless mouse mode exists for at least a
> decade and has worked fine during this entire decade. It also works
> fine when using the legacy (non atomic) DRM_IOCTL_MODE_CURSOR2 ioctl;
> 
> Also this problem reproduces with 2 completely independent VM code-bases,
> it has been seen on both qemu-kvm VMs and on VirtualBox VMs and I would
> not be surprised if other hypervisors are also affected.
> 
> And on the API consumer side this problem has been triggered by both
> mutter and Weston.

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Hans de Goede]

Hi,

On 3/18/20 3:38 PM, Simon Ser wrote:
> Hi,
> 
>> 1) Letting the VM-viewer window-system draw the cursor as it normally
>> would draw it.
> 
> Why is this important? Can't the VM viewer hide the cursor and use a
> sub-surface to manually draw the cursor plane configured by the guest?

Because then moving the cursor as seen by the user requires a round trip
through the VM and that adds latency, esp. when the VM viewer is viewing
a VM which is running somewhere else over the network.

Also note that a subsurface is a Wayland specific solution, where as
the VM-viewer may be running on X11, Windows, Mac OS, etc.

> This would also allow the compositor running inside the VM to correctly
> have control over the cursor position, which is necessary for pointer
> constraints.

Vms basically have 2 mouse modes:

1) Seamless, this works well for all apps which don't do weird things
with the cursor. This is what 99% of users want

2) Grab/confine the mouse on the first click inside the VM-viewer window
and constantly warp it to the center so that it can move "endlessly"
combined with drawing the VM's mouse cursor as a software sprite.

Combined with a special key combo to release the cursor and allow it
to leave the VM window in case the user wants to interact with anything
else on their desktop. AKA the "this user experience sucks" mode which
sometimes is necessary for guests which don't support absolute input
coordinates, or for special use cases.

Mode 2. can be used in case of apps inside the guest which want need
to constrain the pointer to stay inside a part-of the vm-viewer window,
note that the most prominent example of such apps are VM-viewer's
themselves and the whole purpose of seamless mode is to not need this
less then ideal user experience mode.

Anyways as I mentioned in the p.s. to my original mail already, this
is exactly NOT the kind of feedback I'm looking for. Seamless mode
exists, it has done so for at least a decade, probably a lot longer.

It works everywhere, across multiple platforms and hypervisors,
except with the KMS atomic API. The need to set hotspot coordinates
is not something which is up to discussion from my pov. What is up
for discussion is how to extend the KMS atomic API to allow this.

Regards,

Hans

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Daniel Vetter]

On Wed, Mar 18, 2020 at 3:29 PM Hans de Goede <hdegoede@redhat.com> wrote:
>
> Hi All,
>
> I'm not sure if $subject was a conscious design decision, or an oversight,
> but that does not really matter.
>
> ATM the Atomic KMS API lacks the ability to set cursor hot-spot
> coordinates. Mutter (and Weston) have tried to emulate this by shifting
> the coordinates for where to draw the cursor by the hotspot-coordinates
> and always using 0x0 for the hotspot.
>
> But this breaks the so called "seamless mouse mode" for virtual-machines
> and there really is no way to fix this but to allow passing the proper
> hotspot coordinates to the virtual gfx-card.

Fully aware, we simply never had userspace for this. So
- type weston patch
- type kernel patch (iirc this might be some work of moving the
hotspot coordinates from struct drm_plane to drm_plane_state)
- wire up atomic property (it'd just do "CURSOR_HOT_X/Y" and be done
with it, only attached to cursor planes that support it ofc)
- I think all drivers supporting vm cursor hotspot are now atomic, so
would be good to garbage-collect the old legacy infrastructure
completely. legacy cursor ioctl will remap to the atomic call
transparently like we do now already for everything else
- type igt
- get it all reviewed and merged.

0 design reasons for omitting this, simply Not Yet Done (tm).

Cheers, Daniel

> Seamless-mode consists of 2 parts:
>
> 1) Letting the VM-viewer window-system draw the cursor as it normally
> would draw it.
>
> 2) Giving absolute coordinates of the mouse to the VM by e.g. emulating
> an USB drawing tablet. These coordinates come from translating the
> coordinates where the VM-viewer window-system is drawing the cursor
> to an absolute position using the top left corner of the view as 0x0
> and the bottom right corner as max-abs-x,max-abs-y.
>
> 2) Means that any coordinates the window-system inside the VM passes to
> the VM's gfx-card for where to draw the cursor are basically totally
> ignored to avoid lag / flicker (and to not have to grab the cursor /
> confine it to the VM-viewer window and to not have to warp the
> pointer).
>
> This means that the offset added to the coordinates by e.g. mutter to
> emulate the hotspot are ignored. For Seamless mouse mode to keep working
> properly the window-system inside the VM need to pass the VM's gfx-card
> the correct hotspot when setting the cursor. Which currently is not
> possible when restricting oneself to the atomic APIs.
>
> Also see: https://gitlab.gnome.org/GNOME/mutter/issues/1094
> Where this is currently being tracked from the mutter side. Mutter
> internally has both atomic and legacy paths. The plan for now is to
> push the hotspot-emulation by shifting coordinates thing into the
> atomic path, fixing seamless mouse mode when running in legacy mode,
> combined with blacklisting vboxvideo, vmwgfx, qxl and cirrus from
> using atomic mode.
>
> This is of course a workaround, eventually we would like to see
> the atomic API extended to allow passing the cursor hot spot.
>
> I'm not really familiar enough with the atomic API to come up with
> an API design for this, but if there are suggestions on how this
> should look like from the uAPI side then I can take a shot at
> implementing this (and hooking it up in mutter's atomic code
> paths to test it).
>
> Regards,
>
> Hans
>
>
>
> p.s.
>
> Before people start discussing how the VM / VM-viewer is broken here and
> the VM needs to be fixed. Seamless mouse mode exists for at least a
> decade and has worked fine during this entire decade. It also works
> fine when using the legacy (non atomic) DRM_IOCTL_MODE_CURSOR2 ioctl;
>
> Also this problem reproduces with 2 completely independent VM code-bases,
> it has been seen on both qemu-kvm VMs and on VirtualBox VMs and I would
> not be surprised if other hypervisors are also affected.
>
> And on the API consumer side this problem has been triggered by both
> mutter and Weston.
>
> _______________________________________________
> dri-devel mailing list
> dri-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel



-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Simon Ser]

> Hi,
>
> On 3/18/20 3:38 PM, Simon Ser wrote:
> > Hi,
> >
> >> 1) Letting the VM-viewer window-system draw the cursor as it normally
> >> would draw it.
> >
> > Why is this important? Can't the VM viewer hide the cursor and use a
> > sub-surface to manually draw the cursor plane configured by the guest?
>
> Because then moving the cursor as seen by the user requires a round trip
> through the VM and that adds latency, esp. when the VM viewer is viewing
> a VM which is running somewhere else over the network.

The video output has latency anyway. Using the host cursor will make
the primary plane and cursor plane desynchronized.

> Also note that a subsurface is a Wayland specific solution, where as
> the VM-viewer may be running on X11, Windows, Mac OS, etc.

I'm sure other window systems have similar solutions. You could always
blend the cursor yourself.

> > This would also allow the compositor running inside the VM to correctly
> > have control over the cursor position, which is necessary for pointer
> > constraints.
>
> Vms basically have 2 mouse modes:
>
> 1) Seamless, this works well for all apps which don't do weird things
> with the cursor. This is what 99% of users want
>
> 2) Grab/confine the mouse on the first click inside the VM-viewer window
> and constantly warp it to the center so that it can move "endlessly"
> combined with drawing the VM's mouse cursor as a software sprite.
>
> Combined with a special key combo to release the cursor and allow it
> to leave the VM window in case the user wants to interact with anything
> else on their desktop. AKA the "this user experience sucks" mode which
> sometimes is necessary for guests which don't support absolute input
> coordinates, or for special use cases.
>
> Mode 2. can be used in case of apps inside the guest which want need
> to constrain the pointer to stay inside a part-of the vm-viewer window,
> note that the most prominent example of such apps are VM-viewer's
> themselves and the whole purpose of seamless mode is to not need this
> less then ideal user experience mode.

If you don't care about synchronization and breaking pointer constraints
in the guest, yes a new KMS plane property will be required. This sure
sounds like abusing the KMS interface though.

> Anyways as I mentioned in the p.s. to my original mail already, this
> is exactly NOT the kind of feedback I'm looking for. Seamless mode
> exists, it has done so for at least a decade, probably a lot longer.
>
> It works everywhere, across multiple platforms and hypervisors,
> except with the KMS atomic API. The need to set hotspot coordinates
> is not something which is up to discussion from my pov. What is up
> for discussion is how to extend the KMS atomic API to allow this.

That's not how it works.

I'm sorry to say that, but I don't think asking a project to support a
feature just because you want that feature is a good mind set. You'll
need to convince people maintaining the project that adding the feature
is a good idea whether you like it or not.

A new feature is always up for discussion. Atomic KMS is missing the
feature, and this can't be seen as a regression, because it never had
the feature.
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Hans de Goede]

Hi,

On 3/18/20 4:09 PM, Daniel Vetter wrote:
> On Wed, Mar 18, 2020 at 3:29 PM Hans de Goede <hdegoede@redhat.com> wrote:
>>
>> Hi All,
>>
>> I'm not sure if $subject was a conscious design decision, or an oversight,
>> but that does not really matter.
>>
>> ATM the Atomic KMS API lacks the ability to set cursor hot-spot
>> coordinates. Mutter (and Weston) have tried to emulate this by shifting
>> the coordinates for where to draw the cursor by the hotspot-coordinates
>> and always using 0x0 for the hotspot.
>>
>> But this breaks the so called "seamless mouse mode" for virtual-machines
>> and there really is no way to fix this but to allow passing the proper
>> hotspot coordinates to the virtual gfx-card.
> 
> Fully aware, we simply never had userspace for this. So
> - type weston patch
> - type kernel patch (iirc this might be some work of moving the
> hotspot coordinates from struct drm_plane to drm_plane_state)
> - wire up atomic property (it'd just do "CURSOR_HOT_X/Y" and be done
> with it, only attached to cursor planes that support it ofc)
> - I think all drivers supporting vm cursor hotspot are now atomic, so
> would be good to garbage-collect the old legacy infrastructure
> completely. legacy cursor ioctl will remap to the atomic call
> transparently like we do now already for everything else
> - type igt
> - get it all reviewed and merged.
> 
> 0 design reasons for omitting this, simply Not Yet Done (tm).

Ok, sounds good, thank you.  I've put this on my TODO list, but below
a bunch of other higher prio items, so it may be a while before I
get around to this.

Regards,

Hans

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Pekka Paalanen]

[-- Attachment #1.1: Type: text/plain, Size: 688 bytes --]

On Wed, 18 Mar 2020 14:38:48 +0000
Simon Ser <contact@emersion.fr> wrote:

> Hi,
> 
> > 1) Letting the VM-viewer window-system draw the cursor as it normally
> > would draw it.  
> 
> Why is this important? Can't the VM viewer hide the cursor and use a
> sub-surface to manually draw the cursor plane configured by the guest?
> 
> This would also allow the compositor running inside the VM to correctly
> have control over the cursor position, which is necessary for pointer
> constraints.

Aren't pointer motion events delivered in absolute and not relative
coordinates in the "seamless mode"?

Do we do pointer constraints also with absolute motion?


Thanks,
pq

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Pekka Paalanen]

[-- Attachment #1.1: Type: text/plain, Size: 4250 bytes --]

On Wed, 18 Mar 2020 15:28:02 +0100
Hans de Goede <hdegoede@redhat.com> wrote:

> ATM the Atomic KMS API lacks the ability to set cursor hot-spot
> coordinates. Mutter (and Weston) have tried to emulate this by shifting
> the coordinates for where to draw the cursor by the hotspot-coordinates
> and always using 0x0 for the hotspot.
> 
> But this breaks the so called "seamless mouse mode" for virtual-machines
> and there really is no way to fix this but to allow passing the proper
> hotspot coordinates to the virtual gfx-card.
> 
> Seamless-mode consists of 2 parts:
> 
> 1) Letting the VM-viewer window-system draw the cursor as it normally
> would draw it.
> 
> 2) Giving absolute coordinates of the mouse to the VM by e.g. emulating
> an USB drawing tablet. These coordinates come from translating the
> coordinates where the VM-viewer window-system is drawing the cursor
> to an absolute position using the top left corner of the view as 0x0
> and the bottom right corner as max-abs-x,max-abs-y.

Hi,

is the VM-viewer then hiding or autonomously moving what the display
server inside VM has put on the KMS cursor plane?

If so, sounds like hilarity would ensue with Weston.

Weston does not actually know what a cursor is. Weston will happily put
any arbitrary non-cursor content onto the KMS cursor plane if it happens
to fit. It's just that usually there is a small surface top-most that
ends up on the cursor plane, and that surface accidentally happens to
be a cursor by Wayland protocol.

It's not difficult to get e.g. weston-simple-shm window to be shown on
the KMS cursor plane: just hide the real cursor from the client.

No, it's not an oversight. It is called "making maximal use of the
available KMS resources" by using KMS planes as much as possible to
avoid compositing by rendering with Pixman or OpenGL.


Thanks,
pq


> 1) Means that any coordinates the window-system inside the VM passes to
> the VM's gfx-card for where to draw the cursor are basically totally
> ignored to avoid lag / flicker (and to not have to grab the cursor /
> confine it to the VM-viewer window and to not have to warp the
> pointer).
> 
> This means that the offset added to the coordinates by e.g. mutter to
> emulate the hotspot are ignored. For Seamless mouse mode to keep working
> properly the window-system inside the VM need to pass the VM's gfx-card
> the correct hotspot when setting the cursor. Which currently is not
> possible when restricting oneself to the atomic APIs.
> 
> Also see: https://gitlab.gnome.org/GNOME/mutter/issues/1094
> Where this is currently being tracked from the mutter side. Mutter
> internally has both atomic and legacy paths. The plan for now is to
> push the hotspot-emulation by shifting coordinates thing into the
> atomic path, fixing seamless mouse mode when running in legacy mode,
> combined with blacklisting vboxvideo, vmwgfx, qxl and cirrus from
> using atomic mode.
> 
> This is of course a workaround, eventually we would like to see
> the atomic API extended to allow passing the cursor hot spot.
> 
> I'm not really familiar enough with the atomic API to come up with
> an API design for this, but if there are suggestions on how this
> should look like from the uAPI side then I can take a shot at
> implementing this (and hooking it up in mutter's atomic code
> paths to test it).
> 
> Regards,
> 
> Hans
> 
> 
> 
> p.s.
> 
> Before people start discussing how the VM / VM-viewer is broken here and
> the VM needs to be fixed. Seamless mouse mode exists for at least a
> decade and has worked fine during this entire decade. It also works
> fine when using the legacy (non atomic) DRM_IOCTL_MODE_CURSOR2 ioctl;
> 
> Also this problem reproduces with 2 completely independent VM code-bases,
> it has been seen on both qemu-kvm VMs and on VirtualBox VMs and I would
> not be surprised if other hypervisors are also affected.
> 
> And on the API consumer side this problem has been triggered by both
> mutter and Weston.
> 
> _______________________________________________
> dri-devel mailing list
> dri-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Simon Ser]

On Thursday, March 19, 2020 10:57 AM, Pekka Paalanen <ppaalanen@gmail.com> wrote:

> On Wed, 18 Mar 2020 14:38:48 +0000
> Simon Ser contact@emersion.fr wrote:
>
> > Hi,
> >
> > > 1.  Letting the VM-viewer window-system draw the cursor as it normally
> > >     would draw it.
> > >
> >
> > Why is this important? Can't the VM viewer hide the cursor and use a
> > sub-surface to manually draw the cursor plane configured by the guest?
> > This would also allow the compositor running inside the VM to correctly
> > have control over the cursor position, which is necessary for pointer
> > constraints.
>
> Aren't pointer motion events delivered in absolute and not relative
> coordinates in the "seamless mode"?
>
> Do we do pointer constraints also with absolute motion?

Some apps do pointer constraints with absolute motion, e.g. games and
VNC/VM viewers. It's usually used to prevent the user from leaving the
constrained surface by mistake.
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Michel Dänzer]

On 2020-03-18 4:22 p.m., Simon Ser wrote:
>>
>> On 3/18/20 3:38 PM, Simon Ser wrote:
>>>
>>>> 1) Letting the VM-viewer window-system draw the cursor as it normally
>>>> would draw it.
>>>
>>> Why is this important? Can't the VM viewer hide the cursor and use a
>>> sub-surface to manually draw the cursor plane configured by the guest?
>>
>> Because then moving the cursor as seen by the user requires a round trip
>> through the VM and that adds latency, esp. when the VM viewer is viewing
>> a VM which is running somewhere else over the network.
> 
> The video output has latency anyway.

Sounds like you've never tried the two different modes yourself? :) IME
it makes a big difference even with a local VM. Even very little latency
can make the cursor feel awkward, like it's being held back by a rubber
band or something.


-- 
Earthling Michel Dänzer               |               https://redhat.com
Libre software enthusiast             |             Mesa and X developer
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Hans de Goede]

Hi,

On 3/19/20 11:00 AM, Pekka Paalanen wrote:
> On Wed, 18 Mar 2020 15:28:02 +0100
> Hans de Goede <hdegoede@redhat.com> wrote:
> 
>> ATM the Atomic KMS API lacks the ability to set cursor hot-spot
>> coordinates. Mutter (and Weston) have tried to emulate this by shifting
>> the coordinates for where to draw the cursor by the hotspot-coordinates
>> and always using 0x0 for the hotspot.
>>
>> But this breaks the so called "seamless mouse mode" for virtual-machines
>> and there really is no way to fix this but to allow passing the proper
>> hotspot coordinates to the virtual gfx-card.
>>
>> Seamless-mode consists of 2 parts:
>>
>> 1) Letting the VM-viewer window-system draw the cursor as it normally
>> would draw it.
>>
>> 2) Giving absolute coordinates of the mouse to the VM by e.g. emulating
>> an USB drawing tablet. These coordinates come from translating the
>> coordinates where the VM-viewer window-system is drawing the cursor
>> to an absolute position using the top left corner of the view as 0x0
>> and the bottom right corner as max-abs-x,max-abs-y.
> 
> Hi,
> 
> is the VM-viewer then hiding or autonomously moving what the display
> server inside VM has put on the KMS cursor plane?

Yes and no, it is not the VM-viewer which is hiding what the
display-server inside the VM has put on the KMS cursor plane,
the VM-viewer negotiates seamless mouse mode with the hypervisor
and then the hypervisor just ignores the cursor-plane except for
sending "sprite" changes to the VM-viewer.

> If so, sounds like hilarity would ensue with Weston.
> 
> Weston does not actually know what a cursor is. Weston will happily put
> any arbitrary non-cursor content onto the KMS cursor plane if it happens
> to fit. It's just that usually there is a small surface top-most that
> ends up on the cursor plane, and that surface accidentally happens to
> be a cursor by Wayland protocol.
> 
> It's not difficult to get e.g. weston-simple-shm window to be shown on
> the KMS cursor plane: just hide the real cursor from the client.
> 
> No, it's not an oversight. It is called "making maximal use of the
> available KMS resources" by using KMS planes as much as possible to
> avoid compositing by rendering with Pixman or OpenGL.

Yes it sounds like this will break with Weston, note that it already
is broken in Weston, if you run e.g. Fedora 32 beta + its Weston
package inside a VirtualBox VM then start gnome-terminal (so
that you get a caret cursor instead of the default one) and try to
select text. This will result in the wrong text being displayed
because Weston does not relay cursor hotspot info to the GPU,
also see:
https://gitlab.gnome.org/GNOME/mutter/issues/1094

Where the symptoms of this are described in more detail
(they are identical for Weston and mutter).

Fixing this will require the discussed KMS atomic API changes
and also changes on the Weston and mutter side to pass through
the hotspot info.

Mutter used to do this, but mutter's internal API-s have been
reworked to closer match the atomic KMS API and as part of this
passing the hotspot coords was dropped, this is being fixed
for the legacy KMS API code-paths now (which atm are the only
code paths, atomic support has no landed yet):
https://gitlab.gnome.org/GNOME/mutter/merge_requests/1136

Regards,

Hans



>> 1) Means that any coordinates the window-system inside the VM passes to
>> the VM's gfx-card for where to draw the cursor are basically totally
>> ignored to avoid lag / flicker (and to not have to grab the cursor /
>> confine it to the VM-viewer window and to not have to warp the
>> pointer).
>>
>> This means that the offset added to the coordinates by e.g. mutter to
>> emulate the hotspot are ignored. For Seamless mouse mode to keep working
>> properly the window-system inside the VM need to pass the VM's gfx-card
>> the correct hotspot when setting the cursor. Which currently is not
>> possible when restricting oneself to the atomic APIs.
>>
>> Also see: https://gitlab.gnome.org/GNOME/mutter/issues/1094
>> Where this is currently being tracked from the mutter side. Mutter
>> internally has both atomic and legacy paths. The plan for now is to
>> push the hotspot-emulation by shifting coordinates thing into the
>> atomic path, fixing seamless mouse mode when running in legacy mode,
>> combined with blacklisting vboxvideo, vmwgfx, qxl and cirrus from
>> using atomic mode.
>>
>> This is of course a workaround, eventually we would like to see
>> the atomic API extended to allow passing the cursor hot spot.
>>
>> I'm not really familiar enough with the atomic API to come up with
>> an API design for this, but if there are suggestions on how this
>> should look like from the uAPI side then I can take a shot at
>> implementing this (and hooking it up in mutter's atomic code
>> paths to test it).
>>
>> Regards,
>>
>> Hans
>>
>>
>>
>> p.s.
>>
>> Before people start discussing how the VM / VM-viewer is broken here and
>> the VM needs to be fixed. Seamless mouse mode exists for at least a
>> decade and has worked fine during this entire decade. It also works
>> fine when using the legacy (non atomic) DRM_IOCTL_MODE_CURSOR2 ioctl;
>>
>> Also this problem reproduces with 2 completely independent VM code-bases,
>> it has been seen on both qemu-kvm VMs and on VirtualBox VMs and I would
>> not be surprised if other hypervisors are also affected.
>>
>> And on the API consumer side this problem has been triggered by both
>> mutter and Weston.
>>
>> _______________________________________________
>> dri-devel mailing list
>> dri-devel@lists.freedesktop.org
>> https://lists.freedesktop.org/mailman/listinfo/dri-devel
> 

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Hans de Goede]

Hi,

On 3/19/20 12:35 PM, Michel Dänzer wrote:
> On 2020-03-18 4:22 p.m., Simon Ser wrote:
>>>
>>> On 3/18/20 3:38 PM, Simon Ser wrote:
>>>>
>>>>> 1) Letting the VM-viewer window-system draw the cursor as it normally
>>>>> would draw it.
>>>>
>>>> Why is this important? Can't the VM viewer hide the cursor and use a
>>>> sub-surface to manually draw the cursor plane configured by the guest?
>>>
>>> Because then moving the cursor as seen by the user requires a round trip
>>> through the VM and that adds latency, esp. when the VM viewer is viewing
>>> a VM which is running somewhere else over the network.
>>
>> The video output has latency anyway.
> 
> Sounds like you've never tried the two different modes yourself? :) IME
> it makes a big difference even with a local VM. Even very little latency
> can make the cursor feel awkward, like it's being held back by a rubber
> band or something.

Right not to mention that the latency may be variable, so the cursor
moves in a jittery fashion instead of having it move smoothly
matching the smooth way a user normally moves the mouse.

This totally wrecks hand-eye coordination and is just plain awefull.

Regards,

Hans

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Pekka Paalanen]

[-- Attachment #1.1: Type: text/plain, Size: 1907 bytes --]

On Thu, 19 Mar 2020 12:52:14 +0100
Hans de Goede <hdegoede@redhat.com> wrote:

> Hi,
> 
> On 3/19/20 12:35 PM, Michel Dänzer wrote:
> > On 2020-03-18 4:22 p.m., Simon Ser wrote:  
> >>>
> >>> On 3/18/20 3:38 PM, Simon Ser wrote:  
> >>>>  
> >>>>> 1) Letting the VM-viewer window-system draw the cursor as it normally
> >>>>> would draw it.  
> >>>>
> >>>> Why is this important? Can't the VM viewer hide the cursor and use a
> >>>> sub-surface to manually draw the cursor plane configured by the guest?  
> >>>
> >>> Because then moving the cursor as seen by the user requires a round trip
> >>> through the VM and that adds latency, esp. when the VM viewer is viewing
> >>> a VM which is running somewhere else over the network.  
> >>
> >> The video output has latency anyway.  
> > 
> > Sounds like you've never tried the two different modes yourself? :) IME
> > it makes a big difference even with a local VM. Even very little latency
> > can make the cursor feel awkward, like it's being held back by a rubber
> > band or something.  
> 
> Right not to mention that the latency may be variable, so the cursor
> moves in a jittery fashion instead of having it move smoothly
> matching the smooth way a user normally moves the mouse.
> 
> This totally wrecks hand-eye coordination and is just plain awefull.

I have experienced it, and while it is painful, I prefer that pain over
the pain of accidentally clicking something that was not transmitted to
the remote display yet.

Therefore I think the best user experience is to use both types of
cursor at the same time: the remote desktop or VM viewer paints the
local cursor as an aid, like a phantom, and the cursor from inside the
VM is also visible with the latency it naturally has. That means I
could actually see that the screen has caught up with my motions before
I click something.


Thanks,
pq

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Pekka Paalanen]

[-- Attachment #1.1: Type: text/plain, Size: 5056 bytes --]

On Thu, 19 Mar 2020 12:49:27 +0100
Hans de Goede <hdegoede@redhat.com> wrote:

> Hi,
> 
> On 3/19/20 11:00 AM, Pekka Paalanen wrote:
> > On Wed, 18 Mar 2020 15:28:02 +0100
> > Hans de Goede <hdegoede@redhat.com> wrote:
> >   
> >> ATM the Atomic KMS API lacks the ability to set cursor hot-spot
> >> coordinates. Mutter (and Weston) have tried to emulate this by shifting
> >> the coordinates for where to draw the cursor by the hotspot-coordinates
> >> and always using 0x0 for the hotspot.
> >>
> >> But this breaks the so called "seamless mouse mode" for virtual-machines
> >> and there really is no way to fix this but to allow passing the proper
> >> hotspot coordinates to the virtual gfx-card.
> >>
> >> Seamless-mode consists of 2 parts:
> >>
> >> 1) Letting the VM-viewer window-system draw the cursor as it normally
> >> would draw it.
> >>
> >> 2) Giving absolute coordinates of the mouse to the VM by e.g. emulating
> >> an USB drawing tablet. These coordinates come from translating the
> >> coordinates where the VM-viewer window-system is drawing the cursor
> >> to an absolute position using the top left corner of the view as 0x0
> >> and the bottom right corner as max-abs-x,max-abs-y.  
> > 
> > Hi,
> > 
> > is the VM-viewer then hiding or autonomously moving what the display
> > server inside VM has put on the KMS cursor plane?  
> 
> Yes and no, it is not the VM-viewer which is hiding what the
> display-server inside the VM has put on the KMS cursor plane,
> the VM-viewer negotiates seamless mouse mode with the hypervisor
> and then the hypervisor just ignores the cursor-plane except for
> sending "sprite" changes to the VM-viewer.

Hi,

I don't think I understand what you're saying, but I assume that I was
right in that the VM cursor plane content will not be shown always
exactly in the very position the compositor inside the VM puts it.
Maybe the example further below explain the issue I envision.

> > If so, sounds like hilarity would ensue with Weston.
> > 
> > Weston does not actually know what a cursor is. Weston will happily put
> > any arbitrary non-cursor content onto the KMS cursor plane if it happens
> > to fit. It's just that usually there is a small surface top-most that
> > ends up on the cursor plane, and that surface accidentally happens to
> > be a cursor by Wayland protocol.
> > 
> > It's not difficult to get e.g. weston-simple-shm window to be shown on
> > the KMS cursor plane: just hide the real cursor from the client.
> > 
> > No, it's not an oversight. It is called "making maximal use of the
> > available KMS resources" by using KMS planes as much as possible to
> > avoid compositing by rendering with Pixman or OpenGL.  
> 
> Yes it sounds like this will break with Weston, note that it already
> is broken in Weston, if you run e.g. Fedora 32 beta + its Weston
> package inside a VirtualBox VM then start gnome-terminal (so
> that you get a caret cursor instead of the default one) and try to
> select text. This will result in the wrong text being displayed
> because Weston does not relay cursor hotspot info to the GPU,
> also see:
> https://gitlab.gnome.org/GNOME/mutter/issues/1094
> 
> Where the symptoms of this are described in more detail
> (they are identical for Weston and mutter).

Right, that's the problem with the hotspot.

> Fixing this will require the discussed KMS atomic API changes
> and also changes on the Weston and mutter side to pass through
> the hotspot info.

The problem I am referring to is that to the user looking at the
VM-viewer, suddenly an arbitrary application window (e.g.
weston-simple-shm) starts to act as if it was the cursor, when there is
no real cursor shown. You have a random window unexpectedly moving
around, as if you had started dragging it around with your mouse.

The only way to fix that is to stop Weston from putting non-cursor
content on the cursor plane.

It sounds like your VM-viewer makes the assumption that the pointer
input device it delivers to the VM is the one that will control the KMS
cursor plane position inside the VM. Is that right?

What if the desktop inside the VM is controlled by a remote, e.g. VNC?
Then the input events to the VM are completely unrelated to the
expected motion of the cursor. Do you just tell the users to stop using
the seamless mode in that case?

What if display servers stop using the cursor plane completely, because
people may hit a case where a VM-viewer makes the wrong assumption about
which input device is associated to which cursor plane inside the VM?


Thanks,
pq

> Mutter used to do this, but mutter's internal API-s have been
> reworked to closer match the atomic KMS API and as part of this
> passing the hotspot coords was dropped, this is being fixed
> for the legacy KMS API code-paths now (which atm are the only
> code paths, atomic support has no landed yet):
> https://gitlab.gnome.org/GNOME/mutter/merge_requests/1136
> 
> Regards,
> 
> Hans

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Daniel Vetter]

On Thu, Mar 19, 2020 at 1:58 PM Pekka Paalanen <ppaalanen@gmail.com> wrote:
>
> On Thu, 19 Mar 2020 12:49:27 +0100
> Hans de Goede <hdegoede@redhat.com> wrote:
>
> > Hi,
> >
> > On 3/19/20 11:00 AM, Pekka Paalanen wrote:
> > > On Wed, 18 Mar 2020 15:28:02 +0100
> > > Hans de Goede <hdegoede@redhat.com> wrote:
> > >
> > >> ATM the Atomic KMS API lacks the ability to set cursor hot-spot
> > >> coordinates. Mutter (and Weston) have tried to emulate this by shifting
> > >> the coordinates for where to draw the cursor by the hotspot-coordinates
> > >> and always using 0x0 for the hotspot.
> > >>
> > >> But this breaks the so called "seamless mouse mode" for virtual-machines
> > >> and there really is no way to fix this but to allow passing the proper
> > >> hotspot coordinates to the virtual gfx-card.
> > >>
> > >> Seamless-mode consists of 2 parts:
> > >>
> > >> 1) Letting the VM-viewer window-system draw the cursor as it normally
> > >> would draw it.
> > >>
> > >> 2) Giving absolute coordinates of the mouse to the VM by e.g. emulating
> > >> an USB drawing tablet. These coordinates come from translating the
> > >> coordinates where the VM-viewer window-system is drawing the cursor
> > >> to an absolute position using the top left corner of the view as 0x0
> > >> and the bottom right corner as max-abs-x,max-abs-y.
> > >
> > > Hi,
> > >
> > > is the VM-viewer then hiding or autonomously moving what the display
> > > server inside VM has put on the KMS cursor plane?
> >
> > Yes and no, it is not the VM-viewer which is hiding what the
> > display-server inside the VM has put on the KMS cursor plane,
> > the VM-viewer negotiates seamless mouse mode with the hypervisor
> > and then the hypervisor just ignores the cursor-plane except for
> > sending "sprite" changes to the VM-viewer.
>
> Hi,
>
> I don't think I understand what you're saying, but I assume that I was
> right in that the VM cursor plane content will not be shown always
> exactly in the very position the compositor inside the VM puts it.
> Maybe the example further below explain the issue I envision.
>
> > > If so, sounds like hilarity would ensue with Weston.
> > >
> > > Weston does not actually know what a cursor is. Weston will happily put
> > > any arbitrary non-cursor content onto the KMS cursor plane if it happens
> > > to fit. It's just that usually there is a small surface top-most that
> > > ends up on the cursor plane, and that surface accidentally happens to
> > > be a cursor by Wayland protocol.
> > >
> > > It's not difficult to get e.g. weston-simple-shm window to be shown on
> > > the KMS cursor plane: just hide the real cursor from the client.
> > >
> > > No, it's not an oversight. It is called "making maximal use of the
> > > available KMS resources" by using KMS planes as much as possible to
> > > avoid compositing by rendering with Pixman or OpenGL.
> >
> > Yes it sounds like this will break with Weston, note that it already
> > is broken in Weston, if you run e.g. Fedora 32 beta + its Weston
> > package inside a VirtualBox VM then start gnome-terminal (so
> > that you get a caret cursor instead of the default one) and try to
> > select text. This will result in the wrong text being displayed
> > because Weston does not relay cursor hotspot info to the GPU,
> > also see:
> > https://gitlab.gnome.org/GNOME/mutter/issues/1094
> >
> > Where the symptoms of this are described in more detail
> > (they are identical for Weston and mutter).
>
> Right, that's the problem with the hotspot.
>
> > Fixing this will require the discussed KMS atomic API changes
> > and also changes on the Weston and mutter side to pass through
> > the hotspot info.
>
> The problem I am referring to is that to the user looking at the
> VM-viewer, suddenly an arbitrary application window (e.g.
> weston-simple-shm) starts to act as if it was the cursor, when there is
> no real cursor shown. You have a random window unexpectedly moving
> around, as if you had started dragging it around with your mouse.
>
> The only way to fix that is to stop Weston from putting non-cursor
> content on the cursor plane.
>
> It sounds like your VM-viewer makes the assumption that the pointer
> input device it delivers to the VM is the one that will control the KMS
> cursor plane position inside the VM. Is that right?
>
> What if the desktop inside the VM is controlled by a remote, e.g. VNC?
> Then the input events to the VM are completely unrelated to the
> expected motion of the cursor. Do you just tell the users to stop using
> the seamless mode in that case?
>
> What if display servers stop using the cursor plane completely, because
> people may hit a case where a VM-viewer makes the wrong assumption about
> which input device is associated to which cursor plane inside the VM?

Hm that sounds like we should have an additional flag with atomic that
enables/disables the hotspot functionality. Then compositors can
decide what they want to do with that thing.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Simon Ser]

On Thursday, March 19, 2020 2:16 PM, Daniel Vetter <daniel@ffwll.ch> wrote:

> On Thu, Mar 19, 2020 at 1:58 PM Pekka Paalanen ppaalanen@gmail.com wrote:
>
> > On Thu, 19 Mar 2020 12:49:27 +0100
> > Hans de Goede hdegoede@redhat.com wrote:
> >
> > > Hi,
> > > On 3/19/20 11:00 AM, Pekka Paalanen wrote:
> > >
> > > > On Wed, 18 Mar 2020 15:28:02 +0100
> > > > Hans de Goede hdegoede@redhat.com wrote:
> > > >
> > > > > ATM the Atomic KMS API lacks the ability to set cursor hot-spot
> > > > > coordinates. Mutter (and Weston) have tried to emulate this by shifting
> > > > > the coordinates for where to draw the cursor by the hotspot-coordinates
> > > > > and always using 0x0 for the hotspot.
> > > > > But this breaks the so called "seamless mouse mode" for virtual-machines
> > > > > and there really is no way to fix this but to allow passing the proper
> > > > > hotspot coordinates to the virtual gfx-card.
> > > > > Seamless-mode consists of 2 parts:
> > > > >
> > > > > 1.  Letting the VM-viewer window-system draw the cursor as it normally
> > > > >     would draw it.
> > > > >
> > > > > 2.  Giving absolute coordinates of the mouse to the VM by e.g. emulating
> > > > >     an USB drawing tablet. These coordinates come from translating the
> > > > >     coordinates where the VM-viewer window-system is drawing the cursor
> > > > >     to an absolute position using the top left corner of the view as 0x0
> > > > >     and the bottom right corner as max-abs-x,max-abs-y.
> > > > >
> > > >
> > > > Hi,
> > > > is the VM-viewer then hiding or autonomously moving what the display
> > > > server inside VM has put on the KMS cursor plane?
> > >
> > > Yes and no, it is not the VM-viewer which is hiding what the
> > > display-server inside the VM has put on the KMS cursor plane,
> > > the VM-viewer negotiates seamless mouse mode with the hypervisor
> > > and then the hypervisor just ignores the cursor-plane except for
> > > sending "sprite" changes to the VM-viewer.
> >
> > Hi,
> > I don't think I understand what you're saying, but I assume that I was
> > right in that the VM cursor plane content will not be shown always
> > exactly in the very position the compositor inside the VM puts it.
> > Maybe the example further below explain the issue I envision.
> >
> > > > If so, sounds like hilarity would ensue with Weston.
> > > > Weston does not actually know what a cursor is. Weston will happily put
> > > > any arbitrary non-cursor content onto the KMS cursor plane if it happens
> > > > to fit. It's just that usually there is a small surface top-most that
> > > > ends up on the cursor plane, and that surface accidentally happens to
> > > > be a cursor by Wayland protocol.
> > > > It's not difficult to get e.g. weston-simple-shm window to be shown on
> > > > the KMS cursor plane: just hide the real cursor from the client.
> > > > No, it's not an oversight. It is called "making maximal use of the
> > > > available KMS resources" by using KMS planes as much as possible to
> > > > avoid compositing by rendering with Pixman or OpenGL.
> > >
> > > Yes it sounds like this will break with Weston, note that it already
> > > is broken in Weston, if you run e.g. Fedora 32 beta + its Weston
> > > package inside a VirtualBox VM then start gnome-terminal (so
> > > that you get a caret cursor instead of the default one) and try to
> > > select text. This will result in the wrong text being displayed
> > > because Weston does not relay cursor hotspot info to the GPU,
> > > also see:
> > > https://gitlab.gnome.org/GNOME/mutter/issues/1094
> > > Where the symptoms of this are described in more detail
> > > (they are identical for Weston and mutter).
> >
> > Right, that's the problem with the hotspot.
> >
> > > Fixing this will require the discussed KMS atomic API changes
> > > and also changes on the Weston and mutter side to pass through
> > > the hotspot info.
> >
> > The problem I am referring to is that to the user looking at the
> > VM-viewer, suddenly an arbitrary application window (e.g.
> > weston-simple-shm) starts to act as if it was the cursor, when there is
> > no real cursor shown. You have a random window unexpectedly moving
> > around, as if you had started dragging it around with your mouse.
> > The only way to fix that is to stop Weston from putting non-cursor
> > content on the cursor plane.
> > It sounds like your VM-viewer makes the assumption that the pointer
> > input device it delivers to the VM is the one that will control the KMS
> > cursor plane position inside the VM. Is that right?
> > What if the desktop inside the VM is controlled by a remote, e.g. VNC?
> > Then the input events to the VM are completely unrelated to the
> > expected motion of the cursor. Do you just tell the users to stop using
> > the seamless mode in that case?
> > What if display servers stop using the cursor plane completely, because
> > people may hit a case where a VM-viewer makes the wrong assumption about
> > which input device is associated to which cursor plane inside the VM?
>
> Hm that sounds like we should have an additional flag with atomic that
> enables/disables the hotspot functionality. Then compositors can
> decide what they want to do with that thing.

A per-plane prop to enable/disable the hotspot would help.

This still doesn't fix issues with multiple input devices, e.g. VNC.
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Michel Dänzer]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2020-03-19 1:54 p.m., Pekka Paalanen wrote:
> On Thu, 19 Mar 2020 12:52:14 +0100 Hans de Goede
> <hdegoede@redhat.com> wrote:
>> On 3/19/20 12:35 PM, Michel Dänzer wrote:
>>> On 2020-03-18 4:22 p.m., Simon Ser wrote:
>>>>>
>>>>> On 3/18/20 3:38 PM, Simon Ser wrote:
>>>>>>
>>>>>>> 1) Letting the VM-viewer window-system draw the cursor
>>>>>>> as it normally would draw it.
>>>>>>
>>>>>> Why is this important? Can't the VM viewer hide the
>>>>>> cursor and use a sub-surface to manually draw the cursor
>>>>>> plane configured by the guest?
>>>>>
>>>>> Because then moving the cursor as seen by the user requires
>>>>> a round trip through the VM and that adds latency, esp.
>>>>> when the VM viewer is viewing a VM which is running
>>>>> somewhere else over the network.
>>>>
>>>> The video output has latency anyway.
>>>
>>> Sounds like you've never tried the two different modes
>>> yourself? :) IME it makes a big difference even with a local
>>> VM. Even very little latency can make the cursor feel awkward,
>>> like it's being held back by a rubber band or something.
>>
>> Right not to mention that the latency may be variable, so the
>> cursor moves in a jittery fashion instead of having it move
>> smoothly matching the smooth way a user normally moves the
>> mouse.
>>
>> This totally wrecks hand-eye coordination and is just plain
>> awefull.
>
> I have experienced it, and while it is painful, I prefer that pain
> over the pain of accidentally clicking something that was not
> transmitted to the remote display yet.

Unless you mean clicking something while the cursor is moving, not
sure how seamless vs not affects this, since the delay of something
appearing on the remote display should be the same in both cases?


> Therefore I think the best user experience is to use both types of
> cursor at the same time: the remote desktop or VM viewer paints
> the local cursor as an aid, like a phantom, and the cursor from
> inside the VM is also visible with the latency it naturally has.
> That means I could actually see that the screen has caught up with
> my motions before I click something.

I'd expect that to result in "duplicate cursor" bug reports — I'd
certainly consider it a bug with my user hat on.


- -- 
Earthling Michel Dänzer               |               https://redhat.com
Libre software enthusiast             |             Mesa and X developer
-----BEGIN PGP SIGNATURE-----

iF0EARECAB0WIQSwn681vpFFIZgJURRaga+OatuyAAUCXnN45gAKCRBaga+Oatuy
AEMOAJsEtOvBJIA6CGHOmwrp7SFd9M3o2wCfcCs7lCrCNsHxAels/4YUXm5FJas=
=FfIQ
-----END PGP SIGNATURE-----
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Hans de Goede]

Hi,

On 3/19/20 1:58 PM, Pekka Paalanen wrote:
> On Thu, 19 Mar 2020 12:49:27 +0100
> Hans de Goede <hdegoede@redhat.com> wrote:
> 
>> Hi,
>>
>> On 3/19/20 11:00 AM, Pekka Paalanen wrote:
>>> On Wed, 18 Mar 2020 15:28:02 +0100
>>> Hans de Goede <hdegoede@redhat.com> wrote:
>>>    
>>>> ATM the Atomic KMS API lacks the ability to set cursor hot-spot
>>>> coordinates. Mutter (and Weston) have tried to emulate this by shifting
>>>> the coordinates for where to draw the cursor by the hotspot-coordinates
>>>> and always using 0x0 for the hotspot.
>>>>
>>>> But this breaks the so called "seamless mouse mode" for virtual-machines
>>>> and there really is no way to fix this but to allow passing the proper
>>>> hotspot coordinates to the virtual gfx-card.
>>>>
>>>> Seamless-mode consists of 2 parts:
>>>>
>>>> 1) Letting the VM-viewer window-system draw the cursor as it normally
>>>> would draw it.
>>>>
>>>> 2) Giving absolute coordinates of the mouse to the VM by e.g. emulating
>>>> an USB drawing tablet. These coordinates come from translating the
>>>> coordinates where the VM-viewer window-system is drawing the cursor
>>>> to an absolute position using the top left corner of the view as 0x0
>>>> and the bottom right corner as max-abs-x,max-abs-y.
>>>
>>> Hi,
>>>
>>> is the VM-viewer then hiding or autonomously moving what the display
>>> server inside VM has put on the KMS cursor plane?
>>
>> Yes and no, it is not the VM-viewer which is hiding what the
>> display-server inside the VM has put on the KMS cursor plane,
>> the VM-viewer negotiates seamless mouse mode with the hypervisor
>> and then the hypervisor just ignores the cursor-plane except for
>> sending "sprite" changes to the VM-viewer.
> 
> Hi,
> 
> I don't think I understand what you're saying, but I assume that I was
> right in that the VM cursor plane content will not be shown always
> exactly in the very position the compositor inside the VM puts it.

Right, when seamless mouse mode is enabled the VM's cursor plane will
not be shown *at all*, instead the VM-viewer's window-system's
cursor plane is shown.

Sprite changes (normal cursor, caret, window-resize cursor, etc.)
are forwarded from the VM-s cursor-plane to the VM-viewer
which then requests that as cursor to the window-system under
which the VM-viewer is running.

> Maybe the example further below explain the issue I envision.
> 
>>> If so, sounds like hilarity would ensue with Weston.
>>>
>>> Weston does not actually know what a cursor is. Weston will happily put
>>> any arbitrary non-cursor content onto the KMS cursor plane if it happens
>>> to fit. It's just that usually there is a small surface top-most that
>>> ends up on the cursor plane, and that surface accidentally happens to
>>> be a cursor by Wayland protocol.
>>>
>>> It's not difficult to get e.g. weston-simple-shm window to be shown on
>>> the KMS cursor plane: just hide the real cursor from the client.
>>>
>>> No, it's not an oversight. It is called "making maximal use of the
>>> available KMS resources" by using KMS planes as much as possible to
>>> avoid compositing by rendering with Pixman or OpenGL.
>>
>> Yes it sounds like this will break with Weston, note that it already
>> is broken in Weston, if you run e.g. Fedora 32 beta + its Weston
>> package inside a VirtualBox VM then start gnome-terminal (so
>> that you get a caret cursor instead of the default one) and try to
>> select text. This will result in the wrong text being displayed
>> because Weston does not relay cursor hotspot info to the GPU,
>> also see:
>> https://gitlab.gnome.org/GNOME/mutter/issues/1094
>>
>> Where the symptoms of this are described in more detail
>> (they are identical for Weston and mutter).
> 
> Right, that's the problem with the hotspot.

Ack.

>> Fixing this will require the discussed KMS atomic API changes
>> and also changes on the Weston and mutter side to pass through
>> the hotspot info.
> 
> The problem I am referring to is that to the user looking at the
> VM-viewer, suddenly an arbitrary application window (e.g.
> weston-simple-shm) starts to act as if it was the cursor, when there is
> no real cursor shown. You have a random window unexpectedly moving
> around, as if you had started dragging it around with your mouse.

Correct.
> The only way to fix that is to stop Weston from putting non-cursor
> content on the cursor plane.

Correct.

> It sounds like your VM-viewer makes the assumption that the pointer
> input device it delivers to the VM is the one that will control the KMS
> cursor plane position inside the VM. Is that right?

Correct.

> What if the desktop inside the VM is controlled by a remote, e.g. VNC?
> Then the input events to the VM are completely unrelated to the
> expected motion of the cursor. Do you just tell the users to stop using
> the seamless mode in that case?

A VNC viewer (which is not using seamless mode itself) has the same issue
of a mismatch between the cursor position of the window-system it is a client
of and the cursor position of the window system inside the VM.

A VNC viewer typically works around this by changing the window-system
cursor to transparent and drawing its own cursor, the transparent sprite
or disabling of the cursor-plane will get forwarded to the window-system
under which the VM-viewer runs so this will work fine.

A real problem though is the absolute input mode, when e.g. the VNC
viewer is not using something like seamless mouse mode it will want
to do the usual (nasty) tricks of confining the pointer to the window
and warping it to the center after each pointer move so that mouse
can be moved "endlessly' inside the window. This requires relative
input, so emulating a mouse instead of say a drawing tablet, which
breaks seamless mode.

So although the way the cursor is drawn typically does not require
disabling seamless mode, the input problems do require disabling
seamless mode.

In my experience (I have worked on VDI (spice) a couple of years),
running a VM-viewer inside a VM is something which not a lot of
people do. I guess the fact that this works quite poorly might
be one of the reasons people do not do this.

But all VMs I'm familiar with allow disabling seamless mode for
compatibility with corner cases like this. This does lead to a
seriously degraded user-experience though. For just using an
office suit or browser inside a VM seamless mode really is a
lot more pleasant for the user.  I still remember when seamless
mode first got introduced, in the beginning it used to not
always work and fixing the fallback to the confine + warp tricks
then always was a serious itch which I tried to scratch quickly.
Breaking seamless mode also is a good way to quickly accumulate
a lot of bug reports from end users in my experience.

> What if display servers stop using the cursor plane completely, because
> people may hit a case where a VM-viewer makes the wrong assumption about
> which input device is associated to which cursor plane inside the VM?

The confine + warp trick is typically the default mode and only
if the guest indicates through e.g. a guest-agent process that
seamless mode is supported then seamless mode is enabled.

IOW the VM is careful to not enable it when it might not work.

OTOH most distros now ship with the necessary agents installed
by default, so e.g. a F32 beta install will automatically use
seamless mode under QEMU/KVM (spice display protocol) or
VirtualBox.  If a user uses say F32 + a vnc-viewer inside it
which needs to do confine + warp tricks then the user will have
to disable seamless mode manually. In my prior experience
working on VDI this is usually something which users are
understanding of and not a problem.

Regards,

Hans

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Pekka Paalanen]

[-- Attachment #1.1: Type: text/plain, Size: 3211 bytes --]

On Thu, 19 Mar 2020 14:51:41 +0100
Michel Dänzer <michel@daenzer.net> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 2020-03-19 1:54 p.m., Pekka Paalanen wrote:
> > On Thu, 19 Mar 2020 12:52:14 +0100 Hans de Goede
> > <hdegoede@redhat.com> wrote:  
> >> On 3/19/20 12:35 PM, Michel Dänzer wrote:  
> >>> On 2020-03-18 4:22 p.m., Simon Ser wrote:  
> >>>>>
> >>>>> On 3/18/20 3:38 PM, Simon Ser wrote:  
> >>>>>>  
> >>>>>>> 1) Letting the VM-viewer window-system draw the cursor
> >>>>>>> as it normally would draw it.  
> >>>>>>
> >>>>>> Why is this important? Can't the VM viewer hide the
> >>>>>> cursor and use a sub-surface to manually draw the cursor
> >>>>>> plane configured by the guest?  
> >>>>>
> >>>>> Because then moving the cursor as seen by the user requires
> >>>>> a round trip through the VM and that adds latency, esp.
> >>>>> when the VM viewer is viewing a VM which is running
> >>>>> somewhere else over the network.  
> >>>>
> >>>> The video output has latency anyway.  
> >>>
> >>> Sounds like you've never tried the two different modes
> >>> yourself? :) IME it makes a big difference even with a local
> >>> VM. Even very little latency can make the cursor feel awkward,
> >>> like it's being held back by a rubber band or something.  
> >>
> >> Right not to mention that the latency may be variable, so the
> >> cursor moves in a jittery fashion instead of having it move
> >> smoothly matching the smooth way a user normally moves the
> >> mouse.
> >>
> >> This totally wrecks hand-eye coordination and is just plain
> >> awefull.  
> >
> > I have experienced it, and while it is painful, I prefer that pain
> > over the pain of accidentally clicking something that was not
> > transmitted to the remote display yet.  
> 
> Unless you mean clicking something while the cursor is moving, not
> sure how seamless vs not affects this, since the delay of something
> appearing on the remote display should be the same in both cases?

How do you know if the cursor is still moving or not, if you don't see
the real cursor but only the fake cursor?

I move the mouse, then it takes 0.1 - 10 seconds for the real cursor
to reach where I moved it. Only after that I see what the motion
caused. Then I can do the next thing.

The mouse motion / cursor response gives me continuous feedback, so
that I don't run too far ahead of the remote end.

> > Therefore I think the best user experience is to use both types of
> > cursor at the same time: the remote desktop or VM viewer paints
> > the local cursor as an aid, like a phantom, and the cursor from
> > inside the VM is also visible with the latency it naturally has.
> > That means I could actually see that the screen has caught up with
> > my motions before I click something.  
> 
> I'd expect that to result in "duplicate cursor" bug reports — I'd
> certainly consider it a bug with my user hat on.

The point is to make the phantom cursor look like a phantom cursor, so
it cannot be mistaken as the real cursor. It doesn't even need to use
the same cursor image as the real cursor, it could be just a
translucent spot.


Thanks,
pq

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Hans de Goede]

Hi,

On 3/19/20 3:48 PM, Pekka Paalanen wrote:
> On Thu, 19 Mar 2020 14:51:41 +0100
> Michel Dänzer <michel@daenzer.net> wrote:
> 
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> On 2020-03-19 1:54 p.m., Pekka Paalanen wrote:
>>> On Thu, 19 Mar 2020 12:52:14 +0100 Hans de Goede
>>> <hdegoede@redhat.com> wrote:
>>>> On 3/19/20 12:35 PM, Michel Dänzer wrote:
>>>>> On 2020-03-18 4:22 p.m., Simon Ser wrote:
>>>>>>>
>>>>>>> On 3/18/20 3:38 PM, Simon Ser wrote:
>>>>>>>>   
>>>>>>>>> 1) Letting the VM-viewer window-system draw the cursor
>>>>>>>>> as it normally would draw it.
>>>>>>>>
>>>>>>>> Why is this important? Can't the VM viewer hide the
>>>>>>>> cursor and use a sub-surface to manually draw the cursor
>>>>>>>> plane configured by the guest?
>>>>>>>
>>>>>>> Because then moving the cursor as seen by the user requires
>>>>>>> a round trip through the VM and that adds latency, esp.
>>>>>>> when the VM viewer is viewing a VM which is running
>>>>>>> somewhere else over the network.
>>>>>>
>>>>>> The video output has latency anyway.
>>>>>
>>>>> Sounds like you've never tried the two different modes
>>>>> yourself? :) IME it makes a big difference even with a local
>>>>> VM. Even very little latency can make the cursor feel awkward,
>>>>> like it's being held back by a rubber band or something.
>>>>
>>>> Right not to mention that the latency may be variable, so the
>>>> cursor moves in a jittery fashion instead of having it move
>>>> smoothly matching the smooth way a user normally moves the
>>>> mouse.
>>>>
>>>> This totally wrecks hand-eye coordination and is just plain
>>>> awefull.
>>>
>>> I have experienced it, and while it is painful, I prefer that pain
>>> over the pain of accidentally clicking something that was not
>>> transmitted to the remote display yet.
>>
>> Unless you mean clicking something while the cursor is moving, not
>> sure how seamless vs not affects this, since the delay of something
>> appearing on the remote display should be the same in both cases?
> 
> How do you know if the cursor is still moving or not, if you don't see
> the real cursor but only the fake cursor?
> 
> I move the mouse, then it takes 0.1 - 10 seconds for the real cursor
> to reach where I moved it. Only after that I see what the motion
> caused. Then I can do the next thing.
> 
> The mouse motion / cursor response gives me continuous feedback, so
> that I don't run too far ahead of the remote end.
> 
>>> Therefore I think the best user experience is to use both types of
>>> cursor at the same time: the remote desktop or VM viewer paints
>>> the local cursor as an aid, like a phantom, and the cursor from
>>> inside the VM is also visible with the latency it naturally has.
>>> That means I could actually see that the screen has caught up with
>>> my motions before I click something.
>>
>> I'd expect that to result in "duplicate cursor" bug reports — I'd
>> certainly consider it a bug with my user hat on.
> 
> The point is to make the phantom cursor look like a phantom cursor, so
> it cannot be mistaken as the real cursor. It doesn't even need to use
> the same cursor image as the real cursor, it could be just a
> translucent spot.

I think you are trying to fix a non existing problem here, accept
for say a game of whack a mode, the target which the user is trying
to hit is typically static and the VM will receive and process the
coordinates where the user stops the motion, before it will receive
and process the click. So unless say an icon on a toolbar for some
reason is a moving target there is no problem here.

Regards,

Hans

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Pekka Paalanen]

[-- Attachment #1.1: Type: text/plain, Size: 9734 bytes --]

On Thu, 19 Mar 2020 15:30:03 +0100
Hans de Goede <hdegoede@redhat.com> wrote:

> Hi,
> 
> On 3/19/20 1:58 PM, Pekka Paalanen wrote:
> > On Thu, 19 Mar 2020 12:49:27 +0100
> > Hans de Goede <hdegoede@redhat.com> wrote:
> >   
> >> Hi,
> >>
> >> On 3/19/20 11:00 AM, Pekka Paalanen wrote:  
> >>> On Wed, 18 Mar 2020 15:28:02 +0100
> >>> Hans de Goede <hdegoede@redhat.com> wrote:
> >>>      
> >>>> ATM the Atomic KMS API lacks the ability to set cursor hot-spot
> >>>> coordinates. Mutter (and Weston) have tried to emulate this by shifting
> >>>> the coordinates for where to draw the cursor by the hotspot-coordinates
> >>>> and always using 0x0 for the hotspot.
> >>>>
> >>>> But this breaks the so called "seamless mouse mode" for virtual-machines
> >>>> and there really is no way to fix this but to allow passing the proper
> >>>> hotspot coordinates to the virtual gfx-card.
> >>>>
> >>>> Seamless-mode consists of 2 parts:
> >>>>
> >>>> 1) Letting the VM-viewer window-system draw the cursor as it normally
> >>>> would draw it.
> >>>>
> >>>> 2) Giving absolute coordinates of the mouse to the VM by e.g. emulating
> >>>> an USB drawing tablet. These coordinates come from translating the
> >>>> coordinates where the VM-viewer window-system is drawing the cursor
> >>>> to an absolute position using the top left corner of the view as 0x0
> >>>> and the bottom right corner as max-abs-x,max-abs-y.  
> >>>
> >>> Hi,
> >>>
> >>> is the VM-viewer then hiding or autonomously moving what the display
> >>> server inside VM has put on the KMS cursor plane?  
> >>
> >> Yes and no, it is not the VM-viewer which is hiding what the
> >> display-server inside the VM has put on the KMS cursor plane,
> >> the VM-viewer negotiates seamless mouse mode with the hypervisor
> >> and then the hypervisor just ignores the cursor-plane except for
> >> sending "sprite" changes to the VM-viewer.  
> > 
> > Hi,
> > 
> > I don't think I understand what you're saying, but I assume that I was
> > right in that the VM cursor plane content will not be shown always
> > exactly in the very position the compositor inside the VM puts it.  
> 
> Right, when seamless mouse mode is enabled the VM's cursor plane will
> not be shown *at all*, instead the VM-viewer's window-system's
> cursor plane is shown.
> 
> Sprite changes (normal cursor, caret, window-resize cursor, etc.)
> are forwarded from the VM-s cursor-plane to the VM-viewer
> which then requests that as cursor to the window-system under
> which the VM-viewer is running.
> 
> > Maybe the example further below explain the issue I envision.
> >   
> >>> If so, sounds like hilarity would ensue with Weston.
> >>>
> >>> Weston does not actually know what a cursor is. Weston will happily put
> >>> any arbitrary non-cursor content onto the KMS cursor plane if it happens
> >>> to fit. It's just that usually there is a small surface top-most that
> >>> ends up on the cursor plane, and that surface accidentally happens to
> >>> be a cursor by Wayland protocol.
> >>>
> >>> It's not difficult to get e.g. weston-simple-shm window to be shown on
> >>> the KMS cursor plane: just hide the real cursor from the client.
> >>>
> >>> No, it's not an oversight. It is called "making maximal use of the
> >>> available KMS resources" by using KMS planes as much as possible to
> >>> avoid compositing by rendering with Pixman or OpenGL.  
> >>
> >> Yes it sounds like this will break with Weston, note that it already
> >> is broken in Weston, if you run e.g. Fedora 32 beta + its Weston
> >> package inside a VirtualBox VM then start gnome-terminal (so
> >> that you get a caret cursor instead of the default one) and try to
> >> select text. This will result in the wrong text being displayed
> >> because Weston does not relay cursor hotspot info to the GPU,
> >> also see:
> >> https://gitlab.gnome.org/GNOME/mutter/issues/1094
> >>
> >> Where the symptoms of this are described in more detail
> >> (they are identical for Weston and mutter).  
> > 
> > Right, that's the problem with the hotspot.  
> 
> Ack.
> 
> >> Fixing this will require the discussed KMS atomic API changes
> >> and also changes on the Weston and mutter side to pass through
> >> the hotspot info.  
> > 
> > The problem I am referring to is that to the user looking at the
> > VM-viewer, suddenly an arbitrary application window (e.g.
> > weston-simple-shm) starts to act as if it was the cursor, when there is
> > no real cursor shown. You have a random window unexpectedly moving
> > around, as if you had started dragging it around with your mouse.  
> 
> Correct.
> > The only way to fix that is to stop Weston from putting non-cursor
> > content on the cursor plane.  
> 
> Correct.

Is that something that should be done?

If the hotspot property also had a "disabled" value, then Weston could
set the hotspot to disabled when it is using the cursor plane for
non-cursor content and not lose the feature. And of course set hotspot
correctly when it in fact is a cursor (but for what input?).

> 
> > It sounds like your VM-viewer makes the assumption that the pointer
> > input device it delivers to the VM is the one that will control the KMS
> > cursor plane position inside the VM. Is that right?  
> 
> Correct.
> 
> > What if the desktop inside the VM is controlled by a remote, e.g. VNC?
> > Then the input events to the VM are completely unrelated to the
> > expected motion of the cursor. Do you just tell the users to stop using
> > the seamless mode in that case?  
> 
> A VNC viewer (which is not using seamless mode itself) has the same issue
> of a mismatch between the cursor position of the window-system it is a client
> of and the cursor position of the window system inside the VM.
> 
> A VNC viewer typically works around this by changing the window-system
> cursor to transparent and drawing its own cursor, the transparent sprite
> or disabling of the cursor-plane will get forwarded to the window-system
> under which the VM-viewer runs so this will work fine.
> 
> A real problem though is the absolute input mode, when e.g. the VNC
> viewer is not using something like seamless mouse mode it will want
> to do the usual (nasty) tricks of confining the pointer to the window
> and warping it to the center after each pointer move so that mouse
> can be moved "endlessly' inside the window. This requires relative
> input, so emulating a mouse instead of say a drawing tablet, which
> breaks seamless mode.
> 
> So although the way the cursor is drawn typically does not require
> disabling seamless mode, the input problems do require disabling
> seamless mode.
> 
> In my experience (I have worked on VDI (spice) a couple of years),
> running a VM-viewer inside a VM is something which not a lot of
> people do. I guess the fact that this works quite poorly might
> be one of the reasons people do not do this.

Sorry. I meant that you run a desktop in a VM. Then you control that
desktop from some remote via VNC, and you still look at a VM's
VM-viewer.

You can replace VNC with any remote input thing. E.g. USB devices
plumbed directly through to the VM while still looking at the VM-viewer
for output.

Niche? Yes, quite likely.

> But all VMs I'm familiar with allow disabling seamless mode for
> compatibility with corner cases like this. This does lead to a
> seriously degraded user-experience though. For just using an
> office suit or browser inside a VM seamless mode really is a
> lot more pleasant for the user.  I still remember when seamless
> mode first got introduced, in the beginning it used to not
> always work and fixing the fallback to the confine + warp tricks
> then always was a serious itch which I tried to scratch quickly.
> Breaking seamless mode also is a good way to quickly accumulate
> a lot of bug reports from end users in my experience.

Btw. the warp trick is purely a legacy X11 thing, we have relative
motion interfaces nowadays at least in Wayland and I think also with
XI2?

But if you need to grab the pointer to allow endless relative motion,
then you grab, that hasn't changed from end user perspective.

I think one of the major reasons why Wayland pointer relative motion
and confinement extensions were designed was VM- etc. viewers, and of
course games.

> > What if display servers stop using the cursor plane completely, because
> > people may hit a case where a VM-viewer makes the wrong assumption about
> > which input device is associated to which cursor plane inside the VM?  
> 
> The confine + warp trick is typically the default mode and only
> if the guest indicates through e.g. a guest-agent process that
> seamless mode is supported then seamless mode is enabled.
> 
> IOW the VM is careful to not enable it when it might not work.

How would the guest-agent know? Does it check that there is literally
only one pointer input device and that comes from the VM-viewer?
Or does it limit seamless to white-listed display servers perhaps?

> OTOH most distros now ship with the necessary agents installed
> by default, so e.g. a F32 beta install will automatically use
> seamless mode under QEMU/KVM (spice display protocol) or
> VirtualBox.  If a user uses say F32 + a vnc-viewer inside it
> which needs to do confine + warp tricks then the user will have
> to disable seamless mode manually. In my prior experience
> working on VDI this is usually something which users are
> understanding of and not a problem.

Yeah, making the association between the input and the cursor plane
is unsolved.


Thanks,
pq

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Hans de Goede]

Hi,

On 3/19/20 4:16 PM, Pekka Paalanen wrote:
> On Thu, 19 Mar 2020 15:30:03 +0100
> Hans de Goede <hdegoede@redhat.com> wrote:

<snip>

>>> The only way to fix that is to stop Weston from putting non-cursor
>>> content on the cursor plane.
>>
>> Correct.
> 
> Is that something that should be done?
> 
> If the hotspot property also had a "disabled" value, then Weston could
> set the hotspot to disabled when it is using the cursor plane for
> non-cursor content and not lose the feature. And of course set hotspot
> correctly when it in fact is a cursor (but for what input?).

I believe cursor planes in the affected virtual gfx-cards do not
really have a mode where they can actually be used as a generic overlay
plane, certainly not in a useful manner (if anything works it will all
be software emulation), implementing a hotspot disabled mode would be
tricky and this would needs to be duplicated in all virtual-gfx cards
kms drivers.

If I understood Daniel's proposal for how to deal with this properly,
then only cursor planes which actually need them would get the new
hotspot x/y properties.  If we do that then Weston could use the
presence of the hotspot x/y properties to detect if it is dealing
with a proper hw plane which can also be used as a generic
plane; or a virtual-gfx cards cursor-plane, and then just not
bother with trying to use the plane as a generic hw plane.

Would that work?

<snip>

> I think one of the major reasons why Wayland pointer relative motion
> and confinement extensions were designed was VM- etc. viewers, and of
> course games.

ack.

>>> What if display servers stop using the cursor plane completely, because
>>> people may hit a case where a VM-viewer makes the wrong assumption about
>>> which input device is associated to which cursor plane inside the VM?
>>
>> The confine + warp trick is typically the default mode and only
>> if the guest indicates through e.g. a guest-agent process that
>> seamless mode is supported then seamless mode is enabled.
>>
>> IOW the VM is careful to not enable it when it might not work.
> 
> How would the guest-agent know? Does it check that there is literally
> only one pointer input device and that comes from the VM-viewer?
> Or does it limit seamless to white-listed display servers perhaps?

This varies between different solutions. E.g. the spice vdagent has
a desktop component which is started with a /etc/xdg/autostart/foo.desktop
file, assuming that it will work with any desktops honoring
/etc/xdg/autostart.

VirtualBox OTOH assumes it can use seamless mode as soon as
the vboxvideo driver loads inside the guest.

Regards,

Hans

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Simon Ser]

On Thursday, March 19, 2020 7:18 PM, Hans de Goede <hdegoede@redhat.com> wrote:

> > > > The only way to fix that is to stop Weston from putting non-cursor
> > > > content on the cursor plane.
> > >
> > > Correct.
> >
> > Is that something that should be done?
> > If the hotspot property also had a "disabled" value, then Weston could
> > set the hotspot to disabled when it is using the cursor plane for
> > non-cursor content and not lose the feature. And of course set hotspot
> > correctly when it in fact is a cursor (but for what input?).
>
> I believe cursor planes in the affected virtual gfx-cards do not
> really have a mode where they can actually be used as a generic overlay
> plane, certainly not in a useful manner (if anything works it will all
> be software emulation), implementing a hotspot disabled mode would be
> tricky and this would needs to be duplicated in all virtual-gfx cards
> kms drivers.
>
> If I understood Daniel's proposal for how to deal with this properly,
> then only cursor planes which actually need them would get the new
> hotspot x/y properties. If we do that then Weston could use the
> presence of the hotspot x/y properties to detect if it is dealing
> with a proper hw plane which can also be used as a generic
> plane; or a virtual-gfx cards cursor-plane, and then just not
> bother with trying to use the plane as a generic hw plane.
>
> Would that work?

That would need to at least be hidden behind a DRM capability, otherwise
it would break existing user-space ignoring the hotspot props (e.g.
current Weston).
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Hans de Goede]

Hi,

On 3/19/20 9:14 PM, Simon Ser wrote:
> On Thursday, March 19, 2020 7:18 PM, Hans de Goede <hdegoede@redhat.com> wrote:
> 
>>>>> The only way to fix that is to stop Weston from putting non-cursor
>>>>> content on the cursor plane.
>>>>
>>>> Correct.
>>>
>>> Is that something that should be done?
>>> If the hotspot property also had a "disabled" value, then Weston could
>>> set the hotspot to disabled when it is using the cursor plane for
>>> non-cursor content and not lose the feature. And of course set hotspot
>>> correctly when it in fact is a cursor (but for what input?).
>>
>> I believe cursor planes in the affected virtual gfx-cards do not
>> really have a mode where they can actually be used as a generic overlay
>> plane, certainly not in a useful manner (if anything works it will all
>> be software emulation), implementing a hotspot disabled mode would be
>> tricky and this would needs to be duplicated in all virtual-gfx cards
>> kms drivers.
>>
>> If I understood Daniel's proposal for how to deal with this properly,
>> then only cursor planes which actually need them would get the new
>> hotspot x/y properties. If we do that then Weston could use the
>> presence of the hotspot x/y properties to detect if it is dealing
>> with a proper hw plane which can also be used as a generic
>> plane; or a virtual-gfx cards cursor-plane, and then just not
>> bother with trying to use the plane as a generic hw plane.
>>
>> Would that work?
> 
> That would need to at least be hidden behind a DRM capability, otherwise
> it would break existing user-space ignoring the hotspot props (e.g.
> current Weston).

Current Weston is already broken, fixing that is what this whole
thread is about.

The virtual gfx-cards drivers simply must now the hotspot for things to
work; and a capability is not going to help here for 2 reasons:

1) Short of disabling seamless mode there is nothing the virtual
gfx-cards drivers can do when clients do not pass the hotspot info;
and in some cases they cannot even do this as it is under control
of a userspace agent process with its own channel to the hypervisor.

2) Most existing clients which obviously do not set this to-be-introduced
capability already pass the hotspot info using the DRM_IOCTL_MODE_CURSOR2
ioctl. Disabling seamless mode when this to-be-introduced capability is
not set would cause a huge regression for all these existing clients.

Regards,

Hans

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Simon Ser]

> >>> Is that something that should be done?
> >>> If the hotspot property also had a "disabled" value, then Weston could
> >>> set the hotspot to disabled when it is using the cursor plane for
> >>> non-cursor content and not lose the feature. And of course set hotspot
> >>> correctly when it in fact is a cursor (but for what input?).
> >>
> >> I believe cursor planes in the affected virtual gfx-cards do not
> >> really have a mode where they can actually be used as a generic overlay
> >> plane, certainly not in a useful manner (if anything works it will all
> >> be software emulation), implementing a hotspot disabled mode would be
> >> tricky and this would needs to be duplicated in all virtual-gfx cards
> >> kms drivers.
> >>
> >> If I understood Daniel's proposal for how to deal with this properly,
> >> then only cursor planes which actually need them would get the new
> >> hotspot x/y properties. If we do that then Weston could use the
> >> presence of the hotspot x/y properties to detect if it is dealing
> >> with a proper hw plane which can also be used as a generic
> >> plane; or a virtual-gfx cards cursor-plane, and then just not
> >> bother with trying to use the plane as a generic hw plane.
> >>
> >> Would that work?
> >
> > That would need to at least be hidden behind a DRM capability, otherwise
> > it would break existing user-space ignoring the hotspot props (e.g.
> > current Weston).
>
> Current Weston is already broken, fixing that is what this whole
> thread is about.
>
> The virtual gfx-cards drivers simply must now the hotspot for things to
> work; and a capability is not going to help here for 2 reasons:
>
> 1) Short of disabling seamless mode there is nothing the virtual
> gfx-cards drivers can do when clients do not pass the hotspot info;
> and in some cases they cannot even do this as it is under control
> of a userspace agent process with its own channel to the hypervisor.
>
> 2) Most existing clients which obviously do not set this to-be-introduced
> capability already pass the hotspot info using the DRM_IOCTL_MODE_CURSOR2
> ioctl. Disabling seamless mode when this to-be-introduced capability is
> not set would cause a huge regression for all these existing clients.

Compositors which don't support the hotspot prop are and will continue to break
seamless mode. Setting the prop will never be mandatory, because all user-space
today doesn't do it.

Since we need to update all user-space to add support for this prop, it would
be a good idea to recognize user-space that doesn't support it. Doing something
sensible with user-space that doesn't support the prop is up to the virtual
driver.

If we have a capability, the virtual driver could for instance not advertise a
cursor plane at all if user-space won't correctly set the hotspot anyway.

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Thomas Hellström (VMware)]

Hi,

On 3/19/20 10:07 PM, Simon Ser wrote:
>>>>> Is that something that should be done?
>>>>> If the hotspot property also had a "disabled" value, then Weston could
>>>>> set the hotspot to disabled when it is using the cursor plane for
>>>>> non-cursor content and not lose the feature. And of course set hotspot
>>>>> correctly when it in fact is a cursor (but for what input?).
>>>> I believe cursor planes in the affected virtual gfx-cards do not
>>>> really have a mode where they can actually be used as a generic overlay
>>>> plane, certainly not in a useful manner (if anything works it will all
>>>> be software emulation), implementing a hotspot disabled mode would be
>>>> tricky and this would needs to be duplicated in all virtual-gfx cards
>>>> kms drivers.
>>>>
>>>> If I understood Daniel's proposal for how to deal with this properly,
>>>> then only cursor planes which actually need them would get the new
>>>> hotspot x/y properties. If we do that then Weston could use the
>>>> presence of the hotspot x/y properties to detect if it is dealing
>>>> with a proper hw plane which can also be used as a generic
>>>> plane; or a virtual-gfx cards cursor-plane, and then just not
>>>> bother with trying to use the plane as a generic hw plane.
>>>>
>>>> Would that work?
>>> That would need to at least be hidden behind a DRM capability, otherwise
>>> it would break existing user-space ignoring the hotspot props (e.g.
>>> current Weston).
>> Current Weston is already broken, fixing that is what this whole
>> thread is about.
>>
>> The virtual gfx-cards drivers simply must now the hotspot for things to
>> work; and a capability is not going to help here for 2 reasons:
>>
>> 1) Short of disabling seamless mode there is nothing the virtual
>> gfx-cards drivers can do when clients do not pass the hotspot info;
>> and in some cases they cannot even do this as it is under control
>> of a userspace agent process with its own channel to the hypervisor.
>>
>> 2) Most existing clients which obviously do not set this to-be-introduced
>> capability already pass the hotspot info using the DRM_IOCTL_MODE_CURSOR2
>> ioctl. Disabling seamless mode when this to-be-introduced capability is
>> not set would cause a huge regression for all these existing clients.
> Compositors which don't support the hotspot prop are and will continue to break
> seamless mode. Setting the prop will never be mandatory, because all user-space
> today doesn't do it.
>
> Since we need to update all user-space to add support for this prop, it would
> be a good idea to recognize user-space that doesn't support it. Doing something
> sensible with user-space that doesn't support the prop is up to the virtual
> driver.
>
> If we have a capability, the virtual driver could for instance not advertise a
> cursor plane at all if user-space won't correctly set the hotspot anyway.

What compositors do we have around today that wouldn't want to support 
setting the hotspot correctly? Do we like to keep them that way, and do 
we want to encourage any new compositors to also not do this properly?

/Thomas


>
> _______________________________________________
> dri-devel mailing list
> dri-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel


_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Pekka Paalanen]

[-- Attachment #1.1: Type: text/plain, Size: 5362 bytes --]

On Thu, 19 Mar 2020 23:57:09 +0100
Thomas Hellström (VMware) <thomas_os@shipmail.org> wrote:

> Hi,
> 
> On 3/19/20 10:07 PM, Simon Ser wrote:
> >>>>> Is that something that should be done?
> >>>>> If the hotspot property also had a "disabled" value, then Weston could
> >>>>> set the hotspot to disabled when it is using the cursor plane for
> >>>>> non-cursor content and not lose the feature. And of course set hotspot
> >>>>> correctly when it in fact is a cursor (but for what input?).  
> >>>> I believe cursor planes in the affected virtual gfx-cards do not
> >>>> really have a mode where they can actually be used as a generic overlay
> >>>> plane, certainly not in a useful manner (if anything works it will all
> >>>> be software emulation), implementing a hotspot disabled mode would be
> >>>> tricky and this would needs to be duplicated in all virtual-gfx cards
> >>>> kms drivers.
> >>>>
> >>>> If I understood Daniel's proposal for how to deal with this properly,
> >>>> then only cursor planes which actually need them would get the new
> >>>> hotspot x/y properties. If we do that then Weston could use the
> >>>> presence of the hotspot x/y properties to detect if it is dealing
> >>>> with a proper hw plane which can also be used as a generic
> >>>> plane; or a virtual-gfx cards cursor-plane, and then just not
> >>>> bother with trying to use the plane as a generic hw plane.
> >>>>
> >>>> Would that work?  
> >>> That would need to at least be hidden behind a DRM capability, otherwise
> >>> it would break existing user-space ignoring the hotspot props (e.g.
> >>> current Weston).  
> >> Current Weston is already broken, fixing that is what this whole
> >> thread is about.
> >>
> >> The virtual gfx-cards drivers simply must now the hotspot for things to
> >> work; and a capability is not going to help here for 2 reasons:
> >>
> >> 1) Short of disabling seamless mode there is nothing the virtual
> >> gfx-cards drivers can do when clients do not pass the hotspot info;
> >> and in some cases they cannot even do this as it is under control
> >> of a userspace agent process with its own channel to the hypervisor.
> >>
> >> 2) Most existing clients which obviously do not set this to-be-introduced
> >> capability already pass the hotspot info using the DRM_IOCTL_MODE_CURSOR2
> >> ioctl. Disabling seamless mode when this to-be-introduced capability is
> >> not set would cause a huge regression for all these existing clients.  
> > Compositors which don't support the hotspot prop are and will continue to break
> > seamless mode. Setting the prop will never be mandatory, because all user-space
> > today doesn't do it.
> >
> > Since we need to update all user-space to add support for this prop, it would
> > be a good idea to recognize user-space that doesn't support it. Doing something
> > sensible with user-space that doesn't support the prop is up to the virtual
> > driver.
> >
> > If we have a capability, the virtual driver could for instance not advertise a
> > cursor plane at all if user-space won't correctly set the hotspot anyway.  
> 
> What compositors do we have around today that wouldn't want to support 
> setting the hotspot correctly? Do we like to keep them that way, and do 
> we want to encourage any new compositors to also not do this properly?

All compositors that sometimes want to use cursor planes for non-cursor
content. If a cursor plane does not actually have any performance
benefits, then somehow userspace would need to know to not use it in
that case.

I think the capability is a well-designed solution: a driver that
knows the cursor plane will not be always exactly like userspace
programs it should not advertise the cursor plane at all if userspace
does not set the new cap, and the cap is a promise that userspace will
set the hotspot correctly.

This cap will not break userspace that uses DRM_IOCTL_MODE_CURSOR2 with
correct hotspot info because that is a legacy KMS API and we are
talking about atomic here. The legacy ioctl can continue to ignore the
new cap.

It does mean that if legacy KMS userspace uses the cursor plane for
non-cursor content, it keeps being broken for drivers that mangle cursor
planes. I can certainly live with that.

It seems people are also forgetting the problem of associating the
cursor plane with an input device, so that whatever is looking to
mangle the cursor plane behind the KMS app's back would know how to do
it right.

My first thought for that is a new cursor plane property with the value
of major, minor of the kernel input device that userspace is using to
control the cursor plane. This property should be set by userspace only
when there is exactly one kernel input device it uses for controlling
the cursor plane. Setting this property to none/disabled would be a
clear indication that "seamless mode" would be unwanted. The DRM
driver or whatever it talks to could then check if the cursor plane is
indeed controlled by the input it so far has only assumed and
automatically choose correctly between seamless mode or not.

Anyway, these are just ideas. Use them if you wish, I won't NAK
anything if you don't.

Now you know what Weston currently does. If Weston needs to change, I
guess that is fine with me.


Thanks,
pq

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Thomas Hellström (VMware)]

On 3/20/20 10:13 AM, Pekka Paalanen wrote:
> On Thu, 19 Mar 2020 23:57:09 +0100
> Thomas Hellström (VMware) <thomas_os@shipmail.org> wrote:
>
>> Hi,
>>
>> On 3/19/20 10:07 PM, Simon Ser wrote:
>>>>>>> Is that something that should be done?
>>>>>>> If the hotspot property also had a "disabled" value, then Weston could
>>>>>>> set the hotspot to disabled when it is using the cursor plane for
>>>>>>> non-cursor content and not lose the feature. And of course set hotspot
>>>>>>> correctly when it in fact is a cursor (but for what input?).
>>>>>> I believe cursor planes in the affected virtual gfx-cards do not
>>>>>> really have a mode where they can actually be used as a generic overlay
>>>>>> plane, certainly not in a useful manner (if anything works it will all
>>>>>> be software emulation), implementing a hotspot disabled mode would be
>>>>>> tricky and this would needs to be duplicated in all virtual-gfx cards
>>>>>> kms drivers.
>>>>>>
>>>>>> If I understood Daniel's proposal for how to deal with this properly,
>>>>>> then only cursor planes which actually need them would get the new
>>>>>> hotspot x/y properties. If we do that then Weston could use the
>>>>>> presence of the hotspot x/y properties to detect if it is dealing
>>>>>> with a proper hw plane which can also be used as a generic
>>>>>> plane; or a virtual-gfx cards cursor-plane, and then just not
>>>>>> bother with trying to use the plane as a generic hw plane.
>>>>>>
>>>>>> Would that work?
>>>>> That would need to at least be hidden behind a DRM capability, otherwise
>>>>> it would break existing user-space ignoring the hotspot props (e.g.
>>>>> current Weston).
>>>> Current Weston is already broken, fixing that is what this whole
>>>> thread is about.
>>>>
>>>> The virtual gfx-cards drivers simply must now the hotspot for things to
>>>> work; and a capability is not going to help here for 2 reasons:
>>>>
>>>> 1) Short of disabling seamless mode there is nothing the virtual
>>>> gfx-cards drivers can do when clients do not pass the hotspot info;
>>>> and in some cases they cannot even do this as it is under control
>>>> of a userspace agent process with its own channel to the hypervisor.
>>>>
>>>> 2) Most existing clients which obviously do not set this to-be-introduced
>>>> capability already pass the hotspot info using the DRM_IOCTL_MODE_CURSOR2
>>>> ioctl. Disabling seamless mode when this to-be-introduced capability is
>>>> not set would cause a huge regression for all these existing clients.
>>> Compositors which don't support the hotspot prop are and will continue to break
>>> seamless mode. Setting the prop will never be mandatory, because all user-space
>>> today doesn't do it.
>>>
>>> Since we need to update all user-space to add support for this prop, it would
>>> be a good idea to recognize user-space that doesn't support it. Doing something
>>> sensible with user-space that doesn't support the prop is up to the virtual
>>> driver.
>>>
>>> If we have a capability, the virtual driver could for instance not advertise a
>>> cursor plane at all if user-space won't correctly set the hotspot anyway.
>> What compositors do we have around today that wouldn't want to support
>> setting the hotspot correctly? Do we like to keep them that way, and do
>> we want to encourage any new compositors to also not do this properly?
> All compositors that sometimes want to use cursor planes for non-cursor
> content. If a cursor plane does not actually have any performance
> benefits, then somehow userspace would need to know to not use it in
> that case.

Yes, but is that Weston only? Do we know about others?


>
> I think the capability is a well-designed solution: a driver that
> knows the cursor plane will not be always exactly like userspace
> programs it should not advertise the cursor plane at all if userspace
> does not set the new cap, and the cap is a promise that userspace will
> set the hotspot correctly.
>
> This cap will not break userspace that uses DRM_IOCTL_MODE_CURSOR2 with
> correct hotspot info because that is a legacy KMS API and we are
> talking about atomic here. The legacy ioctl can continue to ignore the
> new cap.
>
> It does mean that if legacy KMS userspace uses the cursor plane for
> non-cursor content, it keeps being broken for drivers that mangle cursor
> planes. I can certainly live with that.

What I'm afraid of is that people writing compositors will just leave 
that cap off, ignoring the huge benefit for virtual environments and 
future kms-based remoting solutions.

Couldn't compositors that see hotspot properties on a plane just ignore 
that plane for other stuff than cursors?


>
> It seems people are also forgetting the problem of associating the
> cursor plane with an input device, so that whatever is looking to
> mangle the cursor plane behind the KMS app's back would know how to do
> it right.
>
> My first thought for that is a new cursor plane property with the value
> of major, minor of the kernel input device that userspace is using to
> control the cursor plane. This property should be set by userspace only
> when there is exactly one kernel input device it uses for controlling
> the cursor plane. Setting this property to none/disabled would be a
> clear indication that "seamless mode" would be unwanted. The DRM
> driver or whatever it talks to could then check if the cursor plane is
> indeed controlled by the input it so far has only assumed and
> automatically choose correctly between seamless mode or not.

Are you sure this i really needed? VMware's SVGA device checks whether 
the guest cursor position and host cursor position are reasonably 
aligned, and if not, composites the guest cursor over the display plane. 
So if you, for example,  attach a passthrough USB mouse to the VM while 
running in seamless mode, things "just work". Similarly if you were to 
attach a kms-based vnc solution that behaved in the same way and that 
created a new input device, things would also look fine, except for 
temporary cursor jumps when you switch input device.

>
> Anyway, these are just ideas. Use them if you wish, I won't NAK
> anything if you don't.
>
> Now you know what Weston currently does. If Weston needs to change, I
> guess that is fine with me.
>
>
> Thanks,
> pq

Thanks,

Thomas



>
> _______________________________________________
> dri-devel mailing list
> dri-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel


_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Simon Ser]

On Friday, March 20, 2020 11:59 AM, Thomas Hellström <thomas_os@shipmail.org> wrote:

> On 3/20/20 10:13 AM, Pekka Paalanen wrote:
>
> > On Thu, 19 Mar 2020 23:57:09 +0100
> > Thomas Hellström (VMware) thomas_os@shipmail.org wrote:
> >
> > > Hi,
> > > On 3/19/20 10:07 PM, Simon Ser wrote:
> > >
> > > > > > > > Is that something that should be done?
> > > > > > > > If the hotspot property also had a "disabled" value, then Weston could
> > > > > > > > set the hotspot to disabled when it is using the cursor plane for
> > > > > > > > non-cursor content and not lose the feature. And of course set hotspot
> > > > > > > > correctly when it in fact is a cursor (but for what input?).
> > > > > > > > I believe cursor planes in the affected virtual gfx-cards do not
> > > > > > > > really have a mode where they can actually be used as a generic overlay
> > > > > > > > plane, certainly not in a useful manner (if anything works it will all
> > > > > > > > be software emulation), implementing a hotspot disabled mode would be
> > > > > > > > tricky and this would needs to be duplicated in all virtual-gfx cards
> > > > > > > > kms drivers.
> > > > > > >
> > > > > > > If I understood Daniel's proposal for how to deal with this properly,
> > > > > > > then only cursor planes which actually need them would get the new
> > > > > > > hotspot x/y properties. If we do that then Weston could use the
> > > > > > > presence of the hotspot x/y properties to detect if it is dealing
> > > > > > > with a proper hw plane which can also be used as a generic
> > > > > > > plane; or a virtual-gfx cards cursor-plane, and then just not
> > > > > > > bother with trying to use the plane as a generic hw plane.
> > > > > > > Would that work?
> > > > > > > That would need to at least be hidden behind a DRM capability, otherwise
> > > > > > > it would break existing user-space ignoring the hotspot props (e.g.
> > > > > > > current Weston).
> > > > > > > Current Weston is already broken, fixing that is what this whole
> > > > > > > thread is about.
> > > > >
> > > > > The virtual gfx-cards drivers simply must now the hotspot for things to
> > > > > work; and a capability is not going to help here for 2 reasons:
> > > > >
> > > > > 1.  Short of disabling seamless mode there is nothing the virtual
> > > > >     gfx-cards drivers can do when clients do not pass the hotspot info;
> > > > >     and in some cases they cannot even do this as it is under control
> > > > >     of a userspace agent process with its own channel to the hypervisor.
> > > > >
> > > > > 2.  Most existing clients which obviously do not set this to-be-introduced
> > > > >     capability already pass the hotspot info using the DRM_IOCTL_MODE_CURSOR2
> > > > >     ioctl. Disabling seamless mode when this to-be-introduced capability is
> > > > >     not set would cause a huge regression for all these existing clients.
> > > > >     Compositors which don't support the hotspot prop are and will continue to break
> > > > >     seamless mode. Setting the prop will never be mandatory, because all user-space
> > > > >     today doesn't do it.
> > > > >
> > > >
> > > > Since we need to update all user-space to add support for this prop, it would
> > > > be a good idea to recognize user-space that doesn't support it. Doing something
> > > > sensible with user-space that doesn't support the prop is up to the virtual
> > > > driver.
> > > > If we have a capability, the virtual driver could for instance not advertise a
> > > > cursor plane at all if user-space won't correctly set the hotspot anyway.
> > > > What compositors do we have around today that wouldn't want to support
> > > > setting the hotspot correctly? Do we like to keep them that way, and do
> > > > we want to encourage any new compositors to also not do this properly?
> > > > All compositors that sometimes want to use cursor planes for non-cursor
> > > > content. If a cursor plane does not actually have any performance
> > > > benefits, then somehow userspace would need to know to not use it in
> > > > that case.
>
> Yes, but is that Weston only? Do we know about others?

wlroots plans to do the same as Weston.

> > I think the capability is a well-designed solution: a driver that
> > knows the cursor plane will not be always exactly like userspace
> > programs it should not advertise the cursor plane at all if userspace
> > does not set the new cap, and the cap is a promise that userspace will
> > set the hotspot correctly.
> > This cap will not break userspace that uses DRM_IOCTL_MODE_CURSOR2 with
> > correct hotspot info because that is a legacy KMS API and we are
> > talking about atomic here. The legacy ioctl can continue to ignore the
> > new cap.
> > It does mean that if legacy KMS userspace uses the cursor plane for
> > non-cursor content, it keeps being broken for drivers that mangle cursor
> > planes. I can certainly live with that.
>
> What I'm afraid of is that people writing compositors will just leave
> that cap off, ignoring the huge benefit for virtual environments and
> future kms-based remoting solutions.
>
> Couldn't compositors that see hotspot properties on a plane just ignore
> that plane for other stuff than cursors?

KMS clients will need patching anyway to populate the new prop.
Enabling a cap is a one-liner.

Is the plan to intentionally break KMS clients to force them to add
support for the property? This increases the number of things KMS
clients need to do in order to correctly work. I think the KMS clients'
job is already hard enough as-is.

There will always exist user-space which doesn't set the prop. I'm not
just thinking about old compositors here: KMS clients are not limited
to just compositors. Think about Kodi for instance or DRM lease
clients.

> > It seems people are also forgetting the problem of associating the
> > cursor plane with an input device, so that whatever is looking to
> > mangle the cursor plane behind the KMS app's back would know how to do
> > it right.
> > My first thought for that is a new cursor plane property with the value
> > of major, minor of the kernel input device that userspace is using to
> > control the cursor plane. This property should be set by userspace only
> > when there is exactly one kernel input device it uses for controlling
> > the cursor plane. Setting this property to none/disabled would be a
> > clear indication that "seamless mode" would be unwanted. The DRM
> > driver or whatever it talks to could then check if the cursor plane is
> > indeed controlled by the input it so far has only assumed and
> > automatically choose correctly between seamless mode or not.
>
> Are you sure this i really needed? VMware's SVGA device checks whether
> the guest cursor position and host cursor position are reasonably
> aligned, and if not, composites the guest cursor over the display plane.
> So if you, for example,  attach a passthrough USB mouse to the VM while
> running in seamless mode, things "just work". Similarly if you were to
> attach a kms-based vnc solution that behaved in the same way and that
> created a new input device, things would also look fine, except for
> temporary cursor jumps when you switch input device.

Seems more like a workaround than a real solution.
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Thomas Hellström (VMware)]

On 3/20/20 12:27 PM, Simon Ser wrote:
> On Friday, March 20, 2020 11:59 AM, Thomas Hellström <thomas_os@shipmail.org> wrote:
>
>> On 3/20/20 10:13 AM, Pekka Paalanen wrote:
>>
>>> On Thu, 19 Mar 2020 23:57:09 +0100
>>> Thomas Hellström (VMware) thomas_os@shipmail.org wrote:
>>>
>>>> Hi,
>>>> On 3/19/20 10:07 PM, Simon Ser wrote:
>>>>
>>>>>>>>> Is that something that should be done?
>>>>>>>>> If the hotspot property also had a "disabled" value, then Weston could
>>>>>>>>> set the hotspot to disabled when it is using the cursor plane for
>>>>>>>>> non-cursor content and not lose the feature. And of course set hotspot
>>>>>>>>> correctly when it in fact is a cursor (but for what input?).
>>>>>>>>> I believe cursor planes in the affected virtual gfx-cards do not
>>>>>>>>> really have a mode where they can actually be used as a generic overlay
>>>>>>>>> plane, certainly not in a useful manner (if anything works it will all
>>>>>>>>> be software emulation), implementing a hotspot disabled mode would be
>>>>>>>>> tricky and this would needs to be duplicated in all virtual-gfx cards
>>>>>>>>> kms drivers.
>>>>>>>> If I understood Daniel's proposal for how to deal with this properly,
>>>>>>>> then only cursor planes which actually need them would get the new
>>>>>>>> hotspot x/y properties. If we do that then Weston could use the
>>>>>>>> presence of the hotspot x/y properties to detect if it is dealing
>>>>>>>> with a proper hw plane which can also be used as a generic
>>>>>>>> plane; or a virtual-gfx cards cursor-plane, and then just not
>>>>>>>> bother with trying to use the plane as a generic hw plane.
>>>>>>>> Would that work?
>>>>>>>> That would need to at least be hidden behind a DRM capability, otherwise
>>>>>>>> it would break existing user-space ignoring the hotspot props (e.g.
>>>>>>>> current Weston).
>>>>>>>> Current Weston is already broken, fixing that is what this whole
>>>>>>>> thread is about.
>>>>>> The virtual gfx-cards drivers simply must now the hotspot for things to
>>>>>> work; and a capability is not going to help here for 2 reasons:
>>>>>>
>>>>>> 1.  Short of disabling seamless mode there is nothing the virtual
>>>>>>      gfx-cards drivers can do when clients do not pass the hotspot info;
>>>>>>      and in some cases they cannot even do this as it is under control
>>>>>>      of a userspace agent process with its own channel to the hypervisor.
>>>>>>
>>>>>> 2.  Most existing clients which obviously do not set this to-be-introduced
>>>>>>      capability already pass the hotspot info using the DRM_IOCTL_MODE_CURSOR2
>>>>>>      ioctl. Disabling seamless mode when this to-be-introduced capability is
>>>>>>      not set would cause a huge regression for all these existing clients.
>>>>>>      Compositors which don't support the hotspot prop are and will continue to break
>>>>>>      seamless mode. Setting the prop will never be mandatory, because all user-space
>>>>>>      today doesn't do it.
>>>>>>
>>>>> Since we need to update all user-space to add support for this prop, it would
>>>>> be a good idea to recognize user-space that doesn't support it. Doing something
>>>>> sensible with user-space that doesn't support the prop is up to the virtual
>>>>> driver.
>>>>> If we have a capability, the virtual driver could for instance not advertise a
>>>>> cursor plane at all if user-space won't correctly set the hotspot anyway.
>>>>> What compositors do we have around today that wouldn't want to support
>>>>> setting the hotspot correctly? Do we like to keep them that way, and do
>>>>> we want to encourage any new compositors to also not do this properly?
>>>>> All compositors that sometimes want to use cursor planes for non-cursor
>>>>> content. If a cursor plane does not actually have any performance
>>>>> benefits, then somehow userspace would need to know to not use it in
>>>>> that case.
>> Yes, but is that Weston only? Do we know about others?
> wlroots plans to do the same as Weston.
>
>>> I think the capability is a well-designed solution: a driver that
>>> knows the cursor plane will not be always exactly like userspace
>>> programs it should not advertise the cursor plane at all if userspace
>>> does not set the new cap, and the cap is a promise that userspace will
>>> set the hotspot correctly.
>>> This cap will not break userspace that uses DRM_IOCTL_MODE_CURSOR2 with
>>> correct hotspot info because that is a legacy KMS API and we are
>>> talking about atomic here. The legacy ioctl can continue to ignore the
>>> new cap.
>>> It does mean that if legacy KMS userspace uses the cursor plane for
>>> non-cursor content, it keeps being broken for drivers that mangle cursor
>>> planes. I can certainly live with that.
>> What I'm afraid of is that people writing compositors will just leave
>> that cap off, ignoring the huge benefit for virtual environments and
>> future kms-based remoting solutions.
>>
>> Couldn't compositors that see hotspot properties on a plane just ignore
>> that plane for other stuff than cursors?
> KMS clients will need patching anyway to populate the new prop.
> Enabling a cap is a one-liner.
>
> Is the plan to intentionally break KMS clients to force them to add
> support for the property?

We can't do that. As Hans pointed out, some KMS clients are already 
broken, and hardware planes that doesn't fiddle with the cursor (on 
which Weston works now) don't have to expose the hotspot properties.


> This increases the number of things KMS
> clients need to do in order to correctly work. I think the KMS clients'
> job is already hard enough as-is.

That's exactly the kind of thinking that I'm afraid will leave virtual- 
and remoting solutions unsupported.

> There will always exist user-space which doesn't set the prop. I'm not
> just thinking about old compositors here: KMS clients are not limited
> to just compositors. Think about Kodi for instance or DRM lease
> clients.
>
>>> It seems people are also forgetting the problem of associating the
>>> cursor plane with an input device, so that whatever is looking to
>>> mangle the cursor plane behind the KMS app's back would know how to do
>>> it right.
>>> My first thought for that is a new cursor plane property with the value
>>> of major, minor of the kernel input device that userspace is using to
>>> control the cursor plane. This property should be set by userspace only
>>> when there is exactly one kernel input device it uses for controlling
>>> the cursor plane. Setting this property to none/disabled would be a
>>> clear indication that "seamless mode" would be unwanted. The DRM
>>> driver or whatever it talks to could then check if the cursor plane is
>>> indeed controlled by the input it so far has only assumed and
>>> automatically choose correctly between seamless mode or not.
>> Are you sure this i really needed? VMware's SVGA device checks whether
>> the guest cursor position and host cursor position are reasonably
>> aligned, and if not, composites the guest cursor over the display plane.
>> So if you, for example,  attach a passthrough USB mouse to the VM while
>> running in seamless mode, things "just work". Similarly if you were to
>> attach a kms-based vnc solution that behaved in the same way and that
>> created a new input device, things would also look fine, except for
>> temporary cursor jumps when you switch input device.
> Seems more like a workaround than a real solution.

That may be the case, but still it works and if you have multiple 
clients it always allows the active client run in seamless mode. 
Besides, if people think supporting KMS hotspots is already hard enough, 
the chance of having them supplying the correct input device is small.

/Thomas

> _______________________________________________
> dri-devel mailing list
> dri-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel


_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Thomas Hellström (VMware)]

On 3/20/20 12:47 PM, Thomas Hellström (VMware) wrote:
> On 3/20/20 12:27 PM, Simon Ser wrote:
>> On Friday, March 20, 2020 11:59 AM, Thomas Hellström 
>> <thomas_os@shipmail.org> wrote:
>>
>>> On 3/20/20 10:13 AM, Pekka Paalanen wrote:
>>>
>>>>> It seems people are also forgetting the problem of associating the
>>>>> cursor plane with an input device, so that whatever is looking to
>>>>> mangle the cursor plane behind the KMS app's back would know how 
>>>>> to do
>>>>> it right.
>>>>> My first thought for that is a new cursor plane property with the 
>>>>> value
>>>>> of major, minor of the kernel input device that userspace is using to
>>>>> control the cursor plane. This property should be set by userspace 
>>>>> only
>>>>> when there is exactly one kernel input device it uses for controlling
>>>>> the cursor plane. Setting this property to none/disabled would be a
>>>>> clear indication that "seamless mode" would be unwanted. The DRM
>>>>> driver or whatever it talks to could then check if the cursor 
>>>>> plane is
>>>>> indeed controlled by the input it so far has only assumed and
>>>>> automatically choose correctly between seamless mode or not.
>>>> Are you sure this i really needed? VMware's SVGA device checks whether
>>>> the guest cursor position and host cursor position are reasonably
>>>> aligned, and if not, composites the guest cursor over the display 
>>>> plane.
>>>> So if you, for example,  attach a passthrough USB mouse to the VM 
>>>> while
>>>> running in seamless mode, things "just work". Similarly if you were to
>>>> attach a kms-based vnc solution that behaved in the same way and that
>>>> created a new input device, things would also look fine, except for
>>>> temporary cursor jumps when you switch input device.
>>> Seems more like a workaround than a real solution.
>
> That may be the case, but still it works and if you have multiple 
> clients it always allows the active client run in seamless mode. 
> Besides, if people think supporting KMS hotspots is already hard 
> enough, the chance of having them supplying the correct input device 
> is small.
>
> /Thomas

To be clear about this, I'm not *against* a property associating a 
device with a cursor plane as long as it's just a hint. However in the 
VMware case, it's unlikely that we will try to implement any kernel 
driver support for it since we have a working solution and it also may 
not be possible. (Some remoting solutions only work when seamless mode 
is enabled).

/Thomas


>
>> _______________________________________________
>> dri-devel mailing list
>> dri-devel@lists.freedesktop.org
>> https://lists.freedesktop.org/mailman/listinfo/dri-devel
>
>

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel