vnc clipboard support

vnc clipboard support

[Gerd Hoffmann]

  Hi folks,

I'm looking for a good way to implement cut+paste support for vnc.

The vnc core protocol has support for text/plain cut+paste, and there
is an extension adding support for other formats.  That'll cover one
part of the problem, exchanging cut+paste data between vnc client and
qemu vnc server.

The tricky part is the second: the guest <=> qemu communication.
I see basically two possible approaches here:

  (1) Have some guest agent (spice does it that way).
      Advantage: more flexible, allows more features.
      Disadvantage: requires agent in the guest.
  
  (2) Send text as key events.
      Advantage: no guest agent needed.
      Disadvantage: is translated by guests keyboard map, so qemu
      needs to know the map for proper char -> key event translation.
      Only works for text/plain and only for chars you can easily
      type, anything needing input methods (emoji 😊 for example)
      isn't going to fly.

I think that (1) is clearly the better way.  Given that the agent
would need to run in user wayland/xorg session context the existing
qemu-guest-agent will not work.  Also linking against some UI library
like gtk3 for clipboard handling is not something we want for the
qemu-guest-agent.  So we need another one, I'll name it
qemu-clipboard-agent for the rest of this mail.  And we need a
communication channel.

I'd tend to model the qemu-clipboard-agent simliar to the
qemu-guest-agent, i.e. have some stream as communication path and run
some stream protocol over it.

Stream options I see are (in order of personal preference):

  (1) New virtio-serial port.  virtio-serial likely is there anyway
      for the qemu-guest-agent ...

  (2) Have qemu-clipboard-agent and qemu-guest-agent share the agent
      channel, i.e. qemu-clipboard-agent will proxy everything through
      qemu-guest-agent (spice does it that way).

Protocol options I see are (not sure yet which to prefer, need to have
a closer look at the candidates):

  (1) Add clipboard commands to QMP and use these.

  (2) Reuse the clipboard bits of the vnc protocol (forward
      VNC_MSG_CLIENT_CUT_TEXT messages to the guest agent)

  (3) Reuse the clipboard bits of the spice-agent protocol.

  (4) Reuse the clipboard bits of the wayland protocol.

Once we have sorted the qemu <-> guest communication path it should be
possible to also hook up other UIs (specifically gtk) without too much
effort.  Which probably makes (2) a rather poor choice.

Comments?
Suggestions?
Other ideas?

take care,
  Gerd

Re: vnc clipboard support

[Daniel P. Berrangé]

On Thu, Jan 28, 2021 at 06:12:24PM +0100, Gerd Hoffmann wrote:
>   Hi folks,
> 
> I'm looking for a good way to implement cut+paste support for vnc.
> 
> The vnc core protocol has support for text/plain cut+paste, and there
> is an extension adding support for other formats.  That'll cover one
> part of the problem, exchanging cut+paste data between vnc client and
> qemu vnc server.

NB the VNC cut+paste impl is technically undefined  for non-7-bit ascii
data. In reality though I'd assume most servers/clients and up sending
UTF8. 

> The tricky part is the second: the guest <=> qemu communication.
> I see basically two possible approaches here:
> 
>   (1) Have some guest agent (spice does it that way).
>       Advantage: more flexible, allows more features.
>       Disadvantage: requires agent in the guest.



>   
>   (2) Send text as key events.
>       Advantage: no guest agent needed.
>       Disadvantage: is translated by guests keyboard map, so qemu
>       needs to know the map for proper char -> key event translation.
>       Only works for text/plain and only for chars you can easily
>       type, anything needing input methods (emoji 😊 for example)
>       isn't going to fly.
> 
> I think that (1) is clearly the better way.

I agree - the agent based approach also lets you have a bi-directional
solution. ie copy in the guest, paste in the host, which you can't
do by simulating key presses.  Simulating keypress is insanely slow
too, so useless for more than 100 bytes of text really.

>                                              Given that the agent
> would need to run in user wayland/xorg session context the existing
> qemu-guest-agent will not work.  Also linking against some UI library
> like gtk3 for clipboard handling is not something we want for the
> qemu-guest-agent.  So we need another one, I'll name it
> qemu-clipboard-agent for the rest of this mail.  And we need a
> communication channel.

I've never looked at the spice-guest-agent code, but I wonder if there
is any scope for re-using it with VNC, or is it too closely tangled
up with SPICE ?  The advantage of reuse is that all existing guest
OS with spice-guest-agent installed will "just work".

> I'd tend to model the qemu-clipboard-agent simliar to the
> qemu-guest-agent, i.e. have some stream as communication path and run
> some stream protocol over it.
> 
> Stream options I see are (in order of personal preference):
> 
>   (1) New virtio-serial port.  virtio-serial likely is there anyway
>       for the qemu-guest-agent ...
> 
>   (2) Have qemu-clipboard-agent and qemu-guest-agent share the agent
>       channel, i.e. qemu-clipboard-agent will proxy everything through
>       qemu-guest-agent (spice does it that way).
> 
> Protocol options I see are (not sure yet which to prefer, need to have
> a closer look at the candidates):
> 
>   (1) Add clipboard commands to QMP and use these.
> 
>   (2) Reuse the clipboard bits of the vnc protocol (forward
>       VNC_MSG_CLIENT_CUT_TEXT messages to the guest agent)
> 
>   (3) Reuse the clipboard bits of the spice-agent protocol.
> 
>   (4) Reuse the clipboard bits of the wayland protocol.
> 
> Once we have sorted the qemu <-> guest communication path it should be
> possible to also hook up other UIs (specifically gtk) without too much
> effort.  Which probably makes (2) a rather poor choice.

I think the VNC clipboard stuff is too crude to base a long term
design on. I figure we'll want to be at least inspired by (3) or
(4).

> Comments?
> Suggestions?
> Other ideas?

The QEMU guest agent is a single system level agent. IIUC, with SPICE
you have a single system level agent, and then zero or more session
level agents - one per desktop login basically. I assume we'll want
to support the latter too, at least from a design level, even if we
don't implemnet multi session straight away.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: vnc clipboard support

[Gerd Hoffmann]

On Thu, Jan 28, 2021 at 05:35:04PM +0000, Daniel P. Berrangé wrote:
> On Thu, Jan 28, 2021 at 06:12:24PM +0100, Gerd Hoffmann wrote:
> >   Hi folks,
> > 
> > I'm looking for a good way to implement cut+paste support for vnc.
> > 
> > The vnc core protocol has support for text/plain cut+paste, and there
> > is an extension adding support for other formats.  That'll cover one
> > part of the problem, exchanging cut+paste data between vnc client and
> > qemu vnc server.
> 
> NB the VNC cut+paste impl is technically undefined  for non-7-bit ascii
> data. In reality though I'd assume most servers/clients and up sending
> UTF8. 

rfbspec says it is latin1 for classic cut_text messages and utf-8 with
the extension.  In reality I've seen tigervnc send utf-8 even in classic
cut_text messages.  Given running in utf-8 locale is sort-of standard
these days I'm not surprised.

> I've never looked at the spice-guest-agent code, but I wonder if there
> is any scope for re-using it with VNC, or is it too closely tangled
> up with SPICE ?  The advantage of reuse is that all existing guest
> OS with spice-guest-agent installed will "just work".

Neat idea.  Need to check.  spice tunnels alot of stuff through the
agent channel, but IIRC there is some kind of feature negotiation so
maybe it isn't that hard to cherry-pick cut+paste and say "not
supported" for everything else.

> The QEMU guest agent is a single system level agent. IIUC, with SPICE
> you have a single system level agent, and then zero or more session
> level agents - one per desktop login basically.

Ok, easy multi-session support is one advantage of the proxy model.

take care,
  Gerd

Re: vnc clipboard support

[Christophe de Dinechin]

> On 28 Jan 2021, at 18:12, Gerd Hoffmann <kraxel@redhat.com> wrote:
> 
>  Hi folks,
> 
> I'm looking for a good way to implement cut+paste support for vnc.
> 
> The vnc core protocol has support for text/plain cut+paste, and there
> is an extension adding support for other formats.  That'll cover one
> part of the problem, exchanging cut+paste data between vnc client and
> qemu vnc server.
> 
> The tricky part is the second: the guest <=> qemu communication.
> I see basically two possible approaches here:
> 
>  (1) Have some guest agent (spice does it that way).
>      Advantage: more flexible, allows more features.
>      Disadvantage: requires agent in the guest.

What about running the option to relay data to a VNC server in the
guest if there is one running? In other words, using an existing
VNC server as an agent, with the option of having a stripped-down,
clipboard only VNC server as a later optimization.

The benefit I see is that the work of adjusting between the VNC
clipboard protocol and the local OS clipboard is something that is
likely already done when porting VNC, so you leverage that, even
for exotic operating systems. And in many cases, VNC may already
be part of the distro / OS, or at least easy to find.

Then it's a matter of choice whether you want to talk to some
real guest network, or use a character device and route data
through some other non-network path.

> 
>  (2) Send text as key events.
>      Advantage: no guest agent needed.
>      Disadvantage: is translated by guests keyboard map, so qemu
>      needs to know the map for proper char -> key event translation.
>      Only works for text/plain and only for chars you can easily
>      type, anything needing input methods (emoji 😊 for example)
>      isn't going to fly.


> 
> I think that (1) is clearly the better way.  Given that the agent
> would need to run in user wayland/xorg session context the existing
> qemu-guest-agent will not work.  Also linking against some UI library
> like gtk3 for clipboard handling is not something we want for the
> qemu-guest-agent.  So we need another one, I'll name it
> qemu-clipboard-agent for the rest of this mail.  And we need a
> communication channel.
> 
> I'd tend to model the qemu-clipboard-agent simliar to the
> qemu-guest-agent, i.e. have some stream as communication path and run
> some stream protocol over it.
> 
> Stream options I see are (in order of personal preference):
> 
>  (1) New virtio-serial port.  virtio-serial likely is there anyway
>      for the qemu-guest-agent ...
> 
>  (2) Have qemu-clipboard-agent and qemu-guest-agent share the agent
>      channel, i.e. qemu-clipboard-agent will proxy everything through
>      qemu-guest-agent (spice does it that way).
> 
> Protocol options I see are (not sure yet which to prefer, need to have
> a closer look at the candidates):
> 
>  (1) Add clipboard commands to QMP and use these.
> 
>  (2) Reuse the clipboard bits of the vnc protocol (forward
>      VNC_MSG_CLIENT_CUT_TEXT messages to the guest agent)
> 
>  (3) Reuse the clipboard bits of the spice-agent protocol.
> 
>  (4) Reuse the clipboard bits of the wayland protocol.
> 
> Once we have sorted the qemu <-> guest communication path it should be
> possible to also hook up other UIs (specifically gtk) without too much
> effort.  Which probably makes (2) a rather poor choice.
> 
> Comments?
> Suggestions?
> Other ideas?
> 
> take care,
>  Gerd
> 
> 

Re: vnc clipboard support

[Gerd Hoffmann]

  Hi,

> >  (1) Have some guest agent (spice does it that way).
> >      Advantage: more flexible, allows more features.
> >      Disadvantage: requires agent in the guest.
> 
> What about running the option to relay data to a VNC server in the
> guest if there is one running? In other words, using an existing
> VNC server as an agent, with the option of having a stripped-down,
> clipboard only VNC server as a later optimization.

Well, if you run Xvnc in the guest anyway why use the qemu vnc server
in the first place?

take care,
  Gerd

Re: vnc clipboard support

[Christophe de Dinechin]

> On 29 Jan 2021, at 09:03, Gerd Hoffmann <kraxel@redhat.com> wrote:
> 
>  Hi,
> 
>>> (1) Have some guest agent (spice does it that way).
>>>     Advantage: more flexible, allows more features.
>>>     Disadvantage: requires agent in the guest.
>> 
>> What about running the option to relay data to a VNC server in the
>> guest if there is one running? In other words, using an existing
>> VNC server as an agent, with the option of having a stripped-down,
>> clipboard only VNC server as a later optimization.
> 
> Well, if you run Xvnc in the guest anyway why use the qemu vnc server
> in the first place?

I assume that if you use the qemu VNC, it's because you you don't want to
run Xvnc on some external network, or care about accessing the guest
before Xvnc can be launched. There can be many reasons.

Again, I want to make it clear that my suggestion is _not_ simply to access
the existing Xvnc as is, but rather to stick with some VNC server code to handle
the clipboard if / when possible.

Let me try to imagine a scenario, where I'll use a macOS guest intentionally
to clarify what I was thinking about.

- During early boot, there is no in-guest VNC server, so to address that,
you connect to the qemu VNC. At this stage, all screen refresh is handled
by the qemu VNC, and the best you can do if you want to support any
kind of copy-paste is to convert it to virtual keystrokes. The same would
be true for Linux on a text console.

- Then comes up you macOS guest, and it still has no VNC port open,
so you are stuck with qemu-vnc doing all the work. But now you can
enable screen sharing, and that launches the Apple-maintained macOS
VNC server.

- Let's assume for illustration that this listens on some private network
that qemu can access, but that is not visible externally. In this case,
you could not VNC to the guest, but you can still VNC to qemu.

- What I'm suggesting is that qemu-vnc could then switch to simply
relaying VNC traffic to that in-guest server. You'd get the smart update
algorithm that Apple has put in place to deal with transparency and the
like, as well as a level of guest OS integration that would otherwise be
much harder to replicate.

I believe that the same holds true with X as well. If you can activate or
detect an Xvnc, you are much better off using _that_ to deal with the
idiosyncrasies of Xclipboard rather than try to convert keycodes.

If you don't have it, you fallback to virtual keystrokes, which you can't
dispense with because of the early boot case, but which will at best
deal with simple ASCII (making that work with a clipboard containing
tabs and Control-C/Control-V characters is left as an exercise for the
reader  ;-)

Finally, an optimization I alluded to is to have an agent which is basically
a stripped-down VNC server dealing with only the clipboard aspect, and
that is your agent. In other words, you don't' invent yet another protocol,
but you furiously copy-paste the existing VNC code.


Hope this sounds a bit less crazy explained that way…
Thanks
Christophe

Re: vnc clipboard support

[Daniel P. Berrangé]

On Fri, Jan 29, 2021 at 11:50:10AM +0100, Christophe de Dinechin wrote:
> 
> 
> > On 29 Jan 2021, at 09:03, Gerd Hoffmann <kraxel@redhat.com> wrote:
> > 
> >  Hi,
> > 
> >>> (1) Have some guest agent (spice does it that way).
> >>>     Advantage: more flexible, allows more features.
> >>>     Disadvantage: requires agent in the guest.
> >> 
> >> What about running the option to relay data to a VNC server in the
> >> guest if there is one running? In other words, using an existing
> >> VNC server as an agent, with the option of having a stripped-down,
> >> clipboard only VNC server as a later optimization.
> > 
> > Well, if you run Xvnc in the guest anyway why use the qemu vnc server
> > in the first place?
> 
> I assume that if you use the qemu VNC, it's because you you don't want to
> run Xvnc on some external network, or care about accessing the guest
> before Xvnc can be launched. There can be many reasons.
> 
> Again, I want to make it clear that my suggestion is _not_ simply to access
> the existing Xvnc as is, but rather to stick with some VNC server code to handle
> the clipboard if / when possible.
> 
> Let me try to imagine a scenario, where I'll use a macOS guest intentionally
> to clarify what I was thinking about.
> 
> - During early boot, there is no in-guest VNC server, so to address that,
> you connect to the qemu VNC. At this stage, all screen refresh is handled
> by the qemu VNC, and the best you can do if you want to support any
> kind of copy-paste is to convert it to virtual keystrokes. The same would
> be true for Linux on a text console.
> 
> - Then comes up you macOS guest, and it still has no VNC port open,
> so you are stuck with qemu-vnc doing all the work. But now you can
> enable screen sharing, and that launches the Apple-maintained macOS
> VNC server.
> 
> - Let's assume for illustration that this listens on some private network
> that qemu can access, but that is not visible externally. In this case,
> you could not VNC to the guest, but you can still VNC to qemu.
> 
> - What I'm suggesting is that qemu-vnc could then switch to simply
> relaying VNC traffic to that in-guest server. You'd get the smart update
> algorithm that Apple has put in place to deal with transparency and the
> like, as well as a level of guest OS integration that would otherwise be
> much harder to replicate.

IMHO that's an awful lot of complexity to add to the system
that isn't especially useful and this opens up new attack
vectors for the guest to exploit the host.

If people have VNC/RDP/whatever configured inside their guest
OS, then there's really little to no reason for them to want
to connect to the QEMU VNC server, as viewing initial bootup
phase is not required in normal case. The only time such
people will need to use the QEMU VNC server is if the guest
OS has broken in some way before it fully booted and thus failed
to start the guest VNC server. There is no guest VNC server
to hand off to in this scenario.

The value of the QEMU host side VNC server is that it works
for all possible guest OS, no matter whether they are running
normally or broken, regardless of what the guest admin has
configured in terms of guest level remote desktop.

IOW, if you have a guest remote desktop solution you'll just
use that 99% of the time. If you don't have that, then you'll
use QEMU VNC/SPICE server, and there won't be anything in the
guest for that to proxy to/from.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: vnc clipboard support

[Christophe de Dinechin]

[-- Attachment #1: Type: text/plain, Size: 4478 bytes --]



> On 29 Jan 2021, at 12:08, Daniel P. Berrangé <berrange@redhat.com> wrote:
> 
> On Fri, Jan 29, 2021 at 11:50:10AM +0100, Christophe de Dinechin wrote:
>> 
>> 
>>> On 29 Jan 2021, at 09:03, Gerd Hoffmann <kraxel@redhat.com> wrote:
>>> 
>>> Hi,
>>> 
>>>>> (1) Have some guest agent (spice does it that way).
>>>>>    Advantage: more flexible, allows more features.
>>>>>    Disadvantage: requires agent in the guest.
>>>> 
>>>> What about running the option to relay data to a VNC server in the
>>>> guest if there is one running? In other words, using an existing
>>>> VNC server as an agent, with the option of having a stripped-down,
>>>> clipboard only VNC server as a later optimization.
>>> 
>>> Well, if you run Xvnc in the guest anyway why use the qemu vnc server
>>> in the first place?
>> 
>> I assume that if you use the qemu VNC, it's because you you don't want to
>> run Xvnc on some external network, or care about accessing the guest
>> before Xvnc can be launched. There can be many reasons.
>> 
>> Again, I want to make it clear that my suggestion is _not_ simply to access
>> the existing Xvnc as is, but rather to stick with some VNC server code to handle
>> the clipboard if / when possible.
>> 
>> Let me try to imagine a scenario, where I'll use a macOS guest intentionally
>> to clarify what I was thinking about.
>> 
>> - During early boot, there is no in-guest VNC server, so to address that,
>> you connect to the qemu VNC. At this stage, all screen refresh is handled
>> by the qemu VNC, and the best you can do if you want to support any
>> kind of copy-paste is to convert it to virtual keystrokes. The same would
>> be true for Linux on a text console.
>> 
>> - Then comes up you macOS guest, and it still has no VNC port open,
>> so you are stuck with qemu-vnc doing all the work. But now you can
>> enable screen sharing, and that launches the Apple-maintained macOS
>> VNC server.
>> 
>> - Let's assume for illustration that this listens on some private network
>> that qemu can access, but that is not visible externally. In this case,
>> you could not VNC to the guest, but you can still VNC to qemu.
>> 
>> - What I'm suggesting is that qemu-vnc could then switch to simply
>> relaying VNC traffic to that in-guest server. You'd get the smart update
>> algorithm that Apple has put in place to deal with transparency and the
>> like, as well as a level of guest OS integration that would otherwise be
>> much harder to replicate.
> 
> IMHO that's an awful lot of complexity to add to the system
> that isn't especially useful and this opens up new attack
> vectors for the guest to exploit the host.
> 
> If people have VNC/RDP/whatever configured inside their guest
> OS, then there's really little to no reason for them to want
> to connect to the QEMU VNC server, as viewing initial bootup
> phase is not required in normal case. The only time such
> people will need to use the QEMU VNC server is if the guest
> OS has broken in some way before it fully booted and thus failed
> to start the guest VNC server. There is no guest VNC server
> to hand off to in this scenario.

It's a matter of what you want to do with that qemu vnc.

If it's only a backup when there's nothing in the guest to help,
then maybe trying to support copy-paste is not a good idea.

If it's a standard go-to access point for connecting to your
guest, then making it smart is hard, but useful.

> 
> The value of the QEMU host side VNC server is that it works
> for all possible guest OS, no matter whether they are running
> normally or broken, regardless of what the guest admin has
> configured in terms of guest level remote desktop.

Understood.

The downside is that there are things it can't do. It cannot correctly
determine the keyboard map, because that's controlled in software
in the guest.

Unless we para-virtualize the keyboard?

> 
> IOW, if you have a guest remote desktop solution you'll just
> use that 99% of the time. If you don't have that, then you'll
> use QEMU VNC/SPICE server, and there won't be anything in the
> guest for that to proxy to/from.

If really the assumption is there is nothing in the guest to help
us, then I'd say "paste" should come up with a warning "do you want
me to try and translate that into keystrokes", and I don't see how to
implement copy from a graphical display without OCR…


Cheers
Christophe


[-- Attachment #2: Type: text/html, Size: 22093 bytes --]

Re: vnc clipboard support

[Daniel P. Berrangé]

On Fri, Jan 29, 2021 at 03:19:45PM +0100, Christophe de Dinechin wrote:
> 
> 
> > On 29 Jan 2021, at 12:08, Daniel P. Berrangé <berrange@redhat.com> wrote:
> > 
> > On Fri, Jan 29, 2021 at 11:50:10AM +0100, Christophe de Dinechin wrote:
> >> 
> >> 
> >>> On 29 Jan 2021, at 09:03, Gerd Hoffmann <kraxel@redhat.com> wrote:
> >>> 
> >>> Hi,
> >>> 
> >>>>> (1) Have some guest agent (spice does it that way).
> >>>>>    Advantage: more flexible, allows more features.
> >>>>>    Disadvantage: requires agent in the guest.
> >>>> 
> >>>> What about running the option to relay data to a VNC server in the
> >>>> guest if there is one running? In other words, using an existing
> >>>> VNC server as an agent, with the option of having a stripped-down,
> >>>> clipboard only VNC server as a later optimization.
> >>> 
> >>> Well, if you run Xvnc in the guest anyway why use the qemu vnc server
> >>> in the first place?
> >> 
> >> I assume that if you use the qemu VNC, it's because you you don't want to
> >> run Xvnc on some external network, or care about accessing the guest
> >> before Xvnc can be launched. There can be many reasons.
> >> 
> >> Again, I want to make it clear that my suggestion is _not_ simply to access
> >> the existing Xvnc as is, but rather to stick with some VNC server code to handle
> >> the clipboard if / when possible.
> >> 
> >> Let me try to imagine a scenario, where I'll use a macOS guest intentionally
> >> to clarify what I was thinking about.
> >> 
> >> - During early boot, there is no in-guest VNC server, so to address that,
> >> you connect to the qemu VNC. At this stage, all screen refresh is handled
> >> by the qemu VNC, and the best you can do if you want to support any
> >> kind of copy-paste is to convert it to virtual keystrokes. The same would
> >> be true for Linux on a text console.
> >> 
> >> - Then comes up you macOS guest, and it still has no VNC port open,
> >> so you are stuck with qemu-vnc doing all the work. But now you can
> >> enable screen sharing, and that launches the Apple-maintained macOS
> >> VNC server.
> >> 
> >> - Let's assume for illustration that this listens on some private network
> >> that qemu can access, but that is not visible externally. In this case,
> >> you could not VNC to the guest, but you can still VNC to qemu.
> >> 
> >> - What I'm suggesting is that qemu-vnc could then switch to simply
> >> relaying VNC traffic to that in-guest server. You'd get the smart update
> >> algorithm that Apple has put in place to deal with transparency and the
> >> like, as well as a level of guest OS integration that would otherwise be
> >> much harder to replicate.
> > 
> > IMHO that's an awful lot of complexity to add to the system
> > that isn't especially useful and this opens up new attack
> > vectors for the guest to exploit the host.
> > 
> > If people have VNC/RDP/whatever configured inside their guest
> > OS, then there's really little to no reason for them to want
> > to connect to the QEMU VNC server, as viewing initial bootup
> > phase is not required in normal case. The only time such
> > people will need to use the QEMU VNC server is if the guest
> > OS has broken in some way before it fully booted and thus failed
> > to start the guest VNC server. There is no guest VNC server
> > to hand off to in this scenario.
> 
> It's a matter of what you want to do with that qemu vnc.
> 
> If it's only a backup when there's nothing in the guest to help,
> then maybe trying to support copy-paste is not a good idea.
> 
> If it's a standard go-to access point for connecting to your
> guest, then making it smart is hard, but useful.
> 
> > 
> > The value of the QEMU host side VNC server is that it works
> > for all possible guest OS, no matter whether they are running
> > normally or broken, regardless of what the guest admin has
> > configured in terms of guest level remote desktop.
> 
> Understood.
> 
> The downside is that there are things it can't do. It cannot correctly
> determine the keyboard map, because that's controlled in software
> in the guest.

There is no need for the remote desktop server to care about the
keymap. The remote client takes scancodes and passes them to the
server, which then passes them to the guest.

The person connected to the remote server simply has to configure
their guest OS keymap to match the physical keyboard they currently
have plugged in.

The only reason a remote server would need to know the keymap is
if it was trying to translate from keycodes back into scancodes.
This is what VNC protocol had to do originally, but VNC was since
extended to pass raw scancodes instead of keycodes, precisely to
avoid needing to care about keymaps.

> > IOW, if you have a guest remote desktop solution you'll just
> > use that 99% of the time. If you don't have that, then you'll
> > use QEMU VNC/SPICE server, and there won't be anything in the
> > guest for that to proxy to/from.
> 
> If really the assumption is there is nothing in the guest to help
> us, then I'd say "paste" should come up with a warning "do you want
> me to try and translate that into keystrokes", and I don't see how to
> implement copy from a graphical display without OCR…

I'm not saying we can't use stuff in the guest, just that we should be
pragmatic about what we interact with in the guest and degrade nicely.
I don't think that proxying from a host VNC server to a guest VNC server
is sensible, but making use of a guest agent of some kind is not
unreasonable, especially if we can use one that already exists.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: vnc clipboard support

[Christophe de Dinechin]

[-- Attachment #1: Type: text/plain, Size: 6406 bytes --]



> On 29 Jan 2021, at 15:32, Daniel P. Berrangé <berrange@redhat.com> wrote:
> 
> On Fri, Jan 29, 2021 at 03:19:45PM +0100, Christophe de Dinechin wrote:
>> 
>> 
>>> On 29 Jan 2021, at 12:08, Daniel P. Berrangé <berrange@redhat.com> wrote:
>>> 
>>> On Fri, Jan 29, 2021 at 11:50:10AM +0100, Christophe de Dinechin wrote:
>>>> 
>>>> 
>>>>> On 29 Jan 2021, at 09:03, Gerd Hoffmann <kraxel@redhat.com> wrote:
>>>>> 
>>>>> Hi,
>>>>> 
>>>>>>> (1) Have some guest agent (spice does it that way).
>>>>>>>   Advantage: more flexible, allows more features.
>>>>>>>   Disadvantage: requires agent in the guest.
>>>>>> 
>>>>>> What about running the option to relay data to a VNC server in the
>>>>>> guest if there is one running? In other words, using an existing
>>>>>> VNC server as an agent, with the option of having a stripped-down,
>>>>>> clipboard only VNC server as a later optimization.
>>>>> 
>>>>> Well, if you run Xvnc in the guest anyway why use the qemu vnc server
>>>>> in the first place?
>>>> 
>>>> I assume that if you use the qemu VNC, it's because you you don't want to
>>>> run Xvnc on some external network, or care about accessing the guest
>>>> before Xvnc can be launched. There can be many reasons.
>>>> 
>>>> Again, I want to make it clear that my suggestion is _not_ simply to access
>>>> the existing Xvnc as is, but rather to stick with some VNC server code to handle
>>>> the clipboard if / when possible.
>>>> 
>>>> Let me try to imagine a scenario, where I'll use a macOS guest intentionally
>>>> to clarify what I was thinking about.
>>>> 
>>>> - During early boot, there is no in-guest VNC server, so to address that,
>>>> you connect to the qemu VNC. At this stage, all screen refresh is handled
>>>> by the qemu VNC, and the best you can do if you want to support any
>>>> kind of copy-paste is to convert it to virtual keystrokes. The same would
>>>> be true for Linux on a text console.
>>>> 
>>>> - Then comes up you macOS guest, and it still has no VNC port open,
>>>> so you are stuck with qemu-vnc doing all the work. But now you can
>>>> enable screen sharing, and that launches the Apple-maintained macOS
>>>> VNC server.
>>>> 
>>>> - Let's assume for illustration that this listens on some private network
>>>> that qemu can access, but that is not visible externally. In this case,
>>>> you could not VNC to the guest, but you can still VNC to qemu.
>>>> 
>>>> - What I'm suggesting is that qemu-vnc could then switch to simply
>>>> relaying VNC traffic to that in-guest server. You'd get the smart update
>>>> algorithm that Apple has put in place to deal with transparency and the
>>>> like, as well as a level of guest OS integration that would otherwise be
>>>> much harder to replicate.
>>> 
>>> IMHO that's an awful lot of complexity to add to the system
>>> that isn't especially useful and this opens up new attack
>>> vectors for the guest to exploit the host.
>>> 
>>> If people have VNC/RDP/whatever configured inside their guest
>>> OS, then there's really little to no reason for them to want
>>> to connect to the QEMU VNC server, as viewing initial bootup
>>> phase is not required in normal case. The only time such
>>> people will need to use the QEMU VNC server is if the guest
>>> OS has broken in some way before it fully booted and thus failed
>>> to start the guest VNC server. There is no guest VNC server
>>> to hand off to in this scenario.
>> 
>> It's a matter of what you want to do with that qemu vnc.
>> 
>> If it's only a backup when there's nothing in the guest to help,
>> then maybe trying to support copy-paste is not a good idea.
>> 
>> If it's a standard go-to access point for connecting to your
>> guest, then making it smart is hard, but useful.
>> 
>>> 
>>> The value of the QEMU host side VNC server is that it works
>>> for all possible guest OS, no matter whether they are running
>>> normally or broken, regardless of what the guest admin has
>>> configured in terms of guest level remote desktop.
>> 
>> Understood.
>> 
>> The downside is that there are things it can't do. It cannot correctly
>> determine the keyboard map, because that's controlled in software
>> in the guest.
> 
> There is no need for the remote desktop server to care about the
> keymap. The remote client takes scancodes and passes them to the
> server, which then passes them to the guest.

Aren't we talking about pasting clipboard data here?
I assume that clipboard data is not encoded as scancodes.


> 
> The person connected to the remote server simply has to configure
> their guest OS keymap to match the physical keyboard they currently
> have plugged in.
> 
> The only reason a remote server would need to know the keymap is
> if it was trying to translate from keycodes back into scancodes.
> This is what VNC protocol had to do originally, but VNC was since
> extended to pass raw scancodes instead of keycodes, precisely to
> avoid needing to care about keymaps.
> 
>>> IOW, if you have a guest remote desktop solution you'll just
>>> use that 99% of the time. If you don't have that, then you'll
>>> use QEMU VNC/SPICE server, and there won't be anything in the
>>> guest for that to proxy to/from.
>> 
>> If really the assumption is there is nothing in the guest to help
>> us, then I'd say "paste" should come up with a warning "do you want
>> me to try and translate that into keystrokes", and I don't see how to
>> implement copy from a graphical display without OCR…
> 
> I'm not saying we can't use stuff in the guest, just that we should be
> pragmatic about what we interact with in the guest and degrade nicely.
> I don't think that proxying from a host VNC server to a guest VNC server
> is sensible, but making use of a guest agent of some kind is not
> unreasonable, especially if we can use one that already exists.
> 
> 
> Regards,
> Daniel
> -- 
> |: https://berrange.com <https://berrange.com/>      -o-    https://www.flickr.com/photos/dberrange <https://www.flickr.com/photos/dberrange> :|
> |: https://libvirt.org <https://libvirt.org/>         -o-            https://fstop138.berrange.com <https://fstop138.berrange.com/> :|
> |: https://entangle-photo.org <https://entangle-photo.org/>    -o-    https://www.instagram.com/dberrange <https://www.instagram.com/dberrange> :|


[-- Attachment #2: Type: text/html, Size: 32410 bytes --]

Re: vnc clipboard support

[Daniel P. Berrangé]

On Mon, Feb 01, 2021 at 04:27:43PM +0100, Christophe de Dinechin wrote:
> 
> 
> > On 29 Jan 2021, at 15:32, Daniel P. Berrangé <berrange@redhat.com> wrote:
> > 
> > On Fri, Jan 29, 2021 at 03:19:45PM +0100, Christophe de Dinechin wrote:
> >> 
> >> 
> >>> On 29 Jan 2021, at 12:08, Daniel P. Berrangé <berrange@redhat.com> wrote:
> >>> 
> >>> On Fri, Jan 29, 2021 at 11:50:10AM +0100, Christophe de Dinechin wrote:
> >>>> 
> >>>> 
> >>>>> On 29 Jan 2021, at 09:03, Gerd Hoffmann <kraxel@redhat.com> wrote:
> >>>>> 
> >>>>> Hi,
> >>>>> 
> >>>>>>> (1) Have some guest agent (spice does it that way).
> >>>>>>>   Advantage: more flexible, allows more features.
> >>>>>>>   Disadvantage: requires agent in the guest.
> >>>>>> 
> >>>>>> What about running the option to relay data to a VNC server in the
> >>>>>> guest if there is one running? In other words, using an existing
> >>>>>> VNC server as an agent, with the option of having a stripped-down,
> >>>>>> clipboard only VNC server as a later optimization.
> >>>>> 
> >>>>> Well, if you run Xvnc in the guest anyway why use the qemu vnc server
> >>>>> in the first place?
> >>>> 
> >>>> I assume that if you use the qemu VNC, it's because you you don't want to
> >>>> run Xvnc on some external network, or care about accessing the guest
> >>>> before Xvnc can be launched. There can be many reasons.
> >>>> 
> >>>> Again, I want to make it clear that my suggestion is _not_ simply to access
> >>>> the existing Xvnc as is, but rather to stick with some VNC server code to handle
> >>>> the clipboard if / when possible.
> >>>> 
> >>>> Let me try to imagine a scenario, where I'll use a macOS guest intentionally
> >>>> to clarify what I was thinking about.
> >>>> 
> >>>> - During early boot, there is no in-guest VNC server, so to address that,
> >>>> you connect to the qemu VNC. At this stage, all screen refresh is handled
> >>>> by the qemu VNC, and the best you can do if you want to support any
> >>>> kind of copy-paste is to convert it to virtual keystrokes. The same would
> >>>> be true for Linux on a text console.
> >>>> 
> >>>> - Then comes up you macOS guest, and it still has no VNC port open,
> >>>> so you are stuck with qemu-vnc doing all the work. But now you can
> >>>> enable screen sharing, and that launches the Apple-maintained macOS
> >>>> VNC server.
> >>>> 
> >>>> - Let's assume for illustration that this listens on some private network
> >>>> that qemu can access, but that is not visible externally. In this case,
> >>>> you could not VNC to the guest, but you can still VNC to qemu.
> >>>> 
> >>>> - What I'm suggesting is that qemu-vnc could then switch to simply
> >>>> relaying VNC traffic to that in-guest server. You'd get the smart update
> >>>> algorithm that Apple has put in place to deal with transparency and the
> >>>> like, as well as a level of guest OS integration that would otherwise be
> >>>> much harder to replicate.
> >>> 
> >>> IMHO that's an awful lot of complexity to add to the system
> >>> that isn't especially useful and this opens up new attack
> >>> vectors for the guest to exploit the host.
> >>> 
> >>> If people have VNC/RDP/whatever configured inside their guest
> >>> OS, then there's really little to no reason for them to want
> >>> to connect to the QEMU VNC server, as viewing initial bootup
> >>> phase is not required in normal case. The only time such
> >>> people will need to use the QEMU VNC server is if the guest
> >>> OS has broken in some way before it fully booted and thus failed
> >>> to start the guest VNC server. There is no guest VNC server
> >>> to hand off to in this scenario.
> >> 
> >> It's a matter of what you want to do with that qemu vnc.
> >> 
> >> If it's only a backup when there's nothing in the guest to help,
> >> then maybe trying to support copy-paste is not a good idea.
> >> 
> >> If it's a standard go-to access point for connecting to your
> >> guest, then making it smart is hard, but useful.
> >> 
> >>> 
> >>> The value of the QEMU host side VNC server is that it works
> >>> for all possible guest OS, no matter whether they are running
> >>> normally or broken, regardless of what the guest admin has
> >>> configured in terms of guest level remote desktop.
> >> 
> >> Understood.
> >> 
> >> The downside is that there are things it can't do. It cannot correctly
> >> determine the keyboard map, because that's controlled in software
> >> in the guest.
> > 
> > There is no need for the remote desktop server to care about the
> > keymap. The remote client takes scancodes and passes them to the
> > server, which then passes them to the guest.
> 
> Aren't we talking about pasting clipboard data here?
> I assume that clipboard data is not encoded as scancodes.

The suggestion was that clipboard be sent by injecting key presses
to the guest, so that is scancode based.

If you're not doing it using key press injection, then keymapping
and scancodes are irrelevant.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: vnc clipboard support

[Christophe de Dinechin]

[-- Attachment #1: Type: text/plain, Size: 5617 bytes --]



> On 1 Feb 2021, at 16:51, Daniel P. Berrangé <berrange@redhat.com> wrote:
> 
> On Mon, Feb 01, 2021 at 04:27:43PM +0100, Christophe de Dinechin wrote:
>> 
>> 
>>> On 29 Jan 2021, at 15:32, Daniel P. Berrangé <berrange@redhat.com> wrote:
>>> 
>>> On Fri, Jan 29, 2021 at 03:19:45PM +0100, Christophe de Dinechin wrote:
>>>> 
>>>> 
>>>>> On 29 Jan 2021, at 12:08, Daniel P. Berrangé <berrange@redhat.com> wrote:
>>>>> 
>>>>> On Fri, Jan 29, 2021 at 11:50:10AM +0100, Christophe de Dinechin wrote:
>>>>>> 
>>>>>> 
>>>>>>> On 29 Jan 2021, at 09:03, Gerd Hoffmann <kraxel@redhat.com> wrote:
>>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>>>> (1) Have some guest agent (spice does it that way).
>>>>>>>>>  Advantage: more flexible, allows more features.
>>>>>>>>>  Disadvantage: requires agent in the guest.
>>>>>>>> 
>>>>>>>> What about running the option to relay data to a VNC server in the
>>>>>>>> guest if there is one running? In other words, using an existing
>>>>>>>> VNC server as an agent, with the option of having a stripped-down,
>>>>>>>> clipboard only VNC server as a later optimization.
>>>>>>> 
>>>>>>> Well, if you run Xvnc in the guest anyway why use the qemu vnc server
>>>>>>> in the first place?
>>>>>> 
>>>>>> I assume that if you use the qemu VNC, it's because you you don't want to
>>>>>> run Xvnc on some external network, or care about accessing the guest
>>>>>> before Xvnc can be launched. There can be many reasons.
>>>>>> 
>>>>>> Again, I want to make it clear that my suggestion is _not_ simply to access
>>>>>> the existing Xvnc as is, but rather to stick with some VNC server code to handle
>>>>>> the clipboard if / when possible.
>>>>>> 
>>>>>> Let me try to imagine a scenario, where I'll use a macOS guest intentionally
>>>>>> to clarify what I was thinking about.
>>>>>> 
>>>>>> - During early boot, there is no in-guest VNC server, so to address that,
>>>>>> you connect to the qemu VNC. At this stage, all screen refresh is handled
>>>>>> by the qemu VNC, and the best you can do if you want to support any
>>>>>> kind of copy-paste is to convert it to virtual keystrokes. The same would
>>>>>> be true for Linux on a text console.
>>>>>> 
>>>>>> - Then comes up you macOS guest, and it still has no VNC port open,
>>>>>> so you are stuck with qemu-vnc doing all the work. But now you can
>>>>>> enable screen sharing, and that launches the Apple-maintained macOS
>>>>>> VNC server.
>>>>>> 
>>>>>> - Let's assume for illustration that this listens on some private network
>>>>>> that qemu can access, but that is not visible externally. In this case,
>>>>>> you could not VNC to the guest, but you can still VNC to qemu.
>>>>>> 
>>>>>> - What I'm suggesting is that qemu-vnc could then switch to simply
>>>>>> relaying VNC traffic to that in-guest server. You'd get the smart update
>>>>>> algorithm that Apple has put in place to deal with transparency and the
>>>>>> like, as well as a level of guest OS integration that would otherwise be
>>>>>> much harder to replicate.
>>>>> 
>>>>> IMHO that's an awful lot of complexity to add to the system
>>>>> that isn't especially useful and this opens up new attack
>>>>> vectors for the guest to exploit the host.
>>>>> 
>>>>> If people have VNC/RDP/whatever configured inside their guest
>>>>> OS, then there's really little to no reason for them to want
>>>>> to connect to the QEMU VNC server, as viewing initial bootup
>>>>> phase is not required in normal case. The only time such
>>>>> people will need to use the QEMU VNC server is if the guest
>>>>> OS has broken in some way before it fully booted and thus failed
>>>>> to start the guest VNC server. There is no guest VNC server
>>>>> to hand off to in this scenario.
>>>> 
>>>> It's a matter of what you want to do with that qemu vnc.
>>>> 
>>>> If it's only a backup when there's nothing in the guest to help,
>>>> then maybe trying to support copy-paste is not a good idea.
>>>> 
>>>> If it's a standard go-to access point for connecting to your
>>>> guest, then making it smart is hard, but useful.
>>>> 
>>>>> 
>>>>> The value of the QEMU host side VNC server is that it works
>>>>> for all possible guest OS, no matter whether they are running
>>>>> normally or broken, regardless of what the guest admin has
>>>>> configured in terms of guest level remote desktop.
>>>> 
>>>> Understood.
>>>> 
>>>> The downside is that there are things it can't do. It cannot correctly
>>>> determine the keyboard map, because that's controlled in software
>>>> in the guest.
>>> 
>>> There is no need for the remote desktop server to care about the
>>> keymap. The remote client takes scancodes and passes them to the
>>> server, which then passes them to the guest.
>> 
>> Aren't we talking about pasting clipboard data here?
>> I assume that clipboard data is not encoded as scancodes.
> 
> The suggestion was that clipboard be sent by injecting key presses
> to the guest, so that is scancode based.

But the VNC clipboard protocol is not scancode based. So where
would the clipboard->scancode conversion happen?

Also, any idea how to copy from the guest in that case?

> 
> If you're not doing it using key press injection, then keymapping
> and scancodes are irrelevant.

Indeed, but that requires some guest agent, and Gerd seems to
not like that idea.

BTW, isolating the clipboard sharing protocol from VNC is an
old idea, see https://wiki.xenproject.org/wiki/Clipboard_sharing_protocol <https://wiki.xenproject.org/wiki/Clipboard_sharing_protocol>.




[-- Attachment #2: Type: text/html, Size: 12044 bytes --]

Re: vnc clipboard support

[Daniel P. Berrangé]

On Mon, Feb 01, 2021 at 05:31:52PM +0100, Christophe de Dinechin wrote:
> 
> 
> > On 1 Feb 2021, at 16:51, Daniel P. Berrangé <berrange@redhat.com> wrote:
> > 
> > On Mon, Feb 01, 2021 at 04:27:43PM +0100, Christophe de Dinechin wrote:
> >> 
> >> 
> >>> On 29 Jan 2021, at 15:32, Daniel P. Berrangé <berrange@redhat.com> wrote:
> >>> 
> >>> On Fri, Jan 29, 2021 at 03:19:45PM +0100, Christophe de Dinechin wrote:
> >>>> 
> >>>> 
> >>>>> On 29 Jan 2021, at 12:08, Daniel P. Berrangé <berrange@redhat.com> wrote:
> >>>>> 
> >>>>> On Fri, Jan 29, 2021 at 11:50:10AM +0100, Christophe de Dinechin wrote:
> >>>>>> 
> >>>>>> 
> >>>>>>> On 29 Jan 2021, at 09:03, Gerd Hoffmann <kraxel@redhat.com> wrote:
> >>>>>>> 
> >>>>>>> Hi,
> >>>>>>> 
> >>>>>>>>> (1) Have some guest agent (spice does it that way).
> >>>>>>>>>  Advantage: more flexible, allows more features.
> >>>>>>>>>  Disadvantage: requires agent in the guest.
> >>>>>>>> 
> >>>>>>>> What about running the option to relay data to a VNC server in the
> >>>>>>>> guest if there is one running? In other words, using an existing
> >>>>>>>> VNC server as an agent, with the option of having a stripped-down,
> >>>>>>>> clipboard only VNC server as a later optimization.
> >>>>>>> 
> >>>>>>> Well, if you run Xvnc in the guest anyway why use the qemu vnc server
> >>>>>>> in the first place?
> >>>>>> 
> >>>>>> I assume that if you use the qemu VNC, it's because you you don't want to
> >>>>>> run Xvnc on some external network, or care about accessing the guest
> >>>>>> before Xvnc can be launched. There can be many reasons.
> >>>>>> 
> >>>>>> Again, I want to make it clear that my suggestion is _not_ simply to access
> >>>>>> the existing Xvnc as is, but rather to stick with some VNC server code to handle
> >>>>>> the clipboard if / when possible.
> >>>>>> 
> >>>>>> Let me try to imagine a scenario, where I'll use a macOS guest intentionally
> >>>>>> to clarify what I was thinking about.
> >>>>>> 
> >>>>>> - During early boot, there is no in-guest VNC server, so to address that,
> >>>>>> you connect to the qemu VNC. At this stage, all screen refresh is handled
> >>>>>> by the qemu VNC, and the best you can do if you want to support any
> >>>>>> kind of copy-paste is to convert it to virtual keystrokes. The same would
> >>>>>> be true for Linux on a text console.
> >>>>>> 
> >>>>>> - Then comes up you macOS guest, and it still has no VNC port open,
> >>>>>> so you are stuck with qemu-vnc doing all the work. But now you can
> >>>>>> enable screen sharing, and that launches the Apple-maintained macOS
> >>>>>> VNC server.
> >>>>>> 
> >>>>>> - Let's assume for illustration that this listens on some private network
> >>>>>> that qemu can access, but that is not visible externally. In this case,
> >>>>>> you could not VNC to the guest, but you can still VNC to qemu.
> >>>>>> 
> >>>>>> - What I'm suggesting is that qemu-vnc could then switch to simply
> >>>>>> relaying VNC traffic to that in-guest server. You'd get the smart update
> >>>>>> algorithm that Apple has put in place to deal with transparency and the
> >>>>>> like, as well as a level of guest OS integration that would otherwise be
> >>>>>> much harder to replicate.
> >>>>> 
> >>>>> IMHO that's an awful lot of complexity to add to the system
> >>>>> that isn't especially useful and this opens up new attack
> >>>>> vectors for the guest to exploit the host.
> >>>>> 
> >>>>> If people have VNC/RDP/whatever configured inside their guest
> >>>>> OS, then there's really little to no reason for them to want
> >>>>> to connect to the QEMU VNC server, as viewing initial bootup
> >>>>> phase is not required in normal case. The only time such
> >>>>> people will need to use the QEMU VNC server is if the guest
> >>>>> OS has broken in some way before it fully booted and thus failed
> >>>>> to start the guest VNC server. There is no guest VNC server
> >>>>> to hand off to in this scenario.
> >>>> 
> >>>> It's a matter of what you want to do with that qemu vnc.
> >>>> 
> >>>> If it's only a backup when there's nothing in the guest to help,
> >>>> then maybe trying to support copy-paste is not a good idea.
> >>>> 
> >>>> If it's a standard go-to access point for connecting to your
> >>>> guest, then making it smart is hard, but useful.
> >>>> 
> >>>>> 
> >>>>> The value of the QEMU host side VNC server is that it works
> >>>>> for all possible guest OS, no matter whether they are running
> >>>>> normally or broken, regardless of what the guest admin has
> >>>>> configured in terms of guest level remote desktop.
> >>>> 
> >>>> Understood.
> >>>> 
> >>>> The downside is that there are things it can't do. It cannot correctly
> >>>> determine the keyboard map, because that's controlled in software
> >>>> in the guest.
> >>> 
> >>> There is no need for the remote desktop server to care about the
> >>> keymap. The remote client takes scancodes and passes them to the
> >>> server, which then passes them to the guest.
> >> 
> >> Aren't we talking about pasting clipboard data here?
> >> I assume that clipboard data is not encoded as scancodes.
> > 
> > The suggestion was that clipboard be sent by injecting key presses
> > to the guest, so that is scancode based.
> 
> But the VNC clipboard protocol is not scancode based. So where
> would the clipboard->scancode conversion happen?

If using keyboard injection there's not really much point in
using the VNC clipboard protocol at all. It just complicates
life since now we have to configure QEMU with a keymap so that
it can turn text back into scancodes. We went through alot of
effort precisely to *stop* telling QEMU about keymaps, so that
is not an viable option.

IOW, if we want to use keyboard injection to fake the clipboard
then the VNC Clients should just implemnet this natively and
use the VNC keyboard protocol messages instead, so scancode
conversion is client side. It is still painful, because text
to scancode is inherantly a lossy conversion, but it is better
than doing it on the QEMU code.

> Also, any idea how to copy from the guest in that case?

If using keyboard injection the clipboard will be unidirectional
only.

Bi-directional clipboard needs a guest agent.

> 
> > 
> > If you're not doing it using key press injection, then keymapping
> > and scancodes are irrelevant.
> 
> Indeed, but that requires some guest agent, and Gerd seems to
> not like that idea.
> 
> BTW, isolating the clipboard sharing protocol from VNC is an
> old idea, see https://wiki.xenproject.org/wiki/Clipboard_sharing_protocol <https://wiki.xenproject.org/wiki/Clipboard_sharing_protocol>.

That's not really isolating it from VNC - that's describing a guest
agent the host QEMU can use to support its built-in VNC server.
Confusingly this suggested guest agent happens to use a subset of
the VNC protocol for transferring clipboard data between QEMU and
the guest OS. This part is rather crazy since the VNC clipboard
protocol is incredibly simplistic. Any new guest agent should
learn from the mistakes of VNC and SPICE wrt clipboard.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: vnc clipboard support

[Christophe de Dinechin]

[-- Attachment #1: Type: text/plain, Size: 8664 bytes --]



> On 1 Feb 2021, at 17:56, Daniel P. Berrangé <berrange@redhat.com> wrote:
> 
> On Mon, Feb 01, 2021 at 05:31:52PM +0100, Christophe de Dinechin wrote:
>> 
>> 
>>> On 1 Feb 2021, at 16:51, Daniel P. Berrangé <berrange@redhat.com> wrote:
>>> 
>>> On Mon, Feb 01, 2021 at 04:27:43PM +0100, Christophe de Dinechin wrote:
>>>> 
>>>> 
>>>>> On 29 Jan 2021, at 15:32, Daniel P. Berrangé <berrange@redhat.com> wrote:
>>>>> 
>>>>> On Fri, Jan 29, 2021 at 03:19:45PM +0100, Christophe de Dinechin wrote:
>>>>>> 
>>>>>> 
>>>>>>> On 29 Jan 2021, at 12:08, Daniel P. Berrangé <berrange@redhat.com> wrote:
>>>>>>> 
>>>>>>> On Fri, Jan 29, 2021 at 11:50:10AM +0100, Christophe de Dinechin wrote:
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On 29 Jan 2021, at 09:03, Gerd Hoffmann <kraxel@redhat.com> wrote:
>>>>>>>>> 
>>>>>>>>> Hi,
>>>>>>>>> 
>>>>>>>>>>> (1) Have some guest agent (spice does it that way).
>>>>>>>>>>> Advantage: more flexible, allows more features.
>>>>>>>>>>> Disadvantage: requires agent in the guest.
>>>>>>>>>> 
>>>>>>>>>> What about running the option to relay data to a VNC server in the
>>>>>>>>>> guest if there is one running? In other words, using an existing
>>>>>>>>>> VNC server as an agent, with the option of having a stripped-down,
>>>>>>>>>> clipboard only VNC server as a later optimization.
>>>>>>>>> 
>>>>>>>>> Well, if you run Xvnc in the guest anyway why use the qemu vnc server
>>>>>>>>> in the first place?
>>>>>>>> 
>>>>>>>> I assume that if you use the qemu VNC, it's because you you don't want to
>>>>>>>> run Xvnc on some external network, or care about accessing the guest
>>>>>>>> before Xvnc can be launched. There can be many reasons.
>>>>>>>> 
>>>>>>>> Again, I want to make it clear that my suggestion is _not_ simply to access
>>>>>>>> the existing Xvnc as is, but rather to stick with some VNC server code to handle
>>>>>>>> the clipboard if / when possible.
>>>>>>>> 
>>>>>>>> Let me try to imagine a scenario, where I'll use a macOS guest intentionally
>>>>>>>> to clarify what I was thinking about.
>>>>>>>> 
>>>>>>>> - During early boot, there is no in-guest VNC server, so to address that,
>>>>>>>> you connect to the qemu VNC. At this stage, all screen refresh is handled
>>>>>>>> by the qemu VNC, and the best you can do if you want to support any
>>>>>>>> kind of copy-paste is to convert it to virtual keystrokes. The same would
>>>>>>>> be true for Linux on a text console.
>>>>>>>> 
>>>>>>>> - Then comes up you macOS guest, and it still has no VNC port open,
>>>>>>>> so you are stuck with qemu-vnc doing all the work. But now you can
>>>>>>>> enable screen sharing, and that launches the Apple-maintained macOS
>>>>>>>> VNC server.
>>>>>>>> 
>>>>>>>> - Let's assume for illustration that this listens on some private network
>>>>>>>> that qemu can access, but that is not visible externally. In this case,
>>>>>>>> you could not VNC to the guest, but you can still VNC to qemu.
>>>>>>>> 
>>>>>>>> - What I'm suggesting is that qemu-vnc could then switch to simply
>>>>>>>> relaying VNC traffic to that in-guest server. You'd get the smart update
>>>>>>>> algorithm that Apple has put in place to deal with transparency and the
>>>>>>>> like, as well as a level of guest OS integration that would otherwise be
>>>>>>>> much harder to replicate.
>>>>>>> 
>>>>>>> IMHO that's an awful lot of complexity to add to the system
>>>>>>> that isn't especially useful and this opens up new attack
>>>>>>> vectors for the guest to exploit the host.
>>>>>>> 
>>>>>>> If people have VNC/RDP/whatever configured inside their guest
>>>>>>> OS, then there's really little to no reason for them to want
>>>>>>> to connect to the QEMU VNC server, as viewing initial bootup
>>>>>>> phase is not required in normal case. The only time such
>>>>>>> people will need to use the QEMU VNC server is if the guest
>>>>>>> OS has broken in some way before it fully booted and thus failed
>>>>>>> to start the guest VNC server. There is no guest VNC server
>>>>>>> to hand off to in this scenario.
>>>>>> 
>>>>>> It's a matter of what you want to do with that qemu vnc.
>>>>>> 
>>>>>> If it's only a backup when there's nothing in the guest to help,
>>>>>> then maybe trying to support copy-paste is not a good idea.
>>>>>> 
>>>>>> If it's a standard go-to access point for connecting to your
>>>>>> guest, then making it smart is hard, but useful.
>>>>>> 
>>>>>>> 
>>>>>>> The value of the QEMU host side VNC server is that it works
>>>>>>> for all possible guest OS, no matter whether they are running
>>>>>>> normally or broken, regardless of what the guest admin has
>>>>>>> configured in terms of guest level remote desktop.
>>>>>> 
>>>>>> Understood.
>>>>>> 
>>>>>> The downside is that there are things it can't do. It cannot correctly
>>>>>> determine the keyboard map, because that's controlled in software
>>>>>> in the guest.
>>>>> 
>>>>> There is no need for the remote desktop server to care about the
>>>>> keymap. The remote client takes scancodes and passes them to the
>>>>> server, which then passes them to the guest.
>>>> 
>>>> Aren't we talking about pasting clipboard data here?
>>>> I assume that clipboard data is not encoded as scancodes.
>>> 
>>> The suggestion was that clipboard be sent by injecting key presses
>>> to the guest, so that is scancode based.
>> 
>> But the VNC clipboard protocol is not scancode based. So where
>> would the clipboard->scancode conversion happen?
> 
> If using keyboard injection there's not really much point in
> using the VNC clipboard protocol at all. It just complicates
> life since now we have to configure QEMU with a keymap so that
> it can turn text back into scancodes. We went through alot of
> effort precisely to *stop* telling QEMU about keymaps, so that
> is not an viable option.

How do you get rid of keymaps for local input in qemu?

> 
> IOW, if we want to use keyboard injection to fake the clipboard
> then the VNC Clients should just implemnet this natively and
> use the VNC keyboard protocol messages instead, so scancode
> conversion is client side. It is still painful, because text
> to scancode is inherantly a lossy conversion, but it is better
> than doing it on the QEMU code.

How many brands of VNC clients are there?

What happens if qemu receives clipboard messages from an old client?

Do you plan some new VNC capability to tell the client "I want scancodes"?

What would prevent a client from sending dangerous scancodes,
like Control C or Alt TAB?

How would you prevent the user from typing something in the middle?

How do you rate-limit intelligently to avoid overflowing the guest keyboard buffers?


> 
>> Also, any idea how to copy from the guest in that case?
> 
> If using keyboard injection the clipboard will be unidirectional
> only.
> 
> Bi-directional clipboard needs a guest agent.

OK.

What is the use case for single-directional text-only clipboard?

> 
>> 
>>> 
>>> If you're not doing it using key press injection, then keymapping
>>> and scancodes are irrelevant.
>> 
>> Indeed, but that requires some guest agent, and Gerd seems to
>> not like that idea.
>> 
>> BTW, isolating the clipboard sharing protocol from VNC is an
>> old idea, see https://wiki.xenproject.org/wiki/Clipboard_sharing_protocol <https://wiki.xenproject.org/wiki/Clipboard_sharing_protocol><https://wiki.xenproject.org/wiki/Clipboard_sharing_protocol <https://wiki.xenproject.org/wiki/Clipboard_sharing_protocol>>.
> 
> That's not really isolating it from VNC - that's describing a guest
> agent the host QEMU can use to support its built-in VNC server.
> Confusingly this suggested guest agent happens to use a subset of
> the VNC protocol for transferring clipboard data between QEMU and
> the guest OS. This part is rather crazy since the VNC clipboard
> protocol is incredibly simplistic. Any new guest agent should
> learn from the mistakes of VNC and SPICE wrt clipboard.

Well, to me, the trick is that you got a VNC server in qemu that
receives VNC clipboard data. The question is how you transfer
that to the guest. Indeed, the protocol is simplistic, but you still
need a new data path, e.g. a character device that your in-guest
agent listens to, a bit like the SPICE agent. So the picture becomes:

Normal VNC
Client --> (guest network) --> In-guest VNC server

QEMU VNC:
Client --> (host network) --> qemu-VNC server --> char device --> in-guest VNC clipboard server

Only the data path changes, but the protocol can remain
essentially the same.


[-- Attachment #2: Type: text/html, Size: 30877 bytes --]

Re: vnc clipboard support

[Daniel P. Berrangé]

On Mon, Feb 01, 2021 at 06:28:38PM +0100, Christophe de Dinechin wrote:
> 
> 
> > On 1 Feb 2021, at 17:56, Daniel P. Berrangé <berrange@redhat.com> wrote:
> > 
> > On Mon, Feb 01, 2021 at 05:31:52PM +0100, Christophe de Dinechin wrote:
> >> 
> >> 
> >> 
> >> But the VNC clipboard protocol is not scancode based. So where
> >> would the clipboard->scancode conversion happen?
> > 
> > If using keyboard injection there's not really much point in
> > using the VNC clipboard protocol at all. It just complicates
> > life since now we have to configure QEMU with a keymap so that
> > it can turn text back into scancodes. We went through alot of
> > effort precisely to *stop* telling QEMU about keymaps, so that
> > is not an viable option.
> 
> How do you get rid of keymaps for local input in qemu?

Don't look at the RFC spec for the RFB protocol. Instead look
at the community spec which defines all the extensions that
clients and servers have implemented. In this case, QEMU defined
an extension for scancodes:

  https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#qemu-extended-key-event-message

this is since implemented by alot of VNC clients and severs.


> > IOW, if we want to use keyboard injection to fake the clipboard
> > then the VNC Clients should just implemnet this natively and
> > use the VNC keyboard protocol messages instead, so scancode
> > conversion is client side. It is still painful, because text
> > to scancode is inherantly a lossy conversion, but it is better
> > than doing it on the QEMU code.
> 
> How many brands of VNC clients are there?
> 
> What happens if qemu receives clipboard messages from an old client?
> 
> Do you plan some new VNC capability to tell the client "I want scancodes"?

The ability to send keyboard events as scancodes has already
existed since 2008 :-)

> What would prevent a client from sending dangerous scancodes,
> like Control C or Alt TAB?

Nothing, but then this is largely a case of a user shooting themselves
in the foot, not a security issue.

> How would you prevent the user from typing something in the middle?

You can't.

> How do you rate-limit intelligently to avoid overflowing the guest keyboard buffers?

Like I said earlier in this thread, this is a big problem with
keyboard injection.

IMHO keyboard injection is only barely better than no clipboard
at all, and I don't think we should settle for that as a solution.

My preferred solution is to have QEMU leverage the existing SPICE
guest agent if at all possible, because that's already widely
available in existing guest OS.

> >> Also, any idea how to copy from the guest in that case?
> > 
> > If using keyboard injection the clipboard will be unidirectional
> > only.
> > 
> > Bi-directional clipboard needs a guest agent.
> 
> OK.
> 
> What is the use case for single-directional text-only clipboard?

It is better no clipboard at all.... but only just.


> > That's not really isolating it from VNC - that's describing a guest
> > agent the host QEMU can use to support its built-in VNC server.
> > Confusingly this suggested guest agent happens to use a subset of
> > the VNC protocol for transferring clipboard data between QEMU and
> > the guest OS. This part is rather crazy since the VNC clipboard
> > protocol is incredibly simplistic. Any new guest agent should
> > learn from the mistakes of VNC and SPICE wrt clipboard.
> 
> Well, to me, the trick is that you got a VNC server in qemu that
> receives VNC clipboard data. The question is how you transfer
> that to the guest. Indeed, the protocol is simplistic, but you still
> need a new data path, e.g. a character device that your in-guest
> agent listens to, a bit like the SPICE agent. So the picture becomes:
> 
> Normal VNC
> Client --> (guest network) --> In-guest VNC server
> 
> QEMU VNC:
> Client --> (host network) --> qemu-VNC server --> char device --> in-guest VNC clipboard server
> 
> Only the data path changes, but the protocol can remain
> essentially the same.

We should not be relying on the guest OS using VNC at all. On Windows
it is much more common to use RDP and GNOME looks to be going the same
way for Linux. We none the less want to be able to still use VNC from
the host side.

We need something to be running in the guest, and that should be
agnostic of whatever other software the guest chooses to use for
remote desktop, and should not assume the guest even has remote
desktop setup. 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: vnc clipboard support

[BALATON Zoltan]

[-- Attachment #1: Type: text/plain, Size: 3470 bytes --]

On Mon, 1 Feb 2021, Daniel P. Berrangé wrote:
> IMHO keyboard injection is only barely better than no clipboard
> at all, and I don't think we should settle for that as a solution.

From this discussion it looks like doing keyboard injection is not a good 
solution as it has more problems than what it can solve so we can conclude 
to drop that idea which then means there needs to be some kind of guest 
agent.

> My preferred solution is to have QEMU leverage the existing SPICE
> guest agent if at all possible, because that's already widely
> available in existing guest OS.

I think spice is not as widely available as VNC (or even Synergy) so the 
idea to strip one of those down to just a guest clipboard agent would help 
to get support to the most guests. QEMU already knows about VNC so might 
be the simplest way and you could reuse all kinds of VNC client and server 
code for the guest agent and also make it easy for others to add support 
for their guests. Using spice may not be that easi as it's less widespread 
so probably only works on the guests that already have support for it. And 
then why not just use spice instead of VNC on those guests and then you 
don't have the clipboard problem either.

>> What is the use case for single-directional text-only clipboard?
>
> It is better no clipboard at all.... but only just.

I think the use case Gerd mentioned as a first target: paste a link to the 
guest browser to download your data. But that's very limited and not that 
much useful so I agree more complete support should be targeted than that.

>> Well, to me, the trick is that you got a VNC server in qemu that
>> receives VNC clipboard data. The question is how you transfer
>> that to the guest. Indeed, the protocol is simplistic, but you still
>> need a new data path, e.g. a character device that your in-guest
>> agent listens to, a bit like the SPICE agent. So the picture becomes:
>>
>> Normal VNC
>> Client --> (guest network) --> In-guest VNC server
>>
>> QEMU VNC:
>> Client --> (host network) --> qemu-VNC server --> char device --> in-guest VNC clipboard server
>>
>> Only the data path changes, but the protocol can remain
>> essentially the same.
>
> We should not be relying on the guest OS using VNC at all. On Windows
> it is much more common to use RDP and GNOME looks to be going the same
> way for Linux. We none the less want to be able to still use VNC from
> the host side.
>
> We need something to be running in the guest, and that should be
> agnostic of whatever other software the guest chooses to use for
> remote desktop, and should not assume the guest even has remote
> desktop setup.

As I understood, the idea is not to run a full VNC server on the guest OS 
but to make it easy to write the guest agent that you'll need to run is to 
reuse parts of a VNC server which is already available on almost all 
platforms. So basing the guest agent protocol on that allows you to not 
needing to write a guest agent but just extract the code from an already 
available VNC server that hopefully already solves the guest OS 
integration problems which is the most problematic part. At least where 
open source versions are available but for most platforms there's probably 
a few. This just avoids writing a guest agent for all kinds of guests from 
scratch or pulling in too much unneeded dependencies just to make 
clipboard work (as a full spice guest agent may be overkill for this).

Regards,
BALATON Zoltan

Re: vnc clipboard support

[Gerd Hoffmann]

> > My preferred solution is to have QEMU leverage the existing SPICE
> > guest agent if at all possible, because that's already widely
> > available in existing guest OS.
> 
> I think spice is not as widely available as VNC (or even Synergy) so the
> idea to strip one of those down to just a guest clipboard agent would help
> to get support to the most guests.

Well, we can use the spice agent as-is, then just let it talk to qemu
instead of spice client.  No need to code anything on the guest side,
and the (guest) code packaged up in distros will just work.

> QEMU already knows about VNC so might be the simplest way and you
> could reuse all kinds of VNC client and server code for the guest
> agent and also make it easy for others to add support for their
> guests.

Something vnc-ish doesn't look like a good plan to me.  When coding up
something new I'd go model the protocol after something more modern like
wayland.  When reusing something existing the spice-agent protocol looks
best as most existing linux guests already support it.

Also note that we can hook up clipboard support for the other UIs too
once we have the infrastructure in qemu, so even though there is "vnc"
in the subject line this isn't about vnc only.

> Using spice may not be that easi as it's less widespread so probably
> only works on the guests that already have support for it. And then
> why not just use spice instead of VNC on those guests and then you
> don't have the clipboard problem either.

The guest side is easy.  The spice-vdagent package is small, and the
dependencies it needs are most likely present anyway in a guest with
a graphical desktop installed.

The host side is a bit more tricky.  spice-server has a relatively long
list of dependencies not needed otherwise on a virtualization host.

> > We need something to be running in the guest, and that should be
> > agnostic of whatever other software the guest chooses to use for
> > remote desktop, and should not assume the guest even has remote
> > desktop setup.
> 
> As I understood, the idea is not to run a full VNC server on the guest OS
> but to make it easy to write the guest agent that you'll need to run is to
> reuse parts of a VNC server which is already available on almost all
> platforms.

spice guest agent is already available for linux and windows.  Not fully
sure how the state is on other unix-ish platforms like *bsd is.  In any
case it should not be that hard to port the linux version over.

take care,
  Gerd

Re: vnc clipboard support

[BALATON Zoltan]

On Tue, 2 Feb 2021, Gerd Hoffmann wrote:
>>> My preferred solution is to have QEMU leverage the existing SPICE
>>> guest agent if at all possible, because that's already widely
>>> available in existing guest OS.
>>
>> I think spice is not as widely available as VNC (or even Synergy) so the
>> idea to strip one of those down to just a guest clipboard agent would help
>> to get support to the most guests.
>
> Well, we can use the spice agent as-is, then just let it talk to qemu
> instead of spice client.  No need to code anything on the guest side,
> and the (guest) code packaged up in distros will just work.
>
>> QEMU already knows about VNC so might be the simplest way and you
>> could reuse all kinds of VNC client and server code for the guest
>> agent and also make it easy for others to add support for their
>> guests.
>
> Something vnc-ish doesn't look like a good plan to me.  When coding up
> something new I'd go model the protocol after something more modern like
> wayland.  When reusing something existing the spice-agent protocol looks
> best as most existing linux guests already support it.

Is there anything the vnc protocol lacks over more modern ones that would 
be needed? I don't know what wayland does. Just because vnc is old it 
could be sufficient but I have no strong preference for any of these, just 
wanted to say that reusing something that's available on the most guests 
would make it work for most people with the least effort and vnc is quite 
widely available.

> Also note that we can hook up clipboard support for the other UIs too
> once we have the infrastructure in qemu, so even though there is "vnc"
> in the subject line this isn't about vnc only.

OK in that case using vnc protocol over others would not make it easier to 
connect to the server side but probably would not mmake it harder either.

>> Using spice may not be that easi as it's less widespread so probably
>> only works on the guests that already have support for it. And then
>> why not just use spice instead of VNC on those guests and then you
>> don't have the clipboard problem either.
>
> The guest side is easy.  The spice-vdagent package is small, and the
> dependencies it needs are most likely present anyway in a guest with
> a graphical desktop installed.
>
> The host side is a bit more tricky.  spice-server has a relatively long
> list of dependencies not needed otherwise on a virtualization host.
>
>>> We need something to be running in the guest, and that should be
>>> agnostic of whatever other software the guest chooses to use for
>>> remote desktop, and should not assume the guest even has remote
>>> desktop setup.
>>
>> As I understood, the idea is not to run a full VNC server on the guest OS
>> but to make it easy to write the guest agent that you'll need to run is to
>> reuse parts of a VNC server which is already available on almost all
>> platforms.
>
> spice guest agent is already available for linux and windows.  Not fully
> sure how the state is on other unix-ish platforms like *bsd is.  In any
> case it should not be that hard to port the linux version over.

You scope of possible guests seem to be limited, there are much more that 
can run on QEMU. What about macOS/X, MacOS, MorphOS, etc. OK they are not 
big and may be niche but people I know would like to use clipboard with 
those too. MorphOS has VNC server and Synergy client but not sure it would 
have dependencies for spice agent which is of course not as readily 
available on most platforms as e.g. vnc is. If we plan to allow such 
guests to use this clipboard solution then a protocol that's easy to 
implement without too much dependencies would be best. One which is aleady 
available is even better.

Regards,
BALATON ZOltan

Re: vnc clipboard support

[Daniel P. Berrangé]

On Tue, Feb 02, 2021 at 01:31:23PM +0100, BALATON Zoltan wrote:
> On Tue, 2 Feb 2021, Gerd Hoffmann wrote:
> > > > My preferred solution is to have QEMU leverage the existing SPICE
> > > > guest agent if at all possible, because that's already widely
> > > > available in existing guest OS.
> > > 
> > > I think spice is not as widely available as VNC (or even Synergy) so the
> > > idea to strip one of those down to just a guest clipboard agent would help
> > > to get support to the most guests.
> > 
> > Well, we can use the spice agent as-is, then just let it talk to qemu
> > instead of spice client.  No need to code anything on the guest side,
> > and the (guest) code packaged up in distros will just work.
> > 
> > > QEMU already knows about VNC so might be the simplest way and you
> > > could reuse all kinds of VNC client and server code for the guest
> > > agent and also make it easy for others to add support for their
> > > guests.
> > 
> > Something vnc-ish doesn't look like a good plan to me.  When coding up
> > something new I'd go model the protocol after something more modern like
> > wayland.  When reusing something existing the spice-agent protocol looks
> > best as most existing linux guests already support it.
> 
> Is there anything the vnc protocol lacks over more modern ones that would be
> needed? I don't know what wayland does. Just because vnc is old it could be
> sufficient but I have no strong preference for any of these, just wanted to
> say that reusing something that's available on the most guests would make it
> work for most people with the least effort and vnc is quite widely
> available.

The VNC protocol is way too crude. It is limited to transferring
plain text, and provides no way to specify or negotiate a character
set. The RFB spec says apps should use latin-1. In reality few, if any,
impls do charset conversion so most Linux impls will be sending UTF8
while Windows impls will be sending Windows Codepage 1252. It clearly
shows its heritage of being designed in the 1990s.

Essentially any other protocol relating to clipboard that postdates
RFB will be better.

I expect that once we figure out what we want todo with clipboard
for the QEMU <-> guest communication, then we'll end up wanting to
write a RFB protocol extension to define a better VNC clipboard
protocol that has the modern features people expect.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: vnc clipboard support

[Gerd Hoffmann]

  Hi,

> The VNC protocol is way too crude. It is limited to transferring
> plain text, and provides no way to specify or negotiate a character
> set. The RFB spec says apps should use latin-1. In reality few, if any,
> impls do charset conversion so most Linux impls will be sending UTF8
> while Windows impls will be sending Windows Codepage 1252. It clearly
> shows its heritage of being designed in the 1990s.

Well, there is an extension which supports more than just text.  But it
is a small number of formats which clearly has window-ish background,
for example rich text format (windows wordpad) for formated text and dib
(aka windows device independant bitmap) for images.

Dunno how this looks on windows these days, but linux uses a combination
of legacy x11 naming and mime types for metadata.  When creating
something new I'd tend to also use mime types as they are an
established, os-independent standard.

take care,
  Gerd

Re: vnc clipboard support

[Gerd Hoffmann]

On Tue, Feb 02, 2021 at 02:35:34PM +0100, Gerd Hoffmann wrote:
>   Hi,
> 
> > The VNC protocol is way too crude. It is limited to transferring
> > plain text, and provides no way to specify or negotiate a character
> > set. The RFB spec says apps should use latin-1. In reality few, if any,
> > impls do charset conversion so most Linux impls will be sending UTF8
> > while Windows impls will be sending Windows Codepage 1252. It clearly
> > shows its heritage of being designed in the 1990s.
> 
> Well, there is an extension which supports more than just text.  But it
> is a small number of formats which clearly has window-ish background,
> for example rich text format (windows wordpad) for formated text and dib
> (aka windows device independant bitmap) for images.

Well, spice isn't that much better.  Has a short, fixed list too:

VD_AGENT_CLIPBOARD_UTF8_TEXT
VD_AGENT_CLIPBOARD_IMAGE_PNG (mandatory if images supported)
VD_AGENT_CLIPBOARD_IMAGE_{BMP,TIFF,JPEG} (optional)

There also is ...

VD_AGENT_CLIPBOARD_FILE_LIST

... but that works hand-in-hand with the spice webdav server.

take care,
  Gerd

Re: vnc clipboard support

[Marc-André Lureau]

Hi

On Tue, Feb 2, 2021 at 9:18 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
>
> On Tue, Feb 02, 2021 at 02:35:34PM +0100, Gerd Hoffmann wrote:
> >   Hi,
> >
> > > The VNC protocol is way too crude. It is limited to transferring
> > > plain text, and provides no way to specify or negotiate a character
> > > set. The RFB spec says apps should use latin-1. In reality few, if any,
> > > impls do charset conversion so most Linux impls will be sending UTF8
> > > while Windows impls will be sending Windows Codepage 1252. It clearly
> > > shows its heritage of being designed in the 1990s.
> >
> > Well, there is an extension which supports more than just text.  But it
> > is a small number of formats which clearly has window-ish background,
> > for example rich text format (windows wordpad) for formated text and dib
> > (aka windows device independant bitmap) for images.
>
> Well, spice isn't that much better.  Has a short, fixed list too:
>
> VD_AGENT_CLIPBOARD_UTF8_TEXT
> VD_AGENT_CLIPBOARD_IMAGE_PNG (mandatory if images supported)
> VD_AGENT_CLIPBOARD_IMAGE_{BMP,TIFF,JPEG} (optional)
>
> There also is ...
>
> VD_AGENT_CLIPBOARD_FILE_LIST
>
> ... but that works hand-in-hand with the spice webdav server.
>

There has been some attempts to support generic mime types in Spice clipboard:
https://lists.freedesktop.org/archives/spice-devel/2018-May/043782.html

(If I am not mistaken, I think it was inspired on some earlier attempt
I worked on I lost trace)

In any case, it seems there is just not enough interest for other
things beside the existing Spice clipboard types. Interoperability
between OSes quickly becomes problematic for more advanced types, I am
afraid. (and various OS-specific clipboard & dnd details may raise)

-- 
Marc-André Lureau

Re: vnc clipboard support

[Gerd Hoffmann]

  Hi,

> > Well, spice isn't that much better.  Has a short, fixed list too:
> >
> > VD_AGENT_CLIPBOARD_UTF8_TEXT

> There has been some attempts to support generic mime types in Spice clipboard:
> https://lists.freedesktop.org/archives/spice-devel/2018-May/043782.html
> 
> (If I am not mistaken, I think it was inspired on some earlier attempt
> I worked on I lost trace)
> 
> In any case, it seems there is just not enough interest for other
> things beside the existing Spice clipboard types.

I guess "text/plain; charset=utf-8" covers 95% of the use cases.

Personally I *very* rarely cut+paste anything beside plan text,
and in case it is formated text it typically is something like
markdown ;)

> Interoperability
> between OSes quickly becomes problematic for more advanced types, I am
> afraid. (and various OS-specific clipboard & dnd details may raise)

That too.

I'll guess I start with text only, but of course keep the door open to
add more later on.  So I need some way to specify the type(s).  Obvious
choices are fixed list (like vnc/spice) or mime.  The later is more
flexible and future-proof of course, but I'm not sure we'll actually
need that in practice ...

take care,
  Gerd

Re: vnc clipboard support

[Gerd Hoffmann]

  Hi,

> > How do you rate-limit intelligently to avoid overflowing the guest keyboard buffers?
> 
> Like I said earlier in this thread, this is a big problem with
> keyboard injection.

It's a solved problem though, the qemu vnc server has throttling
implemented already.  There even is an option to configure the delay
(key-delay-ms).

> My preferred solution is to have QEMU leverage the existing SPICE
> guest agent if at all possible, because that's already widely
> available in existing guest OS.

Yep, that is an big advantage.

take care,
  Gerd

Re: vnc clipboard support

[Daniel P. Berrangé]

On Tue, Feb 02, 2021 at 12:10:15PM +0100, Gerd Hoffmann wrote:
>   Hi,
> 
> > > How do you rate-limit intelligently to avoid overflowing the guest keyboard buffers?
> > 
> > Like I said earlier in this thread, this is a big problem with
> > keyboard injection.
> 
> It's a solved problem though, the qemu vnc server has throttling
> implemented already.  There even is an option to configure the delay
> (key-delay-ms).

Depends on your view of "solved" really. If I've highlighted a region
of text in openoffice that is 100 KB in size, and press "Ctrl-V" in
the VNC window for the VM, with a 10 ms delay, i'll be waiting 15
minutes for it to finish injecting keystrokes.

Anything longer than 1 second is unacceptable. If you make the delay
too short the guest keyboard handler will loose keys. So realistically
I think keyboard injection is limted to about 100 characters of text.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: vnc clipboard support

[Gerd Hoffmann]

On Tue, Feb 02, 2021 at 11:17:09AM +0000, Daniel P. Berrangé wrote:
> On Tue, Feb 02, 2021 at 12:10:15PM +0100, Gerd Hoffmann wrote:
> >   Hi,
> > 
> > > > How do you rate-limit intelligently to avoid overflowing the guest keyboard buffers?
> > > 
> > > Like I said earlier in this thread, this is a big problem with
> > > keyboard injection.
> > 
> > It's a solved problem though, the qemu vnc server has throttling
> > implemented already.  There even is an option to configure the delay
> > (key-delay-ms).
> 
> Depends on your view of "solved" really.

"solved" as in "automated qa can send command lines as vnc key events
way without loosing chars".  That is the use case it was added for.
Doing it in qemu works a bit better and faster than doing it in the vnc
client as you don't have to account for stuff like network delays then.

> If I've highlighted a region
> of text in openoffice that is 100 KB in size, and press "Ctrl-V" in
> the VNC window for the VM, with a 10 ms delay, i'll be waiting 15
> minutes for it to finish injecting keystrokes.

Yep.  It's clearly not for really large quantities of text.

But sending "wget $url" to the guest shell works reasonable well.

> Anything longer than 1 second is unacceptable. If you make the delay
> too short the guest keyboard handler will loose keys. So realistically
> I think keyboard injection is limted to about 100 characters of text.

Yes.

take care,
  Gerd

Re: vnc clipboard support

[Gerd Hoffmann]

  Hi,

> Unless we para-virtualize the keyboard?

The main advantage of paste via key events is that it doesn't require
guest support.  Requiring any kind of software in the guest (paravirt
keyboard driver, agent figuring the keymap, whatever else) kills that
advantage.  And if we need guest cooperation anyway it is much better
to just read/write the guest clipboard directly.

Standard keyboard seems to not be an option either.  The HID specs
(https://usb.org/document-library/hid-usage-tables-121) lists a
Unicode Page (Section 18), which looks like it could be used to
send unicode chars to the guest.  Problem is (a) this is 16bit
only (so no emoji) and (b) widely unimplemented in guest OSes.

take care,
  Gerd

Re: vnc clipboard support

[Christophe de Dinechin]

> On 29 Jan 2021, at 16:04, Gerd Hoffmann <kraxel@redhat.com> wrote:
> 
>  Hi,
> 
>> Unless we para-virtualize the keyboard?
> 
> The main advantage of paste via key events is that it doesn't require
> guest support.

The main disadvantage, though, is that it does not work at all ;-)

Imagine pasting Ademar's favorite signature, :wq!
Or pasting "ahs" to a guest that thinks the control key is down
(a not uncommon occurence with VNC, in my personal experience).
So now you pasted "Select All, Delete, Save" in many GUI apps.

(insert infinite variations on the "doom" theme here)

>  Requiring any kind of software in the guest (paravirt
> keyboard driver, agent figuring the keymap, whatever else) kills that
> advantage.

Indeed. I was pointing out that any option that brought any modicum
of reasonable paste behavior (ignoring copy for the moment) required
some in-guest piece of software. So we might as well go for one that
is known to work, and also deals with the copy-from-guest issue.


>  And if we need guest cooperation anyway it is much better
> to just read/write the guest clipboard directly.

… which is exactly what VNC in the guest knows how to do in a
reasonable way, on a large number of platforms. So I don't see
any benefit to reinvent that wheel. I'd rather leverage the VNC
code to build a solid, multi-platform agent. You can extract the
clipboard code into some smaller agent that talks to a specific qemu
character device if you think it's a worthy optimization.


> 
> Standard keyboard seems to not be an option either.  The HID specs
> (https://usb.org/document-library/hid-usage-tables-121) lists a
> Unicode Page (Section 18), which looks like it could be used to
> send unicode chars to the guest.  Problem is (a) this is 16bit
> only (so no emoji) and (b) widely unimplemented in guest OSes.

Some operating systems have a way to enter unicode code points.
On Linux, you can sometimes type shift-control u (hex code), but it's a bit
dependent on the some alignment of mysterious planets.
On Windows, you can type <hex> alt-x. Sometimes. Or alt + <hex>.
On macOS, you need to enable Unicode Hex Input, so unreliable.

Not that I would recommend trying any of that ;-)

Re: vnc clipboard support

[Gerd Hoffmann]

  Hi,

> - What I'm suggesting is that qemu-vnc could then switch to simply
> relaying VNC traffic to that in-guest server. You'd get the smart update
> algorithm that Apple has put in place to deal with transparency and the
> like, as well as a level of guest OS integration that would otherwise be
> much harder to replicate.

Ok, so basically add a vnc proxy mode and allow switching between
internals server and proxy mode (automatically if possible, or
manually).

> If you don't have it, you fallback to virtual keystrokes, which you can't
> dispense with because of the early boot case, but which will at best
> deal with simple ASCII (making that work with a clipboard containing
> tabs and Control-C/Control-V characters is left as an exercise for the
> reader  ;-)

Well, even simple ascii is tricky.  How do you send 'z', as KEY_Z or as
KEY_Y ?

> Finally, an optimization I alluded to is to have an agent which is basically
> a stripped-down VNC server dealing with only the clipboard aspect, and
> that is your agent. In other words, you don't' invent yet another protocol,
> but you furiously copy-paste the existing VNC code.

Third mode: proxy only clipboard messages.

> Hope this sounds a bit less crazy explained that way…

Well, I can't see a way to make all that work nicely fully automatic ...

take care,
  Gerd

Re: vnc clipboard support

[Christophe de Dinechin]

> On 29 Jan 2021, at 12:49, Gerd Hoffmann <kraxel@redhat.com> wrote:
> 
>  Hi,
> 
>> - What I'm suggesting is that qemu-vnc could then switch to simply
>> relaying VNC traffic to that in-guest server. You'd get the smart update
>> algorithm that Apple has put in place to deal with transparency and the
>> like, as well as a level of guest OS integration that would otherwise be
>> much harder to replicate.
> 
> Ok, so basically add a vnc proxy mode and allow switching between
> internals server and proxy mode (automatically if possible, or
> manually).

Yes.

> 
>> If you don't have it, you fallback to virtual keystrokes, which you can't
>> dispense with because of the early boot case, but which will at best
>> deal with simple ASCII (making that work with a clipboard containing
>> tabs and Control-C/Control-V characters is left as an exercise for the
>> reader  ;-)
> 
> Well, even simple ascii is tricky.  How do you send 'z', as KEY_Z or as
> KEY_Y ?

Exactly. Scancode / keymap conversions are no joke. and qemu has
very little information about the current state of the guest in that respect.

> 
>> Finally, an optimization I alluded to is to have an agent which is basically
>> a stripped-down VNC server dealing with only the clipboard aspect, and
>> that is your agent. In other words, you don't' invent yet another protocol,
>> but you furiously copy-paste the existing VNC code.
> 
> Third mode: proxy only clipboard messages.
> 
>> Hope this sounds a bit less crazy explained that way…
> 
> Well, I can't see a way to make all that work nicely fully automatic ...

I see many aspects that are difficult, like disconnecting / reconnecting,
dealing with versions of the protocol, capability changes, and so on.

It was quite hard for SPICE to switch between console and streaming
agent, so I can understand why the prospect does not sound appealing.
Maybe that was just a bad idea.

> 
> take care,
>  Gerd
> 

Re: vnc clipboard support

[Laszlo Ersek]

On 01/28/21 18:12, Gerd Hoffmann wrote:
>   Hi folks,
> 
> I'm looking for a good way to implement cut+paste support for vnc.
> 
> The vnc core protocol has support for text/plain cut+paste, and there
> is an extension adding support for other formats.  That'll cover one
> part of the problem, exchanging cut+paste data between vnc client and
> qemu vnc server.
> 
> The tricky part is the second: the guest <=> qemu communication.
> I see basically two possible approaches here:
> 
>   (1) Have some guest agent (spice does it that way).
>       Advantage: more flexible, allows more features.
>       Disadvantage: requires agent in the guest.
>   
>   (2) Send text as key events.
>       Advantage: no guest agent needed.
>       Disadvantage: is translated by guests keyboard map, so qemu
>       needs to know the map for proper char -> key event translation.
>       Only works for text/plain and only for chars you can easily
>       type, anything needing input methods (emoji 😊 for example)
>       isn't going to fly.
> 
> I think that (1) is clearly the better way.  Given that the agent
> would need to run in user wayland/xorg session context the existing
> qemu-guest-agent will not work.  Also linking against some UI library
> like gtk3 for clipboard handling is not something we want for the
> qemu-guest-agent.  So we need another one, I'll name it
> qemu-clipboard-agent for the rest of this mail.  And we need a
> communication channel.
> 
> I'd tend to model the qemu-clipboard-agent simliar to the
> qemu-guest-agent, i.e. have some stream as communication path and run
> some stream protocol over it.
> 
> Stream options I see are (in order of personal preference):
> 
>   (1) New virtio-serial port.  virtio-serial likely is there anyway
>       for the qemu-guest-agent ...
> 
>   (2) Have qemu-clipboard-agent and qemu-guest-agent share the agent
>       channel, i.e. qemu-clipboard-agent will proxy everything through
>       qemu-guest-agent (spice does it that way).
> 
> Protocol options I see are (not sure yet which to prefer, need to have
> a closer look at the candidates):
> 
>   (1) Add clipboard commands to QMP and use these.
> 
>   (2) Reuse the clipboard bits of the vnc protocol (forward
>       VNC_MSG_CLIENT_CUT_TEXT messages to the guest agent)
> 
>   (3) Reuse the clipboard bits of the spice-agent protocol.
> 
>   (4) Reuse the clipboard bits of the wayland protocol.
> 
> Once we have sorted the qemu <-> guest communication path it should be
> possible to also hook up other UIs (specifically gtk) without too much
> effort.  Which probably makes (2) a rather poor choice.
> 
> Comments?
> Suggestions?
> Other ideas?

Just a random thought: the guest should not be able to sniff, steal, or
overwrite host-side clipboard (selection) content, without the host user
explicitly requesting a clipboard operation. I understand this is
generally a problem between X11 applications (not so on Wayland, if I
understand correctly), but I assume it becomes graver with
virtualization -- there shouldn't be an unintended channel between
host/guest, or even multiple guests. (Analogy: you can "ssh -X -Y" to
two remote machines at the same time, and assuming "xauth" etc on both
of those remote machines, you can run "gedit" on each of those remote
machines, and cut'n'paste will work between them, *even if* you don't
ask for it -- that's why ssh has a separate option called "-Y".
Cut'n'paste does not work without "-Y". So I believe this is something
to consider here.)

Thanks,
Laszlo

Re: vnc clipboard support

[Gerd Hoffmann]

  Hi,

> Just a random thought: the guest should not be able to sniff, steal, or
> overwrite host-side clipboard (selection) content, without the host user
> explicitly requesting a clipboard operation.

Sure.  vncviewer has command line options for that, and you can also
change it at runtime (F8 -> Options -> Input).

Additionally I'd make that optional at qemu level, i.e. something along
the lines of "-vnc $display,clipboard={on,off}" (details depending on
the actual implementation of course).

take care,
  Gerd

Re: vnc clipboard support

[Daniel P. Berrangé]

On Thu, Jan 28, 2021 at 06:57:24PM +0100, Laszlo Ersek wrote:
> On 01/28/21 18:12, Gerd Hoffmann wrote:
> >   Hi folks,
> > 
> > I'm looking for a good way to implement cut+paste support for vnc.
> > 
> > The vnc core protocol has support for text/plain cut+paste, and there
> > is an extension adding support for other formats.  That'll cover one
> > part of the problem, exchanging cut+paste data between vnc client and
> > qemu vnc server.
> > 
> > The tricky part is the second: the guest <=> qemu communication.
> > I see basically two possible approaches here:
> > 
> >   (1) Have some guest agent (spice does it that way).
> >       Advantage: more flexible, allows more features.
> >       Disadvantage: requires agent in the guest.
> >   
> >   (2) Send text as key events.
> >       Advantage: no guest agent needed.
> >       Disadvantage: is translated by guests keyboard map, so qemu
> >       needs to know the map for proper char -> key event translation.
> >       Only works for text/plain and only for chars you can easily
> >       type, anything needing input methods (emoji 😊 for example)
> >       isn't going to fly.
> > 
> > I think that (1) is clearly the better way.  Given that the agent
> > would need to run in user wayland/xorg session context the existing
> > qemu-guest-agent will not work.  Also linking against some UI library
> > like gtk3 for clipboard handling is not something we want for the
> > qemu-guest-agent.  So we need another one, I'll name it
> > qemu-clipboard-agent for the rest of this mail.  And we need a
> > communication channel.
> > 
> > I'd tend to model the qemu-clipboard-agent simliar to the
> > qemu-guest-agent, i.e. have some stream as communication path and run
> > some stream protocol over it.
> > 
> > Stream options I see are (in order of personal preference):
> > 
> >   (1) New virtio-serial port.  virtio-serial likely is there anyway
> >       for the qemu-guest-agent ...
> > 
> >   (2) Have qemu-clipboard-agent and qemu-guest-agent share the agent
> >       channel, i.e. qemu-clipboard-agent will proxy everything through
> >       qemu-guest-agent (spice does it that way).
> > 
> > Protocol options I see are (not sure yet which to prefer, need to have
> > a closer look at the candidates):
> > 
> >   (1) Add clipboard commands to QMP and use these.
> > 
> >   (2) Reuse the clipboard bits of the vnc protocol (forward
> >       VNC_MSG_CLIENT_CUT_TEXT messages to the guest agent)
> > 
> >   (3) Reuse the clipboard bits of the spice-agent protocol.
> > 
> >   (4) Reuse the clipboard bits of the wayland protocol.
> > 
> > Once we have sorted the qemu <-> guest communication path it should be
> > possible to also hook up other UIs (specifically gtk) without too much
> > effort.  Which probably makes (2) a rather poor choice.
> > 
> > Comments?
> > Suggestions?
> > Other ideas?
> 
> Just a random thought: the guest should not be able to sniff, steal, or
> overwrite host-side clipboard (selection) content, without the host user
> explicitly requesting a clipboard operation. I understand this is
> generally a problem between X11 applications (not so on Wayland, if I
> understand correctly), but I assume it becomes graver with
> virtualization -- there shouldn't be an unintended channel between
> host/guest, or even multiple guests. (Analogy: you can "ssh -X -Y" to
> two remote machines at the same time, and assuming "xauth" etc on both
> of those remote machines, you can run "gedit" on each of those remote
> machines, and cut'n'paste will work between them, *even if* you don't
> ask for it -- that's why ssh has a separate option called "-Y".
> Cut'n'paste does not work without "-Y". So I believe this is something
> to consider here.)

Yes, that's a known risk. SPICE lets you disable clipboard synchronization
in QEMU, and any client app should also support doing the same. Basically
only want clipboard sync if you're using a trustworthy guest. eg where
both host and guest admin are the same person, and even then you might
not want it sometimes.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: vnc clipboard support

[Marc-André Lureau]

[-- Attachment #1: Type: text/plain, Size: 5599 bytes --]

Hi

On Thu, Jan 28, 2021 at 9:14 PM Gerd Hoffmann <kraxel@redhat.com> wrote:

>   Hi folks,
>
> I'm looking for a good way to implement cut+paste support for vnc.
>
> The vnc core protocol has support for text/plain cut+paste, and there
> is an extension adding support for other formats.  That'll cover one
> part of the problem, exchanging cut+paste data between vnc client and
> qemu vnc server.
>
> The tricky part is the second: the guest <=> qemu communication.
> I see basically two possible approaches here:
>
>   (1) Have some guest agent (spice does it that way).
>       Advantage: more flexible, allows more features.
>       Disadvantage: requires agent in the guest.
>
>   (2) Send text as key events.
>       Advantage: no guest agent needed.
>       Disadvantage: is translated by guests keyboard map, so qemu
>       needs to know the map for proper char -> key event translation.
>       Only works for text/plain and only for chars you can easily
>       type, anything needing input methods (emoji 😊 for example)
>       isn't going to fly.
>
> I think that (1) is clearly the better way.  Given that the agent
> would need to run in user wayland/xorg session context the existing
> qemu-guest-agent will not work.  Also linking against some UI library
> like gtk3 for clipboard handling is not something we want for the
> qemu-guest-agent.  So we need another one, I'll name it
> qemu-clipboard-agent for the rest of this mail.  And we need a
> communication channel.
>
> I'd tend to model the qemu-clipboard-agent simliar to the
> qemu-guest-agent, i.e. have some stream as communication path and run
> some stream protocol over it.
>
> Stream options I see are (in order of personal preference):
>
>   (1) New virtio-serial port.  virtio-serial likely is there anyway
>       for the qemu-guest-agent ...
>
>   (2) Have qemu-clipboard-agent and qemu-guest-agent share the agent
>       channel, i.e. qemu-clipboard-agent will proxy everything through
>       qemu-guest-agent (spice does it that way).
>
> Protocol options I see are (not sure yet which to prefer, need to have
> a closer look at the candidates):
>
>   (1) Add clipboard commands to QMP and use these.
>
>   (2) Reuse the clipboard bits of the vnc protocol (forward
>       VNC_MSG_CLIENT_CUT_TEXT messages to the guest agent)
>
>   (3) Reuse the clipboard bits of the spice-agent protocol.
>
>   (4) Reuse the clipboard bits of the wayland protocol.
>
> Once we have sorted the qemu <-> guest communication path it should be
> possible to also hook up other UIs (specifically gtk) without too much
> effort.  Which probably makes (2) a rather poor choice.
>
> Comments?
> Suggestions?
> Other ideas?
>


I also had recently some thoughts about how to implement clipboard sharing
in a more general way for QEMU.

I admit I like Christophe's suggestion ("it's somebody else problem"), but
it falls short to me as I don't know of a common open-source remoting
solution for various operating systems, and I don't see how it could
integrate well with our UI and remote protocols. Or look at reusing some
VirtualBox code perhaps?

Some things I keep in mind:
- the spice protocol had a number of iterations to fix some races. It would
be great not to repeat the same mistakes, and I don't know if VNC have the
same flaws or not.
- the spice agent design isn't great: the system agent proxies messages to
the active session. It would be nice if the new solution didn't have such a
middle-man.
- with wayland, clipboard sharing isn't really possible. Or not in a
seamless way at least. Today it kinda works with some X11 compatibility
extensions, but this will eventually go away or change behaviour.
- the GNOME desktop is working on remoting using RDP, and they are
implementing a DBus interface for it (
https://gitlab.gnome.org/jadahl/mutter/-/commits/wip/remote-desktop-clipboard
)
- it's not just about clipboard. We would also want to have some kind of
drag and drop (even if limited to files like Spice atm). We may want some
windowing integration. We may also want to have access to some desktop
services: apps, documents etc.. And what's not.

That leads me to think that virtio-serial is not very well suited, as it
doesn't allow various services / processes to come and go. I see vsock as a
much better alternative. (I also wonder if it handles control flow any
better btw)

I think we shoud work on getting the free desktops our best-class support.
To me, this means we need to speak the lingua franca, which is DBus. The
great thing is that DBus is also equipped to handle services that come and
go, handling discovery, introspection etc. Various services are already
available. As mentioned earlier, that's what the GNOME desktop will offer
for clipboard sharing. There are good chances other desktops will follow if
that design works, as it should be easy for them to implement the same
service. That means good reuse of existing desktop code. Speaking DBus on
Windows, MacOS or Android isn't an issue. However, vsock support may be a
bit tricky atm.

Fwiw, DBus doesn't yet officially support vsock connections:
https://gitlab.freedesktop.org/dbus/dbus/-/merge_requests/200. This a minor
detail, as once you give it a fd for transport, it doesn't really care (I
also took care of glib!1892 and Rust zbus)

Oh and of course, since this is a new daemon, it would be really a shame
not to write it in a modern language (hint! ;-).

Hope that helps,

-- 
Marc-André Lureau

[-- Attachment #2: Type: text/html, Size: 6648 bytes --]

Re: vnc clipboard support

[BALATON Zoltan]

[-- Attachment #1: Type: text/plain, Size: 3265 bytes --]

On Fri, 29 Jan 2021, Marc-André Lureau wrote:
> I also had recently some thoughts about how to implement clipboard sharing
> in a more general way for QEMU.
>
> I admit I like Christophe's suggestion ("it's somebody else problem"), but
> it falls short to me as I don't know of a common open-source remoting
> solution for various operating systems, and I don't see how it could
> integrate well with our UI and remote protocols. Or look at reusing some
> VirtualBox code perhaps?

I also thought about VirtualBox may worth a look at as that supports 
clipboard sharing and there's also Synergy 
[https://en.wikipedia.org/wiki/Synergy_(software)] that I don't know much 
but it's cross platform and may support clipboard sharing. It seems to 
have an open source fork called Barrier: 
https://github.com/debauchee/barrier

Regards,
BALATON Zoltan

> Some things I keep in mind:
> - the spice protocol had a number of iterations to fix some races. It would
> be great not to repeat the same mistakes, and I don't know if VNC have the
> same flaws or not.
> - the spice agent design isn't great: the system agent proxies messages to
> the active session. It would be nice if the new solution didn't have such a
> middle-man.
> - with wayland, clipboard sharing isn't really possible. Or not in a
> seamless way at least. Today it kinda works with some X11 compatibility
> extensions, but this will eventually go away or change behaviour.
> - the GNOME desktop is working on remoting using RDP, and they are
> implementing a DBus interface for it (
> https://gitlab.gnome.org/jadahl/mutter/-/commits/wip/remote-desktop-clipboard
> )
> - it's not just about clipboard. We would also want to have some kind of
> drag and drop (even if limited to files like Spice atm). We may want some
> windowing integration. We may also want to have access to some desktop
> services: apps, documents etc.. And what's not.
>
> That leads me to think that virtio-serial is not very well suited, as it
> doesn't allow various services / processes to come and go. I see vsock as a
> much better alternative. (I also wonder if it handles control flow any
> better btw)
>
> I think we shoud work on getting the free desktops our best-class support.
> To me, this means we need to speak the lingua franca, which is DBus. The
> great thing is that DBus is also equipped to handle services that come and
> go, handling discovery, introspection etc. Various services are already
> available. As mentioned earlier, that's what the GNOME desktop will offer
> for clipboard sharing. There are good chances other desktops will follow if
> that design works, as it should be easy for them to implement the same
> service. That means good reuse of existing desktop code. Speaking DBus on
> Windows, MacOS or Android isn't an issue. However, vsock support may be a
> bit tricky atm.
>
> Fwiw, DBus doesn't yet officially support vsock connections:
> https://gitlab.freedesktop.org/dbus/dbus/-/merge_requests/200. This a minor
> detail, as once you give it a fd for transport, it doesn't really care (I
> also took care of glib!1892 and Rust zbus)
>
> Oh and of course, since this is a new daemon, it would be really a shame
> not to write it in a modern language (hint! ;-).
>
> Hope that helps,
>
>

Re: vnc clipboard support

[Christophe de Dinechin]

[-- Attachment #1: Type: text/plain, Size: 6039 bytes --]



Christophe
(Typos are from my iPhone)

> Le 28 janv. 2021 à 21:24, Marc-André Lureau <marcandre.lureau@gmail.com> a écrit :
> 
> 
> Hi
> 
>> On Thu, Jan 28, 2021 at 9:14 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
>>   Hi folks,
>> 
>> I'm looking for a good way to implement cut+paste support for vnc.
>> 
>> The vnc core protocol has support for text/plain cut+paste, and there
>> is an extension adding support for other formats.  That'll cover one
>> part of the problem, exchanging cut+paste data between vnc client and
>> qemu vnc server.
>> 
>> The tricky part is the second: the guest <=> qemu communication.
>> I see basically two possible approaches here:
>> 
>>   (1) Have some guest agent (spice does it that way).
>>       Advantage: more flexible, allows more features.
>>       Disadvantage: requires agent in the guest.
>> 
>>   (2) Send text as key events.
>>       Advantage: no guest agent needed.
>>       Disadvantage: is translated by guests keyboard map, so qemu
>>       needs to know the map for proper char -> key event translation.
>>       Only works for text/plain and only for chars you can easily
>>       type, anything needing input methods (emoji 😊 for example)
>>       isn't going to fly.
>> 
>> I think that (1) is clearly the better way.  Given that the agent
>> would need to run in user wayland/xorg session context the existing
>> qemu-guest-agent will not work.  Also linking against some UI library
>> like gtk3 for clipboard handling is not something we want for the
>> qemu-guest-agent.  So we need another one, I'll name it
>> qemu-clipboard-agent for the rest of this mail.  And we need a
>> communication channel.
>> 
>> I'd tend to model the qemu-clipboard-agent simliar to the
>> qemu-guest-agent, i.e. have some stream as communication path and run
>> some stream protocol over it.
>> 
>> Stream options I see are (in order of personal preference):
>> 
>>   (1) New virtio-serial port.  virtio-serial likely is there anyway
>>       for the qemu-guest-agent ...
>> 
>>   (2) Have qemu-clipboard-agent and qemu-guest-agent share the agent
>>       channel, i.e. qemu-clipboard-agent will proxy everything through
>>       qemu-guest-agent (spice does it that way).
>> 
>> Protocol options I see are (not sure yet which to prefer, need to have
>> a closer look at the candidates):
>> 
>>   (1) Add clipboard commands to QMP and use these.
>> 
>>   (2) Reuse the clipboard bits of the vnc protocol (forward
>>       VNC_MSG_CLIENT_CUT_TEXT messages to the guest agent)
>> 
>>   (3) Reuse the clipboard bits of the spice-agent protocol.
>> 
>>   (4) Reuse the clipboard bits of the wayland protocol.
>> 
>> Once we have sorted the qemu <-> guest communication path it should be
>> possible to also hook up other UIs (specifically gtk) without too much
>> effort.  Which probably makes (2) a rather poor choice.
>> 
>> Comments?
>> Suggestions?
>> Other ideas?
> 
> 
> 
> I also had recently some thoughts about how to implement clipboard sharing in a more general way for QEMU.
> 
> I admit I like Christophe's suggestion ("it's somebody else problem"), but it falls short to me as I don't know of a common open-source remoting solution for various operating systems, and I don't see how it could integrate well with our UI and remote protocols. Or look at reusing some VirtualBox code perhaps?

Just to clarify, I did not think of my suggestion as “SEP” but more as a way to leverage existing knowledge regarding guest OS clipboard. More a “what should we steal” approach. 

> 
> Some things I keep in mind:
> - the spice protocol had a number of iterations to fix some races. It would be great not to repeat the same mistakes, and I don't know if VNC have the same flaws or not.
> - the spice agent design isn't great: the system agent proxies messages to the active session. It would be nice if the new solution didn't have such a middle-man.
> - with wayland, clipboard sharing isn't really possible. Or not in a seamless way at least. Today it kinda works with some X11 compatibility extensions, but this will eventually go away or change behaviour.
> - the GNOME desktop is working on remoting using RDP, and they are implementing a DBus interface for it (https://gitlab.gnome.org/jadahl/mutter/-/commits/wip/remote-desktop-clipboard)
> - it's not just about clipboard. We would also want to have some kind of drag and drop (even if limited to files like Spice atm). We may want some windowing integration. We may also want to have access to some desktop services: apps, documents etc.. And what's not.
> 
> That leads me to think that virtio-serial is not very well suited, as it doesn't allow various services / processes to come and go. I see vsock as a much better alternative. (I also wonder if it handles control flow any better btw)
> 
> I think we shoud work on getting the free desktops our best-class support. To me, this means we need to speak the lingua franca, which is DBus. The great thing is that DBus is also equipped to handle services that come and go, handling discovery, introspection etc. Various services are already available. As mentioned earlier, that's what the GNOME desktop will offer for clipboard sharing. There are good chances other desktops will follow if that design works, as it should be easy for them to implement the same service. That means good reuse of existing desktop code. Speaking DBus on Windows, MacOS or Android isn't an issue. However, vsock support may be a bit tricky atm.
> 
> Fwiw, DBus doesn't yet officially support vsock connections: https://gitlab.freedesktop.org/dbus/dbus/-/merge_requests/200. This a minor detail, as once you give it a fd for transport, it doesn't really care (I also took care of glib!1892 and Rust zbus)
> 
> Oh and of course, since this is a new daemon, it would be really a shame not to write it in a modern language (hint! ;-).
> 
> Hope that helps,
> 
> -- 
> Marc-André Lureau

[-- Attachment #2: Type: text/html, Size: 7625 bytes --]

Re: vnc clipboard support

[Gerd Hoffmann]

  Hi,

> Some things I keep in mind:
> - the spice protocol had a number of iterations to fix some races. It would
> be great not to repeat the same mistakes, and I don't know if VNC have the
> same flaws or not.

Sure.  One of the reasons I've started this discussion ;)

> - the GNOME desktop is working on remoting using RDP, and they are
> implementing a DBus interface for it (
> https://gitlab.gnome.org/jadahl/mutter/-/commits/wip/remote-desktop-clipboard
> )
> - it's not just about clipboard. We would also want to have some kind of
> drag and drop (even if limited to files like Spice atm). We may want some
> windowing integration. We may also want to have access to some desktop
> services: apps, documents etc.. And what's not.

Well.  That is a completely different level of desktop integration.
It surely makes sense to have that, and dbus-over-vsock looks like
a reasonable choice.

But I'm more after a solution for the "paste that long link into the
guest firefox" problem, without putting everything upside-down 😊

take care,
  Gerd

Re: vnc clipboard support

[Marc-André Lureau]

[-- Attachment #1: Type: text/plain, Size: 680 bytes --]

Hi

On Fri, Jan 29, 2021 at 12:28 PM Gerd Hoffmann <kraxel@redhat.com> wrote:

>
> Well.  That is a completely different level of desktop integration.
> It surely makes sense to have that, and dbus-over-vsock looks like
> a reasonable choice.
>

(using vsock & dbus also goes in the direction where UIs and remote server
can run in different processes and come and go btw)

But I'm more after a solution for the "paste that long link into the
> guest firefox" problem, without putting everything upside-down 😊
>

In this case, I would say an agent-less text-to-type solution (even with
its very limited scope) is quite attractive.

-- 
Marc-André Lureau

[-- Attachment #2: Type: text/html, Size: 1255 bytes --]

Re: vnc clipboard support

[Gerd Hoffmann]

On Fri, Jan 29, 2021 at 12:34:32PM +0400, Marc-André Lureau wrote:
> Hi
> 
> On Fri, Jan 29, 2021 at 12:28 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
> 
> >
> > Well.  That is a completely different level of desktop integration.
> > It surely makes sense to have that, and dbus-over-vsock looks like
> > a reasonable choice.
> >
> 
> (using vsock & dbus also goes in the direction where UIs and remote server
> can run in different processes and come and go btw)

Yep, that would mostly (maybe even completely) implemented outside qemu
thanks to vhost vsock.

> > But I'm more after a solution for the "paste that long link into the
> > guest firefox" problem, without putting everything upside-down 😊
> 
> In this case, I would say an agent-less text-to-type solution (even with
> its very limited scope) is quite attractive.

Any suggestions for the keyboard map problem?

take care,
  Gerd

Re: vnc clipboard support

[Daniel P. Berrangé]

On Fri, Jan 29, 2021 at 12:34:32PM +0400, Marc-André Lureau wrote:
> Hi
> 
> On Fri, Jan 29, 2021 at 12:28 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
> 
> >
> > Well.  That is a completely different level of desktop integration.
> > It surely makes sense to have that, and dbus-over-vsock looks like
> > a reasonable choice.
> >
> 
> (using vsock & dbus also goes in the direction where UIs and remote server
> can run in different processes and come and go btw)
> 
> But I'm more after a solution for the "paste that long link into the
> > guest firefox" problem, without putting everything upside-down 😊
> >
> 
> In this case, I would say an agent-less text-to-type solution (even with
> its very limited scope) is quite attractive.

If you want to implement clipboard paste using text-to-type, then there
is no work required at all in QEMU. That can be done in the VNC client
simply by sending keystrokes. Even for something moderately short like
a URL though this is quite painfully slow because you need a sleep
between each keystroke to avoid overflowing the keyboard driver input
buffers


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: vnc clipboard support

[Daniel P. Berrangé]

On Fri, Jan 29, 2021 at 12:18:19AM +0400, Marc-André Lureau wrote:
> Hi
> 
> On Thu, Jan 28, 2021 at 9:14 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
> 
> >   Hi folks,
> >
> > I'm looking for a good way to implement cut+paste support for vnc.
> >
> > The vnc core protocol has support for text/plain cut+paste, and there
> > is an extension adding support for other formats.  That'll cover one
> > part of the problem, exchanging cut+paste data between vnc client and
> > qemu vnc server.
> >
> > The tricky part is the second: the guest <=> qemu communication.
> > I see basically two possible approaches here:
> >
> >   (1) Have some guest agent (spice does it that way).
> >       Advantage: more flexible, allows more features.
> >       Disadvantage: requires agent in the guest.
> >
> >   (2) Send text as key events.
> >       Advantage: no guest agent needed.
> >       Disadvantage: is translated by guests keyboard map, so qemu
> >       needs to know the map for proper char -> key event translation.
> >       Only works for text/plain and only for chars you can easily
> >       type, anything needing input methods (emoji 😊 for example)
> >       isn't going to fly.
> >
> > I think that (1) is clearly the better way.  Given that the agent
> > would need to run in user wayland/xorg session context the existing
> > qemu-guest-agent will not work.  Also linking against some UI library
> > like gtk3 for clipboard handling is not something we want for the
> > qemu-guest-agent.  So we need another one, I'll name it
> > qemu-clipboard-agent for the rest of this mail.  And we need a
> > communication channel.
> >
> > I'd tend to model the qemu-clipboard-agent simliar to the
> > qemu-guest-agent, i.e. have some stream as communication path and run
> > some stream protocol over it.
> >
> > Stream options I see are (in order of personal preference):
> >
> >   (1) New virtio-serial port.  virtio-serial likely is there anyway
> >       for the qemu-guest-agent ...
> >
> >   (2) Have qemu-clipboard-agent and qemu-guest-agent share the agent
> >       channel, i.e. qemu-clipboard-agent will proxy everything through
> >       qemu-guest-agent (spice does it that way).
> >
> > Protocol options I see are (not sure yet which to prefer, need to have
> > a closer look at the candidates):
> >
> >   (1) Add clipboard commands to QMP and use these.
> >
> >   (2) Reuse the clipboard bits of the vnc protocol (forward
> >       VNC_MSG_CLIENT_CUT_TEXT messages to the guest agent)
> >
> >   (3) Reuse the clipboard bits of the spice-agent protocol.
> >
> >   (4) Reuse the clipboard bits of the wayland protocol.
> >
> > Once we have sorted the qemu <-> guest communication path it should be
> > possible to also hook up other UIs (specifically gtk) without too much
> > effort.  Which probably makes (2) a rather poor choice.
> >
> > Comments?
> > Suggestions?
> > Other ideas?
> >
> 
> 
> I also had recently some thoughts about how to implement clipboard sharing
> in a more general way for QEMU.
> 
> I admit I like Christophe's suggestion ("it's somebody else problem"), but
> it falls short to me as I don't know of a common open-source remoting
> solution for various operating systems, and I don't see how it could
> integrate well with our UI and remote protocols. Or look at reusing some
> VirtualBox code perhaps?
> 
> Some things I keep in mind:
> - the spice protocol had a number of iterations to fix some races. It would
> be great not to repeat the same mistakes, and I don't know if VNC have the
> same flaws or not.
> - the spice agent design isn't great: the system agent proxies messages to
> the active session. It would be nice if the new solution didn't have such a
> middle-man.

I was going to say that's a limtation of the use of virtio-serial, and
to some extent that is true, but it is also not true. There is no reason
it had to limit itselt to 1 virtio-serial device. It would have been
valid to have 1 virtio-serial device for the system level functionality,
and then 1 virtio-serial device per guest seat. Any guest OS configured
for multi-seat is already going to have extra devices added to QEMU
for each seat, so adding a virtio-serial device per seat would have
been easy enough and avoided the crazy proxying.

> - with wayland, clipboard sharing isn't really possible. Or not in a
> seamless way at least. Today it kinda works with some X11 compatibility
> extensions, but this will eventually go away or change behaviour.
> - the GNOME desktop is working on remoting using RDP, and they are
> implementing a DBus interface for it (
> https://gitlab.gnome.org/jadahl/mutter/-/commits/wip/remote-desktop-clipboard
> )
> - it's not just about clipboard. We would also want to have some kind of
> drag and drop (even if limited to files like Spice atm). We may want some
> windowing integration. We may also want to have access to some desktop
> services: apps, documents etc.. And what's not.

Ultimately that all says we need a "remote desktop agent", which is
extensible to other desktop integration features. Essentially that
is what the spice agent was aiming to provide. 

> That leads me to think that virtio-serial is not very well suited, as it
> doesn't allow various services / processes to come and go. I see vsock as a
> much better alternative. (I also wonder if it handles control flow any
> better btw)

virtio-serial does make it a bit easier to have the host backend side
self-contained in QEMU though. I think it is nice that nothing else on
the host ever sees the agent backend, as it is a private internal channel
between SPICE & QEMU's virtio-serial impl via the special chardev type.

vsock gets exposed to the host OS in general, and though you
can have QEMU open the vsock device internally, it feels a bit silly to
have one part of QEMU writing to a vsock device, and then another part
of QEMU reading back from the very same device. Especially because you
have now introduced the extra problem of having to allocate unique
vsock addresses for each instance and deal with possibility of external
apps maliciously trying to interact with your vsock backend.

As above though, I think the way spice used virtio-serial was suboptimal
and it should have had one extra virtio-serial device per seat.

> I think we shoud work on getting the free desktops our best-class support.
> To me, this means we need to speak the lingua franca, which is DBus. The
> great thing is that DBus is also equipped to handle services that come and
> go, handling discovery, introspection etc. Various services are already
> available. As mentioned earlier, that's what the GNOME desktop will offer
> for clipboard sharing. There are good chances other desktops will follow if
> that design works, as it should be easy for them to implement the same
> service. That means good reuse of existing desktop code. Speaking DBus on
> Windows, MacOS or Android isn't an issue. However, vsock support may be a
> bit tricky atm.

Speaking DBus is only useful though if the guest desktop actually implements
the DBus interfaces that we're speaking. The remote desktop clipboard work
shown above is only going to be available on modern Linux guest OS, and
possibly only a subset of them. IOW on the guest side, we're going to have
to have platform specific backends to do the integration in whatever way
is best suited for each OS. I see windows, macos, android, Wayland and
X11 as distinct backends for the guest side.

The same on the host client side, though in that case it is all magically
handled by GTK for us, so we don't need to care about the platform
differences ourselves.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: vnc clipboard support

[Marc-André Lureau]

Hi

On Fri, Jan 29, 2021 at 3:24 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Fri, Jan 29, 2021 at 12:18:19AM +0400, Marc-André Lureau wrote:
> > Hi
> >
> > On Thu, Jan 28, 2021 at 9:14 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
> >
> > >   Hi folks,
> > >
> > > I'm looking for a good way to implement cut+paste support for vnc.
> > >
> > > The vnc core protocol has support for text/plain cut+paste, and there
> > > is an extension adding support for other formats.  That'll cover one
> > > part of the problem, exchanging cut+paste data between vnc client and
> > > qemu vnc server.
> > >
> > > The tricky part is the second: the guest <=> qemu communication.
> > > I see basically two possible approaches here:
> > >
> > >   (1) Have some guest agent (spice does it that way).
> > >       Advantage: more flexible, allows more features.
> > >       Disadvantage: requires agent in the guest.
> > >
> > >   (2) Send text as key events.
> > >       Advantage: no guest agent needed.
> > >       Disadvantage: is translated by guests keyboard map, so qemu
> > >       needs to know the map for proper char -> key event translation.
> > >       Only works for text/plain and only for chars you can easily
> > >       type, anything needing input methods (emoji for example)
> > >       isn't going to fly.
> > >
> > > I think that (1) is clearly the better way.  Given that the agent
> > > would need to run in user wayland/xorg session context the existing
> > > qemu-guest-agent will not work.  Also linking against some UI library
> > > like gtk3 for clipboard handling is not something we want for the
> > > qemu-guest-agent.  So we need another one, I'll name it
> > > qemu-clipboard-agent for the rest of this mail.  And we need a
> > > communication channel.
> > >
> > > I'd tend to model the qemu-clipboard-agent simliar to the
> > > qemu-guest-agent, i.e. have some stream as communication path and run
> > > some stream protocol over it.
> > >
> > > Stream options I see are (in order of personal preference):
> > >
> > >   (1) New virtio-serial port.  virtio-serial likely is there anyway
> > >       for the qemu-guest-agent ...
> > >
> > >   (2) Have qemu-clipboard-agent and qemu-guest-agent share the agent
> > >       channel, i.e. qemu-clipboard-agent will proxy everything through
> > >       qemu-guest-agent (spice does it that way).
> > >
> > > Protocol options I see are (not sure yet which to prefer, need to have
> > > a closer look at the candidates):
> > >
> > >   (1) Add clipboard commands to QMP and use these.
> > >
> > >   (2) Reuse the clipboard bits of the vnc protocol (forward
> > >       VNC_MSG_CLIENT_CUT_TEXT messages to the guest agent)
> > >
> > >   (3) Reuse the clipboard bits of the spice-agent protocol.
> > >
> > >   (4) Reuse the clipboard bits of the wayland protocol.
> > >
> > > Once we have sorted the qemu <-> guest communication path it should be
> > > possible to also hook up other UIs (specifically gtk) without too much
> > > effort.  Which probably makes (2) a rather poor choice.
> > >
> > > Comments?
> > > Suggestions?
> > > Other ideas?
> > >
> >
> >
> > I also had recently some thoughts about how to implement clipboard sharing
> > in a more general way for QEMU.
> >
> > I admit I like Christophe's suggestion ("it's somebody else problem"), but
> > it falls short to me as I don't know of a common open-source remoting
> > solution for various operating systems, and I don't see how it could
> > integrate well with our UI and remote protocols. Or look at reusing some
> > VirtualBox code perhaps?
> >
> > Some things I keep in mind:
> > - the spice protocol had a number of iterations to fix some races. It would
> > be great not to repeat the same mistakes, and I don't know if VNC have the
> > same flaws or not.
> > - the spice agent design isn't great: the system agent proxies messages to
> > the active session. It would be nice if the new solution didn't have such a
> > middle-man.
>
> I was going to say that's a limtation of the use of virtio-serial, and
> to some extent that is true, but it is also not true. There is no reason
> it had to limit itselt to 1 virtio-serial device. It would have been
> valid to have 1 virtio-serial device for the system level functionality,
> and then 1 virtio-serial device per guest seat. Any guest OS configured
> for multi-seat is already going to have extra devices added to QEMU
> for each seat, so adding a virtio-serial device per seat would have
> been easy enough and avoided the crazy proxying.
>
> > - with wayland, clipboard sharing isn't really possible. Or not in a
> > seamless way at least. Today it kinda works with some X11 compatibility
> > extensions, but this will eventually go away or change behaviour.
> > - the GNOME desktop is working on remoting using RDP, and they are
> > implementing a DBus interface for it (
> > https://gitlab.gnome.org/jadahl/mutter/-/commits/wip/remote-desktop-clipboard
> > )
> > - it's not just about clipboard. We would also want to have some kind of
> > drag and drop (even if limited to files like Spice atm). We may want some
> > windowing integration. We may also want to have access to some desktop
> > services: apps, documents etc.. And what's not.
>
> Ultimately that all says we need a "remote desktop agent", which is
> extensible to other desktop integration features. Essentially that
> is what the spice agent was aiming to provide.
>
> > That leads me to think that virtio-serial is not very well suited, as it
> > doesn't allow various services / processes to come and go. I see vsock as a
> > much better alternative. (I also wonder if it handles control flow any
> > better btw)
>
> virtio-serial does make it a bit easier to have the host backend side
> self-contained in QEMU though. I think it is nice that nothing else on
> the host ever sees the agent backend, as it is a private internal channel
> between SPICE & QEMU's virtio-serial impl via the special chardev type.
>
> vsock gets exposed to the host OS in general, and though you

Not necessarily with vhost-user-vsock though.

> can have QEMU open the vsock device internally, it feels a bit silly to
> have one part of QEMU writing to a vsock device, and then another part
> of QEMU reading back from the very same device. Especially because you
> have now introduced the extra problem of having to allocate unique
> vsock addresses for each instance and deal with possibility of external
> apps maliciously trying to interact with your vsock backend.
>
> As above though, I think the way spice used virtio-serial was suboptimal
> and it should have had one extra virtio-serial device per seat.

And per service? and libvirt to hotplug stuff? Sounds insane to me.
And what about services that need to handle several connections in the
guest. For example, the way spice-webdavd works is really a pain, it
has to multiplex guest connections over virtio-serial... All of this
would be so much simpler with a single vsock connection and some kind
of bus.

>
> > I think we shoud work on getting the free desktops our best-class support.
> > To me, this means we need to speak the lingua franca, which is DBus. The
> > great thing is that DBus is also equipped to handle services that come and
> > go, handling discovery, introspection etc. Various services are already
> > available. As mentioned earlier, that's what the GNOME desktop will offer
> > for clipboard sharing. There are good chances other desktops will follow if
> > that design works, as it should be easy for them to implement the same
> > service. That means good reuse of existing desktop code. Speaking DBus on
> > Windows, MacOS or Android isn't an issue. However, vsock support may be a
> > bit tricky atm.
>
> Speaking DBus is only useful though if the guest desktop actually implements
> the DBus interfaces that we're speaking. The remote desktop clipboard work
> shown above is only going to be available on modern Linux guest OS, and
> possibly only a subset of them. IOW on the guest side, we're going to have
> to have platform specific backends to do the integration in whatever way
> is best suited for each OS. I see windows, macos, android, Wayland and
> X11 as distinct backends for the guest side.

Sure, this adaptation to various OS is needed regardless of DBus as a
protocol or not.

But we are in a much better position if we actually shape the future
and put our requirements there rather than playing a catch up game the
way desktop agents usually do. Iow, I think we need deeper integration
in the desktop. The good news is that we have pretty much the same
requirements has remote desktop solutions.


-- 
Marc-André Lureau

Re: vnc clipboard support

[Daniel P. Berrangé]

On Fri, Jan 29, 2021 at 03:58:00PM +0400, Marc-André Lureau wrote:
> Hi
> 
> On Fri, Jan 29, 2021 at 3:24 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> >
> > On Fri, Jan 29, 2021 at 12:18:19AM +0400, Marc-André Lureau wrote:
> > can have QEMU open the vsock device internally, it feels a bit silly to
> > have one part of QEMU writing to a vsock device, and then another part
> > of QEMU reading back from the very same device. Especially because you
> > have now introduced the extra problem of having to allocate unique
> > vsock addresses for each instance and deal with possibility of external
> > apps maliciously trying to interact with your vsock backend.
> >
> > As above though, I think the way spice used virtio-serial was suboptimal
> > and it should have had one extra virtio-serial device per seat.
> 
> And per service? and libvirt to hotplug stuff? Sounds insane to me.
> And what about services that need to handle several connections in the
> guest. For example, the way spice-webdavd works is really a pain, it
> has to multiplex guest connections over virtio-serial... All of this
> would be so much simpler with a single vsock connection and some kind
> of bus.

I wasn't really suggesting it for something like spice-webdavd, just
the spice agent functionality, which is not really connection-oriented
stuff. ie its a simple RPC like service where there's only ever one
client and server endpoint.  spice-webdavd would clearly be better
based on vsock, since it is inherantly a multi-connection architecture.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: vnc clipboard support

[Gerd Hoffmann]

  Hi,

> - the spice protocol had a number of iterations to fix some races. It would
> be great not to repeat the same mistakes, and I don't know if VNC have the
> same flaws or not.

# grep CAP.*CLIPBOARD.*, /usr/include/spice-1/spice/vd_agent.h 
    VD_AGENT_CAP_CLIPBOARD,
    VD_AGENT_CAP_CLIPBOARD_BY_DEMAND,
    VD_AGENT_CAP_CLIPBOARD_SELECTION,
    VD_AGENT_CAP_MAX_CLIPBOARD,
    VD_AGENT_CAP_CLIPBOARD_NO_RELEASE_ON_REGRAB,
    VD_AGENT_CAP_CLIPBOARD_GRAB_SERIAL,

Oh yea, I see.  Is that (and the rest of the protocol) documented
somewhere, other than vd_agent.h?

thanks,
  Gerd

Re: vnc clipboard support

[Marc-André Lureau]

Hi

On Fri, Jan 29, 2021 at 4:03 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
>
>   Hi,
>
> > - the spice protocol had a number of iterations to fix some races. It would
> > be great not to repeat the same mistakes, and I don't know if VNC have the
> > same flaws or not.
>
> # grep CAP.*CLIPBOARD.*, /usr/include/spice-1/spice/vd_agent.h
>     VD_AGENT_CAP_CLIPBOARD,
>     VD_AGENT_CAP_CLIPBOARD_BY_DEMAND,
>     VD_AGENT_CAP_CLIPBOARD_SELECTION,
>     VD_AGENT_CAP_MAX_CLIPBOARD,
>     VD_AGENT_CAP_CLIPBOARD_NO_RELEASE_ON_REGRAB,
>     VD_AGENT_CAP_CLIPBOARD_GRAB_SERIAL,
>
> Oh yea, I see.  Is that (and the rest of the protocol) documented
> somewhere, other than vd_agent.h?

Yes, the git log/blame is probably the best reference: I don't know of
any up to date documentation of the protocol, only some ancient
documents.


-- 
Marc-André Lureau

Re: VNC clipboard support

[Philipp Hahn]

Hello,

Am 29.04.23 um 17:29 schrieb octaveflon@outlook.fr:
> I'm trying to use the copy/paste with VNC.
> 
> I'm launching qemu with:
> 
> $ qemu-system-x86_64 -hda debiandisk.img vnc :1
> 
> I'm using tightvncviewer which has support for copy/paste.
> 
> I try to copy text between guest and host.
> 
> It doesn't work. Neither from host to guest or guest to host.
> 
> As far as I know, there is clipboard support in VNC (ui/vnc-clipboard.c 
> and so on).
> With wireshark, I can see that tightvncviewer send clipboard requests.
> 
> Am I missing some configuration?

Clipboard via VNC to QEMU does not work: Basically the VNC protocol was 
originally used for remote-access to X11 application. There you work on 
a high-level were XKeySyms are used and where clipboard data can be 
exchanged.

In contrast with QEMU you work on a much lower level as you do hardware 
emulation: The operating system inside your QEMU process expects 
low-level hardware events like a USB or PS2 keyboard key Up / Down events.
As the VNC protocol only exchanges "XKeySyms" QEMU has to translate them 
back to those low-level USB/PS2 events expected by the OS.
This back-translation is not unique: For example most full-sized 
keyboards have the number block, so there are 2 keys to enter a digit.

Therefore QEMU added the "extended key event"-extension, which adds the 
low-level "KeyCode" in addition to the "XKeySym" to the VNC protocol: 
When you press a key in your VNC browser the X-Server luckily also gets 
the low-level "KeyCode", but the original VNC protocol did not include 
that information on the protocol level. By including that information 
QEMU no longer has to "fake" it and simply pass that information to the 
inside OS.

Even when your clipboards only contains text (it could also have a 
bitmap picture or some other rich data), you are basically back in the 
situation where you have to translate each character to one (or 
multiple) key-press/release events, again with the same issue mention 
above: You do not have the KeyCode and someone has to fake them.

Therefore it will not work: Albeit some VNC browsers or libraries 
implement that approach, but their result is unreliable, especially if 
the keyboard layout on your client computer does not match the keyboard 
layout configured in QEMU for the backward translation does not match 
the keyboard layout of your operating system running inside of QEMU.

If you want to know more, read this still excellent blog-post from 
Daniel Berrangé:
- 
https://www.berrange.com/posts/2010/07/04/more-than-you-or-i-ever-wanted-to-know-about-virtual-keyboard-handling/
- 
https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#qemu-extended-key-event-message

Philipp