Braille device (chardev/baum.c) is unable to detect the TTY correctly and does not act on graphic console connect/disconnect

Braille device (chardev/baum.c) is unable to detect the TTY correctly and does not act on graphic console connect/disconnect

[Teemu Kuusisto]

Hello,

As a blind developer I would be very happy to use QEMU's baum chardev for a braille display. Unfortunately, this device fails to detect the tty in which 
the spice client is running. I would like to improve this device but I don't yet know how to achieve a better solution.

The current code calls qemu_console_get_window_id() to get the tty. This function returns zero, which causes the code to skip even the default behavior 
of brlapi's brlapi__enterTtyMode() (including checcking some env variables such as CONTROLVT). Furthermore, window id sounds like something different 
than a tty number, maybe a number of X display? The code does not currently consider the fact that the lifetime of the connected graphical consoles is 
not the same as the lifetime of the VM.

I hardcoded a tty number to the sources of QEMU version 3.0.0 to gladly verify that the device is working besides these problems.

So, I have a handful of open questions to start with: Is it possible to get callbacks for connect and disconnect of a graphical console (like spice and 
vnc)? How? Is it further possible to get any information of the connected client to determine the tty, and possibly sub-windows too (see 
brlapi__enterTtyModeWithPath), in which the client is running? Such events should lead to calls of brlapi__EnterTtyMode() and brlapi__leaveTtyMode().

To get this device working properly would be a life changing lifehack for me so I highly appreciate any help!
It would allow the use of Linux side by side with Windows and ChromiumOS, which is a big leap in this rather poorly accessible world.
If this is successful, I'm looking forward to spreading the word and helping others too!

With kind regards and informally on behalf of the Finnish braille authority,
-- Teemu Kuusisto

Re: Braille device (chardev/baum.c) is unable to detect the TTY correctly and does not act on graphic console connect/disconnect

[Samuel Thibault]

Hello,

Teemu Kuusisto, le jeu. 14 nov. 2019 14:09:15 +0200, a ecrit:
> As a blind developer I would be very happy to use QEMU's baum chardev for a braille display. Unfortunately, this device fails to detect the tty in which the spice client is running.

Ah indeed that case was never looked at.

> The current code calls qemu_console_get_window_id() to get the tty.

> The code does not currently consider the fact that the lifetime of the
> connected graphical consoles is not the same as the lifetime of the
> VM.

Indeed, with spice the situation is different.

> This function returns zero, which causes the code to skip even the
> default behavior of brlapi's brlapi__enterTtyMode() (including
> checcking some env variables such as CONTROLVT)

Mmm, indeed that should be fixed into letting brlapi try to find it, so
that CONTROLVT can work.

> window id sounds like something different than a tty number, maybe a
> number of X display?

Yes, under X you need to provide the X window id.

> Is it possible to get callbacks for connect and disconnect of a       
> graphical console (like spice and vnc)? How?
[...]
> Such events should lead to calls of brlapi__EnterTtyMode() and
> brlapi__leaveTtyMode().

That would seem to be the way to go indeed, but see below.

> Is it further possible to get any information of the connected
> client to determine the tty, and possibly sub-windows too (see
> brlapi__enterTtyModeWithPath), in which the client is running?

More precisely you would need to know the X window ID on the front-end
side. The VNC and spice protocols don't currently provide this.  Also
when the VM and the frontend are running on different machines this
would not make any sense anyway so I don't think it will get added to
spice & vnc anyway.

One ugly way to get it to work is to run the spice client and keep it
open, get its X window id through xwininfo or equivalent, and only then
start qemu with CONTROLVT set to that id. But of course you can't close
the client and reopen it later.

The way to properly fix it is to add a brlapi channel to spice: in
addition to main, display, inputs, cursor, playback, and record
channels, we would have a brlapi channel. The brlapi protocol is already
packet-based, so that would work nicely. The server in qemu would list
the brlapi channel in addition to others, and brlapi packets would flow
over. The spice client would just forward brlapi packets over without
modification, except for the enterttymode packet, in which it would just
prepend its own windowpath and window id. The forwarding implementation
could be implemented in libbrlapi so that spice clients don't have
to maintain the support. Similarly, we could modify libbrlapi to not
necessarily connect to the usual brlapi socket, but let the application
provide send/recv functions to exchange the packets.

Samuel

Re: Braille device (chardev/baum.c) is unable to detect the TTY correctly and does not act on graphic console connect/disconnect

[Samuel Thibault]

Samuel Thibault, le jeu. 14 nov. 2019 14:08:41 +0100, a ecrit:
> The way to properly fix it is to add a brlapi channel to spice:

And that would be workable through a spice agent as well, so that
braille management from orca running inside the guest could talk
directly through to brltty running on the host instead of having to go
through the emulation layers.

Samuel

Re: Braille device (chardev/baum.c) is unable to detect the TTY correctly and does not act on graphic console connect/disconnect

[Samuel Thibault]

Samuel Thibault, le jeu. 14 nov. 2019 14:27:12 +0100, a ecrit:
> Samuel Thibault, le jeu. 14 nov. 2019 14:08:41 +0100, a ecrit:
> > The way to properly fix it is to add a brlapi channel to spice:
> 
> And that would be workable through a spice agent as well, so that
> braille management from orca

Or event NVDA or Narrator, actually.

> running inside the guest could talk
> directly through to brltty running on the host instead of having to go
> through the emulation layers.
> 
> Samuel

Re: Braille device (chardev/baum.c) is unable to detect the TTY correctly and does not act on graphic console connect/disconnect

[Teemu Kuusisto]

I totally agree with everything you said! That would allow accessible use of remote virtual machines, which I initially thought of as rather unnecessary but it definitely is not. I just did not dare to even think of the new possibilities it would offer!

Brlapi already has a file descriptor. Could spice just grab that and proxy its contents as a channel? Only brlapi's open/close connection and enter/leave tty needs to be called by spice. That does not sound of too much maintenane for spice. I just took a look at the header files of brlapi and noticed that you are involved in that project too, Samuel.

-- Teemu