[Qemu-devel] curses.c: "We need a terminal output" ?

[Qemu-devel] curses.c: "We need a terminal output" ?

[Joachim Durchholz]

Hi all,

what's the reasoning behind "We need a terminal output" in curses.c?

I don't really see a scenario where it's problematic if qemu emits 
escape sequences to a pipe.
I have experienced one scenario where it is problematic: It doesn't work 
properly with pexpect and expectit (a Python and a Java library that do 
expect). I suspect they fail to use a pty, but then I see no harm in not 
doing so.
I can see another scenario when this is problematic: Unit testing code 
that directly writes to the console. E.g. testing BIOS code, or testing 
operating systems when using the serial port is not an option.

Regards,
Jo

Re: [Qemu-devel] curses.c: "We need a terminal output" ?

[John Snow]

<CC'ing curses maintainer>

On 4/13/19 5:02 AM, Joachim Durchholz wrote:
> Hi all,
> 
> what's the reasoning behind "We need a terminal output" in curses.c?
> 
> I don't really see a scenario where it's problematic if qemu emits
> escape sequences to a pipe.
> I have experienced one scenario where it is problematic: It doesn't work
> properly with pexpect and expectit (a Python and a Java library that do
> expect). I suspect they fail to use a pty, but then I see no harm in not
> doing so.
> I can see another scenario when this is problematic: Unit testing code
> that directly writes to the console. E.g. testing BIOS code, or testing
> operating systems when using the serial port is not an option.
> 
> Regards,
> Jo
> 

Re: [Qemu-devel] curses.c: "We need a terminal output" ?

[Gerd Hoffmann]

On Mon, Apr 15, 2019 at 03:10:09PM -0400, John Snow wrote:
> <CC'ing curses maintainer>
> 
> On 4/13/19 5:02 AM, Joachim Durchholz wrote:
> > Hi all,
> > 
> > what's the reasoning behind "We need a terminal output" in curses.c?

curses needs a terminal.

> > I don't really see a scenario where it's problematic if qemu emits
> > escape sequences to a pipe.

It's not about sending escape sequences.  A pipe simply doesn't support
all the tty ioctls (query terminal size, turn echo on/off, set all kinds
of terminal modes, ...).

> > I can see another scenario when this is problematic: Unit testing code
> > that directly writes to the console. E.g. testing BIOS code, or testing
> > operating systems when using the serial port is not an option.

seabios can work with the serial console just fine, try "qemu -nographic"

cheers,
  Gerd

Re: [Qemu-devel] curses.c: "We need a terminal output" ?

[Laszlo Ersek]

On 04/16/19 09:24, Gerd Hoffmann wrote:
> On Mon, Apr 15, 2019 at 03:10:09PM -0400, John Snow wrote:
>> <CC'ing curses maintainer>
>>
>> On 4/13/19 5:02 AM, Joachim Durchholz wrote:
>>> Hi all,
>>>
>>> what's the reasoning behind "We need a terminal output" in curses.c?
> 
> curses needs a terminal.
> 
>>> I don't really see a scenario where it's problematic if qemu emits
>>> escape sequences to a pipe.
> 
> It's not about sending escape sequences.  A pipe simply doesn't support
> all the tty ioctls (query terminal size, turn echo on/off, set all kinds
> of terminal modes, ...).

(

Indeed, it's called the "General Terminal Interface".

http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap11.html

)

>>> I can see another scenario when this is problematic: Unit testing code
>>> that directly writes to the console. E.g. testing BIOS code, or testing
>>> operating systems when using the serial port is not an option.
> 
> seabios can work with the serial console just fine, try "qemu -nographic"
> 
> cheers,
>   Gerd
> 
> 

Re: [Qemu-devel] curses.c: "We need a terminal output" ?

[Joachim Durchholz]

Am 16.04.19 um 09:24 schrieb Gerd Hoffmann:
> On Mon, Apr 15, 2019 at 03:10:09PM -0400, John Snow wrote:
>> <CC'ing curses maintainer>
>>
>> On 4/13/19 5:02 AM, Joachim Durchholz wrote:
>>> Hi all,
>>>
>>> what's the reasoning behind "We need a terminal output" in curses.c?
> 
> curses needs a terminal.

I commented out the check, and guess what: qemu and the OS in the VM 
would run just fine.

>>> I don't really see a scenario where it's problematic if qemu emits
>>> escape sequences to a pipe.
> 
> It's not about sending escape sequences.  A pipe simply doesn't support
> all the tty ioctls (query terminal size, turn echo on/off, set all kinds
> of terminal modes, ...).

Then I don't understand why it just ran fine.
I don't expect fullscreen output to work perfectly anyway; just enough 
so I can use qemu expect-style.

Also, is qemu -curses actually using all the terminal shenanigans?
I don't think so. If it is, it's not really needed it seems, things work 
fine.

>>> I can see another scenario when this is problematic: Unit testing code
>>> that directly writes to the console. E.g. testing BIOS code, or testing
>>> operating systems when using the serial port is not an option.
> 
> seabios can work with the serial console just fine, try "qemu -nographic"

Well, can you tell me how convince any *BSD to talk to seabios intead of 
to the terminal...

Regards,
Jo

Re: [Qemu-devel] curses.c: "We need a terminal output" ?

[Gerd Hoffmann]

  Hi,

> Then I don't understand why it just ran fine.
> I don't expect fullscreen output to work perfectly anyway;

If you don't expect the output to actually work,
then fine, feel free to comment out that check.

I don't consider that being a sane approach though.

> just enough so I can use qemu expect-style.
> 
> Also, is qemu -curses actually using all the terminal shenanigans?
> I don't think so. If it is, it's not really needed it seems, things work
> fine.

The curses library depends on them, sure.  How else do you think the
curses UI manages to work correctly no matter how big your terminal
window is?

> > > > I can see another scenario when this is problematic: Unit testing code
> > > > that directly writes to the console. E.g. testing BIOS code, or testing
> > > > operating systems when using the serial port is not an option.
> > 
> > seabios can work with the serial console just fine, try "qemu -nographic"
> 
> Well, can you tell me how convince any *BSD to talk to seabios intead of to
> the terminal...

All boot loaders I've seen so far actually talk to seabios.  Which is
enough to enable the kernel's serial console support.  The general
approach is the same on any unix-ish guest operating system.  The
details may vary of course.

FreeBSD:
 (1) qemu-system-x86_64 -nographic -cdrom FreeBSD-11.0-RELEASE-amd64-disc1.iso
 (2) At the bootloader menu hit '3' (Escape to loader prompt).
 (3) set console=comconsole
 (4) boot

Figuring the procedure for other guests is left as an exercise to the
reader.

cheers,
  Gerd

Re: [Qemu-devel] curses.c: "We need a terminal output" ?

[Joachim Durchholz]

Sorry, I'm done having to argue against(!) a person who's stonewalling 
me by wilfully ignoring differences ("doesn't work perfectly" 
interpreted as "does not work at all"), discounting 
just-for-testing-purpose workarounds as if they were actual proposed 
solutions ("not a sane approach"), ignoring the use case (still 
insisting on terminal size mutability), and proposing a nonworking 
solution (-nographic does not give me any output, of course because it 
doesn't have a terminal to talk to).

Reminds me of discussions in the Maven group, there, anything that 
didn't fit their shiny predefined ideas was brushed off in a similar manner.
Choosing the same solution: Use a different workaround, chalk this up 
under the Passive-Aggressive Unhelpfulness Encountered column.

No hard feelings towards the community.

Have a nice day everyone.
Jo

P.S.: Frustrated. In two ways.
Latin "frustra" = "in vain".

P.P.S.: I will stick around a bit so I can answer to questions based on 
genuine interest, but I'll disconnect from the list after that.

P.P.P.S.: No hard feelings towards the community in general. In fact, 
thanks to those two who tried to help.

Re: [Qemu-devel] curses.c: "We need a terminal output" ?

[Laszlo Ersek]

Hi Joachim,

On 04/17/19 19:09, Joachim Durchholz wrote:
> Sorry, I'm done having to argue against(!) a person who's stonewalling
> me by wilfully ignoring differences ("doesn't work perfectly"
> interpreted as "does not work at all"), discounting
> just-for-testing-purpose workarounds as if they were actual proposed
> solutions ("not a sane approach"), ignoring the use case (still
> insisting on terminal size mutability), and proposing a nonworking
> solution (-nographic does not give me any output, of course because it
> doesn't have a terminal to talk to).

you use the word "solution" twice above, but in this thread, I haven't
seen a problem statement from you.

What is the end goal you're trying to achieve?

The source file "ui/curses.c" carries the following comment at the top:

  QEMU curses/ncurses display driver

It is a display backend that is based on Curses.

Curses -- in its originally standardized form, XCURSES -- is an
interface specification for manipulating terminals. Please see:

  https://pubs.opengroup.org/onlinepubs/7908799/xcurses/intov.html

It writes:

    [...] a comprehensive API lets the application perform terminal
    operations. The Curses run-time system receives each terminal
    request and sends appropriate commands to the terminal to achieve
    the desired effect. [...]  Applications using Curses should not also
    control the terminal using capabilities of the general terminal
    interface defined in the XBD specification, General Terminal
    Interface. [...]

(I sent the link to the General Terminal Interface earlier.)

So, let's look at your original question again (which was not a problem
statement):

> what's the reasoning behind "We need a terminal output" in curses.c?

The reasoning is that "curses.c" is a QEMU display backend written in
terms of the Curses [XCURSES] interface specification, and that
interface specification is inherently based on terminals, as defined by
the General Terminal Interface of the Single Unix Specification. (BTW
SUS and POSIX have merged into one set of specs at SUSv3.)

If you tell us what you want to achieve in the end (i.e. you state the
problem), maybe we can tell you what to use *instead of* "ui/curses.c",
as the QEMU display backend. We might also be able to suggest ways to
use "ui/curses.c", so that it give you what you need.


Let me speculate a bit. You mentioned wanting to connect "ui/curses.c"
to a pipe, possibly to transfer the output elsewhere. This is a common
use case -- it's what terminal emulators such as "xterm" do, also what
"ssh" does (when you log in interactively with it). "screen" and "tmux"
are other programs in this class.

The way they all work is, they set up a pseudo-terminal (pty). The pty
has a master end and a slave end. The slave end looks like a real
terminal and provides the General Terminal Interface -- so "ui/curses.c"
would consume that. The kernel driver for pseudo-terminals provides all
the necessary terminal features. On the master end, you get a file
descriptor to a bidirectional stream (similar to a unix domain stream
socket). The various terminal emulators (xterm, ssh, screen, tmux) read
and write the master end, and perform various stuff on the data received
(e.g. "screen" passes the data for display to the other screen process
for display, if there is such an "other" screen process; xterm draws the
characters, clears the screen, changes colors, and beeps; sshd encrypts
the data, transfers it over the network, ssh decrypts the data and
writes it to the local *slave* end; and so on).

One interface to check here is posix_openpt():

http://pubs.opengroup.org/onlinepubs/9699919799/functions/posix_openpt.html

What's important here is that the translation between the master end and
the slave end is *thick*. For example, assuming appropriate settings, if
you write the appropriate character ("INTR") to the master-end file
descriptor, the program for which the slave-end is the "controlling
terminal" -- another term to look up in POSIX -- will get a SIGINT. This
translation (i.e. the raising of SIGINT) is done by the kernel's
pseudo-terminal (pty) driver. This is what happens e.g. when you press
"Ctrl-C" in xterm.

Anyway, you mentioned "pexpect":

https://pexpect.readthedocs.io/en/stable/

"pexpect" is a python facility for automating interactions with programs
that require a terminal (such as the QEMU display backend
"ui/curses.c"). I don't know "pexpect"'s internals, but it will
*unavoidably* have to use a pseudo-terminal.

I think that, if you need "just a pipe", you can satisfy "ui/curses.c"'s
need for a terminal by giving it a slave terminal device with "pexpect".
Then you can read the master-end (pexpect should let you do that as
well) -- the master-end will indeed read similarly to pipe. Just be
aware that all the terminal output goo, such as cursor positioning,
color change sequences, beeps, screen resolution changes, etc, will show
up as binary garbage in the stream.

Thanks
Laszlo

Re: [Qemu-devel] curses.c: "We need a terminal output" ?

[Joachim Durchholz]

Am 17.04.19 um 20:27 schrieb Laszlo Ersek:
> So, let's look at your original question again (which was not a problem
> statement):

So you need an explicit problem statement to know that somebody might 
have a problem?

>> what's the reasoning behind "We need a terminal output" in curses.c?
> 
> The reasoning is that "curses.c" is a QEMU display backend written in
> terms of the Curses [XCURSES] interface specification, and that
> interface specification is inherently based on terminals, as defined by
> the General Terminal Interface of the Single Unix Specification.

Sure.

> If you tell us what you want to achieve in the end (i.e. you state the
> problem),

I described three use cases. I mentioned which of them was mine.
Problem statement: my use case doesn't work. D'oh.

> Let me speculate a bit. You mentioned wanting to connect "ui/curses.c"
> to a pipe, possibly to transfer the output elsewhere. This is a common
> use case -- it's what terminal emulators such as "xterm" do, also what
> "ssh" does (when you log in interactively with it). "screen" and "tmux"
> are other programs in this class.
> 
> The way they all work is, they set up a pseudo-terminal (pty). The pty
> has a master end and a slave end.

I know expect, and yeah it works. It's just that TCL is too old-school 
to cleanly handle all edge cases, and I don't have the time budget to 
really learn all the ins and outs of TCL.
pexpect doesn't work, because it would insist on line buffering (try 
doing a login or responding to a prompt).
expectit doesn't work, because it simply opens a pipe. It's otherwise 
the best-maintained, best-structured API. Being Java, it's a bit 
difficult to interface to C, besides opening ptys is pretty nonportable 
so it would have been a biggish project to give the maintainer a patch 
that is useful to him.

So the workaround is to use expect anyway. So happy :-(

 > (pexpect)
> Then you can read the master-end (pexpect should let you do that as
> well) -- the master-end will indeed read similarly to pipe.

Sure, it fails due to line buffering. Can't wait for the login prompt 
because it isn't sending an eoln.

 > Just be
> aware that all the terminal output goo, such as cursor positioning,
> color change sequences, beeps, screen resolution changes, etc, will show
> up as binary garbage in the stream.

It's just escape sequences, and while they're a bit of a pain, they can 
be filtered out.
If TCL were a sane language, I could do that with a bit of code, but it 
isn't, so I'm doing that mentally - it's a pain, but it works.


I'm still wondering why there's such a stiff opposition to giving that 
with qemu.
Note that I do agree that it's not what qemu should do by default, 
switching that check off should be an option.
I just never got around to making that suggestion because Gerd was so 
insistent on brushing the use cases aside.

Jo

Re: [Qemu-devel] curses.c: "We need a terminal output" ?

[Laszlo Ersek]

On 04/18/19 07:02, Joachim Durchholz wrote:
> Am 17.04.19 um 20:27 schrieb Laszlo Ersek:
>> So, let's look at your original question again (which was not a problem
>> statement):
> 
> So you need an explicit problem statement to know that somebody might
> have a problem?

You're on a technical mailing list. An explicit problem statement is
instrumental for subscribers to understand the particulars of the
problem. "Somebody might have a problem" is a yes/no question and is
basically devoid of information.

>>> what's the reasoning behind "We need a terminal output" in curses.c?
>>
>> The reasoning is that "curses.c" is a QEMU display backend written in
>> terms of the Curses [XCURSES] interface specification, and that
>> interface specification is inherently based on terminals, as defined by
>> the General Terminal Interface of the Single Unix Specification.
> 
> Sure.
> 
>> If you tell us what you want to achieve in the end (i.e. you state the
>> problem),
> 
> I described three use cases. I mentioned which of them was mine.
> Problem statement: my use case doesn't work. D'oh.

I've now re-read your original post again. I certainly wouldn't call it
"d'oh".

Protip: it's always best to insult the people that try to help you. The
more abrasive you are the more willingly they will spend their
nonexistent free time to help you. This is how a development mailing
list works. /s

>> Let me speculate a bit. You mentioned wanting to connect "ui/curses.c"
>> to a pipe, possibly to transfer the output elsewhere. This is a common
>> use case -- it's what terminal emulators such as "xterm" do, also what
>> "ssh" does (when you log in interactively with it). "screen" and "tmux"
>> are other programs in this class.
>>
>> The way they all work is, they set up a pseudo-terminal (pty). The pty
>> has a master end and a slave end.
> 
> I know expect, and yeah it works. It's just that TCL is too old-school
> to cleanly handle all edge cases, and I don't have the time budget to
> really learn all the ins and outs of TCL.
> pexpect doesn't work, because it would insist on line buffering (try
> doing a login or responding to a prompt).

https://pexpect.readthedocs.io/en/stable/FAQ.html

> Q: Can I do screen scraping with this thing?
>
> A: That depends. If your application just does line-oriented output
> then this is easy. If a program emits many terminal sequences, from
> video attributes to screen addressing, such as programs using curses,
> then it may become very difficult to ascertain what text is displayed
> on a screen.
>
> We suggest using the pyte library to screen-scrape. The module
> pexpect.ANSI released with previous versions of pexpect is now marked
> deprecated and may be removed in the future.

Have you tried pyte <https://pyte.readthedocs.io/en/latest/>?

On 04/18/19 07:02, Joachim Durchholz wrote:
> expectit doesn't work, because it simply opens a pipe. It's otherwise
> the best-maintained, best-structured API. Being Java, it's a bit
> difficult to interface to C, besides opening ptys is pretty nonportable
> so it would have been a biggish project to give the maintainer a patch
> that is useful to him.
> 
> So the workaround is to use expect anyway. So happy :-(
> 
>> (pexpect)
>> Then you can read the master-end (pexpect should let you do that as
>> well) -- the master-end will indeed read similarly to pipe.
> 
> Sure, it fails due to line buffering. Can't wait for the login prompt
> because it isn't sending an eoln.
> 
>> Just be
>> aware that all the terminal output goo, such as cursor positioning,
>> color change sequences, beeps, screen resolution changes, etc, will show
>> up as binary garbage in the stream.
> 
> It's just escape sequences, and while they're a bit of a pain, they can
> be filtered out.
> If TCL were a sane language, I could do that with a bit of code, but it
> isn't, so I'm doing that mentally - it's a pain, but it works.
> 
> 
> I'm still wondering why there's such a stiff opposition to giving that
> with qemu.

Because "ui/curses.c" is a display backend written on top of the Curses
API, and that API was *designed* (and so QEMU's display backend was
*designed*) for use on a terminal. That's why the isatty() call is
there, and has been there ever since the introduction of the curses
display backend (commit 4d3b6f6e1265, "Add an ncurses UI.", 2008-02-10).

> Note that I do agree that it's not what qemu should do by default,
> switching that check off should be an option.

Yes, it's also an "option" to ride a bycicle on a busy street at dusk,
without a visibility vest, a helmet, and bike lights. The bike will
still roll, steer and brake.

> I just never got around to making that suggestion because Gerd was so
> insistent on brushing the use cases aside.

Removing the isatty() check, conditionally or unconditionally, is a
suggestion that is counter to the design & purpose of "ui/curses.c", in
my opinion. To me it's a suggestion that is definitely inferior to using
a terminal emulator that works for your purpose.

Adding a different display backend to QEMU, such that it isn't based on
Curses but on a pipe, might make sense. I'm not sure.

Thanks
Laszlo

Re: [Qemu-devel] curses.c: "We need a terminal output" ?

[Joachim Durchholz]

Am 18.04.19 um 13:02 schrieb Laszlo Ersek:
> On 04/18/19 07:02, Joachim Durchholz wrote:
>> Am 17.04.19 um 20:27 schrieb Laszlo Ersek:
>>> So, let's look at your original question again (which was not a problem
>>> statement):
>>
>> So you need an explicit problem statement to know that somebody might
>> have a problem?
> 
> You're on a technical mailing list. An explicit problem statement is
> instrumental for subscribers to understand the particulars of the
> problem.

Yeah okay, another person who prefers to dismiss points raised on the 
basis of formalities.
The technical term for that behaviour is "passive-aggressive behaviour", 
and I have seen enough of that.

As I said, I have stayed to answer questions that show genuine interest 
- that's not the case, so I'm off.

/logoff