Re: [PATCH] tty canonical mode: nicer erase behaviour

Re: [PATCH] tty canonical mode: nicer erase behaviour

[Pete Zaitcev]

> Theoretically every program should be able to determine which erase
> character to accept by looking at terminfo,

Rubbish. Programs get their erase characters from termios(3).

> One of the programs that exhibits the ^H/^? problem is the tty
> line discipline in the Linux kernel.

Read man pages for stty(1) and tset(1).

-- Pete

Re: [PATCH] tty canonical mode: nicer erase behaviour

[zefram]

Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
>Your xterm is not following Linux policy - this is a solved problem in
>Linuxspace. Debian bit the bullet a few years ago and did the neccessary
>deed to make all their terminal emulators and console match.

So Linux policy is to support only terminals that generate ^? for
backspace?  I thought Unix-like systems were supposed to be usable from a
great variety of text terminals.  Debian's policy works when your universe
consists solely of Linux machines configured to Debian specifications,
where your terminals are just the Linux console, xterm and screen,
but it fails to handle glass ttys, or xterms running on other systems.

I recall the reaction when someone proposed changing zsh's line editor
(ZLE) to treat backspace and delete characters differently.  Just among
the developers we have users of a wide variety of terminals, and the
proposed change would have broken the line editor for many of us.
As it is, ZLE just works, with no extra effort, on any text terminal,
and we get no complaints.

>                                                   Original telnet has
>IAC sequences to send a "delete" regardless of keymapping policies that
>are not known at end points.

Yes, it was a nice model, a pity it fell out of use.  Telnet defined a
Network Virtual Terminal, that was intended to be a common terminal type
to be emulated on all Telnet sessions, which would have eliminated the
need for the server to know what terminal type was being used on the
client end.  Of course, the NVT was a product of its time: no cursor
addressing, no display attributes, and on the input side only a single
erase key.  The closest equivalent to this model that is currently in
use is the de facto almost-universal capability to emulate a VT100.

These days telnet connections (and rsh and ssh) are used more as dumb
pipes to talk to the user's actual terminal.  We can't change that,
and the result is that Unix systems have to handle whatever terminal
type the user is using.

-zefram

Re: [PATCH] tty canonical mode: nicer erase behaviour

[Alan Cox]

> proposed change would have broken the line editor for many of us.
> As it is, ZLE just works, with no extra effort, on any text terminal,
> and we get no complaints.

And bash treats the two delete keys differently - something a similarly
religious group believe to be the right thing to do

> pipes to talk to the user's actual terminal.  We can't change that,
> and the result is that Unix systems have to handle whatever terminal
> type the user is using.

Which is why they have termcap, terminfo, and x keyboard maps. The kernel
is not there to cover up for usermode programmers inability to get
things right. It has enough to do covering up for the hardware folk

Alan

Re: [PATCH] tty canonical mode: nicer erase behaviour

[zefram]

Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
>Which is why they have termcap, terminfo, and x keyboard maps. The kernel
>is not there to cover up for usermode programmers inability to get
>things right.

Aha, I believe I now see a substantive difference of opinion.  You argue
that the tty line discipline depends on being configured for the terminal
by usermode code, and that it should remain so, whereas I am proposing
that it adopt a strategy with some additional logic so that it doesn't
require usermode configuration.  Please correct me if this is not a
fair representation of your position.  I'm now going to switch from
correcting inaccurate statements to an attempt to persuade you on this
underlying disagreement.

Most of the tty's control characters are perfectly simple user choices.
The user can sensibly choose to use ^R or ^L to invoke the REPRINT
function, or indeed not to have any such control character, and such
choices can be executed by a very simple invocation of stty.  This kind
of choice is not affected by what terminal the user is using, because the
^R or whatever keystroke is just the same on any terminal.  This kind
of choice extends to the ERASE character, provided that the user is
picking something like the traditional #.  But uniquely among the tty
control characters, the most commonly desired setting for ERASE is not
a particular character, but rather "whatever my backspace key sends".

Theoretically, a user could set up the ERASE key to whatever their
backspace key sends by doing "stty erase $(echotc kd)".  In practice this
doesn't work nearly as well as it should -- it relies on having echotc,
and on the terminfo database being sufficiently complete and accurate
for the current terminal, which is made particularly difficult by things
like the xterm variations previously mentioned.  It is true that this
is principally a usermode problem, and indeed I would like to see the
usermode improvements that would make this a viable solution.

However, this usermode approach, of using terminfo, seems out of
character with the tty canonical mode code.  In every other aspect, it is
terminal-independent.  For output, the only control characters it uses are
^G, ^M, ^J and ^H, which behave the same way on pretty much any terminal.
(Actually, because of this policy of using only terminal-independent
control characters, backspacing past a line wrap doesn't display
correctly on most terminals.)  For input, with the default control
character settings, the only special keys used are Return and Backspace.

When handling the Return key, under default settings, the tty line
discipline accepts both ^M and ^J as indicating the end of the line.
(Actually, internally it translates ^M to ^J.)  This allows for some
variation in terminal behaviour and in translations that might occur
before the input gets to the tty.  The internal translation can be turned
off or modified, but the particular characters that can be accepted are
hardcoded, and the default behaviour is to accept both.

What I propose is that this manner of handling the Return key -- accepting
both possible characters that can result from the key, without requiring
usermode configuration to tell it which one to use -- be extended to the
Backspace key when used for its normal function.  (Obviously the mechanism
is a little different, since the ERASE character is more configurable than
the end-of-line character.)  By doing this, the tty line discipline code
would become fully terminal-independent under its default settings, which
seems a worthwhile goal for a program that is so nearly there already.

I accept that we have here an issue about which reasonable people can
disagree, and so if the disagreement now persists then I don't intend
to pursue it much further in this forum.

-zefram

Re: [PATCH] tty canonical mode: nicer erase behaviour

[Linus Torvalds]

On Sun, 23 Sep 2001 zefram@fysh.org wrote:

> Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
> >Your xterm is not following Linux policy - this is a solved problem in
> >Linuxspace. Debian bit the bullet a few years ago and did the neccessary
> >deed to make all their terminal emulators and console match.
>
> So Linux policy is to support only terminals that generate ^? for
> backspace?

No.

Linux supports everything. Try doing a "man stty".

But the default behaviour is ^?, which makes emacs happy, and also happens
to be the default mode for most real vt100 terminals out there.

If you have a terminal that really really wants ^H, just do

	stty erase ^H

and Linux will happily believe you.

		Linus

Re: [PATCH] tty canonical mode: nicer erase behaviour

[zefram]

Pete Zaitcev <zaitcev@redhat.com> wrote:
>Rubbish. Programs get their erase characters from termios(3).

termios(3) gets its erase character from user setting via stty(1) or
tset(1), neither of which know what character will be generated by the
terminal's main backspace key.  The current situation is that the user
has to manually arrange for the correct setting.  My personal portable
shell setup, for example, contains a script that examines $TERM and sets
the tty erase character accordingly; I've never seen this functionality
available as a standard utility anywhere.

Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
>> One of the long-standing problems preventing Unix from being a
>> user-friendly desktop OS is its handling of erase keys.  There are
>
>Not a kernel space issue

Yes, it's a wider issue, which I wanted to present as background for
the specific change I was proposing.

>                                                      Fix problem apps. 

The Linux tty canonical mode is a problem app.  Let's fix it.

Programs that talk to the tty directly, through the tty raw mode or by
any other means, do need to be fixed separately, if they exhibit the
same problem.  The patch I proposed doesn't affect them.

>They do different things, they are different keys.

On some keyboards they are different keys, and in some programs they do
different things.  Both of these are far from universal.

Programs that want to treat backspace and delete keys differently tend
to get it right already.  Under X, for example, the two keys are both
available, and are distinctly and consistently reported to the program;
many X programs implement text editing with both forward and backward
delete.  There is no problem here.

I am concerned, for the moment, with programs that only need one kind of
erase, and that talk to a physical terminal (or a simulation thereof)
via a character-stream interface.  These programs want to invoke their
backspacing erase function whenever the user presses the backspace key,
and they don't care about any other erase-related key.  Linux's tty line
discipline, in canonical mode, is such a program.

Unfortunately, the programs of which I speak can't see what key the
user is pressing, all they can see is the character that the terminal
decides to generate from the keypress.  And terminals are inconsistent
about which character they generate for the backspace key.  Even before
we look at glass ttys, we have the Linux console (with standard setup)
generating ^? for backspace, and xterm generating ^H for backspace.
Shouldn't the tty line editor work by default on these at least?  Among
glass ttys that I've personally used, the ADM3e and KDS7362 generate ^H,
and VT220 and ND120 generate ^?.

So, *for programs that only care about backspace*, by far the best
solution is to treat both ^H and ^? as backspace.  And as I said before,
the Linux tty line discipline is such a program.  Of course, we then have
the issue that the tty is already somewhat configurable in this regard,
though unfortunately not sufficiently to implement this solution.

>Erase character policy is precisely defined by posix.

It is when the IEXTEN bit is not set.  When it is set, the tty is
permitted to accept other non-standard control characters.  For example,
Linux has a WERASE (word erase) character, which is enabled only in
IEXTEN mode.

>Debian set a policy on this a long time back and have done wonders since

This is interesting.  I wasn't aware of their policy before, but I've
examined it now, after Nadav Har'El provided a URL.

Their policy appears to cover more than one related area, in an incomplete
manner.  For programs that need both forward and backward erase, it says,
basically, "programs shall make it work".  With respect to X programs,
the policy makes perfect sense and is just a statement of common sense.
For programs that interact with a tty through a character stream and
don't want to use terminfo, it insists that terminals generate ^? for
backspace, but ignores the cases where this isn't possible.

The Debian policy is a fine example of the original problem -- it shows
the hoops that people have to jump through (non-standard configuration
of xterm) to get a partial solution.

-zefram

Re: [PATCH] tty canonical mode: nicer erase behaviour

[Alan Cox]

> termios(3) gets its erase character from user setting via stty(1) or
> tset(1), neither of which know what character will be generated by the
> terminal's main backspace key.  The current situation is that the user

Which is correct, because mindreading equipment is currently classified
CIA property at best (hello echelon!)

> The Linux tty canonical mode is a problem app.  Let's fix it.

The Debian people have. There is an official Debian policy which other
vendors adopted on the matter

> we look at glass ttys, we have the Linux console (with standard setup)
> generating ^? for backspace, and xterm generating ^H for backspace.

Your xterm is not following Linux policy - this is a solved problem in
Linuxspace. Debian bit the bullet a few years ago and did the neccessary
deed to make all their terminal emulators and console match.

Alan

[PATCH] tty canonical mode: nicer erase behaviour

[zefram]

problem rationale
-----------------

One of the long-standing problems preventing Unix from being a
user-friendly desktop OS is its handling of erase keys.  There are
often two such keys on a keyboard (Backspace and Delete), and which one
works depends very much on context -- many text editing programs will
only accept one of the erase-related characters (^H and ^?), and the
mapping from keys to characters is terminal-dependent.  Unfortunately,
any solution to this problem has to be implemented in each program that
faces this problem.

Theoretically every program should be able to determine which erase
character to accept by looking at terminfo, but that's very cumbersome,
particularly in programs that might not otherwise need to use terminfo.
It also utterly fails in programs that don't know what kind of terminal
they are interacting with.  The more practical solution, therefore, is
that the program should accept *both* ^H and ^? as erase characters.
Look at bash's and zsh's command line editors for examples of this
approach.

One of the programs that exhibits the ^H/^? problem is the tty
line discipline in the Linux kernel.  It falls into the category of
programs that don't know (and don't care) what kind of terminal they
are interacting with.  This patch implements the accept-both-characters
solution.

patch rationale
---------------

The exact semantics changed are: in canonical mode, if IEXTEN is on
and the ERASE character is set to either ^H or ^?, then both ^H and
^? are treated as ERASE characters.  The standard single-erase-character
behaviour is followed if IEXTEN is off (which prohibits non-POSIX extended
behaviour) or if the ERASE character is neither ^H or ^? (indicating
that the user actually wants erase behaviour other than the usual use
of the erase keys).

At first glance, a more obvious way to implement handling of both erase
characters would be to have an ERASE2 character setting in the tty state,
much as there is EOL and EOL2.  However, this solution would suffer
some annoying problems.  It would rely on the user setting the two erase
characters properly in order to take advantage of it.  Even if they're
set to ^H and ^? by default, user code that tries to set ERASE based on
the terminal type (and doesn't know about ERASE2) might end up with ERASE
and ERASE2 set to the same character, breaking the solution.  Also, code
(and users) that doesn't know about ERASE2 would get surprising results
if they try to set ERASE to something other than ^H/^?.  All of this
points to it being preferable not to change the tty settings interface,
but only change the behaviour to this preferable form.

the patch
---------

This patch is based on kernel version 2.4.8.  It will also apply cleanly
up to at least kernel version 2.4.10-pre14.

--- drivers/char/n_tty.c.orig	Sat Sep 22 22:14:19 2001
+++ drivers/char/n_tty.c	Sun Sep 23 02:59:30 2001
@@ -329,9 +329,11 @@
 	}
 }
 
-static void eraser(unsigned char c, struct tty_struct *tty)
+enum kill_type { ERASE, WERASE, KILL };
+
+static void eraser(int kill_type, unsigned char cc, struct tty_struct *tty)
 {
-	enum { ERASE, WERASE, KILL } kill_type;
+	unsigned char c;
 	int head, seen_alnums;
 	unsigned long flags;
 
@@ -339,11 +341,7 @@
 		/* opost('\a', tty); */		/* what do you think? */
 		return;
 	}
-	if (c == ERASE_CHAR(tty))
-		kill_type = ERASE;
-	else if (c == WERASE_CHAR(tty))
-		kill_type = WERASE;
-	else {
+	if (kill_type == KILL) {
 		if (!L_ECHO(tty)) {
 			spin_lock_irqsave(&tty->read_lock, flags);
 			tty->read_cnt -= ((tty->read_head - tty->canon_head) &
@@ -359,13 +357,12 @@
 			tty->read_head = tty->canon_head;
 			spin_unlock_irqrestore(&tty->read_lock, flags);
 			finish_erasing(tty);
-			echo_char(KILL_CHAR(tty), tty);
+			echo_char(cc, tty);
 			/* Add a newline if ECHOK is on and ECHOKE is off. */
 			if (L_ECHOK(tty))
 				opost('\n', tty);
 			return;
 		}
-		kill_type = KILL;
 	}
 
 	seen_alnums = 0;
@@ -392,7 +389,7 @@
 				}
 				echo_char(c, tty);
 			} else if (kill_type == ERASE && !L_ECHOE(tty)) {
-				echo_char(ERASE_CHAR(tty), tty);
+				echo_char(cc, tty);
 			} else if (c == '\t') {
 				unsigned int col = tty->canon_column;
 				unsigned long tail = tty->canon_head;
@@ -590,9 +587,19 @@
 		}
 	}
 	if (tty->icanon) {
-		if (c == ERASE_CHAR(tty) || c == KILL_CHAR(tty) ||
-		    (c == WERASE_CHAR(tty) && L_IEXTEN(tty))) {
-			eraser(c, tty);
+		if (c == ERASE_CHAR(tty) ||
+		    (L_IEXTEN(tty) &&
+		     (ERASE_CHAR(tty) == 8 || ERASE_CHAR(tty) == 127) &&
+		     (c == 8 || c == 127))) {
+			eraser(ERASE, c, tty);
+			return;
+		}
+		if (c == KILL_CHAR(tty)) {
+			eraser(KILL, c, tty);
+			return;
+		}
+		if (c == WERASE_CHAR(tty) && L_IEXTEN(tty)) {
+			eraser(WERASE, c, tty);
 			return;
 		}
 		if (c == LNEXT_CHAR(tty) && L_IEXTEN(tty)) {
@@ -822,6 +829,11 @@
 			set_bit('\n', &tty->process_char_map);
 			set_bit(EOL_CHAR(tty), &tty->process_char_map);
 			if (L_IEXTEN(tty)) {
+				if (ERASE_CHAR(tty) == 8 ||
+				    ERASE_CHAR(tty) == 127) {
+					set_bit(8, &tty->process_char_map);
+					set_bit(127, &tty->process_char_map);
+				}
 				set_bit(WERASE_CHAR(tty),
 					&tty->process_char_map);
 				set_bit(LNEXT_CHAR(tty),

-zefram

Re: [PATCH] tty canonical mode: nicer erase behaviour

[Alan Cox]

> One of the long-standing problems preventing Unix from being a
> user-friendly desktop OS is its handling of erase keys.  There are

Not a kernel space issue

> often two such keys on a keyboard (Backspace and Delete), and which one
> works depends very much on context -- many text editing programs will
> only accept one of the erase-related characters (^H and ^?), and the

They do different things, they are different keys.

Erase character policy is precisely defined by posix. Fix problem apps. 
Debian set a policy on this a long time back and have done wonders since

Re: [PATCH] tty canonical mode: nicer erase behaviour

[Nadav Har'El]

On Sun, Sep 23, 2001, Alan Cox wrote about "Re: [PATCH] tty canonical mode: nicer erase behaviour":
> Erase character policy is precisely defined by posix. Fix problem apps. 
> Debian set a policy on this a long time back and have done wonders since

Just too bad Debian's policy is to make ^? the erase character - pretty
much the opposite of what most Unix users used before that. Pretending
ASCII BS (^H) doesn't exist any more is an interesting exercise, but
it isn't easy to change habits and standards that existed for a couple of
decades... The same problem exists for the ASCII DEL (^?) which was also used
in many Unix systems, but usually as a intr key, not a "delete-forward" type
of thing...
[see http://www.debian.org/doc/debian-policy/ch-opersys.html#s10.8
for the mentioned Debian policy]

Debian's solution isn't a silver bullet, in my opinion... It just means the
^H/^? confusion stops being a problem in a stand-alone system (if all your
applications and configuration files come as defaults from Debian, they are
consistent) but it just increased the mess when you log in from one system
type to another (one of them being none-Linux Unix)...

P.S.
The only relation any of this has to the kernel is the behavior of the
"cooked" line discipline, as Zefram already said. I think I like his
erase2 idea better than his forget-the-difference-between-^H-and-^? idea.
However, this former idea will work well only if applications like ssh,
for example, will know how to propegate erase2, and they currently don't.
But this is again not really a kernel issue...

-- 
Nadav Har'El                        |       Sunday, Sep 23 2001, 7 Tishri 5762
nyh@math.technion.ac.il             |-----------------------------------------
Phone: +972-53-245868, ICQ 13349191 |Experience is what causes a person to
http://nadav.harel.org.il           |make new mistakes instead of old ones.

Re: [PATCH] tty canonical mode: nicer erase behaviour

[Alan Cox]

> Debian's solution isn't a silver bullet, in my opinion... It just means the
> ^H/^? confusion stops being a problem in a stand-alone system (if all your
> applications and configuration files come as defaults from Debian, they are
> consistent) but it just increased the mess when you log in from one system
> type to another (one of them being none-Linux Unix)...

Thats in many ways a design flaw in the protocols. Original telnet has
IAC sequences to send a "delete" regardless of keymapping policies that
are not known at end points. X also does it right with the keysyms.
Ssh seems to lack this

Re: [PATCH] tty canonical mode: nicer erase behaviour

[Nadav Har'El]

On Sun, Sep 23, 2001, Alan Cox wrote about "Re: [PATCH] tty canonical mode: nicer erase behaviour":
> Thats in many ways a design flaw in the protocols. Original telnet has
> IAC sequences to send a "delete" regardless of keymapping policies that
> are not known at end points. X also does it right with the keysyms.
> Ssh seems to lack this

Consider the letter "a". There is no such problem of "keymapping policies"
or how to send it on telnet or ssh, because ASCII, in 1965, (or even before
that), standardized it. The last two characters in that acronym stand for
"Information Interchange", of course.
Anyway, ASCII's character 010 (^H) is called "BS", i.e., backspace. What's
so wrong with using that as the standard way to send a backspace? Is it
because someone decided that it would be cool to have help be summoned in
Emacs with a ^H? Next thing we know there'll be someone annoyed by the fact
that the Escape key sends ^[ and ruins his ability to map Control-[ to
something in Emacs (running on a tty), and we'll need to change the way Escape
is coded too?

Of course there are opposite arguments, with people calling their backspace
key a "rubout" and then claiming it is justified to use the ASCII rubout
character (0177, DEL) for it. Which makes this whole situation even uglier :(


-- 
Nadav Har'El                        |       Monday, Sep 24 2001, 7 Tishri 5762
nyh@math.technion.ac.il             |-----------------------------------------
Phone: +972-53-245868, ICQ 13349191 |Hardware, n.: The parts of a computer
http://nadav.harel.org.il           |system that can be kicked.

Re: [PATCH] tty canonical mode: nicer erase behaviour

[Matthias Andree]

On Sun, 23 Sep 2001, Nadav Har'El wrote:

> Just too bad Debian's policy is to make ^? the erase character - pretty
> much the opposite of what most Unix users used before that. Pretending
> ASCII BS (^H) doesn't exist any more is an interesting exercise, but
> it isn't easy to change habits and standards that existed for a couple of
> decades... The same problem exists for the ASCII DEL (^?) which was also used
> in many Unix systems, but usually as a intr key, not a "delete-forward" type
> of thing...

[OT RANT]

Well, it's also fun to log in to a Solaris 2.5.1 ksh (which doesn't talk
bind, but just alias and soft-keys) from a Linux xterm which sends funny
Esc [3~ sequences. You can "fix" that with "Del sends Del", but you then
end up having either Del and Backspace both do Backspace or have their
meanings reversed from what the "left array" on the Backspace key
suggests. The Linux TTYs behave the same way (sending Esc [3~), and you
don't easily fix that without confusing EOF detection or things.

As appreciable a solution to consistently make "Delete" do
"delete-forward" and "Backspace" do "delete-backward" would be, as low
are the chances to get this done before the year 2039.

Re: [PATCH] tty canonical mode: nicer erase behaviour

[Kai Henningsen]

nyh@math.technion.ac.il (Nadav Har'El)  wrote on 23.09.01 in <20010923234111.A16873@leeor.math.technion.ac.il>:

> Just too bad Debian's policy is to make ^? the erase character - pretty
> much the opposite of what most Unix users used before that. Pretending

Actually, the main argument for chosing this was that this *was* the  
traditional behaviour - as seen by vtxxx terminals and traditional Unix  
applications like emacs. (If ^H was the traditional erase, how come emacs  
wants ^H for help? Doesn't add up.)

MfG Kai