Re: [PATCH] Try URI quoting for embedded TAB and LF in pathnames

[Linus Torvalds <torvalds@osdl.org>]

On Tue, 11 Oct 2005, Paul Eggert wrote:
> 
> Yes, quite true.  But we don't have to come up with something that's
> perfect in all cases, just something that's good enough to handle
> cases that we expect will be common in practice, in a world where
> UTF-8 is the preferred encoding for non-ASCII characters.

The thing is, I can almost guarantee you that any quoting in the high 
characters is going to be _worse_ than no quoting at all.

Exactly because quoting as UTF-8 is the wrong thing when it isn't actually 
UTF-8, and quoting as non-UTF-8 is the wrong thing when it _is_.

Not quoting at all, on the other hand, is unambigious. If you have a 
mailer that corrupts your text stream (which-ever type it is), then it's 
clearly the mailers problem. The _mailer_ at least has a chance in hell to 
know what character set it is getting mailed as.

The other alternative is to quote _everything_ non-ASCII. That's 
definitely reliable, but it's also unquestionably ugly as hell, especially 
in the long run.

Yes, there are some complex quoting approaches you can do, which quote 
things "correctly" (ie at a byte stream level) _and_ keep it valid UTF-8 
at the same time.

For example, you can read it as a UTF-8 stream, but then quote things at a 
byte level (ie if you quote one "character", you quote _all_ bytes in that 
character). And you quote if:

 - the UTF-8 _character_ is in the 0x80-0x9f control range
 - any _raw_byte_ is in the 0x80-0x9f range (it might not be UTF-8)
 - any _raw_byte_ is 0xfe-0xff (illegal UTF-8 character)
 - misformed UTF-8 (non-shortest sequence, or just generally invalid 
   sequences with missing or wrong high bits)

but quite frankly, that's a pretty painful thing to write. The upside is 
that it's easy to decode: you can _unquote_ it just as a byte stream.

			Linus