[PATCH 1/2] rpmatch.3: remove first-character-only FUD

[PATCH 1/2] rpmatch.3: remove first-character-only FUD

[наб]

[-- Attachment #1: Type: text/plain, Size: 1101 bytes --]

It's plain not true; locales can and do provide longer matches
(Aramaic has a "አዎን" alternative, for example)

Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
---
 man3/rpmatch.3 | 15 ---------------
 1 file changed, 15 deletions(-)

diff --git a/man3/rpmatch.3 b/man3/rpmatch.3
index 846c492b7..e0f92a20d 100644
--- a/man3/rpmatch.3
+++ b/man3/rpmatch.3
@@ -123,21 +123,6 @@ T}	Thread safety	MT-Safe locale
 is not required by any standard, but
 is available on a few other systems.
 .\" It is available on at least AIX 5.1 and FreeBSD 6.0.
-.SH BUGS
-The
-.BR rpmatch ()
-implementation looks at only the first character
-of
-.IR response .
-As a consequence, "nyes" returns 0, and
-"ynever; not in a million years" returns 1.
-It would be preferable to accept input strings much more
-strictly, for example (using the extended regular
-expression notation described in
-.BR regex (7)):
-.B \(ha([yY]|yes|YES)$
-and
-.BR \(ha([nN]|no|NO)$ .
 .SH EXAMPLES
 The following program displays the results when
 .BR rpmatch ()
-- 
2.20.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH 2/2] rpmatch.3: clarify availability to glibc, Fx, and AIX

[наб]

[-- Attachment #1: Type: text/plain, Size: 855 bytes --]

"On a few other systems" is misleading, implying that it's available on
/this/ system; this is not strictly true, seeing as musl doesn't have it

Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
---
 man3/rpmatch.3 | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/man3/rpmatch.3 b/man3/rpmatch.3
index e0f92a20d..8719d4871 100644
--- a/man3/rpmatch.3
+++ b/man3/rpmatch.3
@@ -120,9 +120,8 @@ T}	Thread safety	MT-Safe locale
 .sp 1
 .SH CONFORMING TO
 .BR rpmatch ()
-is not required by any standard, but
-is available on a few other systems.
-.\" It is available on at least AIX 5.1 and FreeBSD 6.0.
+is not required by any standard,
+but available under the GNU C library, FreeBSD, and AIX.
 .SH EXAMPLES
 The following program displays the results when
 .BR rpmatch ()
-- 
2.20.1

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 1/2] rpmatch.3: remove first-character-only FUD

[Alejandro Colomar (man-pages)]

Hi, наб!

On 9/21/21 3:46 PM, наб wrote:
> It's plain not true; locales can and do provide longer matches
> (Aramaic has a "አዎን" alternative, for example)
> 
> Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>

Are you sure?

I just checked the glibc implementation, and it uses YESEXPR and NOEXPR 
(instead of YESSTR and NOSTR, which would match complete strings).


// stdlib/rpmatch.c
int
rpmatch (const char *response)
{
   /* We cache the response patterns and compiled regexps here.  */
   static const char *yesexpr, *noexpr;
   static regex_t yesre, nore;

   return (try (response, YESEXPR, 1, 0, &yesexpr, &yesre) ?:
	  try (response, NOEXPR, 0, -1, &noexpr, &nore));
}


$ grep -rn define.*YESEXPR;
locale/langinfo.h:570:#define YESEXPR			__YESEXPR


$ grep -rn __YESEXPR;
locale/langinfo.h:569:  __YESEXPR = _NL_ITEM (__LC_MESSAGES, 0), /* 
Regex matching ``yes'' input.  */
locale/langinfo.h:570:#define YESEXPR			__YESEXPR


$ grep -rn 'define _NL_ITEM(';
locale/langinfo.h:34:#define _NL_ITEM(category, index)	(((category) << 
16) | (index))


$ grep -rn LC_MESSAGES localedata/ | grep es_ES
localedata/locales/es_ES@euro:37:category "i18n:2012";LC_MESSAGES
localedata/locales/es_ES@euro:53:LC_MESSAGES
localedata/locales/es_ES@euro:55:END LC_MESSAGES
localedata/locales/es_ES:45:category "i18n:2012";LC_MESSAGES
localedata/locales/es_ES:113:LC_MESSAGES
localedata/locales/es_ES:118:END LC_MESSAGES


$ sed -n '/^LC_MESSAGES/,/END LC_MESSAGES/p' localedata/locales/es_ES;
LC_MESSAGES
yesexpr "^[+1sSyY]"
noexpr  "^[-0nN]"
yesstr  "sí"
nostr   "no"
END LC_MESSAGES



So, it seems to me that by using {yes,no}expr and not {yes,no}str, it is 
limiting itself to the first letter, as the current BUGS section 
specifies.  Right?

Thanks,

Alex


> ---
>   man3/rpmatch.3 | 15 ---------------
>   1 file changed, 15 deletions(-)
> 
> diff --git a/man3/rpmatch.3 b/man3/rpmatch.3
> index 846c492b7..e0f92a20d 100644
> --- a/man3/rpmatch.3
> +++ b/man3/rpmatch.3
> @@ -123,21 +123,6 @@ T}	Thread safety	MT-Safe locale
>   is not required by any standard, but
>   is available on a few other systems.
>   .\" It is available on at least AIX 5.1 and FreeBSD 6.0.
> -.SH BUGS
> -The
> -.BR rpmatch ()
> -implementation looks at only the first character
> -of
> -.IR response .
> -As a consequence, "nyes" returns 0, and
> -"ynever; not in a million years" returns 1.
> -It would be preferable to accept input strings much more
> -strictly, for example (using the extended regular
> -expression notation described in
> -.BR regex (7)):
> -.B \(ha([yY]|yes|YES)$
> -and
> -.BR \(ha([nN]|no|NO)$ .
>   .SH EXAMPLES
>   The following program displays the results when
>   .BR rpmatch ()
> 


-- 
Alejandro Colomar
Linux man-pages comaintainer; https://www.kernel.org/doc/man-pages/
http://www.alejandro-colomar.es/

Re: [PATCH 1/2] rpmatch.3: remove first-character-only FUD

[наб]

[-- Attachment #1: Type: text/plain, Size: 2365 bytes --]

Hi!

On Tue, Sep 21, 2021 at 05:20:32PM +0200, Alejandro Colomar (man-pages) wrote:
> Are you sure?
> 
> So, it seems to me that by using {yes,no}expr and not {yes,no}str, it is
> limiting itself to the first letter, as the current BUGS section specifies.
> Right?
Quite sure:
	localedata/locales/am_ET:yesexpr "^([+1yY<U12CE>]|<U12A0><U12CE><U1295>)"
Granted, I, unfortunately, don't strictly read Aramaic
(but a cursory glance at a dictionary shows "አዎን" means yes),
but I do Ukrainian:
	localedata/locales/uk_UA:yesexpr "^([+1Yy]|[<U0422><U0442>][<U0410><U0430>][<U041A><U043A>]?)$"
which works out to
	"^([+1Yy]|[Тт][Аа][Кк]?)$"

This is odd, data-wise, but it's decidedly not just the first letter
(but it does match, what, "^y$", "^та$", and "^так$"? very odd!!).

On current glibc, if I was in a uk_UA locale,
"nyes" is -1, not 0 like this page would lead me to believe,
and, similarly, in an_ET, "አ" (-1) is not the same as "አዎን" (1).

FreeBSD (and, presumably, everyone else) uses CLDR data,
which provides something much more sensible:
  [1] ^(([yY]([eE][sS])?)|([yY]))
  [2] ^(([дД]([аА])?)|([дД])|([yY]([eE][sS])?)|([yY]))

This, admittedly, is not perfect, but the code that generates it [3]
explicitly handles full yesstr words because the data itself [4] is
constructed around yesstr, and yesexpr is a generated expression that
matches yesstr ‒ they're the same.

rpmatch() is a correct (well, /the/ correct) approach to handling this
(or, well, an equivalent on libcs that lack it, it's like seven lines) ‒
if a similar warning were prudent, and I very much believe it is /not/,
it'd belong in nl_langinfo() {YES,NO}EXPR or langinfo.h,
but it'd be a warning /for the end-user/, who, presumably,
knows the language they speak, not for the programmer.

наб

1. https://github.com/freebsd/freebsd-src/blob/373ffc62c158e52cde86a5b934ab4a51307f9f2e/share/msgdef/en_US.UTF-8.src
2. https://github.com/freebsd/freebsd-src/blob/373ffc62c158e52cde86a5b934ab4a51307f9f2e/share/msgdef/ru_RU.UTF-8.src
3. https://github.com/unicode-org/cldr/blob/62c90a357dc25911db60fcdf7d5a80119df27963/tools/cldr-code/src/main/java/org/unicode/cldr/posix/POSIXUtilities.java#L336
4. https://github.com/unicode-org/cldr/blob/62c90a357dc25911db60fcdf7d5a80119df27963/common/main/ru.xml#L15789

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 1/2] rpmatch.3: remove first-character-only FUD

[Alejandro Colomar (man-pages)]

Hi!

On 9/21/21 6:06 PM, наб wrote:
> Hi!
> 
> On Tue, Sep 21, 2021 at 05:20:32PM +0200, Alejandro Colomar (man-pages) wrote:
>> Are you sure?
>>
>> So, it seems to me that by using {yes,no}expr and not {yes,no}str, it is
>> limiting itself to the first letter, as the current BUGS section specifies.
>> Right?
> Quite sure:
> 	localedata/locales/am_ET:yesexpr "^([+1yY<U12CE>]|<U12A0><U12CE><U1295>)"
> Granted, I, unfortunately, don't strictly read Aramaic
> (but a cursory glance at a dictionary shows "አዎን" means yes),
> but I do Ukrainian:
> 	localedata/locales/uk_UA:yesexpr "^([+1Yy]|[<U0422><U0442>][<U0410><U0430>][<U041A><U043A>]?)$"
> which works out to
> 	"^([+1Yy]|[Тт][Аа][Кк]?)$"
> 
> This is odd, data-wise, but it's decidedly not just the first letter
> (but it does match, what, "^y$", "^та$", and "^так$"? very odd!!).
> 
> On current glibc, if I was in a uk_UA locale,
> "nyes" is -1, not 0 like this page would lead me to believe,
> and, similarly, in an_ET, "አ" (-1) is not the same as "አዎን" (1).
> 
> FreeBSD (and, presumably, everyone else) uses CLDR data,
> which provides something much more sensible:
>    [1] ^(([yY]([eE][sS])?)|([yY]))
>    [2] ^(([дД]([аА])?)|([дД])|([yY]([eE][sS])?)|([yY]))
> 
> This, admittedly, is not perfect, but the code that generates it [3]
> explicitly handles full yesstr words because the data itself [4] is
> constructed around yesstr, and yesexpr is a generated expression that
> matches yesstr ‒ they're the same.
> 
> rpmatch() is a correct (well, /the/ correct) approach to handling this
> (or, well, an equivalent on libcs that lack it, it's like seven lines) ‒
> if a similar warning were prudent, and I very much believe it is /not/,
> it'd belong in nl_langinfo() {YES,NO}EXPR or langinfo.h,
> but it'd be a warning /for the end-user/, who, presumably,
> knows the language they speak, not for the programmer.

So, it seems that some locales try to do some extra work, and Ukrainian 
seems to be doing a good job.  I had a bit of bad luck with the Spanish 
one...  However, it seems that the C locale is also unfortunate:


user@sqli:~/src/test$ cat rpmatch.c
#include <locale.h>
#include <stdio.h>
#include <stdlib.h>

int main(void)
{
	const char *str;

	str = "ynever; don't even think about it!";
	printf("%s;; %i;; %s\n", setlocale(LC_MESSAGES, NULL), rpmatch(str), str);
	return 0;
}
user@sqli:~/src/test$ cc -Wall -Wextra -Werror rpmatch.c
user@sqli:~/src/test$ ./a.out
C;; 1;; ynever; don't even think about it!


Since the C locale is the most important one, IMHO, and it is as 
problematic as the BUGS section mentions, I think we should keep the 
warning, and maybe add a mention that it depends on the locale.  What do 
you think?

Thanks,

Alex

> 
> наб
> 
> 1. https://github.com/freebsd/freebsd-src/blob/373ffc62c158e52cde86a5b934ab4a51307f9f2e/share/msgdef/en_US.UTF-8.src
> 2. https://github.com/freebsd/freebsd-src/blob/373ffc62c158e52cde86a5b934ab4a51307f9f2e/share/msgdef/ru_RU.UTF-8.src
> 3. https://github.com/unicode-org/cldr/blob/62c90a357dc25911db60fcdf7d5a80119df27963/tools/cldr-code/src/main/java/org/unicode/cldr/posix/POSIXUtilities.java#L336
> 4. https://github.com/unicode-org/cldr/blob/62c90a357dc25911db60fcdf7d5a80119df27963/common/main/ru.xml#L15789
> 


-- 
Alejandro Colomar
Linux man-pages comaintainer; https://www.kernel.org/doc/man-pages/
http://www.alejandro-colomar.es/

Re: [PATCH 1/2] rpmatch.3: remove first-character-only FUD

[Alejandro Colomar (man-pages)]

Hi наб,

On 9/22/21 1:25 AM, Alejandro Colomar (man-pages) wrote:

> Since the C locale is the most important one, IMHO, and it is as 
> problematic as the BUGS section mentions, I think we should keep the 
> warning, and maybe add a mention that it depends on the locale.  What do 
> you think?

I mean, I agree with you that rpmatch(3) is **the correct** way of 
checking for a yes/no user response, but the programmer should also be 
aware of locale's (and especially the C locale) problems, so that he can 
also do some extra work and try to fix the locale he expects to use.

However, I also see it good if we move this bug to where the problem 
really is (instead of here, which is where the problem is noticed), 
which would be one of the locale pages (nl_langinfo(3) for example, as 
you said).

Thanks,

Alex

-- 
Alejandro Colomar
Linux man-pages comaintainer; https://www.kernel.org/doc/man-pages/
http://www.alejandro-colomar.es/

[PATCH v2 1/2] rpmatch.3: clarify first-character-only FUD

[наб]

[-- Attachment #1: Type: text/plain, Size: 1880 bytes --]

It's plain not true as-written ‒ locales can and do provide longer matches
(Aramaic has a "አዎን" alternative, for example) ‒ but it's important to
note that (a) this may be an issue and (b) nonetheless this is the right
way to process this

Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
---
This should resolve both the unnecessary FUD and the doubts it would
raise, while preserving the note. I'm not sure I'd agree with C locale
being the most important one (I'd put that burden on "the current one"),
but it's mentioned here because English locales (and most other ones for
YESEXPR/NOEXPR) derive from it.

 man3/rpmatch.3 | 20 ++++++++------------
 1 file changed, 8 insertions(+), 12 deletions(-)

diff --git a/man3/rpmatch.3 b/man3/rpmatch.3
index 846c492b7..1f9732e3f 100644
--- a/man3/rpmatch.3
+++ b/man3/rpmatch.3
@@ -125,19 +125,15 @@ is available on a few other systems.
 .\" It is available on at least AIX 5.1 and FreeBSD 6.0.
 .SH BUGS
 The
-.BR rpmatch ()
-implementation looks at only the first character
-of
+.BR YESEXPR " and " NOEXPR
+of some locales (including "C") only inspect the first character of the
 .IR response .
-As a consequence, "nyes" returns 0, and
-"ynever; not in a million years" returns 1.
-It would be preferable to accept input strings much more
-strictly, for example (using the extended regular
-expression notation described in
-.BR regex (7)):
-.B \(ha([yY]|yes|YES)$
-and
-.BR \(ha([nN]|no|NO)$ .
+This can mean that "yno" et al. resolve to
+.BR 1 .
+This is an unfortunate historical side-effect which should be fixed in time
+with proper localisation, and should not deter from
+.BR rpmatch ()
+being the proper way to distinguish between binary answers.
 .SH EXAMPLES
 The following program displays the results when
 .BR rpmatch ()
-- 
2.20.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v2 2/2] rpmatch.3: clarify availability to glibc, Fx, and AIX

[наб]

[-- Attachment #1: Type: text/plain, Size: 819 bytes --]

"On a few other systems" is misleading, implying that it's available on
/this/ system; this is not strictly true, seeing as musl doesn't have it

Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
---
 man3/rpmatch.3 | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/man3/rpmatch.3 b/man3/rpmatch.3
index 1f9732e3f..39e3926d7 100644
--- a/man3/rpmatch.3
+++ b/man3/rpmatch.3
@@ -120,9 +120,8 @@ T}	Thread safety	MT-Safe locale
 .sp 1
 .SH CONFORMING TO
 .BR rpmatch ()
-is not required by any standard, but
-is available on a few other systems.
-.\" It is available on at least AIX 5.1 and FreeBSD 6.0.
+is not required by any standard,
+but available under the GNU C library, FreeBSD, and AIX.
 .SH BUGS
 The
 .BR YESEXPR " and " NOEXPR
-- 
2.20.1

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 1/2] rpmatch.3: clarify first-character-only FUD

[Alejandro Colomar (man-pages)]

Hi наб!

On 10/2/21 2:18 PM, наб wrote:
> It's plain not true as-written ‒ locales can and do provide longer matches
> (Aramaic has a "አዎን" alternative, for example) ‒ but it's important to
> note that (a) this may be an issue and (b) nonetheless this is the right
> way to process this
> 
> Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
> ---
> This should resolve both the unnecessary FUD and the doubts it would
> raise, while preserving the note. I'm not sure I'd agree with C locale
> being the most important one (I'd put that burden on "the current one"),
> but it's mentioned here because English locales (and most other ones for
> YESEXPR/NOEXPR) derive from it.

Yep!  I applied both v2 patches for rpmatch.3.

Thanks,

Alex

> 
>   man3/rpmatch.3 | 20 ++++++++------------
>   1 file changed, 8 insertions(+), 12 deletions(-)
> 
> diff --git a/man3/rpmatch.3 b/man3/rpmatch.3
> index 846c492b7..1f9732e3f 100644
> --- a/man3/rpmatch.3
> +++ b/man3/rpmatch.3
> @@ -125,19 +125,15 @@ is available on a few other systems.
>   .\" It is available on at least AIX 5.1 and FreeBSD 6.0.
>   .SH BUGS
>   The
> -.BR rpmatch ()
> -implementation looks at only the first character
> -of
> +.BR YESEXPR " and " NOEXPR
> +of some locales (including "C") only inspect the first character of the
>   .IR response .
> -As a consequence, "nyes" returns 0, and
> -"ynever; not in a million years" returns 1.
> -It would be preferable to accept input strings much more
> -strictly, for example (using the extended regular
> -expression notation described in
> -.BR regex (7)):
> -.B \(ha([yY]|yes|YES)$
> -and
> -.BR \(ha([nN]|no|NO)$ .
> +This can mean that "yno" et al. resolve to
> +.BR 1 .
> +This is an unfortunate historical side-effect which should be fixed in time
> +with proper localisation, and should not deter from
> +.BR rpmatch ()
> +being the proper way to distinguish between binary answers.
>   .SH EXAMPLES
>   The following program displays the results when
>   .BR rpmatch ()
> 


-- 
Alejandro Colomar
Linux man-pages comaintainer; https://www.kernel.org/doc/man-pages/
http://www.alejandro-colomar.es/