* Re: [bug-diffutils] bug#24116: [platform-testers] new snapshot available: diffutils-3.3.50-0353 [not found] ` <CA+8g5KGack9X8ByfoJEtbQHj-44iG6bvQ0yhguVqQ4vqZh4geA@mail.gmail.com> @ 2016-08-05 12:46 ` Dave Gordon 2016-08-05 13:13 ` Harald van Dijk 0 siblings, 1 reply; 6+ messages in thread From: Dave Gordon @ 2016-08-05 12:46 UTC (permalink / raw) To: Jim Meyering, Assaf Gordon; +Cc: 24116, dash On 01/08/16 01:36, Jim Meyering wrote: > On Sun, Jul 31, 2016 at 10:17 AM, Assaf Gordon <assafgordon@gmail.com> wrote: >> Hello Jim >> >>> On Jul 31, 2016, at 03:08, Jim Meyering <jim@meyering.net> wrote: >>> >>> diffutils snapshot: >>> http://meyering.net/diff/diffutils-3.3.50-0353.tar.xz >> >> The "colors" test seems to succeed on Fedora/CentOS/SUSE systems (of various versions), but fail on others (Ubuntu, Debian, FreeBSD, Mac OS X). >> >> Attached are logs from 3 systems. From a cursory look it seems the exact same failure, but I haven't looked deeper. >> No other test failures found, but I'll have more results later today. > > Hi Assaf, > Thank you for all the speedy testing. > I've looked into the failure on a Debian system for which /bin/sh is > dash 0.5.8-2.2. > dash's printf builtin handles \e differently -- that's easy to work > around: use \033, which *is* portable. > More surprising is that this generates no output: > > dash -c 'f() { local t=$(printf '\''\t\t'\''); printf "$t"; }; f' > > I.e., piping it into wc -c prints 0. > With bash, it prints the expected pair of TAB bytes. > I found that I could work around this nonsensical behavior by hoisting > the "tab=..." definition up/out of those two functions, or by adding > standard-says-never-necessary double quotes like this: > > dash -c 'f() { local t="$(printf '\''\t\t'\'')"; printf "$t"; }; f' > > However, I prefer not to work around it here (and in every other test > script where this comes up), and will insulate all of our test scripts > by rejecting any shell with that misbehavior, so plan to adjust > init.sh to select another shell when it finds this flaw. > > On second thought, I will make the local change now, and sleep on the > idea of making init.sh reject dash. > Done in the attached patch. No, that's definitely a dash(1) bug, and quite a serious one. Here's a variant that makes it more obvious: # Define our test string, without too much complicated quoting $ X='f() { local t=$(printf "abc"); printf "$t"; }; f' $ bash -c "$X" | hd 00000000 61 62 63 |abc| 00000003 $ dash -c "$X" | hd 00000000 61 62 63 |abc| 00000003 # As expected, we get the same result from bash(1) and dash(1). # Now try a different test string: $ X='f() { local t=$(printf "a\tc"); printf "$t"; }; f' $ bash -c "$X" | hd 00000000 61 09 63 |a.c| 00000003 $ dash -c "$X" | hd 00000000 61 |a| 00000001 # Wibble! dash(1) has truncated the string at the TAB :( # In fact it's worse that that $ X='f() { local t=$(printf "a\tc=d"); printf "$t+$c"; }; f' $ bash -c "$X" | hd 00000000 61 09 63 3d 64 2b |a.c=d+| 00000006 $ dash -c "$X" | hd 00000000 61 2b 64 |a+d| 00000003 What dash(1) appears to have done is silently take the TAB as the terminator of the containing double-quoted string, AND of the containing $() construct, as well as a whitespace, so that the "c=d" is taken as the next argument to the 'local' builtin. I suspect this unexpected termination of the inner quoted-string could be quite exploitable! .Dave. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [bug-diffutils] bug#24116: [platform-testers] new snapshot available: diffutils-3.3.50-0353 2016-08-05 12:46 ` [bug-diffutils] bug#24116: [platform-testers] new snapshot available: diffutils-3.3.50-0353 Dave Gordon @ 2016-08-05 13:13 ` Harald van Dijk 2016-08-05 14:09 ` Dave Gordon 2016-08-05 16:21 ` Eric Blake 0 siblings, 2 replies; 6+ messages in thread From: Harald van Dijk @ 2016-08-05 13:13 UTC (permalink / raw) To: Dave Gordon, Jim Meyering, Assaf Gordon; +Cc: 24116, dash On 5-8-2016 14:46, Dave Gordon wrote: > On 01/08/16 01:36, Jim Meyering wrote: >> On Sun, Jul 31, 2016 at 10:17 AM, Assaf Gordon <assafgordon@gmail.com> >> wrote: >>> Hello Jim >>> >>>> On Jul 31, 2016, at 03:08, Jim Meyering <jim@meyering.net> wrote: >>>> >>>> diffutils snapshot: >>>> http://meyering.net/diff/diffutils-3.3.50-0353.tar.xz >>> >>> The "colors" test seems to succeed on Fedora/CentOS/SUSE systems (of >>> various versions), but fail on others (Ubuntu, Debian, FreeBSD, Mac >>> OS X). >>> >>> Attached are logs from 3 systems. From a cursory look it seems the >>> exact same failure, but I haven't looked deeper. >>> No other test failures found, but I'll have more results later today. >> >> Hi Assaf, >> Thank you for all the speedy testing. >> I've looked into the failure on a Debian system for which /bin/sh is >> dash 0.5.8-2.2. >> dash's printf builtin handles \e differently -- that's easy to work >> around: use \033, which *is* portable. >> More surprising is that this generates no output: >> >> dash -c 'f() { local t=$(printf '\''\t\t'\''); printf "$t"; }; f' >> >> I.e., piping it into wc -c prints 0. >> With bash, it prints the expected pair of TAB bytes. >> I found that I could work around this nonsensical behavior by hoisting >> the "tab=..." definition up/out of those two functions, or by adding >> standard-says-never-necessary double quotes like this: >> >> dash -c 'f() { local t="$(printf '\''\t\t'\'')"; printf "$t"; }; f' >> >> However, I prefer not to work around it here (and in every other test >> script where this comes up), and will insulate all of our test scripts >> by rejecting any shell with that misbehavior, so plan to adjust >> init.sh to select another shell when it finds this flaw. >> >> On second thought, I will make the local change now, and sleep on the >> idea of making init.sh reject dash. >> Done in the attached patch. > > No, that's definitely a dash(1) bug, and quite a serious one. Here's a > variant that makes it more obvious: > > # Define our test string, without too much complicated quoting > $ X='f() { local t=$(printf "abc"); printf "$t"; }; f' > $ bash -c "$X" | hd > 00000000 61 62 63 |abc| > 00000003 > $ dash -c "$X" | hd > 00000000 61 62 63 |abc| > 00000003 > # As expected, we get the same result from bash(1) and dash(1). > > # Now try a different test string: > $ X='f() { local t=$(printf "a\tc"); printf "$t"; }; f' > $ bash -c "$X" | hd > 00000000 61 09 63 |a.c| > 00000003 > $ dash -c "$X" | hd > 00000000 61 |a| > 00000001 > # Wibble! dash(1) has truncated the string at the TAB :( > > # In fact it's worse that that > $ X='f() { local t=$(printf "a\tc=d"); printf "$t+$c"; }; f' > $ bash -c "$X" | hd > 00000000 61 09 63 3d 64 2b |a.c=d+| > 00000006 > $ dash -c "$X" | hd > 00000000 61 2b 64 |a+d| > 00000003 > > What dash(1) appears to have done is silently take the TAB as > the terminator of the containing double-quoted string, AND of > the containing $() construct, as well as a whitespace, so that > the "c=d" is taken as the next argument to the 'local' builtin. > > I suspect this unexpected termination of the inner quoted-string > could be quite exploitable! This gets reported relatively frequently. The local command is non-standard but a common extension in shells. In the shells that provide it, it gets treated the same, syntax-wise, as the standard export command, including in dash. Unfortunately, POSIX currently requires the export command to not have any magic quoting, and any POSIX-conforming shell will make a="b c=d" export a=$a set a to b, and c to d. Not so with bash, but that's because bash simply isn't POSIX-conforming, even if invoked as sh. POSIX will require special quoting rules for the export command in the future, similar to what bash does today. When it does, dash is likely to change to match that, and the local command will likely be changed to work the same way. Right now, though, since the special quoting behaviour is non-standard, this is a bug in the script unless the script is explicitly stated to work only with specific shells. If the script is meant to be portable, even if only across shells that provide the local command, quoting $(...) is the right thing to do. Alternatively: local a a=$(...) should work too, including in dash. Since a=$(...) is not an argument to any command here, since it's the shell syntax that says it's an assignment rather than the semantics of a particular command, field splitting won't happen here. Cheers, Harald van Dijk ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [bug-diffutils] bug#24116: [platform-testers] new snapshot available: diffutils-3.3.50-0353 2016-08-05 13:13 ` Harald van Dijk @ 2016-08-05 14:09 ` Dave Gordon 2016-08-05 16:21 ` Eric Blake 1 sibling, 0 replies; 6+ messages in thread From: Dave Gordon @ 2016-08-05 14:09 UTC (permalink / raw) To: Harald van Dijk; +Cc: 24116, dash On 05/08/16 14:13, Harald van Dijk wrote: > On 5-8-2016 14:46, Dave Gordon wrote: >> On 01/08/16 01:36, Jim Meyering wrote: >>> On Sun, Jul 31, 2016 at 10:17 AM, Assaf Gordon <assafgordon@gmail.com> >>> wrote: >>>> Hello Jim >>>> >>>>> On Jul 31, 2016, at 03:08, Jim Meyering <jim@meyering.net> wrote: >>>>> >>>>> diffutils snapshot: >>>>> http://meyering.net/diff/diffutils-3.3.50-0353.tar.xz >>>> >>>> The "colors" test seems to succeed on Fedora/CentOS/SUSE systems (of >>>> various versions), but fail on others (Ubuntu, Debian, FreeBSD, Mac >>>> OS X). >>>> >>>> Attached are logs from 3 systems. From a cursory look it seems the >>>> exact same failure, but I haven't looked deeper. >>>> No other test failures found, but I'll have more results later today. >>> >>> Hi Assaf, >>> Thank you for all the speedy testing. >>> I've looked into the failure on a Debian system for which /bin/sh is >>> dash 0.5.8-2.2. >>> dash's printf builtin handles \e differently -- that's easy to work >>> around: use \033, which *is* portable. >>> More surprising is that this generates no output: >>> >>> dash -c 'f() { local t=$(printf '\''\t\t'\''); printf "$t"; }; f' >>> >>> I.e., piping it into wc -c prints 0. >>> With bash, it prints the expected pair of TAB bytes. >>> I found that I could work around this nonsensical behavior by hoisting >>> the "tab=..." definition up/out of those two functions, or by adding >>> standard-says-never-necessary double quotes like this: >>> >>> dash -c 'f() { local t="$(printf '\''\t\t'\'')"; printf "$t"; }; f' >>> >>> However, I prefer not to work around it here (and in every other test >>> script where this comes up), and will insulate all of our test scripts >>> by rejecting any shell with that misbehavior, so plan to adjust >>> init.sh to select another shell when it finds this flaw. >>> >>> On second thought, I will make the local change now, and sleep on the >>> idea of making init.sh reject dash. >>> Done in the attached patch. >> >> No, that's definitely a dash(1) bug, and quite a serious one. Here's a >> variant that makes it more obvious: >> >> # Define our test string, without too much complicated quoting >> $ X='f() { local t=$(printf "abc"); printf "$t"; }; f' >> $ bash -c "$X" | hd >> 00000000 61 62 63 |abc| >> 00000003 >> $ dash -c "$X" | hd >> 00000000 61 62 63 |abc| >> 00000003 >> # As expected, we get the same result from bash(1) and dash(1). >> >> # Now try a different test string: >> $ X='f() { local t=$(printf "a\tc"); printf "$t"; }; f' >> $ bash -c "$X" | hd >> 00000000 61 09 63 |a.c| >> 00000003 >> $ dash -c "$X" | hd >> 00000000 61 |a| >> 00000001 >> # Wibble! dash(1) has truncated the string at the TAB :( >> >> # In fact it's worse that that >> $ X='f() { local t=$(printf "a\tc=d"); printf "$t+$c"; }; f' >> $ bash -c "$X" | hd >> 00000000 61 09 63 3d 64 2b |a.c=d+| >> 00000006 >> $ dash -c "$X" | hd >> 00000000 61 2b 64 |a+d| >> 00000003 >> >> What dash(1) appears to have done is silently take the TAB as >> the terminator of the containing double-quoted string, AND of >> the containing $() construct, as well as a whitespace, so that >> the "c=d" is taken as the next argument to the 'local' builtin. >> >> I suspect this unexpected termination of the inner quoted-string >> could be quite exploitable! > > This gets reported relatively frequently. The local command is > non-standard but a common extension in shells. In the shells that > provide it, it gets treated the same, syntax-wise, as the standard > export command, including in dash. > > Unfortunately, POSIX currently requires the export command to not have > any magic quoting, and any POSIX-conforming shell will make > > a="b c=d" > export a=$a > > set a to b, and c to d. Not so with bash, but that's because bash simply > isn't POSIX-conforming, even if invoked as sh. > > POSIX will require special quoting rules for the export command in the > future, similar to what bash does today. When it does, dash is likely to > change to match that, and the local command will likely be changed to > work the same way. > > Right now, though, since the special quoting behaviour is non-standard, > this is a bug in the script unless the script is explicitly stated to > work only with specific shells. If the script is meant to be portable, > even if only across shells that provide the local command, quoting > $(...) is the right thing to do. > > Alternatively: > > local a > a=$(...) > > should work too, including in dash. Since a=$(...) is not an argument to > any command here, since it's the shell syntax that says it's an > assignment rather than the semantics of a particular command, field > splitting won't happen here. > > Cheers, > Harald van Dijk Hi, thanks for the explanation :) I had devised a few more tests and realised that dash is applying word-splitting after substitution, as would be expected for ordinary external commands e.g. $ X="256 if=foo" $ dd bs=$X dd: failed to open ‘foo’: No such file or directory where one would always expect to write bs="$X" with quotes if one wanted to ensure that it was taken as a single parameter and without quotes if one wanted it to be broken into multiple words. It was just a surprise to find this (rather than bash's implicit quoting) applying to dash builtin commands! Another variant that does work, this time by escaping rather than quoting the TAB, and deferring conversion of '\t' into a TAB until after the word-splitting: $ X='f() { local t=$(printf a\\\\tc=d); printf "$t+$c"; }; f' $ bash -c "$X" | hd 00000000 61 09 63 3d 64 2b |a.c=d+| 00000006 $ dash -c "$X" | hd 00000000 61 09 63 3d 64 2b |a.c=d+| 00000006 Cheers, .Dave. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [bug-diffutils] bug#24116: [platform-testers] new snapshot available: diffutils-3.3.50-0353 2016-08-05 13:13 ` Harald van Dijk 2016-08-05 14:09 ` Dave Gordon @ 2016-08-05 16:21 ` Eric Blake 2016-08-05 17:15 ` Harald van Dijk 1 sibling, 1 reply; 6+ messages in thread From: Eric Blake @ 2016-08-05 16:21 UTC (permalink / raw) To: Harald van Dijk, Dave Gordon, Jim Meyering, Assaf Gordon; +Cc: 24116, dash [-- Attachment #1: Type: text/plain, Size: 1383 bytes --] On 08/05/2016 07:13 AM, Harald van Dijk wrote: > > Unfortunately, POSIX currently requires the export command to not have > any magic quoting, and any POSIX-conforming shell will make > > a="b c=d" > export a=$a > > set a to b, and c to d. Not so with bash, but that's because bash simply > isn't POSIX-conforming, even if invoked as sh. Not quite so fast. > > POSIX will require special quoting rules for the export command in the > future, similar to what bash does today. When it does, dash is likely to > change to match that, and the local command will likely be changed to > work the same way. POSIX has already issued an interpretation request, and therefore, any CURRENT implementations (including bash) that already behave as permitted by the interpretation ARE conforming. Dash is still currently conforming per the current version of POSIX, but will be non-conforming when the interpretation request goes live in the next revision of POSIX, while bash is already conforming both now and in the future revision, without change. So I would suggest that dash make the change to implement assignment context to 'export', 'readonly', and 'local' sooner, rather than later. http://austingroupbugs.net/view.php?id=351 -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 604 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [bug-diffutils] bug#24116: [platform-testers] new snapshot available: diffutils-3.3.50-0353 2016-08-05 16:21 ` Eric Blake @ 2016-08-05 17:15 ` Harald van Dijk 2016-08-23 22:04 ` declaration utilities (was: [bug-diffutils] bug#24116: [platform-testers] new snapshot available: diffutils-3.3.50-0353) Jilles Tjoelker 0 siblings, 1 reply; 6+ messages in thread From: Harald van Dijk @ 2016-08-05 17:15 UTC (permalink / raw) To: Eric Blake, Dave Gordon, Jim Meyering, Assaf Gordon; +Cc: 24116, dash On 05/08/2016 18:21, Eric Blake wrote: > On 08/05/2016 07:13 AM, Harald van Dijk wrote: >> Unfortunately, POSIX currently requires the export command to not have >> any magic quoting, and any POSIX-conforming shell will make >> >> a="b c=d" >> export a=$a >> >> set a to b, and c to d. Not so with bash, but that's because bash simply >> isn't POSIX-conforming, even if invoked as sh. > > Not quite so fast. > >> >> POSIX will require special quoting rules for the export command in the >> future, similar to what bash does today. When it does, dash is likely to >> change to match that, and the local command will likely be changed to >> work the same way. > > POSIX has already issued an interpretation request, and therefore, any > CURRENT implementations (including bash) that already behave as > permitted by the interpretation ARE conforming. [...] > http://austingroupbugs.net/view.php?id=351 An interpretation request doesn't automatically mean that all current implementations are conforming, does it? It only means that when the interpretation response says so. In this case, the response says that the standard does not allow bash/ksh's behaviour. http://austingroupbugs.net/view.php?id=351#c865: > Interpretation response > ------------------------ > The standard states that the current behavior of ksh93 does not conform, and conforming implementations must conform to this. However, concerns have been raised about this which are being referred to the sponsor. The fact that this is seen as a defect in the standard which will be fixed, rather than a defect in bash/ksh, doesn't make bash/ksh conform, it merely means that bash/ksh shouldn't be changed to conform. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: declaration utilities (was: [bug-diffutils] bug#24116: [platform-testers] new snapshot available: diffutils-3.3.50-0353) 2016-08-05 17:15 ` Harald van Dijk @ 2016-08-23 22:04 ` Jilles Tjoelker 0 siblings, 0 replies; 6+ messages in thread From: Jilles Tjoelker @ 2016-08-23 22:04 UTC (permalink / raw) To: Harald van Dijk; +Cc: Eric Blake, dash [Cc trimmed] On Fri, Aug 05, 2016 at 07:15:02PM +0200, Harald van Dijk wrote: > On 05/08/2016 18:21, Eric Blake wrote: > > On 08/05/2016 07:13 AM, Harald van Dijk wrote: > >> Unfortunately, POSIX currently requires the export command to not have > >> any magic quoting, and any POSIX-conforming shell will make > >> a="b c=d" > >> export a=$a > >> set a to b, and c to d. Not so with bash, but that's because bash simply > >> isn't POSIX-conforming, even if invoked as sh. > > Not quite so fast. > >> POSIX will require special quoting rules for the export command in the > >> future, similar to what bash does today. When it does, dash is likely to > >> change to match that, and the local command will likely be changed to > >> work the same way. > > POSIX has already issued an interpretation request, and therefore, any > > CURRENT implementations (including bash) that already behave as > > permitted by the interpretation ARE conforming. [...] > > http://austingroupbugs.net/view.php?id=351 > An interpretation request doesn't automatically mean that all current > implementations are conforming, does it? It only means that when the > interpretation response says so. In this case, the response says that > the standard does not allow bash/ksh's behaviour. > http://austingroupbugs.net/view.php?id=351#c865: > > Interpretation response > > ------------------------ > > The standard states that the current behavior of ksh93 does not > > conform, and conforming implementations must conform to this. > > However, concerns have been raised about this which are being > > referred to the sponsor. > The fact that this is seen as a defect in the standard which will be > fixed, rather than a defect in bash/ksh, doesn't make bash/ksh conform, > it merely means that bash/ksh shouldn't be changed to conform. Although it is certainly true that the current standard forbids special quoting rules for export and readonly, making this change sooner rather than later will be helpful to users. Looking at ChangeLog.O, I can see dash had special quoting rules for export, readonly and local between 1997 and 2001. They were removed, basically, because there was no specification guidance. The specification guidance is now available in the form of Austin group bug #351. Users of FreeBSD sh have likewise liked having these new rules available. -- Jilles Tjoelker ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-08-23 22:04 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <CA+8g5KEOBs=AtZoBZw7CJ5wW8=Yw88KrvjJK1PeEqK3uj_1wEg@mail.gmail.com> [not found] ` <9C56E56C-4D31-46AB-AC75-1AA8A759BF4D@gmail.com> [not found] ` <CA+8g5KGack9X8ByfoJEtbQHj-44iG6bvQ0yhguVqQ4vqZh4geA@mail.gmail.com> 2016-08-05 12:46 ` [bug-diffutils] bug#24116: [platform-testers] new snapshot available: diffutils-3.3.50-0353 Dave Gordon 2016-08-05 13:13 ` Harald van Dijk 2016-08-05 14:09 ` Dave Gordon 2016-08-05 16:21 ` Eric Blake 2016-08-05 17:15 ` Harald van Dijk 2016-08-23 22:04 ` declaration utilities (was: [bug-diffutils] bug#24116: [platform-testers] new snapshot available: diffutils-3.3.50-0353) Jilles Tjoelker
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).