Re: [Toybox] Regression: commit da029c11e6b1 broke toybox xargs.

[enh <enh@google.com>]

On Fri, Nov 3, 2017 at 4:58 PM, Rob Landley <rob@landley.net> wrote:
> On 11/02/2017 10:40 AM, Linus Torvalds wrote:
>> On Wed, Nov 1, 2017 at 9:28 PM, Linus Torvalds
>> <torvalds@linux-foundation.org> wrote:
>>>
>>> Behavior changed. Things that test particular limits will get different
>>> results. That's not breakage.
>>>
>>> Did an actual user application or script break?
>
> Only due to getting the limit wrong. The actual failure's in the android
> internal bugzilla I've never been able to read:
>
> http://lists.landley.net/pipermail/toybox-landley.net/2017-September/009167.html

there was nothing secret or interesting in the original bug report.
just someone trying to use find/xargs on the whole file system. here's
a copy & paste:

marlin:/ # find / /data -xdev -type f ! -name \*.jpg -exec grep -Fl
belkin.a {} +
find: exec grep: Argument list too long

marlin:/ # find / /data -xdev -type f ! -name \*.jpg -print | xargs
grep -Fl belkin.a
xargs: exec grep: Argument list too long

> But it boils down to "got the limit wrong, the exec failed after the
> fork(), dynamic recovery from which is awkward so I'm trying to figure
> out the right limit".
>
>> Ahh. I should have read that email more carefully. If xargs broke,
>> that _will_ break actual scripts, yes. Do you actually set the stack
>> limit to insane values? Anybody using toybox really shouldn't be doing
>> 32MB stacks.
>
> Toybox is the default command line of android since M, which went 64 bit
> in L, and the Pixel 2 phone has 4 gigs of ram. My goal with toybox is to
> turn android into a self-hosting development environment no longer
> cross-compiled from a PC (http://landley.net/talks/celf-2013.txt) so I'm
> trying to implement a command line that can run the entire AOSP build.
>
> I.E. I have no idea what people will do with it, and try not to get in
> their way.
>
> My problem here is it's hard to figure out what exec size the limit
> _is_. There's a sysconf(_SC_ARG_MAX) which bionic and glibc are
> currently returning as stack_limit/4, which is now too big and exec()
> will error out after the fork. Musl is returning the 131072 limit from
> 2011-ish, meaning "/bin/echo $(printf '%0*d' 131071)" works but
> "printf '%0*d' 131071 | xargs" fails, an inconsistency I was trying to
> avoid. Maybe I don't have that luxury...
>
> Each argument has its own limit separate from the argv+envp total limit,
> but there's only one "size" you can query through sysconf, so the
> querying API is insufficient at the design level.
>
> Meanwhile under bash you can allocate and dirty 256 megabytes from the
> command line with:
>
>   echo $(printf '%0*d' $((1<<28)))
>
> Because it's a shell builtin so there's no actual exec. (And if
> https://sourceware.org/bugzilla/show_bug.cgi?id=17829 ever gets fixed
> it'll go back to allowing INT_MAX.)
>
> Posix is its usual helpful self, read conservatively
> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/xargs.html
> says to break the line at 2048 bytes.
>
>> So I still do wonder if this actually breaks anything real, or just a
>> test-suite or something?
>
> I've cc'd Elliott, who would know. (He's the Android base os userspace
> maintainer, he knows everything. Or can at least decode
> http://b/65818597 .)
>
> But this just broke my _fix_, not the earlier deployed stuff. I removed
> the size measuring code when the 131072 limit went away, the bug was
> there's a new limit I need to not hit, I tried to figure out what the
> limit is now, confirmed that the various libc implementations don't
> agree, then the actual kernel limit changed again while I was looking at it.
>
>>                    Linus
>
> Should I just go back to hardwiring in 131072? It's no _less_ arbitrary
> than 10 megs, and it sounds like getting it _right_ is unachievable.
>
> Thanks,
>
> Rob
>
>
> _______________________________________________
> Toybox mailing list
> Toybox@lists.landley.net
> http://lists.landley.net/listinfo.cgi/toybox-landley.net



-- 
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.