From: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
To: Jeff King <peff@peff.net>
Cc: git@vger.kernel.org, "Junio C Hamano" <gitster@pobox.com>,
"René Scharfe" <l.s.r@web.de>,
"SZEDER Gábor" <szeder.dev@gmail.com>, "Eric Wong" <e@80x24.org>
Subject: Re: [PATCH v4 3/4] Makefile/coccicheck: allow for setting xargs concurrency
Date: Thu, 25 Mar 2021 03:29:11 +0100 [thread overview]
Message-ID: <877dlwotjc.fsf@evledraar.gmail.com> (raw)
In-Reply-To: <YFuSSqbAjTmaEMCB@coredump.intra.peff.net>
On Wed, Mar 24 2021, Jeff King wrote:
> On Mon, Mar 22, 2021 at 01:11:49PM +0100, Ævar Arnfjörð Bjarmason wrote:
>
>> Extend the SPATCH_BATCH_SIZE facility added in
>> 960154b9c17 (coccicheck: optionally batch spatch invocations,
>> 2019-05-06) and bcb4edf7af7 (coccicheck: make batch size of 0 mean
>> "unlimited", 2019-05-08) to allow for both setting
>> SPATCH_BATCH_SIZE=N, and also to have a more advanced SPATCH_XARGS
>> argument.
>>
>> The reason to replace the "$$limit" is that now you actually see under
>> V=1 what argument your program will get invoked with.
>>
>> The reason for the "9999" limit is that if you e.g. try to define an
>> "$(XARGS)" which is conditionally an empty string or not depending on
>> this setting then e.g.:
>>
>> echo $(FILES) | $(XARGS) $(XARGS_FLAGS) $(SPATCH)
>>
>> Over multiple lines with "\" will error out. I think just setting it
>> to "xargs -n 9999" as a trivial workaround is the best solution here.
>
> I don't understand this 9999 comment. The original was sometimes setting
> $limit to the empty string, and then doing:
>
> xargs $limit
>
> How is that any different than setting SPATCH_XARGS to just "xargs" for
> the unlimited case?
The "over multiple lines" is important. But it seems not
anymore. I.e. in an earlier version I had:
$(XARGS) \
$(XARGS_FLAGS) \
$(SPATCH)
And it would brek if XARGS_FLAGS was empty. So I set it to -n 9999 as a
fallback.
>> +# For the 'coccicheck' target; SPATCH_XARGS can be used to manually
>> +# tweak the xargs invocation. By default we invoke "xargs -n 1", and
>> +# "xargs -n 9999" under SPATCH_BATCH_SIZE=0.
>> +#
>> +# Setting SPATCH_XARGS overrides SPATCH_BATCH_SIZE. To get concurrency
>> +# when targeting a single contrib/coccinelle/%.patch use e.g. "-P" if
>> +# your xargs(1) supports it:
>> +#
>> +# make contrib/coccinelle/strbuf.cocci.patch SPATCH_XARGS="xargs -P 8 -n 8"
>> +#
>> +# Or a combination -jN and "xargs -P":
>> +#
>> +# make -j4 coccicheck SPATCH_XARGS="xargs -P 2 -n 8"
>
> As I mentioned in the last round, using "-P" is racy. I'm not sure if
> it's something we should be recommending to people.
Yes, this would need to use N tempfiles we'd cat together at the end to
be POSIX-reliable.
In practice I've never seen this sort of thing be unreliable for a case
like this.
POSIX just guarantees that the output won't be interleaved up to
PIPE_BUF, which is typically 4k. We certainly get patches bigger than
that from spatch in some cases.
But from the OS's perspective emitting this output happens at a glacial
pace. So even if it crosses that boundary it's unlikely to be
interleaved.
Even:
perl -wE 'print "a" for 1..1024*1024*100' >1
perl -wE 'print "b" for 1..1024*1024*100' >2
perl -wE 'print "\n" for 1..1024*1024*100' >3
$ du -shc 1 2 3
100M 1
100M 2
100M 3
300M total
Which at least on this computer I can't get to not print:
$ echo 1 2 3 | xargs -P 3 -n 1 cat|wc -l
104857600
Suggesting that even for output of that size the \n's aren't mixed
up. YMMV.
Anyway. I think I'll just redact this topic. Maybe I'll change my mind
on it, or pick it up some other month/year/whatever.
I appreciate everyone's feedback on it, but for something I thought
would be a rather trivial bugfix I've sunk way too much time into it.
At this point I can't even remember why I was trying to run coccicheck
in the first place.
There's e.g. still your outstanding feedback on what's really going on
with that underlying spatch race I was trying to fix with this
--include-headers-for-types workaround.
I think digging into that would be interesting for someone, but right
now that's not me. I really don't want to spend time on digging into
spatch's OCaml guts to figure that & any other outstanding stuff out.
next prev parent reply other threads:[~2021-03-25 2:30 UTC|newest]
Thread overview: 103+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-04-12 23:48 [RFC PATCH] *.h: remove extern from function declarations Denton Liu
2019-04-13 1:24 ` Jeff King
2019-04-13 5:45 ` Junio C Hamano
2019-04-15 18:24 ` [PATCH v2 0/3] " Denton Liu
2019-04-15 18:24 ` [PATCH v2 1/3] *.[ch]: remove extern from function declarations using spatch Denton Liu
2019-04-15 19:19 ` Thomas Gummerer
2019-04-15 18:24 ` [PATCH v2 2/3] *.[ch]: remove extern from function declarations using sed Denton Liu
2019-04-15 18:24 ` [PATCH v2 3/3] cocci: prevent extern function declarations Denton Liu
2019-04-17 7:58 ` [PATCH v3 0/4] remove extern from " Denton Liu
2019-04-17 7:58 ` [PATCH v3 1/4] *.[ch]: remove extern from function declarations using spatch Denton Liu
2019-04-17 7:58 ` [PATCH v3 2/4] *.[ch]: remove extern from function declarations using sed Denton Liu
2019-04-17 7:58 ` [PATCH v3 3/4] *.[ch]: manually align parameter lists Denton Liu
2019-04-17 7:58 ` [PATCH v3 4/4] cocci: prevent extern function declarations Denton Liu
2019-04-22 5:44 ` [PATCH] cache.h: fix mismerge of 'dl/no-extern-in-func-decl' Denton Liu
2019-04-22 6:30 ` Junio C Hamano
2019-04-22 11:19 ` Junio C Hamano
2019-04-22 21:49 ` [PATCH v3 0/4] remove extern from function declarations Jeff King
2019-04-25 12:07 ` SZEDER Gábor
2019-04-25 18:05 ` Denton Liu
2019-04-30 23:21 ` Johannes Schindelin
2019-05-01 10:01 ` Denton Liu
2019-05-01 18:56 ` Jeff King
2019-05-02 0:04 ` SZEDER Gábor
2019-05-03 9:32 ` Johannes Schindelin
2019-05-03 14:42 ` SZEDER Gábor
2019-05-03 14:58 ` SZEDER Gábor
2019-05-03 17:45 ` Jeff King
2019-05-03 18:44 ` SZEDER Gábor
2019-05-05 5:28 ` Junio C Hamano
2019-05-05 18:09 ` Jacob Keller
2019-05-05 18:08 ` Jacob Keller
2019-05-06 5:11 ` [PATCH] coccicheck: optionally process every source file at once Jeff King
2019-05-06 9:34 ` Duy Nguyen
2019-05-06 23:43 ` [PATCH] coccicheck: optionally batch spatch invocations Jeff King
2019-05-07 1:41 ` Jacob Keller
2019-05-07 2:04 ` Jeff King
2019-05-07 2:42 ` Junio C Hamano
2019-05-07 2:55 ` Jeff King
2019-05-07 3:04 ` Jacob Keller
2019-05-07 4:52 ` Junio C Hamano
2019-05-08 7:07 ` Jeff King
2019-05-08 12:36 ` Denton Liu
2019-05-08 22:39 ` Jeff King
2019-05-07 10:20 ` Duy Nguyen
2019-05-07 11:19 ` SZEDER Gábor
2021-03-02 20:51 ` [PATCH] Makefile: fix bugs in coccicheck and speed it up Ævar Arnfjörð Bjarmason
2021-03-03 9:43 ` Denton Liu
2021-03-03 11:45 ` Ævar Arnfjörð Bjarmason
2021-03-04 23:18 ` Junio C Hamano
2021-03-05 11:17 ` Ævar Arnfjörð Bjarmason
2021-03-05 10:24 ` Jeff King
2021-03-05 17:20 ` Ævar Arnfjörð Bjarmason
2021-03-06 10:59 ` Jeff King
2021-03-05 17:07 ` [PATCH v2 0/4] Makefile/coccicheck: fix bugs " Ævar Arnfjörð Bjarmason
2021-03-05 19:10 ` René Scharfe.
[not found] ` <xmqqim659u57.fsf@gitster.c.googlers.com>
2021-03-06 11:26 ` René Scharfe.
2021-03-06 12:43 ` René Scharfe.
[not found] ` <xmqqft16914r.fsf@gitster.c.googlers.com>
2021-03-13 16:10 ` René Scharfe.
2021-03-06 17:27 ` Ævar Arnfjörð Bjarmason
2021-03-06 17:41 ` René Scharfe.
2021-03-06 17:52 ` Ævar Arnfjörð Bjarmason
2021-03-06 19:08 ` René Scharfe.
2021-03-05 17:07 ` [PATCH v2 1/4] Makefile/coccicheck: add comment heading for all SPATCH flags Ævar Arnfjörð Bjarmason
2021-03-05 17:07 ` [PATCH v2 2/4] Makefile/coccicheck: speed up and fix bug with duplicate hunks Ævar Arnfjörð Bjarmason
2021-03-06 10:45 ` Jeff King
2021-03-06 19:29 ` Ævar Arnfjörð Bjarmason
2021-03-05 17:07 ` [PATCH v2 3/4] Makefile/coccicheck: allow for setting xargs concurrency Ævar Arnfjörð Bjarmason
2021-03-06 10:51 ` Jeff King
2021-03-05 17:07 ` [PATCH v2 4/4] Makefile/coccicheck: set SPATCH_BATCH_SIZE to 8 Ævar Arnfjörð Bjarmason
2021-03-06 19:25 ` [PATCH v2 5/4] Makefile/coccicheck: use --include-headers-for-types Ævar Arnfjörð Bjarmason
2021-03-18 20:49 ` SZEDER Gábor
2021-03-19 10:32 ` Ævar Arnfjörð Bjarmason
2021-03-22 12:11 ` [PATCH v4 0/4] Makefile/coccicheck: fix bugs and speed it up Ævar Arnfjörð Bjarmason
2021-03-22 12:11 ` [PATCH v4 1/4] Makefile/coccicheck: add comment heading for all SPATCH flags Ævar Arnfjörð Bjarmason
2021-03-22 18:04 ` René Scharfe.
2021-03-22 12:11 ` [PATCH v4 2/4] Makefile/coccicheck: speed up and fix bug with duplicate hunks Ævar Arnfjörð Bjarmason
2021-03-22 18:05 ` René Scharfe.
2021-03-24 19:19 ` Jeff King
2021-03-22 19:09 ` Junio C Hamano
2021-03-22 12:11 ` [PATCH v4 3/4] Makefile/coccicheck: allow for setting xargs concurrency Ævar Arnfjörð Bjarmason
2021-03-24 19:26 ` Jeff King
2021-03-25 2:29 ` Ævar Arnfjörð Bjarmason [this message]
2021-03-26 4:11 ` Jeff King
2021-03-22 12:11 ` [PATCH v4 4/4] Makefile/coccicheck: set SPATCH_BATCH_SIZE to 8 Ævar Arnfjörð Bjarmason
2021-03-22 18:05 ` René Scharfe.
2021-03-24 19:27 ` Jeff King
2021-03-27 17:43 ` [PATCH v4 0/4] Makefile/coccicheck: fix bugs and speed it up Junio C Hamano
2021-03-27 19:46 ` Ævar Arnfjörð Bjarmason
2019-05-03 9:40 ` [PATCH v3 0/4] remove extern from function declarations Denton Liu
2019-04-23 23:40 ` [PATCH v4 " Denton Liu
2019-04-23 23:40 ` [PATCH v4 1/4] *.[ch]: remove extern from function declarations using spatch Denton Liu
2019-04-23 23:40 ` [PATCH v4 2/4] *.[ch]: remove extern from function declarations using sed Denton Liu
2019-04-24 4:56 ` Junio C Hamano
2019-04-25 19:00 ` Denton Liu
2019-04-23 23:40 ` [PATCH v4 3/4] *.[ch]: manually align parameter lists Denton Liu
2019-04-23 23:40 ` [PATCH v4 4/4] cocci: prevent extern function declarations Denton Liu
2019-04-29 8:28 ` [PATCH v5 0/3] *** SUBJECT HERE *** Denton Liu
2019-04-29 8:28 ` [PATCH v5 1/3] *.[ch]: remove extern from function declarations using spatch Denton Liu
2019-04-29 8:28 ` [PATCH v5 2/3] *.[ch]: remove extern from function declarations using sed Denton Liu
2019-04-29 8:28 ` [PATCH v5 3/3] *.[ch]: manually align parameter lists Denton Liu
2019-04-29 8:30 ` [PATCH v5 0/3] *** SUBJECT HERE *** Denton Liu
2019-05-06 11:03 ` Ævar Arnfjörð Bjarmason
2019-05-06 15:34 ` Denton Liu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=877dlwotjc.fsf@evledraar.gmail.com \
--to=avarab@gmail.com \
--cc=e@80x24.org \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=l.s.r@web.de \
--cc=peff@peff.net \
--cc=szeder.dev@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).