Re: [PATCH v3] coccicheck: process every source file at once

From: "René Scharfe" <l.s.r@web.de>
To: "Jeff King" <peff@peff.net>, "SZEDER Gábor" <szeder.dev@gmail.com>
Cc: "Keller, Jacob E" <jacob.e.keller@intel.com>,
	Jacob Keller <jacob.keller@gmail.com>,
	Git mailing list <git@vger.kernel.org>
Subject: Re: [PATCH v3] coccicheck: process every source file at once
Date: Sat, 6 Oct 2018 10:42:57 +0200	[thread overview]
Message-ID: <8b51c4c0-df51-7639-07fe-74d6ca3b5944@web.de> (raw)
In-Reply-To: <20181005190045.GA17482@sigill.intra.peff.net>

Am 05.10.2018 um 21:00 schrieb Jeff King:
> On Fri, Oct 05, 2018 at 08:50:50PM +0200, SZEDER Gábor wrote:
> 
>> On Fri, Oct 05, 2018 at 12:59:01PM -0400, Jeff King wrote:
>>> On Fri, Oct 05, 2018 at 04:53:35PM +0000, Keller, Jacob E wrote:
>>>
>>>>> Are we OK with saying 1.3-1.8GB is necessary to run coccicheck? That
>>>>> doesn't feel like an exorbitant request for a developer-only tool these
>>>>> days, but I have noticed some people on the list tend to have lousier
>>>>> machines than I do. ;)
>>>>>
>>>>> -Peff
>>>>
>>>> It's probably not worth trying to make this more complicated and scale
>>>> up how many files we do at once based on the amount of available
>>>> memory on the system...
>>>
>>> Yeah, that sounds too complicated. At most I'd give a Makefile knob to
>>> say "spatch in batches of $(N)". But I'd prefer to avoid even that
>>> complexity if we can.
>>
>> But perhaps one more if-else, e.g.:
>>
>>   if test -n "$(COCCICHECK_ALL_AT_ONCE)"; then \
>>       <all at once from Jacob>
>>   else
>>       <old for loop>
>>   fi
>>
>> would be an acceptable compromise?  Dunno.
> 
> That's OK, too, assuming people would actually want to use it. I'm also
> OK shipping this (with the "make -j" fix you suggested) and seeing if
> anybody actually complains. I assume there are only a handful of people
> running coccicheck in the first place.

FWIW, my development environment is a virtual machine with 1200MB RAM
and 900MB swap space.  coccicheck takes almost eight minutes
sequentially, and four and a half minutes with -j4.

Unsurprisingly, it fails after almost three minutes with the patch,
reporting that it ran out of memory.  With 2900MB it fails after almost
two minutes, with 3000MB it succeeds after a good two minutes.

time(1) says (for -j1):

433.30user 36.17system 7:49.84elapsed 99%CPU (0avgtext+0avgdata 108212maxresident)k
192inputs+1512outputs (0major+16409056minor)pagefaults 0swaps

129.74user 2.06system 2:13.27elapsed 98%CPU (0avgtext+0avgdata 1884568maxresident)k
236896inputs+1096outputs (795major+462129minor)pagefaults 0swaps

So with the patch it's more than three times faster, but needs more
than seventeen times more memory.  And I need a bigger VM. :-/

René