Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained

From: Julia Lawall <julia.lawall@lip6.fr>
To: Jonathan Nieder <jrnieder@gmail.com>
Cc: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>,
	"Jeff King" <peff@peff.net>,
	"SZEDER Gábor" <szeder.dev@gmail.com>,
	git@vger.kernel.org, "René Scharfe" <l.s.r@web.de>,
	"Derrick Stolee" <stolee@gmail.com>,
	"Himanshu Jha" <himanshujha199640@gmail.com>,
	nicolas.palix@univ-grenoble-alpes.fr,
	yamada.masahiro@socionext.com, cocci@systeme.lip6.fr,
	nicolas.palix@imag.fr
Subject: Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained
Date: Fri, 3 Aug 2018 08:52:38 +0200 (CEST)	[thread overview]
Message-ID: <alpine.DEB.2.20.1808030848160.2446@hadrien> (raw)
In-Reply-To: <20180803064419.GC237521@aiede.svl.corp.google.com>

On Thu, 2 Aug 2018, Jonathan Nieder wrote:

> Hi,
>
> Julia Lawall wrote:
>
> > This was already possible.  Make coccicheck is not supposed to be used
> > with -j, but rather with J=n.  That tells Coccinelle to parallelize the
> > treatment of the files internally.  In this case, the semantic patch is
> > only parsed once, and then n worker processes are forked to treat the
> > different files.
>
> Thanks for this hint.
>
> I wonder if we can make this happen automatically under suitable
> conditions.  We can parse -j<num> out of MAKEFLAGS and convert it into a
> J=<num> argument,

It does happen automatically, in the sense that the default with no -j or
J=option is to use all the cores on the machine.

With -j, it seems to have to default to one core internally:

d7059ca0147adcd495f3c5b41f260e1ac55bb679

Also, if it didn't do that, it would end up with the product of the -j
option and the number of cores on the machine, which would give very bad
performance.

> but that only solves half the problem: the "make"
> parent process would think that the coccinelle run only deserves to
> occupy one job slot.

Yes, it seems to be intended to be run by itself.

> Technically we could do all this using a wrapper that pretends to be a
> submake <https://www.gnu.org/software/make/manual/html_node/Job-Slots.html>
> (prefixing the command with '+', parsing jobserver options from the
> MAKEFLAGS envvar) but it gets ugly.
>
> It's likely that the best we can do is just to advertise J more
> prominently.
>
> [...]
> >> On Thu, Aug 02 2018, Jeff King wrote:
>
> >>>   cat contrib/coccinelle/*.cocci >mega.cocci
> >>>   make -j40 coccicheck COCCI_SEM_PATCHES=mega.cocci
> >
> > There was already a COCCI=foo.cocci argument to focus on a single semantic
> > patch.
> >
> > I'm surprised that the above cat command would work.  Semantic patch rules
> > have names, and Coccinelle will not be happy isf two rules have the same
> > name.
>
> Yes, Git's semantic patches (in contrib/coccinelle) tend to be
> relatively undemanding.
>
> >       Some may also have variables declared in initializers, although
> > perhaps the ones in the kernel don't do this.  Causing these variables to
> > be shared would not have a good effect.
>
> ... oh!  You're thinking of the Linux kernel.
>
> It looks like Linux's scripts/coccicheck has a lot we can crib from.
> That's where the J envvar and automatic parallelism you mentioned are
> implemented, too.
>
> So it sounds to me like at a minimum we should use all of that. ;-)

Yes.

julia