From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753757AbdKMQoM (ORCPT ); Mon, 13 Nov 2017 11:44:12 -0500 Received: from mail2-relais-roc.national.inria.fr ([192.134.164.83]:33085 "EHLO mail2-relais-roc.national.inria.fr" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753520AbdKMQoK (ORCPT ); Mon, 13 Nov 2017 11:44:10 -0500 X-IronPort-AV: E=Sophos;i="5.44,389,1505772000"; d="scan'208";a="300603812" Date: Mon, 13 Nov 2017 17:44:04 +0100 (CET) From: Julia Lawall X-X-Sender: jll@hadrien To: Masahiro Yamada cc: Michal Marek , Linux Kbuild mailing list , Nicolas Palix , Linux Kernel Mailing List , cocci@systeme.lip6.fr Subject: Re: [Cocci] [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck In-Reply-To: Message-ID: References: <1510210850-28866-1-git-send-email-yamada.masahiro@socionext.com> User-Agent: Alpine 2.20 (DEB 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, 14 Nov 2017, Masahiro Yamada wrote: > Hi Julia, > > > 2017-11-11 16:30 GMT+09:00 Julia Lawall : > > > > > > On Fri, 10 Nov 2017, Julia Lawall wrote: > > > >> > >> > >> On Thu, 9 Nov 2017, Masahiro Yamada wrote: > >> > >> > The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of > >> > "coccicheck failed" error messages. > >> > >> The question is where parallelism should be specified. Currently, make > >> coccicheck picks up the number of cores on the machine and passes that to > >> Coccinelle. > >> > >> OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1" > >> > >> On my 80 core machine with hyperthreading, this runs 160 jobs in parallel, > >> while in practice that degrades the performance as compared to 40 or 80 > >> cores. > >> > >> On the other hand, if we use the make command line argument (-j), then we > >> will only get parallelism up to the number of semantic patches. Since > >> some finish quickly, there will be a lot of wasted cycles. > >> > >> The best would be that the user knows what works well for his machine, and > >> specifies it on the command line, and then that value gets propagated to > >> Coccinelle, eg so that -j8 would cause not 8 semantic patches to run in > >> parallel but instead would cause Coccinelle to run one semantic patch on 8 > >> files in parallel. But I don't know if that can be done. > > > > Sorry for these fairly nonsensical comments. make -j is going to consider > > every file, then parse and run every semantic patch on that file. If the > > parallelism is pushed down into Coccinelle, each semantic patch will be > > parsed only once, and then Coccinelle will choose the files for which it > > is relevant. If indexing is used (idutils, glimpse), then for semantic > > patches that focus on specific keywords, Coccinelle will efficiently > > ignore files that are not relevant. I don't think there would be many > > cases where make -j would win. Perhaps it would be possible to detect > > its used and abort with an appopriate message? > > > I am afraid you and I are talking different things. > > > For a usual usage of coccicheck, only one thread runs scripts/coccicheck > even if -j is passed from the command line. > > coccicheck provides "J" to specify parallelism. > > if [ -z "$J" ]; then > NPROC=$(getconf _NPROCESSORS_ONLN) > else > NPROC="$J" > fi Even if J is not specified, then it still runs with the maximum number of threads: Coccinelle parallelization --------------------------- By default, coccicheck tries to run as parallel as possible. Indeed, J= does set the number of threads speficied. > My patch addresses a problem where coccicheck is used as CHECK. > The default of CHECK is "sparse", but you can use any checker tool. > > In CHECK=scripts/coccicheck case, if -j is passed, all tasks run in parallel > under control of GNU Make, so scripts/coccicheck is also invoked from > multiple threads. > Passing --jobs to spatch is not sensible because it checks only one file. OK. I tried a simple make coccicheck -j4 and indeed it does not seem to be complaining. The number of spatch processes goes over 160 though. julia