Re: [PATCH v2 3/3] Makefile: run coccicheck on more source files

From: Denton Liu <liu.denton@gmail.com>
To: "SZEDER Gábor" <szeder.dev@gmail.com>
Cc: Junio C Hamano <gitster@pobox.com>,
	Git Mailing List <git@vger.kernel.org>,
	Johannes Schindelin <Johannes.Schindelin@gmx.de>,
	Jeff King <peff@peff.net>
Subject: Re: [PATCH v2 3/3] Makefile: run coccicheck on more source files
Date: Fri, 13 Sep 2019 14:38:18 -0700	[thread overview]
Message-ID: <20190913213818.GA88412@dentonliu-ltm.internal.salesforce.com> (raw)
In-Reply-To: <20190913180014.GQ32087@szeder.dev>

On Fri, Sep 13, 2019 at 08:00:14PM +0200, SZEDER Gábor wrote:
> On Fri, Sep 13, 2019 at 10:14:01AM -0700, Denton Liu wrote:
> > On Fri, Sep 13, 2019 at 01:49:52PM +0200, SZEDER Gábor wrote:
> > > On Thu, Sep 12, 2019 at 11:40:36AM -0700, Junio C Hamano wrote:
> > > > Denton Liu <liu.denton@gmail.com> writes:
> > > > 
> > > > > +FIND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES)))
> > > > > +COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FIND_C_SOURCES))
> > > > 
> > > > The former is somewhat misnamed.  FIND_SOURCE_FILES is *not* a list
> > > > of source files---it is a procedure to list source files to its
> > > > standard output.  FIND_C_SOUCRES sounds as if it is a similar
> > > > procedure, which would be implemented much like
> > > > 
> > > > 	FIND_C_SOURCES = $(FIND_SOURCE_FILES) | sed -n -e '/\.c$/p'
> > > > 
> > > > but that is not what you did and that is not what you want to have.
> > > > Perhaps call it FOUND_C_SOURCES?
> > > > 
> > > > I wonder if we can get rid of FIND_SOURCE_FILES that is a mere
> > > > procedure and replace its use with a true list of source files.
> > > > Would it make the result more pleasant to work with?
> > > > 
> > > > Perhaps something like the attached patch, (which would come before
> > > > this entire thing as a clean-up, and removing the need for 2/3)?
> > > > 
> > > > I dunno.
> > > > 
> > > > Using a procedure whose output is fed to xargs has an advantage that
> > > > a platform with very short command line limit can still work with
> > > > many source files, but the way you create and use COCCI_SOURCES in
> > > > this patch would defeat that advantage anyway,
> > > 
> > > COCCI_SOURCES is only used as an input to 'xargs', so that advantage
> > > is not defeated.
> > 
> > I think it still does matter; the relevant snippet is as follows:
> > 
> > 	if ! echo $(COCCI_SOURCES) | xargs $$limit \
> > 		$(SPATCH) --sp-file $< $(SPATCH_FLAGS) \
> > 		>$@+ 2>$@.log; \
> > 
> > which means that a really big COCCI_SOURCES could exceed the limit.
> 
> Oh, you're both right.
> 
> > That being said, COCCI_SOURCES should be smaller than the future
> > SOURCE_FILES variable since we're only taking %.c files (and filtering
> > out some of them too!).
> 
> We could also argue that Coccinelle only runs on platforms that have a
> reasonably large command line arg limit, and the number of our source
> files is way below that, so it won't matter in the foreseeable future.

Good point.

> 
> (Furthermore, 'echo' is often a shell builtin command, and I don't
> think that the platform's argument size limit applies to them.  At
> least the 'echo' of dash, Bash, ksh, ksh93, mksh, and BusyBox sh can
> deal with at least 10 million arguments; the platform limit is
> somewhere around 147k)
> 
> > > > diff --git a/Makefile b/Makefile
> > > > index f9255344ae..9dddd0e88c 100644
> > > > --- a/Makefile
> > > > +++ b/Makefile
> > > > @@ -2584,7 +2584,7 @@ perl/build/man/man3/Git.3pm: perl/Git.pm
> > > >  	$(QUIET_GEN)mkdir -p $(dir $@) && \
> > > >  	pod2man $< $@
> > > >  
> > > > -FIND_SOURCE_FILES = ( \
> > > > +SOURCE_FILES = $(patsubst ./%,%,$(shell \
> > > >  	git ls-files \
> > > >  		'*.[hcS]' \
> > > >  		'*.sh' \
> > > > @@ -2599,19 +2599,19 @@ FIND_SOURCE_FILES = ( \
> > > >  		-o \( -name 'trash*' -type d -prune \) \
> > > >  		-o \( -name '*.[hcS]' -type f -print \) \
> > > >  		-o \( -name '*.sh' -type f -print \) \
> > > > -	)
> > > > +	))
> > > >  
> > > >  $(ETAGS_TARGET): FORCE
> > > >  	$(RM) $(ETAGS_TARGET)
> > > > -	$(FIND_SOURCE_FILES) | xargs etags -a -o $(ETAGS_TARGET)
> > > > +	etags -a -o $(ETAGS_TARGET) $(SOURCE_FILES)
> > > >  
> > > >  tags: FORCE
> > > >  	$(RM) tags
> > > > -	$(FIND_SOURCE_FILES) | xargs ctags -a
> > > > +	ctags -a $(SOURCE_FILES)
> > > >  
> > > >  cscope:
> > > >  	$(RM) cscope*
> > > > -	$(FIND_SOURCE_FILES) | xargs cscope -b
> > > > +	cscope -b $(SOURCE_FILES)
> 
> Here, however, the list of source files is passed as argument to
> non-builtin commands, that also might be used on
> cmdline-arg-limit-challenged platforms.
> 

After doing a bit of research, I think that I agree with you. It seems
like the max command-line length for CMD on Windows is 8191 characters.

However, after running the following,

	$ git ls-files '*.[hcS]' '*.sh' ':!*[tp][0-9][0-9][0-9][0-9]*' ':!contrib' | wc -c
	   12779

we can see that the command-line length would definitely exceed the max
length so xargs would be required. As a result, we should probably just
keep the existing xargs invocations.