Re: [PATCH] mk: optimize directory dependencies

[Bruce Richardson <bruce.richardson@intel.com>]

On Tue, Jan 24, 2017 at 05:10:15PM +0530, Jerin Jacob wrote:
> On Mon, Jan 23, 2017 at 06:19:13PM +0100, Olivier Matz wrote:
> > Before this patch, the management of dependencies between directories
> > had several issues:
> > 
> > - the generation of .depdirs, done at configuration is slow: it can take
> >   more than one minute on some slow targets (usually ~10s on a standard
> >   PC).
> > 
> > - for instance, it is possible to expressed a dependency like:
> >   - app/foo depends on lib/librte_foo
> >   - and lib/librte_foo depends on app/bar
> >   But this won't work because the directories are traversed with a
> >   depth-first algorithm, so we have to choose between doing 'app' before
> >   or after 'lib'.
> > 
> > - the script depdirs-rule.sh is too complex.
> > 
> > - we cannot use "make -d" for debug, because the output of make is used for
> >   the generation of .depdirs.
> > 
> > This patch moves the DEPDIRS-* variables in the upper Makefile, making
> > the dependencies much easier to calculate. A DEPDIRS variable is still
> > used to process library dependencies in LDLIBS.
> > 
> > After this commit, "make config" is almost immediate.
> > 
> > Signed-off-by: Olivier Matz <olivier.matz@6wind.com>
> 
> Tested both approach on ThunderX. This patch looks better
> 
> Tested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> 
> ➜ [master]GB-2S [dpdk-master] $ time make config T=arm64-thunderx-linuxapp-gcc 
> Configuration done
> 
> real    0m18.112s
> user    0m2.810s
> sys     0m0.660s
> 
> ➜ [master]GB-2S [dpdk-master] $ pwclient git-am 19859
> Applying patch #19859 using 'git am'
> Description: [dpdk-dev] mk: parallelize make config
> Applying: mk: parallelize make config
> 
> ➜ [master]GB-2S [dpdk-master] $ rm -rf build
> ➜ [master]GB-2S [dpdk-master] $ time make config T=arm64-thunderx-linuxapp-gcc -j 8
> Configuration done
> 
> real    0m2.812s
> user    0m3.020s
> sys     0m0.870s
> 
> ➜ [master]GB-2S [dpdk-master] $ rm -rf build
> ➜ [master]GB-2S [dpdk-master] $ time make config T=arm64-thunderx-linuxapp-gcc -j 16
> Configuration done
> 
> real    0m1.748s
> user    0m3.040s
> sys     0m1.020s
> 
> ➜ [master]GB-2S [dpdk-master] $ rm -rf build
> ➜ [master]GB-2S [dpdk-master] $ time make config T=arm64-thunderx-linuxapp-gcc -j 32
> Configuration done
> 
> real    0m1.422s
> user    0m3.380s
> sys     0m1.080s
> 
> ➜ [master]GB-2S [dpdk-master] $ pwclient git-am 19918
> Applying patch #19918 using 'git am'
> Description: [dpdk-dev] mk: optimize directory dependencies
> Applying: mk: optimize directory dependencies
> 
> ➜ [master]GB-2S [dpdk-master] $ rm -rf build
> ➜ [master]GB-2S [dpdk-master] $ time make config T=arm64-thunderx-linuxapp-gcc 
> Configuration done
> 
> real    0m0.064s
> user    0m0.000s
> sys     0m0.000s
> ➜ [master]GB-2S [dpdk-master] $ rm -rf build
> ➜ [master]GB-2S [dpdk-master] $ time make config T=arm64-thunderx-linuxapp-gcc -j 8
> Configuration done
> 
> real    0m0.055s
> user    0m0.000s
> sys     0m0.000s
>

I agree that Olivier's patch is faster. However, I think I prefer having
the library dependencies in the makefiles for the libs themselves rather
than up a level. Given that we are only looking at ~2second of a
difference here in your tests - assuming -j flag - what is the actual
build time differences? My suspicion is that after Ferruh's simpler
patch is applied, the config time becomes such a small part of the
build, that the extra benefits from Oliviers work is not worth the extra
complexity.

Regards,
/Bruce