[PATCH 0/2] make deb-pkg build speed improvements

[PATCH 0/2] make deb-pkg build speed improvements

[Chris J Arges]

These two patches should speed up the make deb-pkg build process by utilizing
the make jobserver properly, and using 'find | xargs' to parallelize debug
module installation.

I've tested with the resulting packages to ensure the symbols work as intended
by crash dumping a machine and using the dbg package symbols.

Using an Ubuntu distro conf + make olddefconfig on a 56-core machine, I can see
the following improvements (make clean && make deb-pkg -j`nproc`):

* Before

real    40m56.240s
user    182m17.292s
sys     140m18.812s

* After

real    36m22.633s
user    182m28.028s
sys     148m17.724s

Chris J Arges (2):
  package: Makefile: ensure $MAKE can use jobserver
  builddeb: parallelize debug module installation

 scripts/package/Makefile |  2 +-
 scripts/package/builddeb | 15 ++++++++-------
 2 files changed, 9 insertions(+), 8 deletions(-)

-- 
1.9.1

[PATCH 1/2] package: Makefile: ensure $MAKE can use jobserver

[Chris J Arges]

When using make deb-pkg, builddeb is called without proper MAKEFLAGS due to the
script being invoked without '+'. This results in the following message when
building:
warning: jobserver unavailable: using -j1. Add `+' to parent make rule

Add the '+' so the make operations can be parallelized.

Signed-off-by: Chris J Arges <chris.j.arges@canonical.com>
---
 scripts/package/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/package/Makefile b/scripts/package/Makefile
index 99ca6e7..0dbfae7 100644
--- a/scripts/package/Makefile
+++ b/scripts/package/Makefile
@@ -88,7 +88,7 @@ quiet_cmd_builddeb = BUILDDEB
 
 deb-pkg: FORCE
 	$(MAKE) KBUILD_SRC=
-	$(call cmd,builddeb)
+	+$(call cmd,builddeb)
 
 clean-dirs += $(objtree)/debian/
 
-- 
1.9.1

[PATCH 2/2] builddeb: parallelize debug module installation

[Chris J Arges]

When building the dbg package, we use a large 'for module in $(find' loop that
can be easily parallelized by using 'find | xargs'. This patch modifies this
loop to use the later paradigm.

Signed-off-by: Chris J Arges <chris.j.arges@canonical.com>
---
 scripts/package/builddeb | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/scripts/package/builddeb b/scripts/package/builddeb
index 88dbf23..d12d062 100755
--- a/scripts/package/builddeb
+++ b/scripts/package/builddeb
@@ -152,16 +152,17 @@ if grep -q '^CONFIG_MODULES=y' $KCONFIG_CONFIG ; then
 		rmdir "$tmpdir/lib/modules/$version"
 	fi
 	if [ -n "$BUILD_DEBUG" ] ; then
-		for module in $(find $tmpdir/lib/modules/ -name *.ko -printf '%P\n'); do
-			module=lib/modules/$module
-			mkdir -p $(dirname $dbg_dir/usr/lib/debug/$module)
+		find $tmpdir/lib/modules/ -name *.ko -printf '%P\n' | xargs -I {} sh -c '
+			mkdir -p $(dirname '"$dbg_dir"'/usr/lib/debug/lib/modules/$1);'"
 			# only keep debug symbols in the debug file
-			$OBJCOPY --only-keep-debug $tmpdir/$module $dbg_dir/usr/lib/debug/$module
+			$OBJCOPY --only-keep-debug $tmpdir/lib/modules/{} \
+				$dbg_dir/usr/lib/debug/lib/modules/{};
 			# strip original module from debug symbols
-			$OBJCOPY --strip-debug $tmpdir/$module
+			$OBJCOPY --strip-debug $tmpdir/lib/modules/{};
 			# then add a link to those
-			$OBJCOPY --add-gnu-debuglink=$dbg_dir/usr/lib/debug/$module $tmpdir/$module
-		done
+			$OBJCOPY --add-gnu-debuglink=$dbg_dir/usr/lib/debug/lib/modules/{} \
+				$tmpdir/lib/modules/{};
+		" -- {}
 	fi
 fi
 
-- 
1.9.1

Re: [PATCH 2/2] builddeb: parallelize debug module installation

[Michal Marek]

On 2015-04-24 20:25, Chris J Arges wrote:
> When building the dbg package, we use a large 'for module in $(find' loop that
> can be easily parallelized by using 'find | xargs'. This patch modifies this
> loop to use the later paradigm.
> 
> Signed-off-by: Chris J Arges <chris.j.arges@canonical.com>
> ---
>  scripts/package/builddeb | 15 ++++++++-------
>  1 file changed, 8 insertions(+), 7 deletions(-)
> 
> diff --git a/scripts/package/builddeb b/scripts/package/builddeb
> index 88dbf23..d12d062 100755
> --- a/scripts/package/builddeb
> +++ b/scripts/package/builddeb
> @@ -152,16 +152,17 @@ if grep -q '^CONFIG_MODULES=y' $KCONFIG_CONFIG ; then
>  		rmdir "$tmpdir/lib/modules/$version"
>  	fi
>  	if [ -n "$BUILD_DEBUG" ] ; then
> -		for module in $(find $tmpdir/lib/modules/ -name *.ko -printf '%P\n'); do
> -			module=lib/modules/$module
> -			mkdir -p $(dirname $dbg_dir/usr/lib/debug/$module)
> +		find $tmpdir/lib/modules/ -name *.ko -printf '%P\n' | xargs -I {} sh -c '

I guess you want to use the -P option here.

Michal

[PATCH v2] builddeb: parallelize debug module installation

[Chris J Arges]

When building the dbg package, we use a large 'for module in $(find' loop that
can be easily parallelized by using 'find | xargs'. This patch modifies this
loop to use the later paradigm.

In addition, ensure we add '-n1 -P0' to xargs to run as many processes as
possible.

Signed-off-by: Chris J Arges <chris.j.arges@canonical.com>
---
 scripts/package/builddeb | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/scripts/package/builddeb b/scripts/package/builddeb
index 88dbf23..538f829 100755
--- a/scripts/package/builddeb
+++ b/scripts/package/builddeb
@@ -152,16 +152,17 @@ if grep -q '^CONFIG_MODULES=y' $KCONFIG_CONFIG ; then
 		rmdir "$tmpdir/lib/modules/$version"
 	fi
 	if [ -n "$BUILD_DEBUG" ] ; then
-		for module in $(find $tmpdir/lib/modules/ -name *.ko -printf '%P\n'); do
-			module=lib/modules/$module
-			mkdir -p $(dirname $dbg_dir/usr/lib/debug/$module)
+		find $tmpdir/lib/modules/ -name *.ko -printf '%P\n' | xargs -n1 -P0 -I {} sh -c '
+			mkdir -p $(dirname '"$dbg_dir"'/usr/lib/debug/lib/modules/$1);'"
 			# only keep debug symbols in the debug file
-			$OBJCOPY --only-keep-debug $tmpdir/$module $dbg_dir/usr/lib/debug/$module
+			$OBJCOPY --only-keep-debug $tmpdir/lib/modules/{} \
+				$dbg_dir/usr/lib/debug/lib/modules/{};
 			# strip original module from debug symbols
-			$OBJCOPY --strip-debug $tmpdir/$module
+			$OBJCOPY --strip-debug $tmpdir/lib/modules/{};
 			# then add a link to those
-			$OBJCOPY --add-gnu-debuglink=$dbg_dir/usr/lib/debug/$module $tmpdir/$module
-		done
+			$OBJCOPY --add-gnu-debuglink=$dbg_dir/usr/lib/debug/lib/modules/{} \
+				$tmpdir/lib/modules/{};
+		" -- {}
 	fi
 fi
 
-- 
1.9.1

Re: [PATCH 1/2] package: Makefile: ensure $MAKE can use jobserver

[Riku Voipio]

On 24 April 2015 at 21:25, Chris J Arges <chris.j.arges@canonical.com> wrote:
> When using make deb-pkg, builddeb is called without proper MAKEFLAGS due to the
> script being invoked without '+'. This results in the following message when
> building:
> warning: jobserver unavailable: using -j1. Add `+' to parent make rule
>
> Add the '+' so the make operations can be parallelized.

Looks good, and works fine in my tests.

Reviewed-by: Riku Voipio <riku.voipio@linaro.org>

> Signed-off-by: Chris J Arges <chris.j.arges@canonical.com>
> ---
>  scripts/package/Makefile | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/scripts/package/Makefile b/scripts/package/Makefile
> index 99ca6e7..0dbfae7 100644
> --- a/scripts/package/Makefile
> +++ b/scripts/package/Makefile
> @@ -88,7 +88,7 @@ quiet_cmd_builddeb = BUILDDEB
>
>  deb-pkg: FORCE
>         $(MAKE) KBUILD_SRC=
> -       $(call cmd,builddeb)
> +       +$(call cmd,builddeb)
>
>  clean-dirs += $(objtree)/debian/
>
> --
> 1.9.1
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] builddeb: parallelize debug module installation

[Riku Voipio]

On 27 April 2015 at 19:43, Chris J Arges <chris.j.arges@canonical.com> wrote:
> When building the dbg package, we use a large 'for module in $(find' loop that
> can be easily parallelized by using 'find | xargs'. This patch modifies this
> loop to use the later paradigm.
>
> In addition, ensure we add '-n1 -P0' to xargs to run as many processes as
> possible.
>
> Signed-off-by: Chris J Arges <chris.j.arges@canonical.com>
> ---
>  scripts/package/builddeb | 15 ++++++++-------
>  1 file changed, 8 insertions(+), 7 deletions(-)
>
> diff --git a/scripts/package/builddeb b/scripts/package/builddeb
> index 88dbf23..538f829 100755
> --- a/scripts/package/builddeb
> +++ b/scripts/package/builddeb
> @@ -152,16 +152,17 @@ if grep -q '^CONFIG_MODULES=y' $KCONFIG_CONFIG ; then
>                 rmdir "$tmpdir/lib/modules/$version"
>         fi
>         if [ -n "$BUILD_DEBUG" ] ; then
> -               for module in $(find $tmpdir/lib/modules/ -name *.ko -printf '%P\n'); do
> -                       module=lib/modules/$module
> -                       mkdir -p $(dirname $dbg_dir/usr/lib/debug/$module)
> +               find $tmpdir/lib/modules/ -name *.ko -printf '%P\n' | xargs -n1 -P0 -I {} sh -c '

I would go with -P`getconf _NPROCESSORS_ONLN`. There can be thousands
of modules (allmodconfig will make 4500).

> +                       mkdir -p $(dirname '"$dbg_dir"'/usr/lib/debug/lib/modules/$1);'"
>                         # only keep debug symbols in the debug file
> -                       $OBJCOPY --only-keep-debug $tmpdir/$module $dbg_dir/usr/lib/debug/$module
> +                       $OBJCOPY --only-keep-debug $tmpdir/lib/modules/{} \
> +                               $dbg_dir/usr/lib/debug/lib/modules/{};
>                         # strip original module from debug symbols
> -                       $OBJCOPY --strip-debug $tmpdir/$module
> +                       $OBJCOPY --strip-debug $tmpdir/lib/modules/{};
>                         # then add a link to those
> -                       $OBJCOPY --add-gnu-debuglink=$dbg_dir/usr/lib/debug/$module $tmpdir/$module
> -               done
> +                       $OBJCOPY --add-gnu-debuglink=$dbg_dir/usr/lib/debug/lib/modules/{} \
> +                               $tmpdir/lib/modules/{};
> +               " -- {}
>         fi
>  fi
>
> --
> 1.9.1
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] builddeb: parallelize debug module installation

[Michal Marek]

On 2015-04-28 10:57, Riku Voipio wrote:
> On 27 April 2015 at 19:43, Chris J Arges <chris.j.arges@canonical.com> wrote:
>> When building the dbg package, we use a large 'for module in $(find' loop that
>> can be easily parallelized by using 'find | xargs'. This patch modifies this
>> loop to use the later paradigm.
>>
>> In addition, ensure we add '-n1 -P0' to xargs to run as many processes as
>> possible.
>>
>> Signed-off-by: Chris J Arges <chris.j.arges@canonical.com>
>> ---
>>  scripts/package/builddeb | 15 ++++++++-------
>>  1 file changed, 8 insertions(+), 7 deletions(-)
>>
>> diff --git a/scripts/package/builddeb b/scripts/package/builddeb
>> index 88dbf23..538f829 100755
>> --- a/scripts/package/builddeb
>> +++ b/scripts/package/builddeb
>> @@ -152,16 +152,17 @@ if grep -q '^CONFIG_MODULES=y' $KCONFIG_CONFIG ; then
>>                 rmdir "$tmpdir/lib/modules/$version"
>>         fi
>>         if [ -n "$BUILD_DEBUG" ] ; then
>> -               for module in $(find $tmpdir/lib/modules/ -name *.ko -printf '%P\n'); do
>> -                       module=lib/modules/$module
>> -                       mkdir -p $(dirname $dbg_dir/usr/lib/debug/$module)
>> +               find $tmpdir/lib/modules/ -name *.ko -printf '%P\n' | xargs -n1 -P0 -I {} sh -c '
> 
> I would go with -P`getconf _NPROCESSORS_ONLN`. There can be thousands
> of modules (allmodconfig will make 4500).

Yep. I was thinking about retrieving the value of the make -j argument
somehow, but this is not possible and we certainly do not want to
implement the jobserver protocol in shell :-). So using the number of
processors is a sensible choice. What can be done is to detect whether
the -j option is in MAKEFLAGS and only then use multiple instances.

Michal

Re: [PATCH v2] builddeb: parallelize debug module installation

[Chris J Arges]

On Tue, Apr 28, 2015 at 12:05:19PM +0200, Michal Marek wrote:
> On 2015-04-28 10:57, Riku Voipio wrote:
> > On 27 April 2015 at 19:43, Chris J Arges <chris.j.arges@canonical.com> wrote:
> >> When building the dbg package, we use a large 'for module in $(find' loop that
> >> can be easily parallelized by using 'find | xargs'. This patch modifies this
> >> loop to use the later paradigm.
> >>
> >> In addition, ensure we add '-n1 -P0' to xargs to run as many processes as
> >> possible.
> >>
> >> Signed-off-by: Chris J Arges <chris.j.arges@canonical.com>
> >> ---
> >>  scripts/package/builddeb | 15 ++++++++-------
> >>  1 file changed, 8 insertions(+), 7 deletions(-)
> >>
> >> diff --git a/scripts/package/builddeb b/scripts/package/builddeb
> >> index 88dbf23..538f829 100755
> >> --- a/scripts/package/builddeb
> >> +++ b/scripts/package/builddeb
> >> @@ -152,16 +152,17 @@ if grep -q '^CONFIG_MODULES=y' $KCONFIG_CONFIG ; then
> >>                 rmdir "$tmpdir/lib/modules/$version"
> >>         fi
> >>         if [ -n "$BUILD_DEBUG" ] ; then
> >> -               for module in $(find $tmpdir/lib/modules/ -name *.ko -printf '%P\n'); do
> >> -                       module=lib/modules/$module
> >> -                       mkdir -p $(dirname $dbg_dir/usr/lib/debug/$module)
> >> +               find $tmpdir/lib/modules/ -name *.ko -printf '%P\n' | xargs -n1 -P0 -I {} sh -c '
> > 
> > I would go with -P`getconf _NPROCESSORS_ONLN`. There can be thousands
> > of modules (allmodconfig will make 4500).
> 
> Yep. I was thinking about retrieving the value of the make -j argument
> somehow, but this is not possible and we certainly do not want to
> implement the jobserver protocol in shell :-). So using the number of
> processors is a sensible choice. What can be done is to detect whether
> the -j option is in MAKEFLAGS and only then use multiple instances.
> 
> Michal
>

Michal,

The MAKEFLAGS variable will have the jobserver fd's in it, if we are building
in parallel with -j. I could use this to detect if we are building in parallel
and adjust the '-P' flag in xargs accordingly. I'll work on v3 of this patch,
thanks for the reviews!

--chris 

[PATCH v3] builddeb: parallelize debug module installation

[Chris J Arges]

When building the dbg package, we use a large 'for module in $(find' loop that
can be easily parallelized by using 'find | xargs'. This patch modifies this
loop to use the later paradigm.

In addition, check if the user has requested a parallel build with make. If so,
add the appropriate flags to xargs to set MAXPROCS (-P) equal to the number of
processing units on the system.

Signed-off-by: Chris J Arges <chris.j.arges@canonical.com>
---
 scripts/package/builddeb | 19 ++++++++++++-------
 1 file changed, 12 insertions(+), 7 deletions(-)

diff --git a/scripts/package/builddeb b/scripts/package/builddeb
index 88dbf23..5849f0c 100755
--- a/scripts/package/builddeb
+++ b/scripts/package/builddeb
@@ -152,16 +152,21 @@ if grep -q '^CONFIG_MODULES=y' $KCONFIG_CONFIG ; then
 		rmdir "$tmpdir/lib/modules/$version"
 	fi
 	if [ -n "$BUILD_DEBUG" ] ; then
-		for module in $(find $tmpdir/lib/modules/ -name *.ko -printf '%P\n'); do
-			module=lib/modules/$module
-			mkdir -p $(dirname $dbg_dir/usr/lib/debug/$module)
+		# If we've invoked make with -j, then parallelize; otherwise
+		# just use a single process.
+		procs=1
+		test "${MAKEFLAGS#*-j}" != "${MAKEFLAGS}" && procs=`getconf _NPROCESSORS_ONLN`
+		find $tmpdir/lib/modules/ -name *.ko -printf '%P\n' | xargs -n1 -P${procs} -I {} sh -c '
+			mkdir -p $(dirname '"$dbg_dir"'/usr/lib/debug/lib/modules/$1);'"
 			# only keep debug symbols in the debug file
-			$OBJCOPY --only-keep-debug $tmpdir/$module $dbg_dir/usr/lib/debug/$module
+			$OBJCOPY --only-keep-debug $tmpdir/lib/modules/{} \
+				$dbg_dir/usr/lib/debug/lib/modules/{};
 			# strip original module from debug symbols
-			$OBJCOPY --strip-debug $tmpdir/$module
+			$OBJCOPY --strip-debug $tmpdir/lib/modules/{};
 			# then add a link to those
-			$OBJCOPY --add-gnu-debuglink=$dbg_dir/usr/lib/debug/$module $tmpdir/$module
-		done
+			$OBJCOPY --add-gnu-debuglink=$dbg_dir/usr/lib/debug/lib/modules/{} \
+				$tmpdir/lib/modules/{};
+		" -- {}
 	fi
 fi
 
-- 
1.9.1