[PATCH v3 0/1] kbuild: deb-pkg: Build parallely with current dpkg-buildpackage

[PATCH v3 0/1] kbuild: deb-pkg: Build parallely with current dpkg-buildpackage

[Bastian Germann]

Make use of DEB_BUILD_OPTIONS' parallel option because that is the only
parallel build option that is set by default in current dpkg-buildpackage
versions.

v2:
 * Clarify that this is for current dpkg-buildpackage versions
 * Evaluate DEB_BUILD_OPTIONS in debian/rules.
v3:
 * Use the Policy snippet which sets MAKEFLAGS and uses patsubst over subst
 * Mention the dpkg commit that changed the -j behaviour.

Bastian Germann (1):
  kbuild: deb-pkg: Build parallely with current dpkg-buildpackage

 scripts/package/mkdebian | 5 +++++
 1 file changed, 5 insertions(+)

-- 
2.39.2

[PATCH v3 1/1] kbuild: deb-pkg: Build parallely with current dpkg-buildpackage

[Bastian Germann]

With dpkg-buildpackage v1.21.10 or later, the only way to build the
deb-pkg generated package parallely is adding -j<N> to the MAKEFLAGS
environment variable or using the --jobs-force option, see dpkg commit
1d0ea9b2ba3f ("dpkg-buildpackage: Change -j, --jobs semantics to
non-force mode"). The package ignores the usual parallel build option
that is described in Debian Policy.

Derive make's -j parameter from the DEB_BUILD_OPTIONS environment variable
that ends up being set by dpkg-buildpackage -j<N>. The snippet is copied
from Debian Policy.

Link: https://www.debian.org/doc/debian-policy/ch-source.html#debian-rules-and-deb-build-options
Signed-off-by: Bastian Germann <bage@linutronix.de>
---
 scripts/package/mkdebian | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/scripts/package/mkdebian b/scripts/package/mkdebian
index f74380036bb5..ed5c2b65798b 100755
--- a/scripts/package/mkdebian
+++ b/scripts/package/mkdebian
@@ -238,6 +238,11 @@ fi
 cat <<EOF > debian/rules
 #!$(command -v $MAKE) -f
 
+ifneq (,\$(filter parallel=%,\$(DEB_BUILD_OPTIONS)))
+	NUMJOBS = \$(patsubst parallel=%,%,\$(filter parallel=%,\$(DEB_BUILD_OPTIONS)))
+	MAKEFLAGS += -j\$(NUMJOBS)
+endif
+
 srctree ?= .
 
 build-indep:
-- 
2.39.2

Re: [PATCH v3 1/1] kbuild: deb-pkg: Build parallely with current dpkg-buildpackage

[Masahiro Yamada]

On Thu, Mar 16, 2023 at 2:35 AM Bastian Germann <bage@linutronix.de> wrote:
>
> With dpkg-buildpackage v1.21.10 or later, the only way to build the
> deb-pkg generated package parallely is adding -j<N> to the MAKEFLAGS
> environment variable or using the --jobs-force option, see dpkg commit
> 1d0ea9b2ba3f ("dpkg-buildpackage: Change -j, --jobs semantics to
> non-force mode"). The package ignores the usual parallel build option
> that is described in Debian Policy.
>
> Derive make's -j parameter from the DEB_BUILD_OPTIONS environment variable
> that ends up being set by dpkg-buildpackage -j<N>. The snippet is copied
> from Debian Policy.
>
> Link: https://www.debian.org/doc/debian-policy/ch-source.html#debian-rules-and-deb-build-options
> Signed-off-by: Bastian Germann <bage@linutronix.de>
> ---


I was about to pick this up, then I stopped it.



If I apply this patch,

"make -j2 bindeb-pkg" shows a new warning.

make[3]: warning: -j16 forced in makefile: resetting jobserver mode.




Please note `nproc` returns 16 on my machine.


"make bindeb-pkg" also runs in 16 threads because
$(DEB_BUILD_OPTIONS) always contains "parallel=16"
even if -j<N> option is given at all.

In other words, this patch will take away user's freedom
to choose the number of threads.









>  scripts/package/mkdebian | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/scripts/package/mkdebian b/scripts/package/mkdebian
> index f74380036bb5..ed5c2b65798b 100755
> --- a/scripts/package/mkdebian
> +++ b/scripts/package/mkdebian
> @@ -238,6 +238,11 @@ fi
>  cat <<EOF > debian/rules
>  #!$(command -v $MAKE) -f
>
> +ifneq (,\$(filter parallel=%,\$(DEB_BUILD_OPTIONS)))
> +       NUMJOBS = \$(patsubst parallel=%,%,\$(filter parallel=%,\$(DEB_BUILD_OPTIONS)))
> +       MAKEFLAGS += -j\$(NUMJOBS)
> +endif
> +
>  srctree ?= .
>
>  build-indep:
> --
> 2.39.2
>


-- 
Best Regards
Masahiro Yamada

Re: [PATCH v3 1/1] kbuild: deb-pkg: Build parallely with current dpkg-buildpackage

[Bastian Germann]

Am 21.03.23 um 11:18 schrieb Masahiro Yamada:
> "make bindeb-pkg" also runs in 16 threads because
> $(DEB_BUILD_OPTIONS) always contains "parallel=16"
> even if -j<N> option is given at all.
> 
> In other words, this patch will take away user's freedom
> to choose the number of threads.

Okay. To solve this, I would forward make's -j option to the dpkg-buildpackage calls,
which is now called with the defaults.

I think you should see the warning with older dpkg-buildpackage versions without the
patch applied as well because their default is dpkg-buildpackage -j`nproc` and is in
effect currently.

Re: [PATCH v3 1/1] kbuild: deb-pkg: Build parallely with current dpkg-buildpackage

[Masahiro Yamada]

On Tue, Mar 21, 2023 at 7:28 PM Bastian Germann <bage@linutronix.de> wrote:
>
> Am 21.03.23 um 11:18 schrieb Masahiro Yamada:
> > "make bindeb-pkg" also runs in 16 threads because
> > $(DEB_BUILD_OPTIONS) always contains "parallel=16"
> > even if -j<N> option is given at all.
> >
> > In other words, this patch will take away user's freedom
> > to choose the number of threads.
>
> Okay. To solve this, I would forward make's -j option to the dpkg-buildpackage calls,
> which is now called with the defaults.

Yes. Agreed.



> I think you should see the warning with older dpkg-buildpackage versions without the
> patch applied as well because their default is dpkg-buildpackage -j`nproc` and is in
> effect currently.



-- 
Best Regards
Masahiro Yamada