[PATCH 0/1] kbuild: deb-pkg: Allow parallel build

[PATCH 0/1] kbuild: deb-pkg: Allow parallel build

[Bastian Germann]

Make use of DEB_BUILD_OPTIONS' parallel option.

Bastian Germann (1):
  kbuild: deb-pkg: Allow parallel build

 scripts/package/deb-build-option | 6 ++++++
 1 file changed, 6 insertions(+)

-- 
2.39.2

[PATCH 1/1] kbuild: deb-pkg: Allow parallel build

[Bastian Germann]

Currently, the only way to build the deb-pkg generated package parallely
is adding -jN to the MAKEFLAGS environment variable. The package ignores
the usual parallel build option that is described in Debian Policy §4.9.1.

Derive make's -j parameter from the DEB_BUILD_OPTIONS environment variable
that ends up being set by Debian's build tools.

Link: https://www.debian.org/doc/debian-policy/ch-source.html
Signed-off-by: Bastian Germann <bage@linutronix.de>
---
 scripts/package/deb-build-option | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/scripts/package/deb-build-option b/scripts/package/deb-build-option
index b079b0d121d4..dd170e2b3018 100755
--- a/scripts/package/deb-build-option
+++ b/scripts/package/deb-build-option
@@ -7,6 +7,12 @@ if [ -z "${CROSS_COMPILE}${cross_compiling}" -a "${DEB_HOST_ARCH}" != "${DEB_BUI
 	echo CROSS_COMPILE=${DEB_HOST_GNU_TYPE}-
 fi
 
+for build_opt in $DEB_BUILD_OPTIONS; do
+	if [ "${build_opt#parallel=}" != "$build_opt" ]; then
+		echo -j${build_opt#parallel=}
+	fi
+done
+
 version=$(dpkg-parsechangelog -S Version)
 version_upstream="${version%-*}"
 debian_revision="${version#${version_upstream}}"
-- 
2.39.2

Re: [PATCH 1/1] kbuild: deb-pkg: Allow parallel build

[Masahiro Yamada]

On Tue, Mar 14, 2023 at 2:10 AM Bastian Germann <bage@linutronix.de> wrote:
>
> Currently, the only way to build the deb-pkg generated package parallely
> is adding -jN to the MAKEFLAGS environment variable. The package ignores
> the usual parallel build option that is described in Debian Policy §4.9.1.



"dpkg-buildpackage -b -j16" worked for me.







> Derive make's -j parameter from the DEB_BUILD_OPTIONS environment variable
> that ends up being set by Debian's build tools.
>
> Link: https://www.debian.org/doc/debian-policy/ch-source.html
> Signed-off-by: Bastian Germann <bage@linutronix.de>
> ---
>  scripts/package/deb-build-option | 6 ++++++
>  1 file changed, 6 insertions(+)
>
> diff --git a/scripts/package/deb-build-option b/scripts/package/deb-build-option
> index b079b0d121d4..dd170e2b3018 100755
> --- a/scripts/package/deb-build-option
> +++ b/scripts/package/deb-build-option
> @@ -7,6 +7,12 @@ if [ -z "${CROSS_COMPILE}${cross_compiling}" -a "${DEB_HOST_ARCH}" != "${DEB_BUI
>         echo CROSS_COMPILE=${DEB_HOST_GNU_TYPE}-
>  fi
>
> +for build_opt in $DEB_BUILD_OPTIONS; do
> +       if [ "${build_opt#parallel=}" != "$build_opt" ]; then
> +               echo -j${build_opt#parallel=}
> +       fi
> +done
> +
>  version=$(dpkg-parsechangelog -S Version)
>  version_upstream="${version%-*}"
>  debian_revision="${version#${version_upstream}}"
> --
> 2.39.2
>


-- 
Best Regards
Masahiro Yamada

Re: [PATCH 1/1] kbuild: deb-pkg: Allow parallel build

[Bastian Germann]

Am 13.03.23 um 18:59 schrieb Masahiro Yamada:
> On Tue, Mar 14, 2023 at 2:10 AM Bastian Germann <bage@linutronix.de> wrote:
>>
>> Currently, the only way to build the deb-pkg generated package parallely
>> is adding -jN to the MAKEFLAGS environment variable. The package ignores
>> the usual parallel build option that is described in Debian Policy §4.9.1.
> 
> 
> 
> "dpkg-buildpackage -b -j16" worked for me.

This ends up in DEB_BUILD_OPTIONS=parallel=16 being set and the call:
/usr/bin/make -f ./Makefile ARCH=x86 	KERNELRELEASE=6.3.0-rc2 KBUILD_BUILD_VERSION=1 	olddefconfig all

So it is not used to run the actual build, just the top level `make -f debian/rules` invocation.
You can set --jobs-force=16, which ends up in MAKEFLAGS but the point of the patch is that one can use
the usual way of parallelizing.

Side note: Without the patch, the build is run with -j1 regardless of being called from the Linux Makefile
or dpkg-buildpackage.

Re: [PATCH 1/1] kbuild: deb-pkg: Allow parallel build

[Masahiro Yamada]

On Tue, Mar 14, 2023 at 3:56 AM Bastian Germann <bage@linutronix.de> wrote:
>
> Am 13.03.23 um 18:59 schrieb Masahiro Yamada:
> > On Tue, Mar 14, 2023 at 2:10 AM Bastian Germann <bage@linutronix.de> wrote:
> >>
> >> Currently, the only way to build the deb-pkg generated package parallely
> >> is adding -jN to the MAKEFLAGS environment variable. The package ignores
> >> the usual parallel build option that is described in Debian Policy §4.9.1.
> >
> >
> >
> > "dpkg-buildpackage -b -j16" worked for me.
>
> This ends up in DEB_BUILD_OPTIONS=parallel=16 being set and the call:
> /usr/bin/make -f ./Makefile ARCH=x86    KERNELRELEASE=6.3.0-rc2 KBUILD_BUILD_VERSION=1  olddefconfig all


"dpkb-buildpackage -j<N>" sets not only DEB_BUILD_OPTIONS
but also MAKEFLAGS.


This is clearly explained in "man dpkb-buildpackage".


 -j, --jobs[=jobs|auto]
   ...
  Will add itself to the MAKEFLAGS environment variable, which should
  cause all subsequent make invocations to inherit the option, thus
  forcing the parallel setting on the packaging ...




Your statement sounds like

 'MAKEFLAGS=-j<N> dpkg-buildpackage'

is the only way to build packages in parallel.

Apparently, dpkg-buildpackage provides a much shorter way
and invokes internal Make in parallel.




>
> So it is not used to run the actual build, just the top level `make -f debian/rules` invocation.
> You can set --jobs-force=16, which ends up in MAKEFLAGS but the point of the patch is that one can use
> the usual way of parallelizing.


What is the "usual" way in this context?


Do you mean
'DEB_BUILD_OPTIONS=parallel=16 dpkg-buildpackage -b'
is the usual way for parallel building?

If so, I agree. This patch caters to this case.


But, I think your code should go to debian/rules
instead of scripts/package/deb-build-option.
Kbuild's "make clean" works much faster with the parallel option.


Also, the commit description should explain the
benefit of this patch.


>
> Side note: Without the patch, the build is run with -j1 regardless of being called from the Linux Makefile
> or dpkg-buildpackage.


Try:

 $ make deb-pkg -j16

and

 $ dpkg-buildpackage -b -j16


Both run the package building in parallel.

You see a big difference in build speed w/wo the -j option.
(and the build logs are interleaved, since they
are emitted by multiple threads)





--
Best Regards
Masahiro Yamada

Re: [PATCH 1/1] kbuild: deb-pkg: Allow parallel build

[Bastian Germann]

Am 15.03.23 um 15:18 schrieb Masahiro Yamada:
> "dpkb-buildpackage -j<N>" sets not only DEB_BUILD_OPTIONS
> but also MAKEFLAGS.
> 
> 
> This is clearly explained in "man dpkb-buildpackage".
> 
> 
>   -j, --jobs[=jobs|auto]
>     ...
>    Will add itself to the MAKEFLAGS environment variable, which should
>    cause all subsequent make invocations to inherit the option, thus
>    forcing the parallel setting on the packaging ...

I see. The testing/unstable dpkg-buildpackage version changed the -j behaviour.
-j only sets DEB_BUILD_OPTIONS=parallel= now.

The man page now says:

        --jobs-force[=jobs|auto]
        This option (since dpkg 1.21.10) is equivalent to the --jobs option except that
        it will enable forced parallel mode, by adding the make -j option with the computed
        number of parallel jobs to the MAKEFLAGS environment variable.

> Your statement sounds like
> 
>   'MAKEFLAGS=-j<N> dpkg-buildpackage'
> 
> is the only way to build packages in parallel.

It is in v1.21.10 or later (or using --jobs-force which does the same thing).

> Apparently, dpkg-buildpackage provides a much shorter way
> and invokes internal Make in parallel.
> 
> 
> 
> 
>>
>> So it is not used to run the actual build, just the top level `make -f debian/rules` invocation.
>> You can set --jobs-force=16, which ends up in MAKEFLAGS but the point of the patch is that one can use
>> the usual way of parallelizing.
> 
> 
> What is the "usual" way in this context?

The usual way is dpkg-buildpackage -jN but that does not work with later versions.

> Do you mean
> 'DEB_BUILD_OPTIONS=parallel=16 dpkg-buildpackage -b'
> is the usual way for parallel building?
> 
> If so, I agree. This patch caters to this case.
> 
> 
> But, I think your code should go to debian/rules
> instead of scripts/package/deb-build-option.
> Kbuild's "make clean" works much faster with the parallel option.
> 
> 
> Also, the commit description should explain the
> benefit of this patch.

I will clarify the dpkg-buildpackage behaviour change and will move the stuff to debian/rules
in v2.

Re: [PATCH 1/1] kbuild: deb-pkg: Allow parallel build

[Sedat Dilek]

On Wed, Mar 15, 2023 at 3:24 PM Masahiro Yamada <masahiroy@kernel.org> wrote:
>
> On Tue, Mar 14, 2023 at 3:56 AM Bastian Germann <bage@linutronix.de> wrote:
> >
> > Am 13.03.23 um 18:59 schrieb Masahiro Yamada:
> > > On Tue, Mar 14, 2023 at 2:10 AM Bastian Germann <bage@linutronix.de> wrote:
> > >>
> > >> Currently, the only way to build the deb-pkg generated package parallely
> > >> is adding -jN to the MAKEFLAGS environment variable. The package ignores
> > >> the usual parallel build option that is described in Debian Policy §4.9.1.
> > >
> > >
> > >
> > > "dpkg-buildpackage -b -j16" worked for me.
> >
> > This ends up in DEB_BUILD_OPTIONS=parallel=16 being set and the call:
> > /usr/bin/make -f ./Makefile ARCH=x86    KERNELRELEASE=6.3.0-rc2 KBUILD_BUILD_VERSION=1  olddefconfig all
>
>
> "dpkb-buildpackage -j<N>" sets not only DEB_BUILD_OPTIONS
> but also MAKEFLAGS.
>
>
> This is clearly explained in "man dpkb-buildpackage".
>
>
>  -j, --jobs[=jobs|auto]
>    ...
>   Will add itself to the MAKEFLAGS environment variable, which should
>   cause all subsequent make invocations to inherit the option, thus
>   forcing the parallel setting on the packaging ...
>
>
>
>
> Your statement sounds like
>
>  'MAKEFLAGS=-j<N> dpkg-buildpackage'
>
> is the only way to build packages in parallel.
>
> Apparently, dpkg-buildpackage provides a much shorter way
> and invokes internal Make in parallel.
>
>
>
>
> >
> > So it is not used to run the actual build, just the top level `make -f debian/rules` invocation.
> > You can set --jobs-force=16, which ends up in MAKEFLAGS but the point of the patch is that one can use
> > the usual way of parallelizing.
>
>
> What is the "usual" way in this context?
>
>
> Do you mean
> 'DEB_BUILD_OPTIONS=parallel=16 dpkg-buildpackage -b'
> is the usual way for parallel building?
>
> If so, I agree. This patch caters to this case.
>
>
> But, I think your code should go to debian/rules
> instead of scripts/package/deb-build-option.
> Kbuild's "make clean" works much faster with the parallel option.
>
>
> Also, the commit description should explain the
> benefit of this patch.
>
>
> >
> > Side note: Without the patch, the build is run with -j1 regardless of being called from the Linux Makefile
> > or dpkg-buildpackage.
>
>
> Try:
>
>  $ make deb-pkg -j16
>
> and
>
>  $ dpkg-buildpackage -b -j16
>
>
> Both run the package building in parallel.
>
> You see a big difference in build speed w/wo the -j option.
> (and the build logs are interleaved, since they
> are emitted by multiple threads)
>
>

I have no strong opinion in having a DEB_BUILD_OPTIONS...

As you stated Masahiro I have seen it mostly in debian/rules.

See the example from [1]:

ifneq (,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
    NUMJOBS = $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
    MAKEFLAGS += -j$(NUMJOBS)
endif

Normally, I use in my kernel build-script:

$ make deb-pkg -j${NUMJOBS}

My €0,02.

-Sedat-


[1] https://www.debian.org/doc/debian-policy/ch-source.html#s-debianrules-options

Re: [PATCH 1/1] kbuild: deb-pkg: Allow parallel build

[Masahiro Yamada]

On Thu, Mar 16, 2023 at 12:51 AM Bastian Germann <bage@linutronix.de> wrote:
>
> Am 15.03.23 um 15:18 schrieb Masahiro Yamada:
> > "dpkb-buildpackage -j<N>" sets not only DEB_BUILD_OPTIONS
> > but also MAKEFLAGS.
> >
> >
> > This is clearly explained in "man dpkb-buildpackage".
> >
> >
> >   -j, --jobs[=jobs|auto]
> >     ...
> >    Will add itself to the MAKEFLAGS environment variable, which should
> >    cause all subsequent make invocations to inherit the option, thus
> >    forcing the parallel setting on the packaging ...
>
> I see. The testing/unstable dpkg-buildpackage version changed the -j behaviour.
> -j only sets DEB_BUILD_OPTIONS=parallel= now.
>
> The man page now says:
>
>         --jobs-force[=jobs|auto]
>         This option (since dpkg 1.21.10) is equivalent to the --jobs option except that
>         it will enable forced parallel mode, by adding the make -j option with the computed
>         number of parallel jobs to the MAKEFLAGS environment variable.
>
> > Your statement sounds like
> >
> >   'MAKEFLAGS=-j<N> dpkg-buildpackage'
> >
> > is the only way to build packages in parallel.
>
> It is in v1.21.10 or later (or using --jobs-force which does the same thing).
>
> > Apparently, dpkg-buildpackage provides a much shorter way
> > and invokes internal Make in parallel.
> >
> >
> >
> >
> >>
> >> So it is not used to run the actual build, just the top level `make -f debian/rules` invocation.
> >> You can set --jobs-force=16, which ends up in MAKEFLAGS but the point of the patch is that one can use
> >> the usual way of parallelizing.
> >
> >
> > What is the "usual" way in this context?
>
> The usual way is dpkg-buildpackage -jN but that does not work with later versions.
>
> > Do you mean
> > 'DEB_BUILD_OPTIONS=parallel=16 dpkg-buildpackage -b'
> > is the usual way for parallel building?
> >
> > If so, I agree. This patch caters to this case.
> >
> >
> > But, I think your code should go to debian/rules
> > instead of scripts/package/deb-build-option.
> > Kbuild's "make clean" works much faster with the parallel option.
> >
> >
> > Also, the commit description should explain the
> > benefit of this patch.
>
> I will clarify the dpkg-buildpackage behaviour change and will move the stuff to debian/rules
> in v2.


Please do so.


Now I understand the point of this patch.
Indeed, my dpkg-buildpackage is old.
(v1.21.1 on Ubuntu 22.04)

Thanks.





--
Best Regards
Masahiro Yamada

Re: [PATCH 1/1] kbuild: deb-pkg: Allow parallel build

[Masahiro Yamada]

On Thu, Mar 16, 2023 at 12:57 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>
> On Wed, Mar 15, 2023 at 3:24 PM Masahiro Yamada <masahiroy@kernel.org> wrote:
> >
> > On Tue, Mar 14, 2023 at 3:56 AM Bastian Germann <bage@linutronix.de> wrote:
> > >
> > > Am 13.03.23 um 18:59 schrieb Masahiro Yamada:
> > > > On Tue, Mar 14, 2023 at 2:10 AM Bastian Germann <bage@linutronix.de> wrote:
> > > >>
> > > >> Currently, the only way to build the deb-pkg generated package parallely
> > > >> is adding -jN to the MAKEFLAGS environment variable. The package ignores
> > > >> the usual parallel build option that is described in Debian Policy §4.9.1.
> > > >
> > > >
> > > >
> > > > "dpkg-buildpackage -b -j16" worked for me.
> > >
> > > This ends up in DEB_BUILD_OPTIONS=parallel=16 being set and the call:
> > > /usr/bin/make -f ./Makefile ARCH=x86    KERNELRELEASE=6.3.0-rc2 KBUILD_BUILD_VERSION=1  olddefconfig all
> >
> >
> > "dpkb-buildpackage -j<N>" sets not only DEB_BUILD_OPTIONS
> > but also MAKEFLAGS.
> >
> >
> > This is clearly explained in "man dpkb-buildpackage".
> >
> >
> >  -j, --jobs[=jobs|auto]
> >    ...
> >   Will add itself to the MAKEFLAGS environment variable, which should
> >   cause all subsequent make invocations to inherit the option, thus
> >   forcing the parallel setting on the packaging ...
> >
> >
> >
> >
> > Your statement sounds like
> >
> >  'MAKEFLAGS=-j<N> dpkg-buildpackage'
> >
> > is the only way to build packages in parallel.
> >
> > Apparently, dpkg-buildpackage provides a much shorter way
> > and invokes internal Make in parallel.
> >
> >
> >
> >
> > >
> > > So it is not used to run the actual build, just the top level `make -f debian/rules` invocation.
> > > You can set --jobs-force=16, which ends up in MAKEFLAGS but the point of the patch is that one can use
> > > the usual way of parallelizing.
> >
> >
> > What is the "usual" way in this context?
> >
> >
> > Do you mean
> > 'DEB_BUILD_OPTIONS=parallel=16 dpkg-buildpackage -b'
> > is the usual way for parallel building?
> >
> > If so, I agree. This patch caters to this case.
> >
> >
> > But, I think your code should go to debian/rules
> > instead of scripts/package/deb-build-option.
> > Kbuild's "make clean" works much faster with the parallel option.
> >
> >
> > Also, the commit description should explain the
> > benefit of this patch.
> >
> >
> > >
> > > Side note: Without the patch, the build is run with -j1 regardless of being called from the Linux Makefile
> > > or dpkg-buildpackage.
> >
> >
> > Try:
> >
> >  $ make deb-pkg -j16
> >
> > and
> >
> >  $ dpkg-buildpackage -b -j16
> >
> >
> > Both run the package building in parallel.
> >
> > You see a big difference in build speed w/wo the -j option.
> > (and the build logs are interleaved, since they
> > are emitted by multiple threads)
> >
> >
>
> I have no strong opinion in having a DEB_BUILD_OPTIONS...
>
> As you stated Masahiro I have seen it mostly in debian/rules.
>
> See the example from [1]:
>
> ifneq (,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
>     NUMJOBS = $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
>     MAKEFLAGS += -j$(NUMJOBS)
> endif
>

Yup, this code looks good to me.




-- 
Best Regards
Masahiro Yamada