Re: [OE-core] [RFC PATCH 10/14] package_ipk/deb/rpm: Drop recursive do_build task dependencies

From: "Khem Raj" <raj.khem@gmail.com>
To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Cc: Richard Purdie <richard.purdie@linuxfoundation.org>,
	 "openembedded-core@lists.openembedded.org"
	<openembedded-core@lists.openembedded.org>
Subject: Re: [OE-core] [RFC PATCH 10/14] package_ipk/deb/rpm: Drop recursive do_build task dependencies
Date: Thu, 23 Sep 2021 21:50:37 -0700	[thread overview]
Message-ID: <CAMKF1sqeyw09suOzauHvqhMfieBedsKa-JH5Qe5H+M8R0cCbgg@mail.gmail.com> (raw)
In-Reply-To: <c6c0603017b848fea5650c2a2273e66b@axis.com>

On Thu, Sep 23, 2021 at 2:42 PM Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:
>
> > -----Original Message-----
> > From: openembedded-core@lists.openembedded.org <openembedded-
> > core@lists.openembedded.org> On Behalf Of Richard Purdie
> > Sent: den 20 september 2021 14:46
> > To: openembedded-core@lists.openembedded.org
> > Subject: [OE-core] [RFC PATCH 10/14] package_ipk/deb/rpm: Drop recursive
> > do_build task dependencies
> >
> > This is a controversial change which removes the recursive dependencies
> > from the do_build target of packaging tasks of recipes.
> >
> > Currently this means when you "bitbake <image>" or "bitbake <recipe>",
> > the packaging tasks run for all packaging backends enabled for all recipes
> > in the dependency chain. The same therefore then applies to images.
> >
> > We don't actually need that, it is a convinience thing. Removing it
> > massively simplifies the task graph and causes much fewer tasks to execute
> > in many common scenarios. It also means less sstate is fetched for
> > example when building an image.
> >
> > This means when building a recipe all package formats would be built
> > but when building an image, only the format used by the image would be
> > generated. This should be an improvement in most cases but some CI systems
> > may need to be explict about what they're building.
> >
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > ---
> >  meta/classes/package_deb.bbclass | 5 +----
> >  meta/classes/package_ipk.bbclass | 4 +---
> >  meta/classes/package_rpm.bbclass | 4 +---
> >  3 files changed, 3 insertions(+), 10 deletions(-)
> >
> > diff --git a/meta/classes/package_deb.bbclass
> > b/meta/classes/package_deb.bbclass
> > index eca43e17876..77caaa55c8f 100644
> > --- a/meta/classes/package_deb.bbclass
> > +++ b/meta/classes/package_deb.bbclass
> > @@ -315,10 +315,7 @@ python do_package_write_deb () {
> >  do_package_write_deb[dirs] = "${PKGWRITEDIRDEB}"
> >  do_package_write_deb[cleandirs] = "${PKGWRITEDIRDEB}"
> >  do_package_write_deb[depends] +=
> > "${@oe.utils.build_depends_string(d.getVar('PACKAGE_WRITE_DEPS'),
> > 'do_populate_sysroot')}"
> > -addtask package_write_deb after do_packagedata do_package
> > -
> > +addtask package_write_deb after do_packagedata do_package before do_build
> >
> >  PACKAGEINDEXDEPS += "dpkg-native:do_populate_sysroot"
> >  PACKAGEINDEXDEPS += "apt-native:do_populate_sysroot"
> > -
> > -do_build[recrdeptask] += "do_package_write_deb"
> > diff --git a/meta/classes/package_ipk.bbclass
> > b/meta/classes/package_ipk.bbclass
> > index c3b53854e8b..998e18ba6c1 100644
> > --- a/meta/classes/package_ipk.bbclass
> > +++ b/meta/classes/package_ipk.bbclass
> > @@ -274,9 +274,7 @@ python do_package_write_ipk () {
> >  do_package_write_ipk[dirs] = "${PKGWRITEDIRIPK}"
> >  do_package_write_ipk[cleandirs] = "${PKGWRITEDIRIPK}"
> >  do_package_write_ipk[depends] +=
> > "${@oe.utils.build_depends_string(d.getVar('PACKAGE_WRITE_DEPS'),
> > 'do_populate_sysroot')}"
> > -addtask package_write_ipk after do_packagedata do_package
> > +addtask package_write_ipk after do_packagedata do_package before do_build
> >
> >  PACKAGEINDEXDEPS += "opkg-utils-native:do_populate_sysroot"
> >  PACKAGEINDEXDEPS += "opkg-native:do_populate_sysroot"
> > -
> > -do_build[recrdeptask] += "do_package_write_ipk"
> > diff --git a/meta/classes/package_rpm.bbclass
> > b/meta/classes/package_rpm.bbclass
> > index 88d861c0e75..60f8299d965 100644
> > --- a/meta/classes/package_rpm.bbclass
> > +++ b/meta/classes/package_rpm.bbclass
> > @@ -748,9 +748,7 @@ python do_package_write_rpm () {
> >  do_package_write_rpm[dirs] = "${PKGWRITEDIRRPM}"
> >  do_package_write_rpm[cleandirs] = "${PKGWRITEDIRRPM}"
> >  do_package_write_rpm[depends] +=
> > "${@oe.utils.build_depends_string(d.getVar('PACKAGE_WRITE_DEPS'),
> > 'do_populate_sysroot')}"
> > -addtask package_write_rpm after do_packagedata do_package
> > +addtask package_write_rpm after do_packagedata do_package before do_build
> >
> >  PACKAGEINDEXDEPS += "rpm-native:do_populate_sysroot"
> >  PACKAGEINDEXDEPS += "createrepo-c-native:do_populate_sysroot"
> > -
> > -do_build[recrdeptask] += "do_package_write_rpm"
> > --
> > 2.32.0
>
> I just wanted to give some positive feedback now that this has been
> integrated. This patch alone reduced the number of edges in the task
> graph by 50% for one of our typical products (from 127721 to 64785).
> That is pretty impressive!
>
> And to add to this, the later patch that removes the use of the meta
> class reduced the number of nodes in the task graph by 10% (from 17122
> to 15313). Also pretty impressive for such a small change.
>

this is quite encouraging, dont go by number of lines of change :)
btw. did you also try with other change which reduces native dependencies,

> //Peter
>
> 
>