is there an easy to prevent *creation* of some recipe's packages?

is there an easy to prevent *creation* of some recipe's packages?

[Robert P. J. Day]

  pretty sure i know the answer to this one, but was asked the other
day and wanted to make sure. in order to speed up the normal OE build,
someone suggested bypassing the creation of packages that weren't
going to be relevant, such as -dev, -doc and so on -- the idea was
that that could make a significant difference in build time.

  i pointed out that, based on default package definition as found in
bitbake.conf:

  PACKAGE_BEFORE_PN ?= ""
  PACKAGES = "${PN}-src ${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc
  ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
  PACKAGES_DYNAMIC = "^${PN}-locale-.*"
  FILES = ""

  FILES_${PN} = "${bindir}/* ${sbindir}/* ${libexecdir}/*   ${libdir}/lib*${SOLIBS} \
            ${sysconfdir} ${sharedstatedir} ${localstatedir} \
     ... snip ...

for every recipe you build, those *are* the packages that will be
built and populated, short of overriding those values. i suggested
that, in normal behaviour, if you check the generated WORKDIR for any
recipe, you would first find the full package breakdown under
packages-split/, where individual package contents can be examined,
and you could further check out the rpms under deploy-rpms/, which
would be built regardless of whether you chose to install them or not.
(that's correct, yes?)

  i finally suggested that, sure, if you do some ugly hacking, i
guess you could save time by not even creating some of those packages,
but i speculated that that wouldn't be a significant saving and that
it wasn't worth trying to cut corners like that and you were more
likely to break things than save any meaningful build time.

  thoughts?

rday

Re: [OE-core] is there an easy to prevent *creation* of some recipe's packages?

[Konrad Weihmann]

On 14.05.21 13:16, Robert P. J. Day wrote:
> 
>    pretty sure i know the answer to this one, but was asked the other
> day and wanted to make sure. in order to speed up the normal OE build,
> someone suggested bypassing the creation of packages that weren't
> going to be relevant, such as -dev, -doc and so on -- the idea was
> that that could make a significant difference in build time.

I pretty much doubt it will speed up anything.
If for instance the -dev package will be removed from PACKAGES it is 
very likely that the files of that package will end up in the base 
package ${PN}, which results in larger target packages that need to be 
installed (and potentially further RDEPENDS)

Furthermore this will break DEPENDS for recipes that do rely on the -dev 
package - so chances are high that this will break (or magically change) 
something further down the line.

One thing that you can do to speed up the build process in terms of 
packaging time is to delete any **really** not needed files in a 
do_install_append, if no proper configuration option (cmake flag, make 
flag, PACKAGECONFIG) does exist which prevents the creation of these 
files in the first place.

I would recommend not to mess with the packaging defaults for reasons 
such as speeding things up - as this is IMHO addressing the issue at the 
wrong end - just make sure the files in <workdir>/image won't get created

> 
>    i pointed out that, based on default package definition as found in
> bitbake.conf:
> 
>    PACKAGE_BEFORE_PN ?= ""
>    PACKAGES = "${PN}-src ${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc
>    ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
>    PACKAGES_DYNAMIC = "^${PN}-locale-.*"
>    FILES = ""
> 
>    FILES_${PN} = "${bindir}/* ${sbindir}/* ${libexecdir}/*   ${libdir}/lib*${SOLIBS} \
>              ${sysconfdir} ${sharedstatedir} ${localstatedir} \
>       ... snip ...
> 
> for every recipe you build, those *are* the packages that will be
> built and populated, short of overriding those values. i suggested
> that, in normal behaviour, if you check the generated WORKDIR for any
> recipe, you would first find the full package breakdown under
> packages-split/, where individual package contents can be examined,
> and you could further check out the rpms under deploy-rpms/, which
> would be built regardless of whether you chose to install them or not.
> (that's correct, yes?)
> 
>    i finally suggested that, sure, if you do some ugly hacking, i
> guess you could save time by not even creating some of those packages,
> but i speculated that that wouldn't be a significant saving and that
> it wasn't worth trying to cut corners like that and you were more
> likely to break things than save any meaningful build time.
> 
>    thoughts?
> 
> rday
> 
> 
> 
> 
> 

Re: [OE-core] is there an easy to prevent *creation* of some recipe's packages?

[Robert P. J. Day]

On Fri, 14 May 2021, Konrad Weihmann wrote:

> On 14.05.21 13:16, Robert P. J. Day wrote:
> >
> >    pretty sure i know the answer to this one, but was asked the
> > other day and wanted to make sure. in order to speed up the normal
> > OE build, someone suggested bypassing the creation of packages
> > that weren't going to be relevant, such as -dev, -doc and so on --
> > the idea was that that could make a significant difference in
> > build time.
>
> I pretty much doubt it will speed up anything. If for instance the
> -dev package will be removed from PACKAGES it is very likely that
> the files of that package will end up in the base package ${PN},
> which results in larger target packages that need to be installed
> (and potentially further RDEPENDS)
>
> Furthermore this will break DEPENDS for recipes that do rely on the
> -dev package - so chances are high that this will break (or
> magically change) something further down the line.
>
> One thing that you can do to speed up the build process in terms of
> packaging time is to delete any **really** not needed files in a
> do_install_append, if no proper configuration option (cmake flag,
> make flag, PACKAGECONFIG) does exist which prevents the creation of
> these files in the first place.
>
> I would recommend not to mess with the packaging defaults for
> reasons such as speeding things up - as this is IMHO addressing the
> issue at the wrong end - just make sure the files in <workdir>/image
> won't get created

  that's the general idea i was trying to get across (so let's confirm
that i was speaking accurately). i suggested that the majority of time
spent in processing a recipe was doing all the fetching and unpacking
and patching and (mostly) compiling, the end result of which was to
populate that recipe's WORKDIR/image directory, at which time the
packaging step took over. so unless one wanted to dangerously mangle
the package definitions, if it ended up in ${D}, it was going to get
packaged, correct?

  so unless one wanted to go back and mess with the recipe's build
configuration to, perhaps, not generate some artifacts, those packages
were going to be built, whether they were subsequently installed or
not. so short of mangling each recipe file to change its
configuration, just leave it alone as the payoff just isn't worth it.

  sound about right?

rday

Re: [OE-core] is there an easy to prevent *creation* of some recipe's packages?

[Konrad Weihmann]

On 14.05.21 13:52, Robert P. J. Day wrote:
> On Fri, 14 May 2021, Konrad Weihmann wrote:
> 
>> On 14.05.21 13:16, Robert P. J. Day wrote:
>>>
>>>     pretty sure i know the answer to this one, but was asked the
>>> other day and wanted to make sure. in order to speed up the normal
>>> OE build, someone suggested bypassing the creation of packages
>>> that weren't going to be relevant, such as -dev, -doc and so on --
>>> the idea was that that could make a significant difference in
>>> build time.
>>
>> I pretty much doubt it will speed up anything. If for instance the
>> -dev package will be removed from PACKAGES it is very likely that
>> the files of that package will end up in the base package ${PN},
>> which results in larger target packages that need to be installed
>> (and potentially further RDEPENDS)
>>
>> Furthermore this will break DEPENDS for recipes that do rely on the
>> -dev package - so chances are high that this will break (or
>> magically change) something further down the line.
>>
>> One thing that you can do to speed up the build process in terms of
>> packaging time is to delete any **really** not needed files in a
>> do_install_append, if no proper configuration option (cmake flag,
>> make flag, PACKAGECONFIG) does exist which prevents the creation of
>> these files in the first place.
>>
>> I would recommend not to mess with the packaging defaults for
>> reasons such as speeding things up - as this is IMHO addressing the
>> issue at the wrong end - just make sure the files in <workdir>/image
>> won't get created
> 
>    that's the general idea i was trying to get across (so let's confirm
> that i was speaking accurately). i suggested that the majority of time
> spent in processing a recipe was doing all the fetching and unpacking
> and patching and (mostly) compiling, the end result of which was to
> populate that recipe's WORKDIR/image directory, at which time the
> packaging step took over. so unless one wanted to dangerously mangle
> the package definitions, if it ended up in ${D}, it was going to get
> packaged, correct?

yes... *but* what the packaging does it to loop over PACKAGES and for 
each item pick up the corresponding FILES_<pkg-name> var and "move" (I 
think it's actually a copy, but that's just details) all matching files 
from image to packages-split.
If files ending up in image that don't match *any* of the 
FILES-glob-pattern one would get a QA error.

> 
>    so unless one wanted to go back and mess with the recipe's build
> configuration to, perhaps, not generate some artifacts, those packages
> were going to be built, whether they were subsequently installed or
> not. so short of mangling each recipe file to change its
> configuration, just leave it alone as the payoff just isn't worth it.
> 
>    sound about right?

yes - the results of trimming down the input to /image will much more 
have a positive impact.

AFAIK packaging speed is directly linked to the number of files in 
/image (worst example I've seen is a ruby package containing 250k+ 
files, which takes round about 20min to package on my machine) - 
concluding from that, reduction of the number of files in /image is will 
do the trick

> 
> rday
> 

Re: [OE-core] is there an easy to prevent *creation* of some recipe's packages?

[Robert Berger]

Let me just throw in one idea with respect to Open Source License 
Compliance and packaging - with "traditional" programming languages.

Let's say we ship a product and are in the phase of an Open Source 
License Compliance audit.

We have a couple of interesting issues here

1) Actually we just need to audit the "main package", which is installed 
on the target. Unfortunately we completely lost the connection between 
what's in the various packages and where those things come from. What I 
mean is, that we typically audit the whole source repo, which contains 
e.g. test cases as well, but those test cases don't even end up in the 
target. Which leads to "over reporting".

1.1) Now we could just say we need some license files which are in the 
repo plus the -src or -dbg package, depending on how things are built.
Ideally the debug sources are those which correspond to what's installed 
on the target.

2) What else is kind of lost, or not taken into account with licensing 
is linking. We only see each package/repo individually, but the right 
thing to do would be to look at the combined work.

If we look at the total project just auditing what's actually needed 
would help a lot time-wise as well ;)

Regards

(another) Robert ;)

Re: [OE-core] is there an easy to prevent *creation* of some recipe's packages?

[Andre McCurdy]

On Fri, May 14, 2021 at 5:39 AM Konrad Weihmann <kweihmann@outlook.com> wrote:
>
> On 14.05.21 13:52, Robert P. J. Day wrote:
> > On Fri, 14 May 2021, Konrad Weihmann wrote:
> >
> >> On 14.05.21 13:16, Robert P. J. Day wrote:
> >>>
> >>>     pretty sure i know the answer to this one, but was asked the
> >>> other day and wanted to make sure. in order to speed up the normal
> >>> OE build, someone suggested bypassing the creation of packages
> >>> that weren't going to be relevant, such as -dev, -doc and so on --
> >>> the idea was that that could make a significant difference in
> >>> build time.
> >>
> >> I pretty much doubt it will speed up anything. If for instance the
> >> -dev package will be removed from PACKAGES it is very likely that
> >> the files of that package will end up in the base package ${PN},
> >> which results in larger target packages that need to be installed
> >> (and potentially further RDEPENDS)
> >>
> >> Furthermore this will break DEPENDS for recipes that do rely on the
> >> -dev package - so chances are high that this will break (or
> >> magically change) something further down the line.
> >>
> >> One thing that you can do to speed up the build process in terms of
> >> packaging time is to delete any **really** not needed files in a
> >> do_install_append, if no proper configuration option (cmake flag,
> >> make flag, PACKAGECONFIG) does exist which prevents the creation of
> >> these files in the first place.
> >>
> >> I would recommend not to mess with the packaging defaults for
> >> reasons such as speeding things up - as this is IMHO addressing the
> >> issue at the wrong end - just make sure the files in <workdir>/image
> >> won't get created
> >
> >    that's the general idea i was trying to get across (so let's confirm
> > that i was speaking accurately). i suggested that the majority of time
> > spent in processing a recipe was doing all the fetching and unpacking
> > and patching and (mostly) compiling, the end result of which was to
> > populate that recipe's WORKDIR/image directory, at which time the
> > packaging step took over. so unless one wanted to dangerously mangle
> > the package definitions, if it ended up in ${D}, it was going to get
> > packaged, correct?
>
> yes... *but* what the packaging does it to loop over PACKAGES and for
> each item pick up the corresponding FILES_<pkg-name> var and "move" (I
> think it's actually a copy, but that's just details) all matching files
> from image to packages-split.
> If files ending up in image that don't match *any* of the
> FILES-glob-pattern one would get a QA error.
>
> >
> >    so unless one wanted to go back and mess with the recipe's build
> > configuration to, perhaps, not generate some artifacts, those packages
> > were going to be built, whether they were subsequently installed or
> > not. so short of mangling each recipe file to change its
> > configuration, just leave it alone as the payoff just isn't worth it.
> >
> >    sound about right?
>
> yes - the results of trimming down the input to /image will much more
> have a positive impact.
>
> AFAIK packaging speed is directly linked to the number of files in
> /image (worst example I've seen is a ruby package containing 250k+
> files, which takes round about 20min to package on my machine) -
> concluding from that, reduction of the number of files in /image is will
> do the trick

This thread seems to have covered the main point already (if you want
to save time by not packaging unwanted files then don't install the
files in the first place).

It's perhaps worth noting one very common case where files are
installed and packaged unnecessarily and that's having the ptest
distro feature enabled even though you don't intend to install any
ptest packages. For most users ptest should be something you enable
selectively when you want to prove the functionality of recipes in
oe-core (e.g. after upgrading to a new OE release or when bringing up
a new machine) and then disable again afterwards.

Re: [OE-core] is there an easy to prevent *creation* of some recipe's packages?

[Ross Burton]

On Fri, 14 May 2021 at 12:17, Robert P. J. Day <rpjday@crashcourse.ca> wrote:
>   pretty sure i know the answer to this one, but was asked the other
> day and wanted to make sure. in order to speed up the normal OE build,
> someone suggested bypassing the creation of packages that weren't
> going to be relevant, such as -dev, -doc and so on -- the idea was
> that that could make a significant difference in build time.

General case: no. Absolute waste of time.

There are common exceptions: if the recipe has e.g. API documentation
that takes a long time to build then add a PACKAGECONFIG for it and
respect the api-documentation DISTRO_FEATURE.

There are limited edge cases of course: Piglit is 99% auto-generated
test cases, the binaries are *huge* and the need to debug them is
limited, so the recipe installs striped binaries so there is no -dbg
package to generate.  We're talking minutes of time spent stripping
gigabytes of libraries here and there's no hackery or magic, the
recipe just tells the makefile to strip on install.

Ross