* [PATCH 0/3] rm_work enhancements @ 2017-01-06 9:55 Patrick Ohly 2017-01-06 9:55 ` [PATCH 1/3] gcc-source.inc: cleanly disable do_rm_work Patrick Ohly ` (2 more replies) 0 siblings, 3 replies; 18+ messages in thread From: Patrick Ohly @ 2017-01-06 9:55 UTC (permalink / raw) To: openembedded-core This is the OE-core side of the rm_work.bbclass enhancements. Depends on the corresponding bitbake patch series. See the OE-core "rm_work + pybootchart enhancements" mail thread for further information. The following changes since commit af330fc0087692039d4256a6fb1e122742e8ec1a: packagegroup-core-lsb: add opengl to REQUIRED_DISTRO_FEATURES (2017-01-05 13:53:45 +0000) are available in the git repository at: git://github.com/pohly/openembedded-core rmwork https://github.com/pohly/openembedded-core/tree/rmwork Patrick Ohly (3): gcc-source.inc: cleanly disable do_rm_work rm_work_and_downloads.bbclass: more aggressively minimize disk usage rm_work.bbclass: clean up sooner meta/classes/rm_work.bbclass | 31 ++++++++++++++++------------ meta/classes/rm_work_and_downloads.bbclass | 33 ++++++++++++++++++++++++++++++ meta/recipes-devtools/gcc/gcc-source.inc | 2 +- 3 files changed, 52 insertions(+), 14 deletions(-) create mode 100644 meta/classes/rm_work_and_downloads.bbclass -- 2.1.4 ^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH 1/3] gcc-source.inc: cleanly disable do_rm_work 2017-01-06 9:55 [PATCH 0/3] rm_work enhancements Patrick Ohly @ 2017-01-06 9:55 ` Patrick Ohly 2017-01-09 18:47 ` Khem Raj 2017-01-06 9:55 ` [PATCH 2/3] rm_work_and_downloads.bbclass: more aggressively minimize disk usage Patrick Ohly 2017-01-06 9:55 ` [PATCH 3/3] rm_work.bbclass: clean up sooner Patrick Ohly 2 siblings, 1 reply; 18+ messages in thread From: Patrick Ohly @ 2017-01-06 9:55 UTC (permalink / raw) To: openembedded-core Using "deltask" assumes that do_rm_work has been added already, which won't be the case anymore in the upcoming improved rm_work.bbclass, because then an anonymous python method will add do_rm_work. Setting RM_WORK_EXCLUDE works with the current and upcoming rm_work.bbclass and is the API that is meant to be used for excluding recipes from cleaning, so use that. --- meta/recipes-devtools/gcc/gcc-source.inc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/recipes-devtools/gcc/gcc-source.inc b/meta/recipes-devtools/gcc/gcc-source.inc index 49bde92..0d0edb5 100644 --- a/meta/recipes-devtools/gcc/gcc-source.inc +++ b/meta/recipes-devtools/gcc/gcc-source.inc @@ -3,7 +3,7 @@ deltask do_compile deltask do_install deltask do_populate_sysroot deltask do_populate_lic -deltask do_rm_work +RM_WORK_EXCLUDE += "${PN}" inherit nopackages -- 2.1.4 ^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH 1/3] gcc-source.inc: cleanly disable do_rm_work 2017-01-06 9:55 ` [PATCH 1/3] gcc-source.inc: cleanly disable do_rm_work Patrick Ohly @ 2017-01-09 18:47 ` Khem Raj 0 siblings, 0 replies; 18+ messages in thread From: Khem Raj @ 2017-01-09 18:47 UTC (permalink / raw) To: Patrick Ohly; +Cc: Patches and discussions about the oe-core layer On Fri, Jan 6, 2017 at 1:55 AM, Patrick Ohly <patrick.ohly@intel.com> wrote: > Using "deltask" assumes that do_rm_work has been added already, which > won't be the case anymore in the upcoming improved rm_work.bbclass, > because then an anonymous python method will add do_rm_work. > > Setting RM_WORK_EXCLUDE works with the current and upcoming > rm_work.bbclass and is the API that is meant to be used for excluding > recipes from cleaning, so use that. > --- > meta/recipes-devtools/gcc/gcc-source.inc | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/meta/recipes-devtools/gcc/gcc-source.inc b/meta/recipes-devtools/gcc/gcc-source.inc > index 49bde92..0d0edb5 100644 > --- a/meta/recipes-devtools/gcc/gcc-source.inc > +++ b/meta/recipes-devtools/gcc/gcc-source.inc > @@ -3,7 +3,7 @@ deltask do_compile > deltask do_install > deltask do_populate_sysroot > deltask do_populate_lic > -deltask do_rm_work > +RM_WORK_EXCLUDE += "${PN}" seems ok > > inherit nopackages > > -- > 2.1.4 > > -- > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core ^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH 2/3] rm_work_and_downloads.bbclass: more aggressively minimize disk usage 2017-01-06 9:55 [PATCH 0/3] rm_work enhancements Patrick Ohly 2017-01-06 9:55 ` [PATCH 1/3] gcc-source.inc: cleanly disable do_rm_work Patrick Ohly @ 2017-01-06 9:55 ` Patrick Ohly 2017-01-06 18:31 ` Randy Witt 2017-01-06 9:55 ` [PATCH 3/3] rm_work.bbclass: clean up sooner Patrick Ohly 2 siblings, 1 reply; 18+ messages in thread From: Patrick Ohly @ 2017-01-06 9:55 UTC (permalink / raw) To: openembedded-core rm_work.bbclass never deletes downloaded files, even if they are not going to be needed again during the build. rm_work_and_downloads.bbclass is more aggressive in minimizing the used disk space during a build, but has other disadvantages: - sources required by different recipes need to be fetched once per recipe, not once per build - incremental builds do not work reliably because sources get removed without ensuring that sources gets fetched again That makes rm_work_and_downloads.bbclass useful for one-time builds in a constrained environment (like a CI system), but not for general use. Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> --- meta/classes/rm_work_and_downloads.bbclass | 33 ++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100644 meta/classes/rm_work_and_downloads.bbclass diff --git a/meta/classes/rm_work_and_downloads.bbclass b/meta/classes/rm_work_and_downloads.bbclass new file mode 100644 index 0000000..7c00bea --- /dev/null +++ b/meta/classes/rm_work_and_downloads.bbclass @@ -0,0 +1,33 @@ +# Author: Patrick Ohly <patrick.ohly@intel.com> +# Copyright: Copyright (C) 2015 Intel Corporation +# +# This file is licensed under the MIT license, see COPYING.MIT in +# this source distribution for the terms. + +# This class is used like rm_work: +# INHERIT += "rm_work_and_downloads" +# +# In addition to removing local build directories of a recipe, it also +# removes the downloaded source. This is achieved by making the DL_DIR +# recipe-specific. While reducing disk usage, it increases network usage (for +# example, compiling the same source for target and host implies downloading +# the source twice). +# +# Because the "do_fetch" task does not get re-run after removing the downloaded +# sources, this class is also not suitable for incremental builds. +# +# Where it works well is in well-connected build environments with limited +# disk space (like TravisCI). + +inherit rm_work + +# This would ensure that the existing do_rm_work() removes the downloads, +# but does not work because some recipes have a circular dependency between +# WORKDIR and DL_DIR (via ${SRCPV}?). +# DL_DIR = "${WORKDIR}/downloads" + +# Instead go up one level and remove ourself. +DL_DIR = "${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}/${PN}/downloads" +do_rm_work_append () { + rm -rf ${DL_DIR} +} -- 2.1.4 ^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH 2/3] rm_work_and_downloads.bbclass: more aggressively minimize disk usage 2017-01-06 9:55 ` [PATCH 2/3] rm_work_and_downloads.bbclass: more aggressively minimize disk usage Patrick Ohly @ 2017-01-06 18:31 ` Randy Witt 2017-01-06 19:22 ` Patrick Ohly 0 siblings, 1 reply; 18+ messages in thread From: Randy Witt @ 2017-01-06 18:31 UTC (permalink / raw) To: Patrick Ohly; +Cc: OE-core [-- Attachment #1: Type: text/plain, Size: 3250 bytes --] Hi Patrick, On Fri, Jan 6, 2017 at 1:55 AM, Patrick Ohly <patrick.ohly@intel.com> wrote: > rm_work.bbclass never deletes downloaded files, even if they are not > going to be needed again during the > build. rm_work_and_downloads.bbclass is more aggressive in minimizing > the used disk space during a build, but has other disadvantages: > - sources required by different recipes need to be fetched once per > recipe, not once per build > - incremental builds do not work reliably because sources get > removed without ensuring that sources gets fetched again > > That makes rm_work_and_downloads.bbclass useful for one-time builds in > a constrained environment (like a CI system), but not for general use. > > I'd rather see this be a new class independent from rm_work. People can always in inherit both rm_work and rm_download. That being said, since this doesn't operate at the fetch level, i.e. only remove things that were fetched as part of the current invocation of bitbake, I don't see why the user couldn't just "rm -rf downloads" themselves as part of the ci teardown. > Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> > --- > meta/classes/rm_work_and_downloads.bbclass | 33 > ++++++++++++++++++++++++++++++ > 1 file changed, 33 insertions(+) > create mode 100644 meta/classes/rm_work_and_downloads.bbclass > > diff --git a/meta/classes/rm_work_and_downloads.bbclass > b/meta/classes/rm_work_and_downloads.bbclass > new file mode 100644 > index 0000000..7c00bea > --- /dev/null > +++ b/meta/classes/rm_work_and_downloads.bbclass > @@ -0,0 +1,33 @@ > +# Author: Patrick Ohly <patrick.ohly@intel.com> > +# Copyright: Copyright (C) 2015 Intel Corporation > +# > +# This file is licensed under the MIT license, see COPYING.MIT in > +# this source distribution for the terms. > + > +# This class is used like rm_work: > +# INHERIT += "rm_work_and_downloads" > +# > +# In addition to removing local build directories of a recipe, it also > +# removes the downloaded source. This is achieved by making the DL_DIR > +# recipe-specific. While reducing disk usage, it increases network usage > (for > +# example, compiling the same source for target and host implies > downloading > +# the source twice). > +# > +# Because the "do_fetch" task does not get re-run after removing the > downloaded > +# sources, this class is also not suitable for incremental builds. > +# > +# Where it works well is in well-connected build environments with limited > +# disk space (like TravisCI). > + > +inherit rm_work > + > +# This would ensure that the existing do_rm_work() removes the downloads, > +# but does not work because some recipes have a circular dependency > between > +# WORKDIR and DL_DIR (via ${SRCPV}?). > +# DL_DIR = "${WORKDIR}/downloads" > + > +# Instead go up one level and remove ourself. > +DL_DIR = "${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}/${PN}/downloads" > +do_rm_work_append () { > + rm -rf ${DL_DIR} > +} > -- > 2.1.4 > > -- > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core > [-- Attachment #2: Type: text/html, Size: 4283 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 2/3] rm_work_and_downloads.bbclass: more aggressively minimize disk usage 2017-01-06 18:31 ` Randy Witt @ 2017-01-06 19:22 ` Patrick Ohly 2017-01-06 21:29 ` Randy Witt 0 siblings, 1 reply; 18+ messages in thread From: Patrick Ohly @ 2017-01-06 19:22 UTC (permalink / raw) To: Randy Witt; +Cc: OE-core On Fri, 2017-01-06 at 10:31 -0800, Randy Witt wrote: > > I'd rather see this be a new class independent from rm_work. People > can always in inherit both rm_work and rm_download. I'm not sure whether it's worth supporting that mode: what would be the motivation for removing downloads, but not the work directory? I'm a bit worried about unexpected breakages when using just rm_download. > That being said, since this doesn't operate at the fetch level, i.e. > only remove things that were fetched as part of the current invocation > of bitbake, I don't see why the user couldn't just "rm -rf downloads" > themselves as part of the ci teardown. The motivation for rm_work_and_downloads.bbclass is the same as for rm_work.bbclass: reducing the required disk space *during* the build, to get builds to succeed which otherwise would run out of disk space. A "rm -rf downloads" at the end of the build doesn't help achieve that goal. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 2/3] rm_work_and_downloads.bbclass: more aggressively minimize disk usage 2017-01-06 19:22 ` Patrick Ohly @ 2017-01-06 21:29 ` Randy Witt 2017-01-07 8:09 ` Patrick Ohly 0 siblings, 1 reply; 18+ messages in thread From: Randy Witt @ 2017-01-06 21:29 UTC (permalink / raw) To: Patrick Ohly; +Cc: OE-core [-- Attachment #1: Type: text/plain, Size: 1969 bytes --] On Fri, Jan 6, 2017 at 11:22 AM, Patrick Ohly <patrick.ohly@intel.com> wrote: > On Fri, 2017-01-06 at 10:31 -0800, Randy Witt wrote: > > > > > I'd rather see this be a new class independent from rm_work. People > > can always in inherit both rm_work and rm_download. > > I'm not sure whether it's worth supporting that mode: what would be the > motivation for removing downloads, but not the work directory? > > There are times that the work directories help with the debugging of build failures. The logs aren't always enough. So a person might want to do something like remove the downloads, but preserve the actual work, especially in the case of failures. I'll admit it is a fabricated scenario, but in general I'm against one knob requiring the turning of another knob. > The motivation for rm_work_and_downloads.bbclass is the same as for > rm_work.bbclass: reducing the required disk space *during* the build, to > get builds to succeed which otherwise would run out of disk space. > > Disk space isn't the only motivation for rm_work. It also allows for deletion of large amounts of inodes before they've been flushed to disk. So rather than taking minutes worth of time to clean up post build, and thrashing media that other tasks may be using, it is almost instantaneous because everything is still in the page cache. Also the build itself can be faster for the same reason, there is no waiting for data to be flushed out to disk( because it was deleted while still in the page cache) and thus impacting reads in other parts of the build. > A "rm -rf downloads" at the end of the build doesn't help achieve that > goal. > > That's true, I hadn't yet processed it completely. Ignore that comment. :) As an overall comment though, I really like the idea of rm_work not being deferred for so long. It always seemed strange to me to see the thundering herd of rm_works after certain do_builds would finish. [-- Attachment #2: Type: text/html, Size: 2844 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 2/3] rm_work_and_downloads.bbclass: more aggressively minimize disk usage 2017-01-06 21:29 ` Randy Witt @ 2017-01-07 8:09 ` Patrick Ohly 2017-01-09 10:25 ` Patrick Ohly 0 siblings, 1 reply; 18+ messages in thread From: Patrick Ohly @ 2017-01-07 8:09 UTC (permalink / raw) To: Randy Witt; +Cc: OE-core On Fri, 2017-01-06 at 13:29 -0800, Randy Witt wrote: > There are times that the work directories help with the debugging of > build failures. The logs aren't always enough. So a person might want > to do something like remove the downloads, but preserve the actual > work, especially in the case of failures. > > > I'll admit it is a fabricated scenario, but in general I'm against one > knob requiring the turning of another knob. In principle I agree. Okay, let's separate the two. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 2/3] rm_work_and_downloads.bbclass: more aggressively minimize disk usage 2017-01-07 8:09 ` Patrick Ohly @ 2017-01-09 10:25 ` Patrick Ohly 2017-01-09 18:17 ` Randy Witt 0 siblings, 1 reply; 18+ messages in thread From: Patrick Ohly @ 2017-01-09 10:25 UTC (permalink / raw) To: Randy Witt; +Cc: OE-core On Sat, 2017-01-07 at 09:09 +0100, Patrick Ohly wrote: > On Fri, 2017-01-06 at 13:29 -0800, Randy Witt wrote: > > > > There are times that the work directories help with the debugging of > > build failures. The logs aren't always enough. So a person might want > > to do something like remove the downloads, but preserve the actual > > work, especially in the case of failures. > > > > > > I'll admit it is a fabricated scenario, but in general I'm against one > > knob requiring the turning of another knob. > > In principle I agree. Okay, let's separate the two. I hit one snag when starting to do that: the implementation of download removal depends on do_rm_work from rm_work.bbclass for hooking into the build of a recipe at the right place. I can't think of an easy way of avoiding that without duplicating the complex logic from rm_work.bbclass, which itself is currently about to change, so I'd prefer to keep rm_work_and_downloads.bbclass as proposed. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 2/3] rm_work_and_downloads.bbclass: more aggressively minimize disk usage 2017-01-09 10:25 ` Patrick Ohly @ 2017-01-09 18:17 ` Randy Witt 0 siblings, 0 replies; 18+ messages in thread From: Randy Witt @ 2017-01-09 18:17 UTC (permalink / raw) To: Patrick Ohly; +Cc: OE-core [-- Attachment #1: Type: text/plain, Size: 695 bytes --] On Mon, Jan 9, 2017 at 2:25 AM, Patrick Ohly <patrick.ohly@intel.com> wrote: > > > > > In principle I agree. Okay, let's separate the two. > > I hit one snag when starting to do that: the implementation of download > removal depends on do_rm_work from rm_work.bbclass for hooking into the > build of a recipe at the right place. > > I can't think of an easy way of avoiding that without duplicating the > complex logic from rm_work.bbclass, which itself is currently about to > change, so I'd prefer to keep rm_work_and_downloads.bbclass as proposed. > Sounds good to me. If it turns out someone would like them separate in the future, we can schedule the time to do it then. [-- Attachment #2: Type: text/html, Size: 1086 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH 3/3] rm_work.bbclass: clean up sooner 2017-01-06 9:55 [PATCH 0/3] rm_work enhancements Patrick Ohly 2017-01-06 9:55 ` [PATCH 1/3] gcc-source.inc: cleanly disable do_rm_work Patrick Ohly 2017-01-06 9:55 ` [PATCH 2/3] rm_work_and_downloads.bbclass: more aggressively minimize disk usage Patrick Ohly @ 2017-01-06 9:55 ` Patrick Ohly 2017-02-15 18:32 ` Martin Jansa 2 siblings, 1 reply; 18+ messages in thread From: Patrick Ohly @ 2017-01-06 9:55 UTC (permalink / raw) To: openembedded-core Having do_rm_work depend on do_build had one major disadvantage: do_build depends on the do_build of other recipes, to ensure that runtime dependencies also get built. The effect is that when work on a recipe is complete and it could get cleaned up, do_rm_work still doesn't run because it waits for those other recipes, thus leading to more temporary disk space usage than really needed. The right solution is to inject do_rm_work before do_build and after all tasks of the recipe. Achieving that depends on the new bitbake support for prioritizing anonymous functions to ensure that rm_work.bbclass gets to see a full set of existing tasks when adding its own one. This is relevant, for example, for do_analyseimage in meta-security-isafw's isafw.bbclass. In addition, the new "rm_work" scheduler is used by default. It prioritizes finishing recipes over continuing with the more important recipes (with "importance" determined by the number of reverse-dependencies). Benchmarking (see "rm_work + pybootchart enhancements" on the OE-core mailing list) showed that builds with the modified rm_work.bbclass were both faster (albeit not by much) and required considerably less disk space (14230MiB instead of 18740MiB for core-image-sato). Interestingly enough, builds with rm_work.bbclass were also faster than those without. Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> --- meta/classes/rm_work.bbclass | 31 ++++++++++++++++++------------- 1 file changed, 18 insertions(+), 13 deletions(-) diff --git a/meta/classes/rm_work.bbclass b/meta/classes/rm_work.bbclass index 3516c7e..1205104 100644 --- a/meta/classes/rm_work.bbclass +++ b/meta/classes/rm_work.bbclass @@ -11,16 +11,13 @@ # RM_WORK_EXCLUDE += "icu-native icu busybox" # -# Use the completion scheduler by default when rm_work is active +# Use the dedicated rm_work scheduler by default when rm_work is active # to try and reduce disk usage -BB_SCHEDULER ?= "completion" +BB_SCHEDULER ?= "rm_work" # Run the rm_work task in the idle scheduling class BB_TASK_IONICE_LEVEL_task-rm_work = "3.0" -RMWORK_ORIG_TASK := "${BB_DEFAULT_TASK}" -BB_DEFAULT_TASK = "rm_work_all" - do_rm_work () { # If the recipe name is in the RM_WORK_EXCLUDE, skip the recipe. for p in ${RM_WORK_EXCLUDE}; do @@ -97,13 +94,6 @@ do_rm_work () { rm -f $i done } -addtask rm_work after do_${RMWORK_ORIG_TASK} - -do_rm_work_all () { - : -} -do_rm_work_all[recrdeptask] = "do_rm_work" -addtask rm_work_all after do_rm_work do_populate_sdk[postfuncs] += "rm_work_populatesdk" rm_work_populatesdk () { @@ -117,7 +107,7 @@ rm_work_rootfs () { } rm_work_rootfs[cleandirs] = "${WORKDIR}/rootfs" -python () { +python __anonymous_rm_work() { if bb.data.inherits_class('kernel', d): d.appendVar("RM_WORK_EXCLUDE", ' ' + d.getVar("PN")) # If the recipe name is in the RM_WORK_EXCLUDE, skip the recipe. @@ -126,4 +116,19 @@ python () { if pn in excludes: d.delVarFlag('rm_work_rootfs', 'cleandirs') d.delVarFlag('rm_work_populatesdk', 'cleandirs') + else: + # Inject do_rm_work into the tasks of the current recipe such that do_build + # depends on it and that it runs after all other tasks that block do_build, + # i.e. after all work on the current recipe is done. The reason for taking + # this approach instead of making do_rm_work depend on do_build is that + # do_build inherits additional runtime dependencies on + # other recipes and thus will typically run much later than completion of + # work in the recipe itself. + deps = bb.build.preceedtask('do_build', True, d) + if 'do_build' in deps: + deps.remove('do_build') + bb.build.addtask('do_rm_work', 'do_build', ' '.join(deps), d) } +# Higher priority than the normal 100, and thus we run after other +# classes like package_rpm.bbclass which also add custom tasks. +__anonymous_rm_work[__anonprio] = "1000" -- 2.1.4 ^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH 3/3] rm_work.bbclass: clean up sooner 2017-01-06 9:55 ` [PATCH 3/3] rm_work.bbclass: clean up sooner Patrick Ohly @ 2017-02-15 18:32 ` Martin Jansa 2017-02-16 10:26 ` Patrick Ohly 0 siblings, 1 reply; 18+ messages in thread From: Martin Jansa @ 2017-02-15 18:32 UTC (permalink / raw) To: Patrick Ohly; +Cc: Patches and discussions about the oe-core layer [-- Attachment #1: Type: text/plain, Size: 9807 bytes --] Are all changes necessary for this to work already in master? Yesterday I've noticed that rm_work for some components which are early in the dependency (like qtbase) are executed relatively late (together with do_package_qa). So I've tried very naive way to find out if the rm_work tasks are executed sooner or not just by comparing Task IDs in build of the same image built from scratch (without sstate) with Dizzy, Morty and current master. First I've stripped unnecessary prefix and names of proprietary components (in case someone wants me to share these lists): grep "^NOTE: Running task .*, do_rm_work)$" log.build |sed 's#/jenkins/mjansa/build-[^/]*/##; s#meta-lg-webos/[^:]*:#private-component:#g; s#^NOTE: Running task ##g' > rm_work.tasks.dizzy with slightly different regexp for morty and master: grep "^NOTE: Running task .*:do_rm_work)$" ld.gold/log.m16p |sed 's#/jenkins/mjansa/build-[^/]*/##; s#meta-lg-webos/[^:]*:#private-component:#g; s#^NOTE: Running task ##g' > rm_work.tasks.morty and then I did even more naive thing to compare average task id of rm-work jobs with following results: for i in rm_work.tasks.*; do echo $i; export COUNT=0 SUM=0; for TASK in `cat $i | cut -f 1 -d\ `; do COUNT=`expr $COUNT + 1`; SUM=`expr $SUM + $TASK`; done; echo "AVG = `expr $SUM / $C OUNT`; COUNT = $COUNT"; done rm_work.tasks.dizzy AVG = 6429; COUNT = 764 rm_work.tasks.master AVG = 7570; COUNT = 891 rm_work.tasks.master.qemux86 AVG = 5527; COUNT = 665 rm_work.tasks.morty AVG = 6689; COUNT = 786 rm_work.tasks.morty.gold AVG = 6764; COUNT = 786 rm_work.tasks.morty.gold is the same build as in rm_work.tasks.morty just with ld-is-gold added to DISTRO_FEATUREs (as I was testing build time to compare ld.bfd and ld.gold in our images). rm_work.tasks.master.qemux86 is the same build as rm_work.tasks.master but for qemux86, all other builds are for some ARM board we use Then few interesting steps: gcc-cross looks good (not available for dizzy build which is using external toolchain) $ grep gcc-cross_ rm_work.tasks.* rm_work.tasks.master:510 of 14470 (oe-core/meta/recipes-devtools/gcc/gcc-cross_6.3.bb:do_rm_work) rm_work.tasks.master.qemux86:515 of 10296 (oe-core/meta/recipes-devtools/gcc/gcc-cross_6.3.bb:do_rm_work) rm_work.tasks.morty:2592 of 12021 (oe-core/meta/recipes-devtools/gcc/gcc-cross_6.2.bb:do_rm_work) rm_work.tasks.morty.gold:2734 of 12021 (oe-core/meta/recipes-devtools/gcc/gcc-cross_6.2.bb:do_rm_work) qtdeclarative-native got rm_work a bit later, whcih might be caused only by the increased number of tasks thanks to RSS $ grep native.*qtdeclarative rm_work.tasks.* rm_work.tasks.dizzy:2101 of 11766 (ID: 11128, virtual:native:meta-qt5/recipes-qt/qt5/qtdeclarative_git.bb, do_rm_work) rm_work.tasks.master:2614 of 14470 (virtual:native:meta-qt5/recipes-qt/qt5/qtdeclarative_git.bb:do_rm_work) rm_work.tasks.master.qemux86:2521 of 10296 (virtual:native:meta-qt5/recipes-qt/qt5/qtdeclarative_git.bb:do_rm_work) rm_work.tasks.morty:1513 of 12021 (virtual:native:meta-qt5/recipes-qt/qt5/qtdeclarative_git.bb:do_rm_work) rm_work.tasks.morty.gold:1514 of 12021 (virtual:native:meta-qt5/recipes-qt/qt5/qtdeclarative_git.bb:do_rm_work) and here is the target qtdeclarative which trigered this whole naive analysis: $ grep qtdeclarative rm_work.tasks.* | grep -v native rm_work.tasks.dizzy:4952 of 11766 (ID: 6670, meta-qt5/recipes-qt/qt5/ qtdeclarative_git.bb, do_rm_work) rm_work.tasks.master:4317 of 14470 (meta-qt5/recipes-qt/qt5/qtdeclarative_git.bb:do_rm_work) rm_work.tasks.master.qemux86:10142 of 10296 (meta-qt5/recipes-qt/qt5/qtdeclarative_git.bb:do_rm_work) rm_work.tasks.morty:6753 of 12021 (meta-qt5/recipes-qt/qt5/qtdeclarative_git.bb:do_rm_work) rm_work.tasks.morty.gold:6883 of 12021 (meta-qt5/recipes-qt/qt5/qtdeclarative_git.bb:do_rm_work) If we dismiss the strange case in rm_work.tasks.master.qemux86 then it seems to perform at least as good as old completion BB_SCHEDULER. But I wanted to ask if there is something else we can do or you were planing to do, because IIRC you shared some longer analysis of what could be improved here and I'm not sure if you managed to implement it all. It feels to me that rm_work has high priority, but still it's "blocked" by e.g. do_package_qa which gets executed late and then immediately followed by rm_work. in ideal case I would really like to have a switch which will force rm_work to take absolute priority over other tasks, it doesn't take very long to delete the files in tmpfs and would allow me to do tmpfs builds on builders with smaller RAM. The "state of bitbake world" builds are performed in 74G tmpfs (for whole tmpdir-glibc) and yesterday's builds started to fail again (when it happens to run chromium and chromium-wayland at the same time) - the manual solution for this I'm using for last couple years is to build in "steps" which force to run rm_work for all included components, so e.g. bitbake gcc-cross-arm && bitbake small-image && bitbake chromium && bitbake chromium-wayland && bitbake big-image && bitbake world will keep the tmpfs usage peaks much lower than running just bitbake world On Fri, Jan 6, 2017 at 10:55 AM, Patrick Ohly <patrick.ohly@intel.com> wrote: > Having do_rm_work depend on do_build had one major disadvantage: > do_build depends on the do_build of other recipes, to ensure that > runtime dependencies also get built. The effect is that when work on a > recipe is complete and it could get cleaned up, do_rm_work still > doesn't run because it waits for those other recipes, thus leading to > more temporary disk space usage than really needed. > > The right solution is to inject do_rm_work before do_build and after > all tasks of the recipe. Achieving that depends on the new bitbake > support for prioritizing anonymous functions to ensure that > rm_work.bbclass gets to see a full set of existing tasks when adding > its own one. This is relevant, for example, for do_analyseimage in > meta-security-isafw's isafw.bbclass. > > In addition, the new "rm_work" scheduler is used by default. It > prioritizes finishing recipes over continuing with the more > important recipes (with "importance" determined by the number of > reverse-dependencies). > > Benchmarking (see "rm_work + pybootchart enhancements" on the OE-core > mailing list) showed that builds with the modified rm_work.bbclass > were both faster (albeit not by much) and required considerably less > disk space (14230MiB instead of 18740MiB for core-image-sato). > Interestingly enough, builds with rm_work.bbclass were also faster > than those without. > > Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> > --- > meta/classes/rm_work.bbclass | 31 ++++++++++++++++++------------- > 1 file changed, 18 insertions(+), 13 deletions(-) > > diff --git a/meta/classes/rm_work.bbclass b/meta/classes/rm_work.bbclass > index 3516c7e..1205104 100644 > --- a/meta/classes/rm_work.bbclass > +++ b/meta/classes/rm_work.bbclass > @@ -11,16 +11,13 @@ > # RM_WORK_EXCLUDE += "icu-native icu busybox" > # > > -# Use the completion scheduler by default when rm_work is active > +# Use the dedicated rm_work scheduler by default when rm_work is active > # to try and reduce disk usage > -BB_SCHEDULER ?= "completion" > +BB_SCHEDULER ?= "rm_work" > > # Run the rm_work task in the idle scheduling class > BB_TASK_IONICE_LEVEL_task-rm_work = "3.0" > > -RMWORK_ORIG_TASK := "${BB_DEFAULT_TASK}" > -BB_DEFAULT_TASK = "rm_work_all" > - > do_rm_work () { > # If the recipe name is in the RM_WORK_EXCLUDE, skip the recipe. > for p in ${RM_WORK_EXCLUDE}; do > @@ -97,13 +94,6 @@ do_rm_work () { > rm -f $i > done > } > -addtask rm_work after do_${RMWORK_ORIG_TASK} > - > -do_rm_work_all () { > - : > -} > -do_rm_work_all[recrdeptask] = "do_rm_work" > -addtask rm_work_all after do_rm_work > > do_populate_sdk[postfuncs] += "rm_work_populatesdk" > rm_work_populatesdk () { > @@ -117,7 +107,7 @@ rm_work_rootfs () { > } > rm_work_rootfs[cleandirs] = "${WORKDIR}/rootfs" > > -python () { > +python __anonymous_rm_work() { > if bb.data.inherits_class('kernel', d): > d.appendVar("RM_WORK_EXCLUDE", ' ' + d.getVar("PN")) > # If the recipe name is in the RM_WORK_EXCLUDE, skip the recipe. > @@ -126,4 +116,19 @@ python () { > if pn in excludes: > d.delVarFlag('rm_work_rootfs', 'cleandirs') > d.delVarFlag('rm_work_populatesdk', 'cleandirs') > + else: > + # Inject do_rm_work into the tasks of the current recipe such > that do_build > + # depends on it and that it runs after all other tasks that block > do_build, > + # i.e. after all work on the current recipe is done. The reason > for taking > + # this approach instead of making do_rm_work depend on do_build > is that > + # do_build inherits additional runtime dependencies on > + # other recipes and thus will typically run much later than > completion of > + # work in the recipe itself. > + deps = bb.build.preceedtask('do_build', True, d) > + if 'do_build' in deps: > + deps.remove('do_build') > + bb.build.addtask('do_rm_work', 'do_build', ' '.join(deps), d) > } > +# Higher priority than the normal 100, and thus we run after other > +# classes like package_rpm.bbclass which also add custom tasks. > +__anonymous_rm_work[__anonprio] = "1000" > -- > 2.1.4 > > -- > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core > [-- Attachment #2: Type: text/html, Size: 12462 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 3/3] rm_work.bbclass: clean up sooner 2017-02-15 18:32 ` Martin Jansa @ 2017-02-16 10:26 ` Patrick Ohly 2017-03-01 15:52 ` Martin Jansa 0 siblings, 1 reply; 18+ messages in thread From: Patrick Ohly @ 2017-02-16 10:26 UTC (permalink / raw) To: Martin Jansa; +Cc: Patches and discussions about the oe-core layer On Wed, 2017-02-15 at 19:32 +0100, Martin Jansa wrote: > Are all changes necessary for this to work already in master? Yes. > Yesterday I've noticed that rm_work for some components which are > early in the dependency (like qtbase) are executed relatively late > (together with do_package_qa). Could do_rm_work run before do_package_qa? rm_work.bbclass doesn't know that, and therefore schedules do_rm_work after do_package_qa. If yes, then adding a list of tasks that can be ignored would be trivial. This can be a variable, so a recipe can even add their own ones, if necessary. > So I've tried very naive way to find out if the rm_work tasks are > executed sooner or not just by comparing Task IDs in build of the same > image built from scratch (without sstate) with Dizzy, Morty and > current master. Interesting, I hadn't thought of testing it like that. > If we dismiss the strange case in rm_work.tasks.master.qemux86 then it > seems to perform at least as good as old completion BB_SCHEDULER. > > > But I wanted to ask if there is something else we can do or you were > planing to do, because IIRC you shared some longer analysis of what > could be improved here and I'm not sure if you managed to implement it > all. The other ideas that I mentioned at some point didn't pan out as intended. In particular allowing do_rm_work tasks to run when the normal task limit was reached didn't have a big effect and the implementation was a hack, so I dropped that. > It feels to me that rm_work has high priority, but still it's > "blocked" by e.g. do_package_qa which gets executed late and then > immediately followed by rm_work. That should be easy to change, perhaps like this (untested): RM_WORK_TASKS_WHITELIST = "do_build do_package_qa" deps = set(bb.build.preceedtask('do_build', True, d)) whitelist = d.getVar('RM_WORK_TASKS_WHITELIST').split() deps.difference_update(whitelist) # In practice, addtask() here merely updates the dependencies. bb.build.addtask('do_rm_work', 'do_build', ' '.join(deps), d) -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 3/3] rm_work.bbclass: clean up sooner 2017-02-16 10:26 ` Patrick Ohly @ 2017-03-01 15:52 ` Martin Jansa 2017-03-01 16:44 ` Patrick Ohly 2017-03-02 9:38 ` Patrick Ohly 0 siblings, 2 replies; 18+ messages in thread From: Martin Jansa @ 2017-03-01 15:52 UTC (permalink / raw) To: Patrick Ohly; +Cc: Patches and discussions about the oe-core layer [-- Attachment #1: Type: text/plain, Size: 5264 bytes --] On Thu, Feb 16, 2017 at 11:26:54AM +0100, Patrick Ohly wrote: > On Wed, 2017-02-15 at 19:32 +0100, Martin Jansa wrote: > > Are all changes necessary for this to work already in master? > > Yes. > > > Yesterday I've noticed that rm_work for some components which are > > early in the dependency (like qtbase) are executed relatively late > > (together with do_package_qa). > > Could do_rm_work run before do_package_qa? rm_work.bbclass doesn't know > that, and therefore schedules do_rm_work after do_package_qa. > > If yes, then adding a list of tasks that can be ignored would be > trivial. This can be a variable, so a recipe can even add their own > ones, if necessary. That's now what I've meant. I believe that rm_work needs to be executed after do_package_qa, but I don't understand the scheduler code enough (at least not yet) to say that higher priority of rm_work task also makes all the tasks rm_work depends on e.g. do_package_qa to be executed sooner. From my observation it looked like do_package_qa is still executed "late", but immediately followed by rm_work thanks to its high priority (so it's executed as soon as it can, but it's still late in progress of the whole build). Another interesting test from today was to run: # rm -rf tmp-glibc/* # bitbake -n zlib | tee log.zlib.rm_work # cd oe-core; git revert -1 936179754c8d0f98e1196ddc6796fdfd72c0c3b4; cd .. # rm -rf tmp-glibc/* # bitbake -n zlib | tee log.zlib.rm_work.revert and it shows interesting difference that many rm_work tasks aren't executed at all: # grep rm_work log.zlib.rm_work* | grep zlib_ log.zlib.rm_work:NOTE: Running task 526 of 527 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work) log.zlib.rm_work.revert:NOTE: Running task 128 of 721 (virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work) log.zlib.rm_work.revert:NOTE: Running task 717 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work) log.zlib.rm_work.revert:NOTE: Running task 721 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work_all) # grep rm_work log.zlib.rm_work* | grep gcc log.zlib.rm_work.revert:NOTE: Running task 2 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-source_6.3.bb:do_rm_work) log.zlib.rm_work.revert:NOTE: Running task 240 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-cross-initial_6.3.bb:do_rm_work) log.zlib.rm_work.revert:NOTE: Running task 250 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/libgcc-initial_6.3.bb:do_rm_work) log.zlib.rm_work.revert:NOTE: Running task 634 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-cross_6.3.bb:do_rm_work) log.zlib.rm_work.revert:NOTE: Running task 674 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/libgcc_6.3.bb:do_rm_work) log.zlib.rm_work.revert:NOTE: Running task 678 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-runtime_6.3.bb:do_rm_work) # grep -c rm_work log.zlib.rm_work* log.zlib.rm_work:1 log.zlib.rm_work.revert:63 I'll check if it's something in my setup or if this happens everywhere now. > > So I've tried very naive way to find out if the rm_work tasks are > > executed sooner or not just by comparing Task IDs in build of the same > > image built from scratch (without sstate) with Dizzy, Morty and > > current master. > > Interesting, I hadn't thought of testing it like that. > > > If we dismiss the strange case in rm_work.tasks.master.qemux86 then it > > seems to perform at least as good as old completion BB_SCHEDULER. > > > > > > But I wanted to ask if there is something else we can do or you were > > planing to do, because IIRC you shared some longer analysis of what > > could be improved here and I'm not sure if you managed to implement it > > all. > > The other ideas that I mentioned at some point didn't pan out as > intended. In particular allowing do_rm_work tasks to run when the normal > task limit was reached didn't have a big effect and the implementation > was a hack, so I dropped that. > > > It feels to me that rm_work has high priority, but still it's > > "blocked" by e.g. do_package_qa which gets executed late and then > > immediately followed by rm_work. > > That should be easy to change, perhaps like this (untested): > > RM_WORK_TASKS_WHITELIST = "do_build do_package_qa" > > deps = set(bb.build.preceedtask('do_build', True, d)) > whitelist = d.getVar('RM_WORK_TASKS_WHITELIST').split() > deps.difference_update(whitelist) > # In practice, addtask() here merely updates the dependencies. > bb.build.addtask('do_rm_work', 'do_build', ' '.join(deps), d) > > > -- > Best Regards, Patrick Ohly > > The content of this message is my personal opinion only and although > I am an employee of Intel, the statements I make here in no way > represent Intel's position on the issue, nor am I authorized to speak > on behalf of Intel on this matter. > > > -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 201 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 3/3] rm_work.bbclass: clean up sooner 2017-03-01 15:52 ` Martin Jansa @ 2017-03-01 16:44 ` Patrick Ohly 2017-03-01 18:47 ` Martin Jansa 2017-03-02 9:38 ` Patrick Ohly 1 sibling, 1 reply; 18+ messages in thread From: Patrick Ohly @ 2017-03-01 16:44 UTC (permalink / raw) To: Martin Jansa; +Cc: Patches and discussions about the oe-core layer On Wed, 2017-03-01 at 16:52 +0100, Martin Jansa wrote: > On Thu, Feb 16, 2017 at 11:26:54AM +0100, Patrick Ohly wrote: > > On Wed, 2017-02-15 at 19:32 +0100, Martin Jansa wrote: > > > Are all changes necessary for this to work already in master? > > > > Yes. > > > > > Yesterday I've noticed that rm_work for some components which are > > > early in the dependency (like qtbase) are executed relatively late > > > (together with do_package_qa). > > > > Could do_rm_work run before do_package_qa? rm_work.bbclass doesn't know > > that, and therefore schedules do_rm_work after do_package_qa. > > > > If yes, then adding a list of tasks that can be ignored would be > > trivial. This can be a variable, so a recipe can even add their own > > ones, if necessary. > > That's now what I've meant. > > I believe that rm_work needs to be executed after do_package_qa, but I > don't understand the scheduler code enough (at least not yet) to say > that higher priority of rm_work task also makes all the tasks rm_work > depends on e.g. do_package_qa to be executed sooner. I see. do_package_qa should have a high priority, but it's still blocked by its dependencies. Building those dependencies only gets an indirect boost from scheduling recipes that many other recipes depend on first (a feature of the normal scheduler). I suspect the problem with do_package_qa is similar to the problem with do_build before: it depends on do_package_qa of everything a recipe depends on, and thus finishing the build of those other recipes also blocks finishing the current recipe. That creates very deep dependency chains and thus increases the number of recipes which are "in progress" at the same time. > Another interesting test from today was to run: > # rm -rf tmp-glibc/* > # bitbake -n zlib | tee log.zlib.rm_work > # cd oe-core; git revert -1 936179754c8d0f98e1196ddc6796fdfd72c0c3b4; cd .. > # rm -rf tmp-glibc/* > # bitbake -n zlib | tee log.zlib.rm_work.revert > > and it shows interesting difference that many rm_work tasks aren't > executed at all: > > # grep rm_work log.zlib.rm_work* | grep zlib_ > log.zlib.rm_work:NOTE: Running task 526 of 527 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work) > log.zlib.rm_work.revert:NOTE: Running task 128 of 721 (virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work) > log.zlib.rm_work.revert:NOTE: Running task 717 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work) > log.zlib.rm_work.revert:NOTE: Running task 721 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work_all) > > # grep rm_work log.zlib.rm_work* | grep gcc > log.zlib.rm_work.revert:NOTE: Running task 2 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-source_6.3.bb:do_rm_work) > log.zlib.rm_work.revert:NOTE: Running task 240 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-cross-initial_6.3.bb:do_rm_work) > log.zlib.rm_work.revert:NOTE: Running task 250 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/libgcc-initial_6.3.bb:do_rm_work) > log.zlib.rm_work.revert:NOTE: Running task 634 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-cross_6.3.bb:do_rm_work) > log.zlib.rm_work.revert:NOTE: Running task 674 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/libgcc_6.3.bb:do_rm_work) > log.zlib.rm_work.revert:NOTE: Running task 678 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-runtime_6.3.bb:do_rm_work) > > # grep -c rm_work log.zlib.rm_work* > log.zlib.rm_work:1 > log.zlib.rm_work.revert:63 > > I'll check if it's something in my setup or if this happens everywhere now. I'll try to double-check this myself, too. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 3/3] rm_work.bbclass: clean up sooner 2017-03-01 16:44 ` Patrick Ohly @ 2017-03-01 18:47 ` Martin Jansa 2017-03-02 10:13 ` Patrick Ohly 0 siblings, 1 reply; 18+ messages in thread From: Martin Jansa @ 2017-03-01 18:47 UTC (permalink / raw) To: Patrick Ohly; +Cc: Patches and discussions about the oe-core layer [-- Attachment #1: Type: text/plain, Size: 15225 bytes --] On Wed, Mar 01, 2017 at 05:44:53PM +0100, Patrick Ohly wrote: > On Wed, 2017-03-01 at 16:52 +0100, Martin Jansa wrote: > > On Thu, Feb 16, 2017 at 11:26:54AM +0100, Patrick Ohly wrote: > > > On Wed, 2017-02-15 at 19:32 +0100, Martin Jansa wrote: > > > > Are all changes necessary for this to work already in master? > > > > > > Yes. > > > > > > > Yesterday I've noticed that rm_work for some components which are > > > > early in the dependency (like qtbase) are executed relatively late > > > > (together with do_package_qa). > > > > > > Could do_rm_work run before do_package_qa? rm_work.bbclass doesn't know > > > that, and therefore schedules do_rm_work after do_package_qa. > > > > > > If yes, then adding a list of tasks that can be ignored would be > > > trivial. This can be a variable, so a recipe can even add their own > > > ones, if necessary. > > > > That's now what I've meant. > > > > I believe that rm_work needs to be executed after do_package_qa, but I > > don't understand the scheduler code enough (at least not yet) to say > > that higher priority of rm_work task also makes all the tasks rm_work > > depends on e.g. do_package_qa to be executed sooner. > > I see. do_package_qa should have a high priority, but it's still blocked > by its dependencies. Building those dependencies only gets an indirect > boost from scheduling recipes that many other recipes depend on first (a > feature of the normal scheduler). > > I suspect the problem with do_package_qa is similar to the problem with > do_build before: it depends on do_package_qa of everything a recipe > depends on, and thus finishing the build of those other recipes also > blocks finishing the current recipe. That creates very deep dependency > chains and thus increases the number of recipes which are "in progress" > at the same time. Another (surprising for me) output from bitbake -D -D -D: OE qemux86@ ~/build/oe-core $ head log.zlib.rm_work.debug3.prio.* ==> log.zlib.rm_work.debug3.prio.completion <== DEBUG: completion priorities (most important first): 1. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_build 2. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work 3. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_package_write_ipk 4. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/expat/expat_2.2.0.bb:do_package_write_ipk 5. ID /OE/build/oe-core/openembedded-core/meta/recipes-kernel/linux-libc-headers/linux-libc-headers_4.9.bb:do_package_write_ipk 6. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/libtool/libtool-cross_2.4.6.bb:do_package_write_ipk 7. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/make/make_4.2.1.bb:do_package_write_ipk 8. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/gettext/gettext_0.19.8.1.bb:do_package_write_ipk 9. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/libgcc_6.3.bb:do_package_write_ipk ==> log.zlib.rm_work.debug3.prio.normal <== DEBUG: original priorities (most important first): 1. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_fetch 2. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_unpack 3. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_prepare_recipe_sysroot 4. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_patch 5. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_configure 6. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_compile 7. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_install 8. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_populate_sysroot 9. ID /OE/build/oe-core/openembedded-core/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb:do_fetch From these 2 I would expect the later one to be completion (to finish whole quilt-native before starting anything else except when all quilt-native tasks are blocked waiting for dependencies and BB_NUMBER_THREADS allows to run more task). But I understand that this isn't the actual order of tasks as explained in bitbake comment: # Group tasks of the same kind before tasks of less important # kinds at the head of the queue (because earlier = lower # priority number = runs earlier), while preserving the # ordering by recipe. If recipe foo is more important than # bar, then the goal is to work on foo's do_populate_sysroot # before bar's do_populate_sysroot and on the more important # tasks of foo before any of the less important tasks in any # other recipe (if those other recipes are more important than # foo). # # All of this only applies when tasks are runable. Explicit # dependencies still override this ordering by priority. # # Here's an example why this priority re-ordering helps with # minimizing disk usage. Consider a recipe foo with a higher # priority than bar where foo DEPENDS on bar. Then the # implicit rule (from base.bbclass) is that foo's do_configure # depends on bar's do_populate_sysroot. This ensures that # bar's do_populate_sysroot gets done first. Normally the # tasks from foo would continue to run once that is done, and # bar only gets completed and cleaned up later. By ordering # bar's task that depend on bar's do_populate_sysroot before foo's # do_configure, that problem gets avoided. and the later one still doesn't list rm_worl task for quilt-native which is the real one I'm interested in. When I do the same with last rm_work.bbclass change reverted I get this: $ head -n 12 log.zlib.rm_work.revert.debug3.prio.* ==> log.zlib.rm_work.revert.debug3.prio.completion <== DEBUG: completion priorities (most important first): 1. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work_all 2. ID /OE/build/oe-core/openembedded-core/meta/recipes-support/ptest-runner/ptest-runner_2.0.bb:do_rm_work 3. ID virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-support/libpcre/libpcre_8.40.bb:do_rm_work 4. ID virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-graphics/xorg-util/util-macros_1.19.0.bb:do_rm_work 5. ID virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-extended/gperf/gperf_3.0.4.bb:do_rm_work 6. ID virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-support/attr/attr_2.4.47.bb:do_rm_work 7. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/python/python-native_2.7.12.bb:do_rm_work 8. ID virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work 9. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/expat/expat_2.2.0.bb:do_rm_work 10. ID virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gnu-config/gnu-config_git.bb:do_rm_work 11. ID virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-extended/pigz/pigz_2.3.4.bb:do_rm_work ==> log.zlib.rm_work.revert.debug3.prio.normal <== DEBUG: original priorities (most important first): 1. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_fetch 2. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_unpack 3. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_patch 4. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_prepare_recipe_sysroot 5. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_configure 6. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_compile 7. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_install 8. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_populate_sysroot 9. ID /OE/build/oe-core/openembedded-core/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb:do_fetch 10. ID virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gnu-config/gnu-config_git.bb:do_fetch 11. ID /OE/build/oe-core/openembedded-core/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb:do_unpack Here it doesn't look so good for quilt-native:rm_work, it's included in zlib dependencies (thanks to the revert), but with "speed" scheduler it's far, while "completion" puts it early: log.zlib.rm_work.revert.debug3.prio.completion:40. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_rm_work log.zlib.rm_work.revert.debug3.prio.completion:66. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_build log.zlib.rm_work.revert.debug3.prio.completion:159. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_populate_lic log.zlib.rm_work.revert.debug3.prio.completion:229. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_populate_sysroot log.zlib.rm_work.revert.debug3.prio.completion:299. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_install log.zlib.rm_work.revert.debug3.prio.completion:361. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_compile log.zlib.rm_work.revert.debug3.prio.completion:423. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_configure log.zlib.rm_work.revert.debug3.prio.completion:484. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_prepare_recipe_sysroot log.zlib.rm_work.revert.debug3.prio.completion:546. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_patch log.zlib.rm_work.revert.debug3.prio.completion:603. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_unpack log.zlib.rm_work.revert.debug3.prio.completion:660. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_fetch log.zlib.rm_work.revert.debug3.prio.normal:1. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_fetch log.zlib.rm_work.revert.debug3.prio.normal:2. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_unpack log.zlib.rm_work.revert.debug3.prio.normal:3. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_patch log.zlib.rm_work.revert.debug3.prio.normal:4. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_prepare_recipe_sysroot log.zlib.rm_work.revert.debug3.prio.normal:5. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_configure log.zlib.rm_work.revert.debug3.prio.normal:6. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_compile log.zlib.rm_work.revert.debug3.prio.normal:7. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_install log.zlib.rm_work.revert.debug3.prio.normal:8. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_populate_sysroot log.zlib.rm_work.revert.debug3.prio.normal:561. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_populate_lic log.zlib.rm_work.revert.debug3.prio.normal:599. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_build log.zlib.rm_work.revert.debug3.prio.normal:697. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_rm_work From this it looks like setting the weight of do_rm_work above do_populate_sysroot would work with "speed" scheduler the way I was hoping for. I'll retry the same on even smaller build and with BB_NUMBER_THREADS = "1" so that it's easier to grasp what the completion scheduler really tries to do and how it influences the actual order of runqueue in the end. > > > Another interesting test from today was to run: > > # rm -rf tmp-glibc/* > > # bitbake -n zlib | tee log.zlib.rm_work > > # cd oe-core; git revert -1 936179754c8d0f98e1196ddc6796fdfd72c0c3b4; cd .. > > # rm -rf tmp-glibc/* > > # bitbake -n zlib | tee log.zlib.rm_work.revert > > > > and it shows interesting difference that many rm_work tasks aren't > > executed at all: > > > > # grep rm_work log.zlib.rm_work* | grep zlib_ > > log.zlib.rm_work:NOTE: Running task 526 of 527 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work) > > log.zlib.rm_work.revert:NOTE: Running task 128 of 721 (virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work) > > log.zlib.rm_work.revert:NOTE: Running task 717 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work) > > log.zlib.rm_work.revert:NOTE: Running task 721 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work_all) > > > > # grep rm_work log.zlib.rm_work* | grep gcc > > log.zlib.rm_work.revert:NOTE: Running task 2 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-source_6.3.bb:do_rm_work) > > log.zlib.rm_work.revert:NOTE: Running task 240 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-cross-initial_6.3.bb:do_rm_work) > > log.zlib.rm_work.revert:NOTE: Running task 250 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/libgcc-initial_6.3.bb:do_rm_work) > > log.zlib.rm_work.revert:NOTE: Running task 634 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-cross_6.3.bb:do_rm_work) > > log.zlib.rm_work.revert:NOTE: Running task 674 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/libgcc_6.3.bb:do_rm_work) > > log.zlib.rm_work.revert:NOTE: Running task 678 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-runtime_6.3.bb:do_rm_work) > > > > # grep -c rm_work log.zlib.rm_work* > > log.zlib.rm_work:1 > > log.zlib.rm_work.revert:63 > > > > I'll check if it's something in my setup or if this happens everywhere now. > > I'll try to double-check this myself, too. > > -- > Best Regards, Patrick Ohly > > The content of this message is my personal opinion only and although > I am an employee of Intel, the statements I make here in no way > represent Intel's position on the issue, nor am I authorized to speak > on behalf of Intel on this matter. > > > -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 201 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 3/3] rm_work.bbclass: clean up sooner 2017-03-01 18:47 ` Martin Jansa @ 2017-03-02 10:13 ` Patrick Ohly 0 siblings, 0 replies; 18+ messages in thread From: Patrick Ohly @ 2017-03-02 10:13 UTC (permalink / raw) To: Martin Jansa; +Cc: Patches and discussions about the oe-core layer On Wed, 2017-03-01 at 19:47 +0100, Martin Jansa wrote: > Another (surprising for me) output from bitbake -D -D -D: > > OE qemux86@ ~/build/oe-core $ head log.zlib.rm_work.debug3.prio.* > ==> log.zlib.rm_work.debug3.prio.completion <== > DEBUG: completion priorities (most important first): > 1. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_build > 2. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work > 3. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_package_write_ipk > 4. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/expat/expat_2.2.0.bb:do_package_write_ipk > 5. ID /OE/build/oe-core/openembedded-core/meta/recipes-kernel/linux-libc-headers/linux-libc-headers_4.9.bb:do_package_write_ipk > 6. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/libtool/libtool-cross_2.4.6.bb:do_package_write_ipk > 7. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/make/make_4.2.1.bb:do_package_write_ipk > 8. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/gettext/gettext_0.19.8.1.bb:do_package_write_ipk > 9. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/libgcc_6.3.bb:do_package_write_ipk > > ==> log.zlib.rm_work.debug3.prio.normal <== > DEBUG: original priorities (most important first): > 1. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_fetch > 2. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_unpack > 3. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_prepare_recipe_sysroot > 4. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_patch > 5. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_configure > 6. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_compile > 7. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_install > 8. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_populate_sysroot > 9. ID /OE/build/oe-core/openembedded-core/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb:do_fetch > > From these 2 I would expect the later one to be completion (to finish whole quilt-native before starting anything else > except when all quilt-native tasks are blocked waiting for dependencies and BB_NUMBER_THREADS allows to run > more task). I've not fully understood what the surprise was. I suppose log.zlib.rm_work.debug3.prio.normal was generated with BB_SCHEDULER="speed" and log.zlib.rm_work.debug3.prio.completion with BB_SCHEDULER="completion". For speed, quilt-native comes first because it's the recipe that most recipes depend on. For completion, it's not near the top because it has none of "important" tasks. It should be the first recipe for do_populate_sysroot. The patch from my other mail should activate quilt-native_0.65.bb:do_rm_work and then I'd expect it also at the top for completion. > But I understand that this isn't the actual order of tasks as explained in bitbake comment Exactly. > and the later one still doesn't list rm_worl task for quilt-native which is the real > one I'm interested in. That's the problem with recursive rm_work. > When I do the same with last rm_work.bbclass change reverted I get this: > $ head -n 12 log.zlib.rm_work.revert.debug3.prio.* > ==> log.zlib.rm_work.revert.debug3.prio.completion <== [...] > Here it doesn't look so good for quilt-native:rm_work, it's included in zlib dependencies (thanks to the revert), but > with "speed" scheduler it's far, while "completion" puts it early: [...] Here's how it looks with the patch: # bitbake -n -D -D -D zlib | tee log.zlib.debug3.rm_work.patched # less log.zlib.debug3.rm_work.patched ... DEBUG: merged task list: ['do_fetch', 'do_unpack', 'do_patch', 'do_preconfigure', 'do_prepare_recipe_sysroot', 'do_configure', 'do_configure_ptest_base', 'do_compile', 'do_compile_ptest_base', 'do_gcc_stash_builddir', 'do_install', 'do_multilib_install', 'do_extra_symlinks', 'do_install_ptest_base', 'do_stash_locale', 'do_poststash_install_cleanup', 'do_package', 'do_packagedata', 'do_package_write_ipk', 'do_package_qa', 'do_populate_sysroot', 'do_populate_lic', 'do_rm_work', 'do_rm_work_all', 'do_build'] DEBUG: original priorities (most important first): 1. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_fetch 2. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_unpack 3. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_patch 4. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_prepare_recipe_sysroot 5. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_configure 6. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_compile 7. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_install 8. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_populate_sysroot 9. ID /work/iot-ref-kit/openembedded-core/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb:do_fetch ... 597. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-core/gettext/gettext_0.19.8.1.bb:do_populate_lic 598. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-devtools/flex/flex_2.6.2.bb:do_rm_work 599. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-core/ncurses/ncurses_6.0+20161126.bb:do_rm_work 600. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-graphics/xorg-proto/xproto_7.0.31.bb:do_rm_work 601. ID /work/iot-ref-kit/openembedded-core/meta/recipes-core/glibc/glibc_2.25.bb:do_rm_work 602. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-support/popt/popt_1.16.bb:do_rm_work 603. ID /work/iot-ref-kit/openembedded-core/meta/recipes-core/systemd/systemd-systemctl-native.bb:do_rm_work 604. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-extended/gperf/gperf_3.0.4.bb:do_rm_work 605. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/gcc/gcc-cross-initial_6.3.bb:do_rm_work 606. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/libtool/libtool-native_2.4.6.bb:do_rm_work 607. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-devtools/ossp-uuid/ossp-uuid_1.6.2.bb:do_rm_work 608. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-graphics/xorg-util/util-macros_1.19.1.bb:do_rm_work 609. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-support/attr/acl_2.2.52.bb:do_rm_work 610. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work 611. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_rm_work ... DEBUG: completion priorities (most important first): 1. ID /work/iot-ref-kit/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_build 2. ID /work/iot-ref-kit/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work_all 3. ID /work/iot-ref-kit/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work 4. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-devtools/flex/flex_2.6.2.bb:do_rm_work 5. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-core/ncurses/ncurses_6.0+20161126.bb:do_rm_work 6. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-graphics/xorg-proto/xproto_7.0.31.bb:do_rm_work 7. ID /work/iot-ref-kit/openembedded-core/meta/recipes-core/glibc/glibc_2.25.bb:do_rm_work 8. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-support/popt/popt_1.16.bb:do_rm_work 9. ID /work/iot-ref-kit/openembedded-core/meta/recipes-core/systemd/systemd-systemctl-native.bb:do_rm_work 10. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-extended/gperf/gperf_3.0.4.bb:do_rm_work 11. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/gcc/gcc-cross-initial_6.3.bb:do_rm_work 12. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/libtool/libtool-native_2.4.6.bb:do_rm_work 13. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-devtools/ossp-uuid/ossp-uuid_1.6.2.bb:do_rm_work 14. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-graphics/xorg-util/util-macros_1.19.1.bb:do_rm_work 15. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-support/attr/acl_2.2.52.bb:do_rm_work 16. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work 17. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_rm_work 18. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb:do_rm_work ... That the speed scheduler (aka "original priorities" above) only adds quilt-native_0.65.bb:do_rm_work at the end is a bit unexpected, I thought it added all tasks of a recipe in sequence. As a result of that, the reordered completion priorities do not have quilt-native_0.65.bb:do_rm_work quite as high as it would be otherwise. I don't think it matters in practice, though. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 3/3] rm_work.bbclass: clean up sooner 2017-03-01 15:52 ` Martin Jansa 2017-03-01 16:44 ` Patrick Ohly @ 2017-03-02 9:38 ` Patrick Ohly 1 sibling, 0 replies; 18+ messages in thread From: Patrick Ohly @ 2017-03-02 9:38 UTC (permalink / raw) To: Martin Jansa; +Cc: Patches and discussions about the oe-core layer On Wed, 2017-03-01 at 16:52 +0100, Martin Jansa wrote: > On Thu, Feb 16, 2017 at 11:26:54AM +0100, Patrick Ohly wrote: > > On Wed, 2017-02-15 at 19:32 +0100, Martin Jansa wrote: > > > Are all changes necessary for this to work already in master? > > > > Yes. > > > > > Yesterday I've noticed that rm_work for some components which are > > > early in the dependency (like qtbase) are executed relatively late > > > (together with do_package_qa). > > > > Could do_rm_work run before do_package_qa? rm_work.bbclass doesn't know > > that, and therefore schedules do_rm_work after do_package_qa. > > > > If yes, then adding a list of tasks that can be ignored would be > > trivial. This can be a variable, so a recipe can even add their own > > ones, if necessary. > > That's now what I've meant. > > I believe that rm_work needs to be executed after do_package_qa, but I > don't understand the scheduler code enough (at least not yet) to say > that higher priority of rm_work task also makes all the tasks rm_work > depends on e.g. do_package_qa to be executed sooner. > > From my observation it looked like do_package_qa is still executed > "late", but immediately followed by rm_work thanks to its high priority > (so it's executed as soon as it can, but it's still late in progress of > the whole build). > > Another interesting test from today was to run: > # rm -rf tmp-glibc/* > # bitbake -n zlib | tee log.zlib.rm_work > # cd oe-core; git revert -1 936179754c8d0f98e1196ddc6796fdfd72c0c3b4; cd .. > # rm -rf tmp-glibc/* > # bitbake -n zlib | tee log.zlib.rm_work.revert > > and it shows interesting difference that many rm_work tasks aren't > executed at all: > > # grep rm_work log.zlib.rm_work* | grep zlib_ > log.zlib.rm_work:NOTE: Running task 526 of 527 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work) > log.zlib.rm_work.revert:NOTE: Running task 128 of 721 (virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work) > log.zlib.rm_work.revert:NOTE: Running task 717 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work) > log.zlib.rm_work.revert:NOTE: Running task 721 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work_all) > > # grep rm_work log.zlib.rm_work* | grep gcc > log.zlib.rm_work.revert:NOTE: Running task 2 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-source_6.3.bb:do_rm_work) > log.zlib.rm_work.revert:NOTE: Running task 240 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-cross-initial_6.3.bb:do_rm_work) > log.zlib.rm_work.revert:NOTE: Running task 250 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/libgcc-initial_6.3.bb:do_rm_work) > log.zlib.rm_work.revert:NOTE: Running task 634 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-cross_6.3.bb:do_rm_work) > log.zlib.rm_work.revert:NOTE: Running task 674 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/libgcc_6.3.bb:do_rm_work) > log.zlib.rm_work.revert:NOTE: Running task 678 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-runtime_6.3.bb:do_rm_work) I can reproduce that. The root cause for the current behavior is that zlib:do_build does not depend on zlib-native:do_build and thus does not enable zlib-native:do_rm_work: # bitbake -g zlib # grep -e '->.*zlib-native' task-depends.dot | grep -v '^.zlib-native' "binutils-cross-x86_64.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot" "rpm-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot" "gcc-cross-initial-x86_64.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot" "binutils-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot" "gcc-cross-x86_64.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot" "file-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot" "pigz-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot" "elfutils-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot" "python-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot" "libpcre-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot" Only tasks that are required by the current build target(s) are executed. From that perspective, the current behavior makes sense (although it is of course not desirable). More interesting is why it worked before. The answer is this: # grep rm_work_all log.zlib.rm_work.revert NOTE: Running task 721 of 721 (/work/iot-ref-kit/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work_all) # grep -e '->.*zlib-native' task-depends.dot | grep -v '^.zlib-native' "elfutils-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot" "binutils-cross-x86_64.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot" "pigz-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot" "gcc-cross-x86_64.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot" "binutils-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot" "gcc-cross-initial-x86_64.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot" "python-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot" "rpm-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot" "zlib.do_rm_work_all" -> "zlib-native.do_rm_work" "libpcre-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot" "file-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot" In other words, do_rm_work_all (the default task for "bitbake zlib" in the previous approach) was responsible for triggering do_rm_work in everything that zlib depended upon. A side effect of that was that do_rm_work itself also caused more work to be done than strictly necessary for the build: # grep Running log.zlib.rm_work | sed 's/ [0-9]* of [0-9]*//' | grep -v rm_work | grep -v 'noexec task' | wc -l 54 # grep Running log.zlib.rm_work.revert | sed 's/ [0-9]* of [0-9]*//' | grep -v rm_work | grep -v 'noexec task' | wc -l 156 $ diff <(grep Running log.zlib.rm_work | sed 's/ [0-9]* of [0-9]*//' | grep -v rm_work | grep -v 'noexec task' | sort) <(grep Running log.zlib.rm_work.revert | sed 's/ [0-9]* of [0-9]*//' | grep -v rm_work | grep -v 'noexec task' | sort) 0a1,11 > NOTE: Running setscene task (virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-connectivity/openssl/openssl_1.0.2k.bb:do_populate_lic_setscene) > NOTE: Running setscene task (virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-connectivity/openssl/openssl_1.0.2k.bb:do_populate_sysroot_setscene) > NOTE: Running setscene task (virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-core/expat/expat_2.2.0.bb:do_populate_lic_setscene) > NOTE: Running setscene task (virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-core/expat/expat_2.2.0.bb:do_populate_sysroot_setscene) ... > NOTE: Running setscene task (/work/iot-ref-kit/openembedded-core/meta/recipes-core/glibc/glibc_2.25.bb:do_package_qa_setscene) 20a94 > NOTE: Running setscene task (/work/iot-ref-kit/openembedded-core/meta/recipes-core/glibc/glibc_2.25.bb:do_populate_lic_setscene) 22a97 > NOTE: Running setscene task (/work/iot-ref-kit/openembedded-core/meta/recipes-core/glibc/glibc-initial_2.25.bb:do_populate_lic_setscene) ... > NOTE: Running setscene task (/work/iot-ref-kit/openembedded-core/meta/recipes-support/ptest-runner/ptest-runner_2.0.bb:do_populate_sysroot_setscene) When rm_work.bbclass is not active, the old code did not execute these tasks for "bitbake zlib". Ideally, enabling rm_work.bbclass should trigger the recursive do_rm_work (as before) but without triggering extra work (unlike before). I just can't think of a way to do that - not with the kind of dependencies we have in bitbake at the moment. We would have to distinguish between "strong" dependencies (if A is active and depends on B, then B must be executed first) and "weak" ordering (if A and B both must run and A must run before B, then execute A first). Given that most tasks triggered by do_rm_work on top of do_populate_sysroot seem to be fairly light, enabling them in addition to do_rm_work itself is probably better. Here's a tentative patch achieving that: # git diff diff --git a/meta/classes/rm_work.bbclass b/meta/classes/rm_work.bbclass index 054c93716e..eeb83d386a 100644 --- a/meta/classes/rm_work.bbclass +++ b/meta/classes/rm_work.bbclass @@ -102,6 +102,14 @@ do_rm_work () { rm -f $i done } +do_rm_work_all () { + : +} +do_rm_work_all[recrdeptask] = "do_rm_work do_rm_work_all" +do_rm_work_all[noexec] = "1" +addtask rm_work_all after before do_build + +do_rm_work_all[recrdeptask] = "do_rm_work" do_populate_sdk[postfuncs] += "rm_work_populatesdk" rm_work_populatesdk () { @@ -138,9 +146,8 @@ python inject_rm_work() { # do_build inherits additional runtime dependencies on # other recipes and thus will typically run much later than completion of # work in the recipe itself. - deps = bb.build.preceedtask('do_build', True, d) - if 'do_build' in deps: - deps.remove('do_build') + deps = set(bb.build.preceedtask('do_build', True, d)) + deps.difference_update(('do_build', 'do_rm_work_all')) # In practice, addtask() here merely updates the dependencies. bb.build.addtask('do_rm_work', 'do_build', ' '.join(deps), d) } # bitbake -n zlib | tee log.zlib.rm_work.patched $ diff <(grep Running log.zlib.rm_work.patched | sed 's/ [0-9]* of [0-9]*//' | grep -v 'noexec task' | sort) <(grep Running log.zlib.rm_work.revert | sed 's/ [0-9]* of [0-9]*//' | grep -v 'noexec task' | sort) 202a203 > NOTE: Running task (/work/iot-ref-kit/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work_all) The only difference compared to before is because of the slight do_rm_work_all[noexec] tweak. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ^ permalink raw reply related [flat|nested] 18+ messages in thread
end of thread, other threads:[~2017-03-02 10:13 UTC | newest] Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-01-06 9:55 [PATCH 0/3] rm_work enhancements Patrick Ohly 2017-01-06 9:55 ` [PATCH 1/3] gcc-source.inc: cleanly disable do_rm_work Patrick Ohly 2017-01-09 18:47 ` Khem Raj 2017-01-06 9:55 ` [PATCH 2/3] rm_work_and_downloads.bbclass: more aggressively minimize disk usage Patrick Ohly 2017-01-06 18:31 ` Randy Witt 2017-01-06 19:22 ` Patrick Ohly 2017-01-06 21:29 ` Randy Witt 2017-01-07 8:09 ` Patrick Ohly 2017-01-09 10:25 ` Patrick Ohly 2017-01-09 18:17 ` Randy Witt 2017-01-06 9:55 ` [PATCH 3/3] rm_work.bbclass: clean up sooner Patrick Ohly 2017-02-15 18:32 ` Martin Jansa 2017-02-16 10:26 ` Patrick Ohly 2017-03-01 15:52 ` Martin Jansa 2017-03-01 16:44 ` Patrick Ohly 2017-03-01 18:47 ` Martin Jansa 2017-03-02 10:13 ` Patrick Ohly 2017-03-02 9:38 ` Patrick Ohly
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.