From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Greylist: delayed 439 seconds by postgrey-1.34 at layers.openembedded.org; Fri, 06 Jan 2017 18:39:30 UTC Received: from p3plsmtpa06-01.prod.phx3.secureserver.net (p3plsmtpa06-01.prod.phx3.secureserver.net [173.201.192.102]) by mail.openembedded.org (Postfix) with ESMTP id A11CE60125 for ; Fri, 6 Jan 2017 18:39:30 +0000 (UTC) Received: from mail-lf0-f52.google.com ([209.85.215.52]) by :SMTPAUTH: with SMTP id PZIicH9UBWbEfPZIjc6JgD; Fri, 06 Jan 2017 11:31:41 -0700 Received: by mail-lf0-f52.google.com with SMTP id o140so4073595lff.3 for ; Fri, 06 Jan 2017 10:31:40 -0800 (PST) X-Gm-Message-State: AIkVDXKyzg+BkKzVrb6NS1ejJgi6ona7DHjE+SkCpR4tOLZwFvJgi3eljkRQPojSUKbmkqM/sTRXY39sVdqu6w== X-Received: by 10.46.77.17 with SMTP id a17mr2942621ljb.2.1483727499945; Fri, 06 Jan 2017 10:31:39 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.223.84 with HTTP; Fri, 6 Jan 2017 10:31:09 -0800 (PST) In-Reply-To: <8002ec38c29d685bd369fd1d2382298ca33f7b68.1483696378.git.patrick.ohly@intel.com> References: <8002ec38c29d685bd369fd1d2382298ca33f7b68.1483696378.git.patrick.ohly@intel.com> From: Randy Witt Date: Fri, 6 Jan 2017 10:31:09 -0800 X-Gmail-Original-Message-ID: Message-ID: To: Patrick Ohly X-CMAE-Envelope: MS4wfBepeGVSWdWp6YunoTUklEGH4PBUhlVrd1cN0xUS3J7il0CT0ZfJJ0wNulnBa51AX4BXcqLjORHOiPMQ2PZbc+O0EqtpPasDpvYfb8tIRRk+rN/W1rj/ tvZJYDhmTkI9SU4Q5FMjFU+QLxYzmrOHBVgkKtfqFvoqbKGRV34OnfVtmhYbQaD7HU6L+DX41DWHbwNj5v/DEoUL7yabKfzFiEc= Cc: OE-core Subject: Re: [PATCH 2/3] rm_work_and_downloads.bbclass: more aggressively minimize disk usage X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Jan 2017 18:39:30 -0000 Content-Type: multipart/alternative; boundary=94eb2c1abbc493ee0a0545713ae6 --94eb2c1abbc493ee0a0545713ae6 Content-Type: text/plain; charset=UTF-8 Hi Patrick, On Fri, Jan 6, 2017 at 1:55 AM, Patrick Ohly wrote: > rm_work.bbclass never deletes downloaded files, even if they are not > going to be needed again during the > build. rm_work_and_downloads.bbclass is more aggressive in minimizing > the used disk space during a build, but has other disadvantages: > - sources required by different recipes need to be fetched once per > recipe, not once per build > - incremental builds do not work reliably because sources get > removed without ensuring that sources gets fetched again > > That makes rm_work_and_downloads.bbclass useful for one-time builds in > a constrained environment (like a CI system), but not for general use. > > I'd rather see this be a new class independent from rm_work. People can always in inherit both rm_work and rm_download. That being said, since this doesn't operate at the fetch level, i.e. only remove things that were fetched as part of the current invocation of bitbake, I don't see why the user couldn't just "rm -rf downloads" themselves as part of the ci teardown. > Signed-off-by: Patrick Ohly > --- > meta/classes/rm_work_and_downloads.bbclass | 33 > ++++++++++++++++++++++++++++++ > 1 file changed, 33 insertions(+) > create mode 100644 meta/classes/rm_work_and_downloads.bbclass > > diff --git a/meta/classes/rm_work_and_downloads.bbclass > b/meta/classes/rm_work_and_downloads.bbclass > new file mode 100644 > index 0000000..7c00bea > --- /dev/null > +++ b/meta/classes/rm_work_and_downloads.bbclass > @@ -0,0 +1,33 @@ > +# Author: Patrick Ohly > +# Copyright: Copyright (C) 2015 Intel Corporation > +# > +# This file is licensed under the MIT license, see COPYING.MIT in > +# this source distribution for the terms. > + > +# This class is used like rm_work: > +# INHERIT += "rm_work_and_downloads" > +# > +# In addition to removing local build directories of a recipe, it also > +# removes the downloaded source. This is achieved by making the DL_DIR > +# recipe-specific. While reducing disk usage, it increases network usage > (for > +# example, compiling the same source for target and host implies > downloading > +# the source twice). > +# > +# Because the "do_fetch" task does not get re-run after removing the > downloaded > +# sources, this class is also not suitable for incremental builds. > +# > +# Where it works well is in well-connected build environments with limited > +# disk space (like TravisCI). > + > +inherit rm_work > + > +# This would ensure that the existing do_rm_work() removes the downloads, > +# but does not work because some recipes have a circular dependency > between > +# WORKDIR and DL_DIR (via ${SRCPV}?). > +# DL_DIR = "${WORKDIR}/downloads" > + > +# Instead go up one level and remove ourself. > +DL_DIR = "${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}/${PN}/downloads" > +do_rm_work_append () { > + rm -rf ${DL_DIR} > +} > -- > 2.1.4 > > -- > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core > --94eb2c1abbc493ee0a0545713ae6 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Patrick,

On Fri, Jan 6, 2017 at 1:55 AM, Patrick Ohly <patric= k.ohly@intel.com> wrote:
rm= _work.bbclass never deletes downloaded files, even if they are not
going to be needed again during the
build. rm_work_and_downloads.bbclass is more aggressive in minimizing
the used disk space during a build, but has other disadvantages:
- sources required by different recipes need to be fetched once per
=C2=A0 recipe, not once per build
- incremental builds do not work reliably because sources get
=C2=A0 removed without ensuring that sources gets fetched again

That makes rm_work_and_downloads.bbclass useful for one-time builds in
a constrained environment (like a CI system), but not for general use.


I'd rather see this be a new class= independent from rm_work. People can always in inherit both rm_work and rm= _download.

That being said, since this doesn't= operate at the fetch level, i.e. only remove things that were fetched as p= art of the current invocation of bitbake, I don't see why the user coul= dn't just "rm -rf downloads" themselves as part of the ci tea= rdown.
=C2=A0
Signed-off-by: Patrick Ohly <p= atrick.ohly@intel.com>
---
=C2=A0meta/classes/rm_work_and_downloads.bbclass | 33 ++++++++++++++++= ++++++++++++++
=C2=A01 file changed, 33 insertions(+)
=C2=A0create mode 100644 meta/classes/rm_work_and_downloads.bbclass
diff --git a/meta/classes/rm_work_and_downloads.bbclass b/meta/classes= /rm_work_and_downloads.bbclass
new file mode 100644
index 0000000..7c00bea
--- /dev/null
+++ b/meta/classes/rm_work_and_downloads.bbclass
@@ -0,0 +1,33 @@
+# Author:=C2=A0 =C2=A0 =C2=A0 =C2=A0Patrick Ohly <patrick.ohly@intel.com>
+# Copyright:=C2=A0 =C2=A0 Copyright (C) 2015 Intel Corporation
+#
+# This file is licensed under the MIT license, see COPYING.MIT in
+# this source distribution for the terms.
+
+# This class is used like rm_work:
+# INHERIT +=3D "rm_work_and_downloads"
+#
+# In addition to removing local build directories of a recipe, it also
+# removes the downloaded source. This is achieved by making the DL_DIR
+# recipe-specific. While reducing disk usage, it increases network usage (= for
+# example, compiling the same source for target and host implies downloadi= ng
+# the source twice).
+#
+# Because the "do_fetch" task does not get re-run after removing= the downloaded
+# sources, this class is also not suitable for incremental builds.
+#
+# Where it works well is in well-connected build environments with limited=
+# disk space (like TravisCI).
+
+inherit rm_work
+
+# This would ensure that the existing do_rm_work() removes the downloads,<= br> +# but does not work because some recipes have a circular dependency betwee= n
+# WORKDIR and DL_DIR (via ${SRCPV}?).
+# DL_DIR =3D "${WORKDIR}/downloads"
+
+# Instead go up one level and remove ourself.
+DL_DIR =3D "${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}/${PN}/downlo= ads"
+do_rm_work_append () {
+=C2=A0 =C2=A0 rm -rf ${DL_DIR}
+}
--
2.1.4

--
_______________________________________________
Openembedded-core mailing list
Openembedded-co= re@lists.openembedded.org
http://lists.openembedded.org/m= ailman/listinfo/openembedded-core

--94eb2c1abbc493ee0a0545713ae6--