[PATCH] base: Rework and improve do_unpack WORKDIR cleanup from previous builds

[PATCH] base: Rework and improve do_unpack WORKDIR cleanup from previous builds

[Richard Purdie]

We have multiple bugs ocurring because when do_unpack reruns, it just
dumps the data base into WORKDIR. This means if a file was removed from
a directory in a file://subdir/ url, that file would be left in the
build confusing users.

Also if files are entirely removed from SRC_URI, they would still exist
in WORKDIR. Read the shadow recipe and the /etc/pam.d/ conditionals
for an example of how this can break in interesting ways.

To work around these issues, improve the cleaning code to remove anything
except temp/ since it contains logs of the currently running task and
*recipe-sysroot* since do_fetch and do_unpack have dependencies which may
be added there already. That directory is setup to look after itself.

We also have to create ${S} since that is a side effect of the current
[cleandirs] which is no longer needed after this change.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/base.bbclass | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 4c681cc870d..97b7e8feca6 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -166,9 +166,14 @@ python base_do_fetch() {
 addtask unpack after do_fetch
 do_unpack[dirs] = "${WORKDIR}"
 
-do_unpack[cleandirs] = "${@d.getVar('S') if os.path.normpath(d.getVar('S')) != os.path.normpath(d.getVar('WORKDIR')) else os.path.join('${S}', 'patches')}"
-
 python base_do_unpack() {
+    workdir = d.getVar("WORKDIR")
+    for p in os.listdir(workdir):
+        if p == "temp" or "recipe-sysroot" in p:
+            continue
+        bb.utils.remove(workdir + "/" + p, recurse=True)
+    bb.utils.mkdirhier(d.getVar("S"))
+
     src_uri = (d.getVar('SRC_URI') or "").split()
     if len(src_uri) == 0:
         return
-- 
2.25.1

Re: [OE-core] [PATCH] base: Rework and improve do_unpack WORKDIR cleanup from previous builds

[Otavio Salvador]

Em sex., 25 de set. de 2020 às 20:30, Richard Purdie
<richard.purdie@linuxfoundation.org> escreveu:
>
> We have multiple bugs ocurring because when do_unpack reruns, it just
> dumps the data base into WORKDIR. This means if a file was removed from
> a directory in a file://subdir/ url, that file would be left in the
> build confusing users.
>
> Also if files are entirely removed from SRC_URI, they would still exist
> in WORKDIR. Read the shadow recipe and the /etc/pam.d/ conditionals
> for an example of how this can break in interesting ways.
>
> To work around these issues, improve the cleaning code to remove anything
> except temp/ since it contains logs of the currently running task and
> *recipe-sysroot* since do_fetch and do_unpack have dependencies which may
> be added there already. That directory is setup to look after itself.
>
> We also have to create ${S} since that is a side effect of the current
> [cleandirs] which is no longer needed after this change.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Cool; this seems to address some annoying corner cases :-)


-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9 9981-7854          Mobile: +1 (347) 903-9750

Re: [OE-core] [PATCH] base: Rework and improve do_unpack WORKDIR cleanup from previous builds

[Richard Purdie]

On Fri, 2020-09-25 at 22:44 -0300, Otavio Salvador wrote:
> Em sex., 25 de set. de 2020 às 20:30, Richard Purdie
> <richard.purdie@linuxfoundation.org> escreveu:
> > We have multiple bugs ocurring because when do_unpack reruns, it
> > just
> > dumps the data base into WORKDIR. This means if a file was removed
> > from
> > a directory in a file://subdir/ url, that file would be left in the
> > build confusing users.
> > 
> > Also if files are entirely removed from SRC_URI, they would still
> > exist
> > in WORKDIR. Read the shadow recipe and the /etc/pam.d/ conditionals
> > for an example of how this can break in interesting ways.
> > 
> > To work around these issues, improve the cleaning code to remove
> > anything
> > except temp/ since it contains logs of the currently running task
> > and
> > *recipe-sysroot* since do_fetch and do_unpack have dependencies
> > which may
> > be added there already. That directory is setup to look after
> > itself.
> > 
> > We also have to create ${S} since that is a side effect of the
> > current
> > [cleandirs] which is no longer needed after this change.
> > 
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> 
> Cool; this seems to address some annoying corner cases :-)

Sadly testing shows we simply can't do this. I've tried adding all
kinds of tweaks in to make it work and there are just too many corner
cases.

For example I just saw a recipe on the autobuilder fail as it did this
(from log.task_order):

do_package_write_deb_setscene (22542): log.do_package_write_deb_setscene.22542
do_populate_lic_setscene (22563): log.do_populate_lic_setscene.22563
do_packagedata_setscene (22912): log.do_packagedata_setscene.22912
do_package_setscene (28295): log.do_package_setscene.28295
do_deploy_source_date_epoch_setscene (29226): log.do_deploy_source_date_epoch_setscene.29226
do_fetch (14380): log.do_fetch.14380
do_unpack (19794): log.do_unpack.19794
do_patch (23663): log.do_patch.23663
do_prepare_recipe_sysroot (48174): log.do_prepare_recipe_sysroot.48174
do_configure (21716): log.do_configure.21716
do_compile (2203): log.do_compile.2203
do_install (48352): log.do_install.48352
do_package_write_ipk (34415): log.do_package_write_ipk.34415
do_package_write_rpm (34825): log.do_package_write_rpm.34825
do_package_qa (6590): log.do_package_qa.6590
do_populate_sysroot (6894): log.do_populate_sysroot.6894

i.e. the do_unpack then wiped out the do_package_setscene and more.

Ultimately I think we may have to inject a directory, WORKDIR/sources/
where everything would be downloaded to. That will mean updating
WORKDIR references in recipes.

For 3.2 we may be able to unpack to a different directory
(WORKDIR/soruces?) then move things into position, *but* log a list of
what was done so we can delete it later if we rerun do_unpack.

That is rather ugly though.

Cheers,

Richard