[PATCH] sstatesig.py: make it fatal error when sstate manifest isn't found

[PATCH] sstatesig.py: make it fatal error when sstate manifest isn't found

[Martin Jansa]

* all known issues in this area were fixed, make it fatal that it
  cannot be overlooked if someone triggers this issue again

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/lib/oe/sstatesig.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
index 6cd6e11acc..e86a09b332 100644
--- a/meta/lib/oe/sstatesig.py
+++ b/meta/lib/oe/sstatesig.py
@@ -453,7 +453,7 @@ def find_sstate_manifest(taskdata, taskdata2, taskname, d, multilibcache):
         manifest = d2.expand("${SSTATE_MANIFESTS}/manifest-%s-%s.%s" % (pkgarch, taskdata, taskname))
         if os.path.exists(manifest):
             return manifest, d2
-    bb.error("Manifest %s not found in %s (variant '%s')?" % (manifest, d2.expand(" ".join(pkgarchs)), variant))
+    bb.fatal("Manifest %s not found in %s (variant '%s')?" % (manifest, d2.expand(" ".join(pkgarchs)), variant))
     return None, d2
 
 def OEOuthashBasic(path, sigfile, task, d):
-- 
2.30.2

Re: [OE-core] [PATCH] sstatesig.py: make it fatal error when sstate manifest isn't found

[Konrad Weihmann]

On 19.05.21 12:23, Martin Jansa wrote:
> * all known issues in this area were fixed, make it fatal that it
>    cannot be overlooked if someone triggers this issue again
> 
> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
>   meta/lib/oe/sstatesig.py | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
> index 6cd6e11acc..e86a09b332 100644
> --- a/meta/lib/oe/sstatesig.py
> +++ b/meta/lib/oe/sstatesig.py
> @@ -453,7 +453,7 @@ def find_sstate_manifest(taskdata, taskdata2, taskname, d, multilibcache):
>           manifest = d2.expand("${SSTATE_MANIFESTS}/manifest-%s-%s.%s" % (pkgarch, taskdata, taskname))
>           if os.path.exists(manifest):
>               return manifest, d2
> -    bb.error("Manifest %s not found in %s (variant '%s')?" % (manifest, d2.expand(" ".join(pkgarchs)), variant))
> +    bb.fatal("Manifest %s not found in %s (variant '%s')?" % (manifest, d2.expand(" ".join(pkgarchs)), variant))
>       return None, d2

doesn't that make the return statement obsolete? also how can an error 
be overlooked? It does make the task triggering this issue fail, so how 
is a fatal better in this case (besides abnormally terminating the task)

>   
>   def OEOuthashBasic(path, sigfile, task, d):
> 
> 
> 
> 
> 

Re: [OE-core] [PATCH] sstatesig.py: make it fatal error when sstate manifest isn't found

[Richard Purdie]

On Wed, 2021-05-19 at 12:47 +0200, Konrad Weihmann wrote:
> 
> On 19.05.21 12:23, Martin Jansa wrote:
> > * all known issues in this area were fixed, make it fatal that it
> >    cannot be overlooked if someone triggers this issue again
> > 
> > Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> > ---
> >   meta/lib/oe/sstatesig.py | 2 +-
> >   1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
> > index 6cd6e11acc..e86a09b332 100644
> > --- a/meta/lib/oe/sstatesig.py
> > +++ b/meta/lib/oe/sstatesig.py
> > @@ -453,7 +453,7 @@ def find_sstate_manifest(taskdata, taskdata2, taskname, d, multilibcache):
> >           manifest = d2.expand("${SSTATE_MANIFESTS}/manifest-%s-%s.%s" % (pkgarch, taskdata, taskname))
> >           if os.path.exists(manifest):
> >               return manifest, d2
> > -    bb.error("Manifest %s not found in %s (variant '%s')?" % (manifest, d2.expand(" ".join(pkgarchs)), variant))
> > +    bb.fatal("Manifest %s not found in %s (variant '%s')?" % (manifest, d2.expand(" ".join(pkgarchs)), variant))
> >       return None, d2
> 
> doesn't that make the return statement obsolete? also how can an error 
> be overlooked? It does make the task triggering this issue fail, so how 
> is a fatal better in this case (besides abnormally terminating the task)

This will fail the task rather than just setting the exit code so it should highlight there 
is a real problem to the user.

Cheers,

Richard

Re: [OE-core] [PATCH] sstatesig.py: make it fatal error when sstate manifest isn't found

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 5971 bytes --]

On Wed, May 19, 2021 at 12:47 PM Konrad Weihmann <kweihmann@outlook.com>
wrote:

>
>
> On 19.05.21 12:23, Martin Jansa wrote:
> > * all known issues in this area were fixed, make it fatal that it
> >    cannot be overlooked if someone triggers this issue again
> >
> > Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> > ---
> >   meta/lib/oe/sstatesig.py | 2 +-
> >   1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
> > index 6cd6e11acc..e86a09b332 100644
> > --- a/meta/lib/oe/sstatesig.py
> > +++ b/meta/lib/oe/sstatesig.py
> > @@ -453,7 +453,7 @@ def find_sstate_manifest(taskdata, taskdata2,
> taskname, d, multilibcache):
> >           manifest = d2.expand("${SSTATE_MANIFESTS}/manifest-%s-%s.%s" %
> (pkgarch, taskdata, taskname))
> >           if os.path.exists(manifest):
> >               return manifest, d2
> > -    bb.error("Manifest %s not found in %s (variant '%s')?" % (manifest,
> d2.expand(" ".join(pkgarchs)), variant))
> > +    bb.fatal("Manifest %s not found in %s (variant '%s')?" % (manifest,
> d2.expand(" ".join(pkgarchs)), variant))
> >       return None, d2
>
> doesn't that make the return statement obsolete? also how can an error
> be overlooked? It does make the task triggering this issue fail, so how
> is a fatal better in this case (besides abnormally terminating the task)
>

bb.error doesn't make the task to fail, it only shows an error (which
results in non-zero exit code), but lets depending tasks to continue and
fail with other (often more misleading) errors, because their dependency is
really missing in RSS

without this patch:
ERROR: adwaita-icon-theme-3.34.3-r0 do_prepare_recipe_sysroot: Manifest
/OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot
not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
ERROR: adwaita-icon-theme-3.34.3-r0 do_prepare_recipe_sysroot: Manifest
/OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot
not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
ERROR: adwaita-icon-theme-3.34.3-r0 do_configure: Manifest
/OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot
not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
ERROR: adwaita-icon-theme-3.34.3-r0 do_install: Manifest
/OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot
not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
ERROR: adwaita-icon-theme-3.34.3-r0 do_package: Manifest
/OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot
not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
ERROR: adwaita-icon-theme-3.34.3-r0 do_package_write_ipk: Manifest
/OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot
not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
NOTE: Tasks Summary: Attempted 2819 tasks of which 2803 didn't need to be
rerun and all succeeded.
Summary: There were 6 ERROR messages shown, returning a non-zero exit code.

with this patch:
ERROR: adwaita-icon-theme-3.34.3-r0 do_prepare_recipe_sysroot: Manifest
/OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot
not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
ERROR: Logfile of failure stored in:
/OE/build/oe-core/tmp-glibc/work/all-oe-linux/adwaita-icon-theme/3.34.3-r0/temp/log.do_prepare_recipe_sysroot.1638704
ERROR: Task
(/OE/build/oe-core/openembedded-core/meta/recipes-gnome/gnome/adwaita-icon-theme_3.34.3.bb:do_prepare_recipe_sysroot)
failed with exit code '1'
NOTE: Tasks Summary: Attempted 2805 tasks of which 2800 didn't need to be
rerun and 1 failed.
Summary: 1 task failed:

/OE/build/oe-core/openembedded-core/meta/recipes-gnome/gnome/adwaita-icon-theme_3.34.3.bb:
do_prepare_recipe_sysroot
Summary: There was 1 ERROR message shown, returning a non-zero exit code.

Without this fix:
https://git.openembedded.org/openembedded-core/commit/?id=63da9a4f889c5b0e41bc8ec08abe0acea1546479

I was seeing various native dependencies missing and as you can see in
the adwaita-icon-theme example, missing at-spi2-core-native isn't fatal for
the build, but were adwaita-icon-theme packages and sstate built correctly?
If you don't investigate all these ERROR messages and non-zero exit code
(e.g. just because you have thousands of builds per day populating the same
sstate mirror), you might miss some of these breakages. While using fatal
prevents adwaita-icon-theme to even start building and producing possibly
broken packages, images, sstate.

In some other cases e.g. qtwayland-native missing when building target
qtwayland you will get do_compile failure complaining that the the binary
from qtwayland-native is missing, but the root cause of this issue is
written many lines above do_compile log. You can see some other examples in
https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/master&id=817f1c14cec4d66504a5028908a4ebac147f3b52
and before changing this from bb.warn to bb.error in previous patch most of
these issues were just "some random builddir breakage which was resolved by
doing clean build from sstate".

And this happens in multi-machine builds with dunfell/gatesgarth/hardknott
and only master is safe now. I plan to ask Steve to backport this and
related fixes to dunfell after a bit more time in master:
https://git.openembedded.org/openembedded-core/commit/?id=0eb95212a7e6b6bdc1243094072dea432cb48f0f
https://git.openembedded.org/openembedded-core/commit/?id=63da9a4f889c5b0e41bc8ec08abe0acea1546479
https://git.openembedded.org/openembedded-core/commit/?id=95607a26854d873399d2b9d7e5fcbffc0cbdba4c
and then this one.

Regards,

[-- Attachment #2: Type: text/html, Size: 7853 bytes --]

Re: [OE-core] [PATCH] sstatesig.py: make it fatal error when sstate manifest isn't found

[Konrad Weihmann]

On 19.05.21 17:02, Martin Jansa wrote:
> 
> 
> On Wed, May 19, 2021 at 12:47 PM Konrad Weihmann <kweihmann@outlook.com 
> <mailto:kweihmann@outlook.com>> wrote:
> 
> 
> 
>     On 19.05.21 12:23, Martin Jansa wrote:
>      > * all known issues in this area were fixed, make it fatal that it
>      >    cannot be overlooked if someone triggers this issue again
>      >
>      > Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com
>     <mailto:Martin.Jansa@gmail.com>>
>      > ---
>      >   meta/lib/oe/sstatesig.py | 2 +-
>      >   1 file changed, 1 insertion(+), 1 deletion(-)
>      >
>      > diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
>      > index 6cd6e11acc..e86a09b332 100644
>      > --- a/meta/lib/oe/sstatesig.py
>      > +++ b/meta/lib/oe/sstatesig.py
>      > @@ -453,7 +453,7 @@ def find_sstate_manifest(taskdata, taskdata2,
>     taskname, d, multilibcache):
>      >           manifest =
>     d2.expand("${SSTATE_MANIFESTS}/manifest-%s-%s.%s" % (pkgarch,
>     taskdata, taskname))
>      >           if os.path.exists(manifest):
>      >               return manifest, d2
>      > -    bb.error("Manifest %s not found in %s (variant '%s')?" %
>     (manifest, d2.expand(" ".join(pkgarchs)), variant))
>      > +    bb.fatal("Manifest %s not found in %s (variant '%s')?" %
>     (manifest, d2.expand(" ".join(pkgarchs)), variant))
>      >       return None, d2
> 
>     doesn't that make the return statement obsolete? also how can an error
>     be overlooked? It does make the task triggering this issue fail, so how
>     is a fatal better in this case (besides abnormally terminating the task)
> 
> 
> bb.error doesn't make the task to fail, it only shows an error (which 
> results in non-zero exit code), but lets depending tasks to continue and 
> fail with other (often more misleading) errors, because their dependency 
> is really missing in RSS
> 
> without this patch:
> ERROR: adwaita-icon-theme-3.34.3-r0 do_prepare_recipe_sysroot: Manifest 
> /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot 
> not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
> ERROR: adwaita-icon-theme-3.34.3-r0 do_prepare_recipe_sysroot: Manifest 
> /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot 
> not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
> ERROR: adwaita-icon-theme-3.34.3-r0 do_configure: Manifest 
> /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot 
> not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
> ERROR: adwaita-icon-theme-3.34.3-r0 do_install: Manifest 
> /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot 
> not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
> ERROR: adwaita-icon-theme-3.34.3-r0 do_package: Manifest 
> /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot 
> not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
> ERROR: adwaita-icon-theme-3.34.3-r0 do_package_write_ipk: Manifest 
> /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot 
> not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
> NOTE: Tasks Summary: Attempted 2819 tasks of which 2803 didn't need to 
> be rerun and all succeeded.
> Summary: There were 6 ERROR messages shown, returning a non-zero exit code.
> 
> with this patch:
> ERROR: adwaita-icon-theme-3.34.3-r0 do_prepare_recipe_sysroot: Manifest 
> /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot 
> not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
> ERROR: Logfile of failure stored in: 
> /OE/build/oe-core/tmp-glibc/work/all-oe-linux/adwaita-icon-theme/3.34.3-r0/temp/log.do_prepare_recipe_sysroot.1638704
> ERROR: Task 
> (/OE/build/oe-core/openembedded-core/meta/recipes-gnome/gnome/adwaita-icon-theme_3.34.3.bb:do_prepare_recipe_sysroot) 
> failed with exit code '1'
> NOTE: Tasks Summary: Attempted 2805 tasks of which 2800 didn't need to 
> be rerun and 1 failed.
> Summary: 1 task failed:
>    
> /OE/build/oe-core/openembedded-core/meta/recipes-gnome/gnome/adwaita-icon-theme_3.34.3.bb:do_prepare_recipe_sysroot
> Summary: There was 1 ERROR message shown, returning a non-zero exit code.
> 
> Without this fix:
> https://git.openembedded.org/openembedded-core/commit/?id=63da9a4f889c5b0e41bc8ec08abe0acea1546479 
> <https://git.openembedded.org/openembedded-core/commit/?id=63da9a4f889c5b0e41bc8ec08abe0acea1546479>
> 
> I was seeing various native dependencies missing and as you can see in 
> the adwaita-icon-theme example, missing at-spi2-core-native isn't fatal 
> for the build, but were adwaita-icon-theme packages and sstate built 
> correctly? If you don't investigate all these ERROR messages and 
> non-zero exit code (e.g. just because you have thousands of builds per 
> day populating the same sstate mirror), you might miss some of these 
> breakages. While using fatal prevents adwaita-icon-theme to even start 
> building and producing possibly broken packages, images, sstate.
> 
> In some other cases e.g. qtwayland-native missing when building target 
> qtwayland you will get do_compile failure complaining that the the 
> binary from qtwayland-native is missing, but the root cause of this 
> issue is written many lines above do_compile log. You can see some other 
> examples in 
> https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/master&id=817f1c14cec4d66504a5028908a4ebac147f3b52 
> <https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/master&id=817f1c14cec4d66504a5028908a4ebac147f3b52> 
> and before changing this from bb.warn to bb.error in previous patch most 
> of these issues were just "some random builddir breakage which was 
> resolved by doing clean build from sstate".
> 
> And this happens in multi-machine builds with 
> dunfell/gatesgarth/hardknott and only master is safe now. I plan to ask 
> Steve to backport this and related fixes to dunfell after a bit more 
> time in master:
> https://git.openembedded.org/openembedded-core/commit/?id=0eb95212a7e6b6bdc1243094072dea432cb48f0f 
> <https://git.openembedded.org/openembedded-core/commit/?id=0eb95212a7e6b6bdc1243094072dea432cb48f0f>
> https://git.openembedded.org/openembedded-core/commit/?id=63da9a4f889c5b0e41bc8ec08abe0acea1546479 
> <https://git.openembedded.org/openembedded-core/commit/?id=63da9a4f889c5b0e41bc8ec08abe0acea1546479>
> https://git.openembedded.org/openembedded-core/commit/?id=95607a26854d873399d2b9d7e5fcbffc0cbdba4c 
> <https://git.openembedded.org/openembedded-core/commit/?id=95607a26854d873399d2b9d7e5fcbffc0cbdba4c>
> and then this one.

Thanks for the detailed explanation, if it reduces the noise (incl false 
positives) I'm all for it

> 
> Regards,

Re: [OE-core] [PATCH] sstatesig.py: make it fatal error when sstate manifest isn't found

[Andreas Müller]

On Wed, May 19, 2021 at 6:46 PM Konrad Weihmann <kweihmann@outlook.com> wrote:
If I am not mistaken this breaks wide areas of meta-freescale +
meta-qt5-extra. I have seen the manifest warning using meta-freescale
for long time (and many licensed not found in do_rootfs) but now
builds fail.
Most recipes failing are native recipes inheriting meta-qt5's
cmake_qt5-bbcalss  - have many of them in meta-qt5-extra.
To analyse I did:

[superandy@thanks-buchmuellers oe-core]$ bitbake -e
extra-cmake-modules-native | grep ^DEPENDS
DEPENDS="qtbase qtbase-native cmake-native  qtbase-native
qtbase-native cmake-native qttools-native python3-native
python3-native  ninja-native"

As soon as I remove (target) qtbase from DEPENDS, build is fine.
Have the strong feeling that the dynamic-packagearch magic [1] is the
culprit confusing a native recipe finding target recipe's manifest if
that recipe is marked as SO-SPECIFIC  - as qtbase [2] is.

Wrote here because I hope the audience is wider - will open an issue
at meta-freescale tomorrow after zzz...

Cheers,

Andreas

[1] https://github.com/Freescale/meta-freescale/blob/master/classes/fsl-dynamic-packagearch.bbclass
[2] https://github.com/Freescale/meta-freescale/blob/733e1683ce8895a960c01a3ee1fce40118d82982/conf/machine/include/imx-base.inc#L246

Re: [OE-core] [PATCH] sstatesig.py: make it fatal error when sstate manifest isn't found

[Andreas Müller]

On Mon, May 31, 2021 at 3:25 AM Andreas Müller via
lists.openembedded.org
<schnitzeltony=gmail.com@lists.openembedded.org> wrote:
>
> On Wed, May 19, 2021 at 6:46 PM Konrad Weihmann <kweihmann@outlook.com> wrote:
> If I am not mistaken this breaks wide areas of meta-freescale +
> meta-qt5-extra. I have seen the manifest warning using meta-freescale
> for long time (and many licensed not found in do_rootfs) but now
> builds fail.
> Most recipes failing are native recipes inheriting meta-qt5's
> cmake_qt5-bbcalss  - have many of them in meta-qt5-extra.
> To analyse I did:
>
> [superandy@thanks-buchmuellers oe-core]$ bitbake -e
> extra-cmake-modules-native | grep ^DEPENDS
> DEPENDS="qtbase qtbase-native cmake-native  qtbase-native
> qtbase-native cmake-native qttools-native python3-native
> python3-native  ninja-native"
>
> As soon as I remove (target) qtbase from DEPENDS, build is fine.
> Have the strong feeling that the dynamic-packagearch magic [1] is the
> culprit confusing a native recipe finding target recipe's manifest if
> that recipe is marked as SO-SPECIFIC  - as qtbase [2] is.
>
> Wrote here because I hope the audience is wider - will open an issue
> at meta-freescale tomorrow after zzz...
see https://github.com/Freescale/meta-freescale/issues/798
>
> Cheers,
>
> Andreas
>
> [1] https://github.com/Freescale/meta-freescale/blob/master/classes/fsl-dynamic-packagearch.bbclass
> [2] https://github.com/Freescale/meta-freescale/blob/733e1683ce8895a960c01a3ee1fce40118d82982/conf/machine/include/imx-base.inc#L246
>
> 
>