On Wed, May 19, 2021 at 12:47 PM Konrad Weihmann <kweihmann@outlook.com> wrote:


On 19.05.21 12:23, Martin Jansa wrote:
> * all known issues in this area were fixed, make it fatal that it
>    cannot be overlooked if someone triggers this issue again
>
> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
>   meta/lib/oe/sstatesig.py | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
> index 6cd6e11acc..e86a09b332 100644
> --- a/meta/lib/oe/sstatesig.py
> +++ b/meta/lib/oe/sstatesig.py
> @@ -453,7 +453,7 @@ def find_sstate_manifest(taskdata, taskdata2, taskname, d, multilibcache):
>           manifest = d2.expand("${SSTATE_MANIFESTS}/manifest-%s-%s.%s" % (pkgarch, taskdata, taskname))
>           if os.path.exists(manifest):
>               return manifest, d2
> -    bb.error("Manifest %s not found in %s (variant '%s')?" % (manifest, d2.expand(" ".join(pkgarchs)), variant))
> +    bb.fatal("Manifest %s not found in %s (variant '%s')?" % (manifest, d2.expand(" ".join(pkgarchs)), variant))
>       return None, d2

doesn't that make the return statement obsolete? also how can an error
be overlooked? It does make the task triggering this issue fail, so how
is a fatal better in this case (besides abnormally terminating the task)

bb.error doesn't make the task to fail, it only shows an error (which results in non-zero exit code), but lets depending tasks to continue and fail with other (often more misleading) errors, because their dependency is really missing in RSS

without this patch:
ERROR: adwaita-icon-theme-3.34.3-r0 do_prepare_recipe_sysroot: Manifest /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
ERROR: adwaita-icon-theme-3.34.3-r0 do_prepare_recipe_sysroot: Manifest /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
ERROR: adwaita-icon-theme-3.34.3-r0 do_configure: Manifest /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
ERROR: adwaita-icon-theme-3.34.3-r0 do_install: Manifest /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
ERROR: adwaita-icon-theme-3.34.3-r0 do_package: Manifest /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
ERROR: adwaita-icon-theme-3.34.3-r0 do_package_write_ipk: Manifest /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
NOTE: Tasks Summary: Attempted 2819 tasks of which 2803 didn't need to be rerun and all succeeded.
Summary: There were 6 ERROR messages shown, returning a non-zero exit code.

with this patch:
ERROR: adwaita-icon-theme-3.34.3-r0 do_prepare_recipe_sysroot: Manifest /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
ERROR: Logfile of failure stored in: /OE/build/oe-core/tmp-glibc/work/all-oe-linux/adwaita-icon-theme/3.34.3-r0/temp/log.do_prepare_recipe_sysroot.1638704
ERROR: Task (/OE/build/oe-core/openembedded-core/meta/recipes-gnome/gnome/adwaita-icon-theme_3.34.3.bb:do_prepare_recipe_sysroot) failed with exit code '1'
NOTE: Tasks Summary: Attempted 2805 tasks of which 2800 didn't need to be rerun and 1 failed.
Summary: 1 task failed:
  /OE/build/oe-core/openembedded-core/meta/recipes-gnome/gnome/adwaita-icon-theme_3.34.3.bb:do_prepare_recipe_sysroot
Summary: There was 1 ERROR message shown, returning a non-zero exit code.

Without this fix:
https://git.openembedded.org/openembedded-core/commit/?id=63da9a4f889c5b0e41bc8ec08abe0acea1546479

I was seeing various native dependencies missing and as you can see in the adwaita-icon-theme example, missing at-spi2-core-native isn't fatal for the build, but were adwaita-icon-theme packages and sstate built correctly? If you don't investigate all these ERROR messages and non-zero exit code (e.g. just because you have thousands of builds per day populating the same sstate mirror), you might miss some of these breakages. While using fatal prevents adwaita-icon-theme to even start building and producing possibly broken packages, images, sstate.

In some other cases e.g. qtwayland-native missing when building target qtwayland you will get do_compile failure complaining that the the binary from qtwayland-native is missing, but the root cause of this issue is written many lines above do_compile log. You can see some other examples in https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/master&id=817f1c14cec4d66504a5028908a4ebac147f3b52 and before changing this from bb.warn to bb.error in previous patch most of these issues were just "some random builddir breakage which was resolved by doing clean build from sstate".

And this happens in multi-machine builds with dunfell/gatesgarth/hardknott and only master is safe now. I plan to ask Steve to backport this and related fixes to dunfell after a bit more time in master:
https://git.openembedded.org/openembedded-core/commit/?id=0eb95212a7e6b6bdc1243094072dea432cb48f0f
https://git.openembedded.org/openembedded-core/commit/?id=63da9a4f889c5b0e41bc8ec08abe0acea1546479
https://git.openembedded.org/openembedded-core/commit/?id=95607a26854d873399d2b9d7e5fcbffc0cbdba4c
and then this one.

Regards,