sstate-cache issue with SPDX enabled

* sstate-cache issue with SPDX enabled
@ 2023-06-07 15:32 mwen
  2023-06-08 20:15 ` [yocto] " Richard Purdie
  0 siblings, 1 reply; 3+ messages in thread
From: mwen @ 2023-06-07 15:32 UTC (permalink / raw)
  To: yocto

[-- Attachment #1: Type: text/plain, Size: 11167 bytes --]

We’re using Poky Kirkstone(LTS) with the latest version for our SDK. Recently, we're trying to enable the feature of creating SBOM in SPDX. But we met a sstate-cache checking failure issue when enabling SPDX to generate SBOM. It will be appreciated if you have any idea for the root cause. Refer to below details for the issue.

https://docs.yoctoproject.org/4.0.10/singleindex.html#creating-a-software-bill-of-materials ( https://urldefense.com/v3/__https:/docs.yoctoproject.org/4.0.10/singleindex.html*creating-a-software-bill-of-materials__;Iw!!PeEy7nZLVv0!iEYzUu9NzMxHXrypN88tmMw1XE3I6xJIUHhsKfEvEXx92zX6oPews42VwnTYHjVn4vmJR2pk0CEn$ )

*==== Here is an easy way to reproduce the issue ====*

*1. pull cbuild-ng project*

jleng@shbuild8:/yocto/system-team/jleng$ git clone https://github.com/lengjingzju/cbuild-ng.git ( https://urldefense.com/v3/__https:/github.com/lengjingzju/cbuild-ng.git__;!!PeEy7nZLVv0!iEYzUu9NzMxHXrypN88tmMw1XE3I6xJIUHhsKfEvEXx92zX6oPews42VwnTYHjVn4vmJRzxUjhi5$ )

jleng@shbuild8:/yocto/system-team/jleng$ cd cbuild-ng

*2. Use the latest code of Yocto kirkstone and enable SBOM*

jleng@shbuild8:/yocto/system-team/jleng/cbuild-ng$ git diff

diff --git a/scripts/yocto.env b/ scripts/yocto.env

index ed8bb35..b7f1316 100755

--- a/scripts/yocto.env

+++ b/scripts/yocto.env

@@ -60,7 +60,7 @@ export ENV_TOP_OUT ENV_CFG_ROOT

POKY_URL=git:// git.yoctoproject.org/poky ( https://urldefense.com/v3/__http:/git.yoctoproject.org/poky__;!!PeEy7nZLVv0!iEYzUu9NzMxHXrypN88tmMw1XE3I6xJIUHhsKfEvEXx92zX6oPews42VwnTYHjVn4vmJR6uBUfNh$ )

POKY_BRANCH=kirkstone

-POKY_REV=30656ea54de82cabb818f07af711b96211421c95

+POKY_REV=1e6f147c686a91b9a5996f529e32ee1852f946b7

POKY_PATH=${ENV_TOP_OUT}/poky

if [ ! -e "${POKY_PATH}" ]; then

@@ -101,6 +101,9 @@ EXTERNALSRC_SYMLINKS = ""

ENV_TOP_DIR = "${ENV_TOP_DIR}"

ENV_CFG_ROOT = "${ENV_CFG_ROOT}"

ENV_BUILD_MODE = "${ENV_BUILD_MODE}"

+

+INHERIT += "create-spdx"

+SPDX_PRETTY = "1"

EOF

*3. First build, success*

jleng@shbuild8:/yocto/system-team/jleng/cbuild-ng$ source scripts/yocto.env

jleng@shbuild8:/yocto/system-team/jleng/cbuild-ng/build/qemux86-64$ bitbake kconfig-native

*4. Remove output and re-build, report an error*

jleng@shbuild8:/yocto/system-team/jleng/cbuild-ng/build/qemux86-64$ cd -

jleng@shbuild8:/yocto/system-team/jleng/cbuild-ng$ rm -rf build/qemux86-64

jleng@shbuild8:/yocto/system-team/jleng/cbuild-ng$ source scripts/yocto.env

jleng@shbuild8:/yocto/system-team/jleng/cbuild-ng/build/qemux86-64$ bitbake kconfig-native

/usr/lib/python3/dist-packages/html5lib/_trie/_base.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working

from collections import Mapping

WARNING: You are using a local hash equivalence server but have configured an sstate mirror. This will likely mean no sstate will match from the mirror. You may wish to disable the hash equivalence use (BB_HASHSERVE), or use a hash equivalence server alongside the sstate mirror.

Loading cache: 100% |                                                                                                                           | ETA:  --:--:--

Loaded 0 entries from dependency cache.

Parsing recipes: 100% |##########################################################################################################################| Time: 0:00:05

Parsing of 885 .bb files complete (0 cached, 885 parsed). 1647 targets, 44 skipped, 0 masked, 0 errors.

NOTE: Resolving any missing task queue dependencies

Build Configuration:

BB_VERSION           = "2.0.0"

BUILD_SYS            = "x86_64-linux"

NATIVELSBSTRING      = "ubuntu-20.04"

TARGET_SYS           = "x86_64-poky-linux"

MACHINE              = "qemux86-64"

DISTRO               = "poky"

DISTRO_VERSION       = "4.0.10"

TUNE_FEATURES        = "m64 core2"

TARGET_FPU           = ""

meta

meta-poky

meta-yocto-bsp       = "kirkstone:1e6f147c686a91b9a5996f529e32ee1852f946b7"

meta-cbuild          = "main:10f9581f393bf67cd06e8849bad409ba91d4d9ad"

Initialising tasks: 100% |#######################################################################################################################| Time: 0:00:00

Sstate summary: Wanted 37 Local 37 Mirrors 0 Missed 0 Current 0 (100% match, 0% complete)

NOTE: Executing Tasks

ERROR: kconfig-native-1.0-r0 do_create_spdx: Error executing a python function in exec_func_python() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:

File: 'exec_func_python() autogenerated', lineno: 2, function: <module>

0001:

*** 0002:do_create_spdx(d)

0003:

File: '/yocto/system-team/jleng/cbuild-ng/build/poky/meta/classes/create-spdx.bbclass', lineno: 521, function: do_create_spdx

0517:

0518:            if archive is not None:

0519:                recipe.packageFileName = str( recipe_archive.name ( https://urldefense.com/v3/__http:/recipe_archive.name__;!!PeEy7nZLVv0!iEYzUu9NzMxHXrypN88tmMw1XE3I6xJIUHhsKfEvEXx92zX6oPews42VwnTYHjVn4vmJRzKP95UT$ ) )

0520:

*** 0521:    dep_recipes = collect_dep_recipes(d, doc, recipe)

0522:

0523:    doc_sha1 = oe.sbom.write_doc(d, doc, "recipes", indent=get_json_indent(d))

0524:    dep_recipes.append(oe.sbom.DepRecipe(doc, doc_sha1, recipe))

0525:

File: '/yocto/system-team/jleng/cbuild-ng/build/poky/meta/classes/create-spdx.bbclass', lineno: 350, function: collect_dep_recipes

0346:    ))

0347:    for dep_pn in deps:

0348:        dep_recipe_path = deploy_dir_spdx / "recipes" / ("recipe-%s.spdx.json" % dep_pn)

0349:

*** 0350:        spdx_dep_doc, spdx_dep_sha1 = oe.sbom.read_doc(dep_recipe_path)

0351:

0352:        for pkg in spdx_dep_doc.packages:

0353:            if pkg.name ( https://urldefense.com/v3/__http:/pkg.name__;!!PeEy7nZLVv0!iEYzUu9NzMxHXrypN88tmMw1XE3I6xJIUHhsKfEvEXx92zX6oPews42VwnTYHjVn4vmJR1_uoEsE$ ) == dep_pn:

0354:                spdx_dep_recipe = pkg

File: '/yocto/system-team/jleng/cbuild-ng/build/poky/meta/lib/oe/sbom.py', lineno: 67, function: read_doc

0063:        else:

0064:            with fn.open("rb") as f:

0065:                yield f

0066:

*** 0067:    with get_file() as f:

0068:        sha1 = hashlib.sha1()

0069:        while True:

0070:            chunk = f.read(4096)

0071:            if not chunk:

File: '/usr/lib/python3.8/contextlib.py', lineno: 113, function: __enter__

0109:        # do not keep args and kwds alive unnecessarily

0110:        # they are only needed for recreation, which is not possible anymore

0111:        del self.args, self.kwds, self.func

0112:        try:

*** 0113:            return next(self.gen)

0114:        except StopIteration:

0115:            raise RuntimeError("generator didn't yield") from None

0116:

0117:    def __exit__(self, type, value, traceback):

File: '/yocto/system-team/jleng/cbuild-ng/build/poky/meta/lib/oe/sbom.py', lineno: 64, function: get_file

0060:    def get_file():

0061:        if isinstance(fn, io.IOBase):

0062:            yield fn

0063:        else:

*** 0064:            with fn.open("rb") as f:

0065:                yield f

0066:

0067:    with get_file() as f:

0068:        sha1 = hashlib.sha1()

File: '/usr/lib/python3.8/pathlib.py', lineno: 1222, function: open

1218:        the built-in open() function does.

1219:        """

1220:        if self._closed:

1221:            self._raise_closed()

*** 1222:        return io.open(self, mode, buffering, encoding, errors, newline,

1223:                       opener=self._opener)

1224:

1225:    def read_bytes(self):

1226:        """

File: '/usr/lib/python3.8/pathlib.py', lineno: 1078, function: _opener

1074:        raise ValueError("I/O operation on closed path")

1075:

1076:    def _opener(self, name, flags, mode=0o666):

1077:        # A stub for the opener argument to built-in open()

*** 1078:        return self._accessor.open(self, flags, mode)

1079:

1080:    def _raw_open(self, flags, mode=0o777):

1081:        """

1082:        Open the file pointed by this path and return a file descriptor,

Exception: FileNotFoundError: [Errno 2] No such file or directory: '/yocto/system-team/jleng/cbuild-ng/build/qemux86-64/tmp/deploy/spdx/qemux86-64/recipes/recipe-autoconf-native.spdx.json'

ERROR: Logfile of failure stored in: /yocto/system-team/jleng/cbuild-ng/build/qemux86-64/tmp/work/x86_64-linux/kconfig-native/1.0-r0/temp/log.do_create_spdx.1001451

ERROR: Task (virtual:native:/yocto/system-team/jleng/cbuild-ng/scripts/meta-cbuild/recipes-devtools/kconfig/kconfig.bb:do_create_spdx) failed with exit code '1'

NOTE: Tasks Summary: Attempted 133 tasks of which 128 didn't need to be rerun and 1 failed.

Summary: 1 task failed:

virtual:native:/yocto/system-team/jleng/cbuild-ng/scripts/meta-cbuild/recipes-devtools/kconfig/kconfig.bb:do_create_spdx

Summary: There was 1 WARNING message.

Summary: There was 1 ERROR message, returning a non-zero exit code.

jleng@shbuild8:/yocto/system-team/jleng/cbuild-ng/build/qemux86-64$ ls tmp/work/x86_64-linux/

bison-native  gettext-minimal-native  kconfig-native  m4-native       pkgconfig-native  texinfo-dummy-native

flex-native   gnu-config-native       libtool-native  ncurses-native  quilt-native      xz-native

*==== Currently, we found a workaround solution to fix this issue ====*

jleng@shbuild8:/yocto/system-team/jleng/cbuild-ng/build/poky$ git diff

diff --git a/meta/classes/sstate.bbclass b/ meta/classes/sstate.bbclass

index dd6cf12920..f47febeb32 100644

--- a/meta/classes/sstate.bbclass

+++ b/meta/classes/sstate.bbclass

@@ -1084,7 +1084,7 @@ def setscene_depvalid(task, taskdependees, notneeded, d, log=None):

logit("Considering setscene task: %s" % (str(taskdependees[task])), log)

-    directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx" , "do_deploy_archives"]

+    directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_deploy_archives"]

def isNativeCross(x):

return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross")

[-- Attachment #2: Type: text/html, Size: 40251 bytes --]

^ permalink raw reply	[flat|nested] 3+ messages in thread