[PATCH V2 0/3] bbclass/sstate: only allowed sstate-cache objects are allowed in a build (read-only sstate-cache)

[PATCH V2 0/3] bbclass/sstate: only allowed sstate-cache objects are allowed in a build (read-only sstate-cache)

[Hongxu Jia]

Changed in V2:

- add hook SSTATE_CHECK_FUNCTIONS for sstate checking, and
  add a new bbclass/sstate_readonly, rather than directly
  modify bbclass/sstate

- terminate build safely while RunQueueExecuteScenequeue
  init failed, the previous bb.msg.fatal is too hack.

Test steps:

1) vim local.conf
INHERIT += 'sstate_readonly'

2) Create a sstate cache for testing
$ bitbake db-native

3) Remove tmp dir and build db-native from sstate-cache
$ mv tmp tmp-back && bitbake db-native

4) Add gzip to SSTATECACHE_WHITELIST, it enabled read-only sstate-cache,
vim local.conf
...
SSTATECACHE_WHITELIST = 'gzip'
...

5) Remove tmp dir and build db-native from sstate-cache
$ mv tmp tmp-back && bitbake db-native

6) tweak db's do_configure task by adding comments
--- a/meta/recipes-support/db/db_6.0.30.bb
+++ b/meta/recipes-support/db/db_6.0.30.bb
@@ -82,6 +82,7 @@ do_configure() {
        gnu-configize --force ${S}
        export STRIP="true"
        oe_runconf
+       echo "hello"
 }


7) build db-native and there is a build failure
$ bitbake db-native
...
NOTE: Preparing runqueue
ERROR: Read-only sstate-cache is enabled, the build of 
"db-native"
did not come from sstate-cache. Only the recipe listed in
SSTATECACHE_WHITELIST is allowed to build from source
...

8) clean db-native failed
$ bitbake db-native -ccleansstate
...
ERROR: Read-only sstate-cache is enabled, the clean of 
db-native is not allowed. Only the recipe listed in
SSTATECACHE_WHITELIST is allowed to clean sstate-cache
...

$ bitbake db-native -ccleanall
...
ERROR: Read-only sstate-cache is enabled, the clean of 
db-native is not allowed. Only the recipe listed in
SSTATECACHE_WHITELIST is allowed to clean sstate-cache
...

9) Add db-native to SSTATECACHE_WHITELIST
vim local.conf
...
SSTATECACHE_WHITELIST = 'gzip db-native'
...

10) build/clean db-native succee
$ bitbake db-native
$ bitbake db-native -ccleansstate
$ bitbake db-native -ccleanall

//Hongxu


The following changes since commit 52b788c6df9201764130ab744bf67b770b24b896:

  autogen-native: inherit pkgconfig to fix a build failure (2014-08-25 10:26:00 +0100)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib hongxu/readonly-sstatecache
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/readonly-sstatecache

Hongxu Jia (3):
  bitbake: runqueue: terminate build safely while
    RunQueueExecuteScenequeue init failed
  bbclass/sstate: add hook SSTATE_CHECK_FUNCTIONS for sstate checking
  bbclass/sstate_readonly: only allowed sstate-cache objects are allowed
    in a build (read-only sstate-cache)

 bitbake/lib/bb/runqueue.py           |  7 ++++-
 meta/classes/sstate.bbclass          | 26 ++++++++++++++++++
 meta/classes/sstate_readonly.bbclass | 51 ++++++++++++++++++++++++++++++++++++
 3 files changed, 83 insertions(+), 1 deletion(-)
 create mode 100644 meta/classes/sstate_readonly.bbclass

-- 
1.9.1

[PATCH 1/3] bitbake: runqueue: terminate build safely while RunQueueExecuteScenequeue init failed

[Hongxu Jia]

While RunQueueExecuteScenequeue init failed, the exit of build is mess.
Here is the example while bb.fatal invoked in RunQueueExecuteScenequeue:
...
Traceback (most recent call last):
  File "/home/jiahongxu/yocto/poky/bitbake/lib/bb/runqueue.py", line 1824, in RunQueueExecuteScenequeue.__init__(rq=<bb.runqueue.RunQueue instance at 0x7f87dd7d5050>):
                 locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname, "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data }
    >            valid = bb.utils.better_eval(call, locs)

  File "/home/jiahongxu/yocto/poky/bitbake/lib/bb/utils.py", line 374, in better_eval(source='sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d)', locals={'sq_fn': ['virtual:native:/home/jiahongxu/yocto/poky/meta/recipes-support/db/db_6.0.30.bb', 'virtual:native:/home/jiahongxu/yocto/poky/meta/recipes-support/db/db_6.0.30.bb', 'virtual:native:/home/jiahongxu/yocto/poky/meta/recipes-support/db/db_6.0.30.bb'], 'sq_task': ['do_populate_sysroot', 'do_populate_lic', 'do_package_qa'], 'sq_hash': ['7d132e7705b5772accce5544cf785c70', '52387c5751c018ddfe2c6a5eecf59e36', '315bcde98908fd36442830f9ae7469c1'], 'sq_hashfn': ['Ubuntu-14.04/ sstate:db-native:x86_64-linux:6.0.30:r0:x86_64:3: sstate:db::6.0.30:r0::3:', 'Ubuntu-14.04/ sstate:db-native:x86_64-linux:6.0.30:r0:x86_64:3: sstate:db::6.0.30:r0::3:', 'Ubuntu-14.04/ sstate:db-native:x86_64-linux:6.0.30:r0:x86_64:3: sstate:db::6.0.30:r0::3:'], 'd': <bb.data_smart.DataSmart object at 0x7f87e15cde10>}):
     def better_eval(source, locals):
    >    return eval(source, get_context(), locals)

  File "<string>", line 1, in <module>
  File "sstate.bbclass", line 110, in sstate_checkhashes(sq_fn=['virtual:native:/home/jiahongxu/yocto/poky/meta/recipes-support/db/db_6.0.30.bb', 'virtual:native:/home/jiahongxu/yocto/poky/meta/recipes-support/db/db_6.0.30.bb', 'virtual:native:/home/jiahongxu/yocto/poky/meta/recipes-support/db/db_6.0.30.bb'], sq_task=['do_populate_sysroot', 'do_populate_lic', 'do_package_qa'], sq_hash=['7d132e7705b5772accce5544cf785c70', '52387c5751c018ddfe2c6a5eecf59e36', '315bcde98908fd36442830f9ae7469c1'], sq_hashfn=['Ubuntu-14.04/ sstate:db-native:x86_64-linux:6.0.30:r0:x86_64:3: sstate:db::6.0.30:r0::3:', 'Ubuntu-14.04/ sstate:db-native:x86_64-linux:6.0.30:r0:x86_64:3: sstate:db::6.0.30:r0::3:', 'Ubuntu-14.04/ sstate:db-native:x86_64-linux:6.0.30:r0:x86_64:3: sstate:db::6.0.30:r0::3:'], d=<bb.data_smart.DataSmart object at 0x7f87e15cde10>)
  File "/home/jiahongxu/yocto/poky/bitbake/lib/bb/__init__.py", line 102, in fatal:
         logger.critical(''.join(args))
    >    raise BBHandledException()

BBHandledException
...

We should terminate build safely while RunQueueExecuteScenequeue init failed.

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 bitbake/lib/bb/runqueue.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index e13dc57..c7b1dc0 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -1046,7 +1046,12 @@ class RunQueue:
                 self.state = runQueueComplete
             else:
                 self.start_worker()
-                self.rqexe = RunQueueExecuteScenequeue(self)
+                try:
+                    self.rqexe = RunQueueExecuteScenequeue(self)
+                except:
+                    # The RunQueueExecuteScenequeue init failure
+                    # terminated the build safely
+                    self.state = runQueueComplete
 
         if self.state in [runQueueSceneRun, runQueueRunning, runQueueCleanUp]:
             self.dm.check(self)
-- 
1.9.1

[PATCH 2/3] bbclass/sstate: add hook SSTATE_CHECK_FUNCTIONS for sstate checking

[Hongxu Jia]

Provide hook for sstate checking, the checking failure will
break the building immediately. Pass input parameters by
bb.data (d.getVar/d.setVar).

Hook  : SSTATE_CHECK_FUNCTIONS
Input : missed    - list of missing sstate file index in sq
      : found     - list of existing sstate file index in sq
      : sq_fn     - list of filenames in the sq
      : sq_task   - list of task names in the sq
      : sq_hash   - list of task hash in the sq
      : sq_hashfn - list of hashfn in the sq

[YOCTO #6639]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/classes/sstate.bbclass | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 4eec6bd..466fdf4 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -36,6 +36,18 @@ SSTATEPREINSTFUNCS ?= ""
 SSTATEPOSTINSTFUNCS ?= ""
 EXTRA_STAGING_FIXMES ?= ""
 
+# Provide hook for sstate checking, the checking failure will
+# break the building immediately. Pass input parameters by
+# bb.data (d.getVar).
+# Input : missed    - list of missing sstate file index in sq
+#       : found     - list of existing sstate file index in sq
+#       : sq_fn     - list of filenames in the sq
+#       : sq_task   - list of task names in the sq
+#       : sq_hash   - list of task hash in the sq
+#       : sq_hashfn - list of hashfn in the sq
+#
+SSTATE_CHECK_FUNCTIONS ?= ""
+
 # Specify dirs in which the shell function is executed and don't use ${B}
 # as default dirs to avoid possible race about ${B} with other task.
 sstate_create_package[dirs] = "${SSTATE_BUILDDIR}"
@@ -704,6 +716,20 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
             evdata['found'].append( (sq_fn[task], sq_task[task], sq_hash[task], sstatefile ) )
         bb.event.fire(bb.event.MetadataEvent("MissedSstate", evdata), d)
 
+    localdata = bb.data.createCopy(d)
+    bb.data.update_data(localdata)
+    for f in (localdata.getVar('SSTATE_CHECK_FUNCTIONS', True) or '').split():
+        localdata = bb.data.createCopy(d)
+        bb.data.update_data(localdata)
+        localdata.setVar('missed', missed)
+        localdata.setVar('found', ret)
+        localdata.setVar('sq_fn', sq_fn)
+        localdata.setVar('sq_task', sq_task)
+        localdata.setVar('sq_hash', sq_hash)
+        localdata.setVar('sq_hashfn', sq_hashfn)
+
+        bb.build.exec_func(f, localdata)
+
     return ret
 
 BB_SETSCENE_DEPVALID = "setscene_depvalid"
-- 
1.9.1

[PATCH 3/3] bbclass/sstate_readonly: only allowed sstate-cache objects are allowed in a build (read-only sstate-cache)

[Hongxu Jia]

The requirement is the developer who demand only the "new" software
they write is allowed to be compiled from source, they only want to
reuse binaries from an existed sstate-cache, if the developer makes
a change that triggers a rebuild, it should be an instant error.

The purpose of this is for the sstate-cache to check if the item
exists or not. If it doesn't the item needs to be in a whitelist
or we need to fail.

If read-only sstate-cache enable, and the recipe's ${PN} not in the
${SSTATECACHE_WHITELIST}, it trigered an instant error.

...
$ bitbake db
ERROR: Read-only sstate-cache is enabled, the build of
"db rpm-native gcc-runtime eglibc linux-libc-headers libgcc"
did not come from sstate-cache. Only the recipe listed in
SSTATECACHE_WHITELIST is allowed to build from source

Summary: There was 1 ERROR message shown, returning a non-zero exit code.
...

[YOCTO #6639]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/classes/sstate_readonly.bbclass | 51 ++++++++++++++++++++++++++++++++++++
 1 file changed, 51 insertions(+)
 create mode 100644 meta/classes/sstate_readonly.bbclass

diff --git a/meta/classes/sstate_readonly.bbclass b/meta/classes/sstate_readonly.bbclass
new file mode 100644
index 0000000..9eabf81
--- /dev/null
+++ b/meta/classes/sstate_readonly.bbclass
@@ -0,0 +1,51 @@
+SSTATE_CHECK_FUNCTIONS_append = " sstate_readonly_check"
+
+# 1) If ${SSTATECACHE_WHITELIST} is "", it means read-only sstate-cache
+#    disabled;
+#
+# 2) If read-only sstate-cache enabled and the recipe's ${PN} not listed
+#    in ${SSTATECACHE_WHITELIST}, the build from source will triger an
+#    instant error;
+SSTATECACHE_WHITELIST ?= ""
+
+python sstate_readonly_check(){
+    whitelist = (d.getVar('SSTATECACHE_WHITELIST', True) or '').split()
+    if whitelist:
+        sq_fn = d.getVar('sq_fn', True) or []
+        missed = d.getVar('missed', True) or []
+        missed_pn = []
+        for task in missed:
+            fn = sq_fn[task]
+            data = bb.cache.Cache.loadDataFull(fn, '', d)
+            pn = data.getVar('PN', True)
+            if pn and pn not in missed_pn:
+                missed_pn.append(pn)
+
+        if missed_pn:
+            blacklist = [pn for pn in missed_pn if pn not in whitelist]
+            if blacklist:
+                msg =  'Read-only sstate-cache is enabled, the build of \n'
+                msg += '"' + ' '.join(blacklist) + '"\n'
+                msg += 'did not come from sstate-cache. Only the recipe listed in\n'
+                msg += 'SSTATECACHE_WHITELIST is allowed to build from source'
+                bb.fatal(msg)
+}
+
+def _sstate_readonly_clean_check(d):
+    whitelist = (d.getVar('SSTATECACHE_WHITELIST', True) or '').split()
+    if whitelist:
+        pn = d.getVar('PN', True)
+        if pn not in whitelist:
+            msg =  'Read-only sstate-cache is enabled, the clean of \n'
+            msg += '%s is not allowed. Only the recipe listed in\n' % pn
+            msg += 'SSTATECACHE_WHITELIST is allowed to clean sstate-cache'
+            bb.fatal(msg)
+
+python do_cleansstate_prepend() {
+        _sstate_readonly_clean_check(d)
+}
+
+python do_cleanall_prepend() {
+    _sstate_readonly_clean_check(d)
+}
+
-- 
1.9.1

Re: [PATCH 1/3] bitbake: runqueue: terminate build safely while RunQueueExecuteScenequeue init failed

[Hongxu Jia]

I will resend this patch to bitbake mailing list 
bitbake-devel@lists.openembedded.org,
sorry for the wrong place.

//Hongxu

On 08/27/2014 03:53 PM, Hongxu Jia wrote:
> While RunQueueExecuteScenequeue init failed, the exit of build is mess.
> Here is the example while bb.fatal invoked in RunQueueExecuteScenequeue:
> ...
> Traceback (most recent call last):
>    File "/home/jiahongxu/yocto/poky/bitbake/lib/bb/runqueue.py", line 1824, in RunQueueExecuteScenequeue.__init__(rq=<bb.runqueue.RunQueue instance at 0x7f87dd7d5050>):
>                   locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname, "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data }
>      >            valid = bb.utils.better_eval(call, locs)
>
>    File "/home/jiahongxu/yocto/poky/bitbake/lib/bb/utils.py", line 374, in better_eval(source='sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d)', locals={'sq_fn': ['virtual:native:/home/jiahongxu/yocto/poky/meta/recipes-support/db/db_6.0.30.bb', 'virtual:native:/home/jiahongxu/yocto/poky/meta/recipes-support/db/db_6.0.30.bb', 'virtual:native:/home/jiahongxu/yocto/poky/meta/recipes-support/db/db_6.0.30.bb'], 'sq_task': ['do_populate_sysroot', 'do_populate_lic', 'do_package_qa'], 'sq_hash': ['7d132e7705b5772accce5544cf785c70', '52387c5751c018ddfe2c6a5eecf59e36', '315bcde98908fd36442830f9ae7469c1'], 'sq_hashfn': ['Ubuntu-14.04/ sstate:db-native:x86_64-linux:6.0.30:r0:x86_64:3: sstate:db::6.0.30:r0::3:', 'Ubuntu-14.04/ sstate:db-native:x86_64-linux:6.0.30:r0:x86_64:3: sstate:db::6.0.30:r0::3:', 'Ubuntu-14.04/ sstate:db-native:x86_64-linux:6.0.30:r0:x86_64:3: sstate:db::6.0.30:r0::3:'], 'd': <bb.data_smart.DataSmart object at 0x7f87e15cde10>}):
>       def better_eval(source, locals):
>      >    return eval(source, get_context(), locals)
>
>    File "<string>", line 1, in <module>
>    File "sstate.bbclass", line 110, in sstate_checkhashes(sq_fn=['virtual:native:/home/jiahongxu/yocto/poky/meta/recipes-support/db/db_6.0.30.bb', 'virtual:native:/home/jiahongxu/yocto/poky/meta/recipes-support/db/db_6.0.30.bb', 'virtual:native:/home/jiahongxu/yocto/poky/meta/recipes-support/db/db_6.0.30.bb'], sq_task=['do_populate_sysroot', 'do_populate_lic', 'do_package_qa'], sq_hash=['7d132e7705b5772accce5544cf785c70', '52387c5751c018ddfe2c6a5eecf59e36', '315bcde98908fd36442830f9ae7469c1'], sq_hashfn=['Ubuntu-14.04/ sstate:db-native:x86_64-linux:6.0.30:r0:x86_64:3: sstate:db::6.0.30:r0::3:', 'Ubuntu-14.04/ sstate:db-native:x86_64-linux:6.0.30:r0:x86_64:3: sstate:db::6.0.30:r0::3:', 'Ubuntu-14.04/ sstate:db-native:x86_64-linux:6.0.30:r0:x86_64:3: sstate:db::6.0.30:r0::3:'], d=<bb.data_smart.DataSmart object at 0x7f87e15cde10>)
>    File "/home/jiahongxu/yocto/poky/bitbake/lib/bb/__init__.py", line 102, in fatal:
>           logger.critical(''.join(args))
>      >    raise BBHandledException()
>
> BBHandledException
> ...
>
> We should terminate build safely while RunQueueExecuteScenequeue init failed.
>
> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
> ---
>   bitbake/lib/bb/runqueue.py | 7 ++++++-
>   1 file changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
> index e13dc57..c7b1dc0 100644
> --- a/bitbake/lib/bb/runqueue.py
> +++ b/bitbake/lib/bb/runqueue.py
> @@ -1046,7 +1046,12 @@ class RunQueue:
>                   self.state = runQueueComplete
>               else:
>                   self.start_worker()
> -                self.rqexe = RunQueueExecuteScenequeue(self)
> +                try:
> +                    self.rqexe = RunQueueExecuteScenequeue(self)
> +                except:
> +                    # The RunQueueExecuteScenequeue init failure
> +                    # terminated the build safely
> +                    self.state = runQueueComplete
>   
>           if self.state in [runQueueSceneRun, runQueueRunning, runQueueCleanUp]:
>               self.dm.check(self)