[zeus][ 0/2] Fix for two issues related to sstate-cache

[zeus][ 0/2] Fix for two issues related to sstate-cache

[Mark Hatle]

Patch 1 is a backport from master (it's already in Dunfell).  If you have a
locked-sigs file, and the sstate for that item isn't available the .siginfo and
sigdata files were not being written.  Patch 1/2 resolved that with the
fragment in "dump_sigtask".

Patch 2 was sent for master as well, and this is just to get it applied to
Zeus.  Due to problem #1, we were generating sstate-cache that did not have
siginfo files.  Due to this we were getting failures later on using that
sstate-cache in a subsequent build.  This patch ensures if this happens
again (of if someone intentionally strips those) it won't fail.

Mark Hatle (1):
  sstate.bbclass: When siginfo or sig files are missing, stop fetcher
    errors

Richard Purdie (1):
  sstatesig: Optimise get_taskhash for hashequiv

 meta/classes/sstate.bbclass |  6 +++++-
 meta/lib/oe/sstatesig.py    | 13 +++++++++++--
 2 files changed, 16 insertions(+), 3 deletions(-)

-- 
2.17.1

[zeus][ 1/2] sstatesig: Optimise get_taskhash for hashequiv

[Mark Hatle]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

With hashequiv the get_taskhash function is called much more regularly
and contains expensive operations. This these don't change based upon
hash in a given build, improve the caching within the function to
reduce overhead.

(From OE-Core rev: de98cfe3cde4b8d5f4b163b5fba3f129651ef06a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
---
 meta/lib/oe/sstatesig.py | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
index b2316b12b8..f1abff0c45 100644
--- a/meta/lib/oe/sstatesig.py
+++ b/meta/lib/oe/sstatesig.py
@@ -151,6 +151,13 @@ class SignatureGeneratorOEBasicHashMixIn(object):
 
     def get_taskhash(self, tid, deps, dataCache):
         h = super(bb.siggen.SignatureGeneratorBasicHash, self).get_taskhash(tid, deps, dataCache)
+        if tid in self.lockedhashes:
+            if self.lockedhashes[tid]:
+                return self.lockedhashes[tid]
+            else:
+                return h
+
+        h = super(bb.siggen.SignatureGeneratorBasicHash, self).get_taskhash(tid, deps, dataCache)
 
         (mc, _, task, fn) = bb.runqueue.split_tid_mcfn(tid)
 
@@ -187,17 +194,19 @@ class SignatureGeneratorOEBasicHashMixIn(object):
                                           % (recipename, task, h, h_locked, var))
 
                 return h_locked
+
+        self.lockedhashes[tid] = False
         #bb.warn("%s %s %s" % (recipename, task, h))
         return h
 
     def get_unihash(self, tid):
-        if tid in self.lockedhashes:
+        if tid in self.lockedhashes and self.lockedhashes[tid]:
             return self.lockedhashes[tid]
         return super().get_unihash(tid)
 
     def dump_sigtask(self, fn, task, stampbase, runtime):
         tid = fn + ":" + task
-        if tid in self.lockedhashes:
+        if tid in self.lockedhashes and self.lockedhashes[tid]:
             return
         super(bb.siggen.SignatureGeneratorBasicHash, self).dump_sigtask(fn, task, stampbase, runtime)
 
-- 
2.17.1

[zeus][ 2/2] sstate.bbclass: When siginfo or sig files are missing, stop fetcher errors

[Mark Hatle]

From: Mark Hatle <mark.hatle@xilinx.com>

Prior to fetching, the system checks if the sstate file is present
either locally or on the mirror.  If it is, then it goes to the fetch
stage.  Up to three files can be fetched, sstate, sstate.siginfo and
sstate.sig (if signature validation is enabled).

The previous pstaging_fetch function would iterate over these, and if
a download error occurred would spew forth a great amount of fetcher
failure messages as well as stop fetching the next item in the set.

This was resolved by adding a fetcher.checkstatus() call prior to
the download.  If the file isn't present, then the exception will
be triggered, and no fetcher failure messages will reach the user.

The exception handler is then modified to be a pass so that it will
loop and pull the rest of the files that that are requested.

Additionally, a check for the existance of the .sig file was added
to the sstate_installpkg to avoid an error trying to load the .sig
if it wasn't downloaded.

Signed-off-by: Mark Hatle <mark.hatle@xilinx.com>
---
 meta/classes/sstate.bbclass | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index c0329cd5d1..3528908727 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -333,6 +333,9 @@ def sstate_installpkg(ss, d):
     d.setVar('SSTATE_INSTDIR', sstateinst)
 
     if bb.utils.to_boolean(d.getVar("SSTATE_VERIFY_SIG"), False):
+        if not os.path.isfile(sstatepkg + '.sig'):
+            bb.warn("Not signature file for sstate package %s, skipping acceleration..." % sstatepkg)
+            return False
         signer = get_signer(d, 'local')
         if not signer.verify(sstatepkg + '.sig'):
             bb.warn("Cannot verify signature on sstate package %s, skipping acceleration..." % sstatepkg)
@@ -703,10 +706,11 @@ def pstaging_fetch(sstatefetch, d):
         localdata.setVar('SRC_URI', srcuri)
         try:
             fetcher = bb.fetch2.Fetch([srcuri], localdata, cache=False)
+            fetcher.checkstatus()
             fetcher.download()
 
         except bb.fetch2.BBFetchException:
-            break
+            pass
 
 def sstate_setscene(d):
     shared_state = sstate_state_fromvars(d)
-- 
2.17.1