About the sstate cache directory hierarchy

* About the sstate cache directory hierarchy
@ 2022-01-28  7:06 ChenQi
  2022-01-28 10:29 ` [OE-core] " Alexander Kanavin
  2022-05-16  5:30 ` Jacob Kroon
  0 siblings, 2 replies; 6+ messages in thread
From: ChenQi @ 2022-01-28  7:06 UTC (permalink / raw)
  To: openembedded-core, Richard Purdie

[-- Attachment #1: Type: text/plain, Size: 5412 bytes --]

Hi All,

I'm sending out this email because I'm wondering if we can change the 
sstate cache directory to use ${PN} and taskname as subditories. Hope to 
hear your opinions.

Below is the long story.

Recently I noticed that running `bitbake xxx -c cleansstate' usually 
takes more than 10 minutes.

After some investigation, I can see that most of the time is spent on 
file searching. This is because we have:

SSTATE_PATHSPEC   = 
"${SSTATE_DIR}/${SSTATE_EXTRAPATHWILDCARD}${PN}/${SSTATE_PATH_CURRTASK}/${SSTATE_PKGSPEC}*_${SSTATE_PATH_CURRTASK}.tar.zst*"

And our sstate cache directory's hierarchy uses hash[:2]/hash[2:4]/ as 
sub-directories.

This essentially means that all sub-directories are searched. This would 
take a long time, especially when run for the first time. I made some 
changes to  output the time and the logs are as below.

$ bitbake glibc -c cleansstate
WARNING: glibc-2.34-r0 do_cleansstate: Removing 
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst*
WARNING: glibc-2.34-r0 do_cleansstate: Took 611.8865714073181 seconds
WARNING: glibc-2.34-r0 do_cleansstate: Removing 
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package.tar.zst*
WARNING: glibc-2.34-r0 do_cleansstate: Took 1.3219327926635742 seconds
WARNING: glibc-2.34-r0 do_cleansstate: Removing 
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package_qa.tar.zst*
WARNING: glibc-2.34-r0 do_cleansstate: Took 1.470815658569336 seconds
WARNING: glibc-2.34-r0 do_cleansstate: Removing 
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package_write_rpm.tar.zst*
WARNING: glibc-2.34-r0 do_cleansstate: Took 1.251939058303833 seconds
WARNING: glibc-2.34-r0 do_cleansstate: Removing 
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_packagedata.tar.zst*
WARNING: glibc-2.34-r0 do_cleansstate: Took 1.2369801998138428 seconds
WARNING: glibc-2.34-r0 do_cleansstate: Removing 
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc::2.34:r0::7:*_populate_lic.tar.zst*
WARNING: glibc-2.34-r0 do_cleansstate: Took 1.1668426990509033 seconds
WARNING: glibc-2.34-r0 do_cleansstate: Removing 
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_populate_sysroot.tar.zst*
WARNING: glibc-2.34-r0 do_cleansstate: Took 1.385568380355835 seconds
WARNING: glibc-2.34-r0 do_cleansstate: Removing 
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_stash_locale.tar.zst*
WARNING: glibc-2.34-r0 do_cleansstate: Took 1.4884181022644043 seconds

I figured that unlike git, we do have knowledge on our sstate objects. 
It does not seem necessary to use hash value as sub directory. So I 
changed the sstate directory hierarchy to use ${PN}/taskname/ as sub 
directories, and here's the result.

$ bitbake libgcc -c cleansstate

WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing 
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/deploy_source_date_epoch/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst*
WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.020630598068237305 seconds
WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing 
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package.tar.zst*
WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0011608600616455078 seconds
WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing 
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_qa/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_qa.tar.zst*
WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0007557868957519531 seconds
WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing 
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_write_rpm/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_write_rpm.tar.zst*
WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0013995170593261719 seconds
WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing 
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/packagedata/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_packagedata.tar.zst*
WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0007488727569580078 seconds
WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing 
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_lic/sstate:libgcc::11.2.0:r0::7:*_populate_lic.tar.zst*
WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0005896091461181641 seconds
WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing 
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_sysroot/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_populate_sysroot.tar.zst*
WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.00080108642578125 seconds

It's much faster.

In addition, the sub dirs now give more info, which should potentially 
make sstate cache easier to manage.

Attached is the patch to quickly try things out. Hope to hear your opinions.

Best Regards,

Chen Qi


[-- Attachment #2: 0001-sstate.bbclasss-use-taskname-as-sub-dirs.patch --]
[-- Type: text/x-patch, Size: 2571 bytes --]

From 6f148d399c2ac575a9079eae682a56802dfac3ca Mon Sep 17 00:00:00 2001
From: Chen Qi <Qi.Chen@windriver.com>
Date: Thu, 27 Jan 2022 22:57:57 -0800
Subject: [PATCH] sstate.bbclasss: use /taskname as sub dirs

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/classes/sstate.bbclass | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 17dcf4cc17..c657d40151 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -17,6 +17,7 @@ def generate_sstatefn(spec, hash, taskname, siginfo, d):
     if not hash:
         hash = "INVALID"
     fn = spec + hash + "_" + taskname + extension
+    pn = spec.split(':')[1]
     # If the filename is too long, attempt to reduce it
     if len(fn) > limit:
         components = spec.split(":")
@@ -30,7 +31,7 @@ def generate_sstatefn(spec, hash, taskname, siginfo, d):
         fn = spec + hash + "_" + taskname + extension
         if len(fn) > limit:
             bb.fatal("Unable to reduce sstate name to less than 255 chararacters")
-    return hash[:2] + "/" + hash[2:4] + "/" + fn
+    return pn + "/" + taskname + "/" + fn
 
 SSTATE_PKGARCH    = "${PACKAGE_ARCH}"
 SSTATE_PKGSPEC    = "sstate:${PN}:${PACKAGE_ARCH}${TARGET_VENDOR}-${TARGET_OS}:${PV}:${PR}:${SSTATE_PKGARCH}:${SSTATE_VERSION}:"
@@ -39,7 +40,7 @@ SSTATE_PKGNAME    = "${SSTATE_EXTRAPATH}${@generate_sstatefn(d.getVar('SSTATE_PK
 SSTATE_PKG        = "${SSTATE_DIR}/${SSTATE_PKGNAME}"
 SSTATE_EXTRAPATH   = ""
 SSTATE_EXTRAPATHWILDCARD = ""
-SSTATE_PATHSPEC   = "${SSTATE_DIR}/${SSTATE_EXTRAPATHWILDCARD}*/*/${SSTATE_PKGSPEC}*_${SSTATE_PATH_CURRTASK}.tar.zst*"
+SSTATE_PATHSPEC   = "${SSTATE_DIR}/${SSTATE_EXTRAPATHWILDCARD}${PN}/${SSTATE_PATH_CURRTASK}/${SSTATE_PKGSPEC}*_${SSTATE_PATH_CURRTASK}.tar.zst*"
 
 # explicitly make PV to depend on evaluated value of PV variable
 PV[vardepvalue] = "${PV}"
@@ -482,12 +483,16 @@ python sstate_hardcode_path_unpack () {
 
 def sstate_clean_cachefile(ss, d):
     import oe.path
+    import time
 
     if d.getVarFlag('do_%s' % ss['task'], 'task'):
         d.setVar("SSTATE_PATH_CURRTASK", ss['task'])
         sstatepkgfile = d.getVar('SSTATE_PATHSPEC')
-        bb.note("Removing %s" % sstatepkgfile)
+        bb.warn("Removing %s" % sstatepkgfile)
+        start = time.time()
         oe.path.remove(sstatepkgfile)
+        end = time.time()
+        bb.warn("Took %s seconds" % (end-start))
 
 def sstate_clean_cachefiles(d):
     for task in (d.getVar('SSTATETASKS') or "").split():
-- 
2.33.0


^ permalink raw reply related	[flat|nested] 6+ messages in thread