* [PATCH] package_rpm: Only rebuild the indexes if the packages have changed
@ 2012-04-27 15:10 Richard Purdie
2012-04-29 0:03 ` Khem Raj
0 siblings, 1 reply; 2+ messages in thread
From: Richard Purdie @ 2012-04-27 15:10 UTC (permalink / raw)
To: openembedded-core
This change farms the solvedb creation out to a separate script which
handles creation of the index, only if mtime of any of the packages
has changed.
For a core-image-minimal set of rpm's this saves ~20s of a 45s rootfs
build. For core-image-sato it saves 1 minute of a 5 minute rootfs build.
The more packages in the system, the bigger the saving will be.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index 2476c16..cf66a51 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -40,12 +40,13 @@ package_update_index_rpm_common () {
rpmconf_base="$1"
shift
+ createdirs=""
for archvar in "$@"; do
eval archs=\${${archvar}}
packagedirs=""
for arch in $archs; do
packagedirs="${DEPLOY_DIR_RPM}/$arch $packagedirs"
- rm -rf ${DEPLOY_DIR_RPM}/$arch/solvedb
+ rm -rf ${DEPLOY_DIR_RPM}/$arch/solvedb.done
done
cat /dev/null > ${rpmconf_base}-${archvar}.conf
@@ -53,22 +54,11 @@ package_update_index_rpm_common () {
if [ -e $pkgdir/ ]; then
echo "Generating solve db for $pkgdir..."
echo $pkgdir/solvedb >> ${rpmconf_base}-${archvar}.conf
- if [ -d $pkgdir/solvedb ]; then
- # We've already processed this and it's a duplicate
- continue
- fi
- mkdir -p $pkgdir/solvedb
- echo "# Dynamically generated solve manifest" >> $pkgdir/solvedb/manifest
- find $pkgdir -maxdepth 1 -type f >> $pkgdir/solvedb/manifest
- ${RPM} -i --replacepkgs --replacefiles --oldpackage \
- -D "_dbpath $pkgdir/solvedb" --justdb \
- --noaid --nodeps --noorder --noscripts --notriggers --noparentdirs --nolinktos --stats \
- --ignoresize --nosignature --nodigest \
- -D "__dbi_txn create nofsync" \
- $pkgdir/solvedb/manifest
+ createdirs="$createdirs $pkgdir"
fi
done
done
+ rpm-createsolvedb.py "${RPM}" $createdirs
}
#
diff --git a/scripts/rpm-createsolvedb.py b/scripts/rpm-createsolvedb.py
new file mode 100755
index 0000000..0d5f219
--- a/dev/null
+++ b/scripts/rpm-createsolvedb.py
@@ -0,0 +1,63 @@
+#!/usr/bin/env python
+#
+# This script generates a solution database for a directory containing rpm packages
+# but tries to be efficient about this, only doing so when the packages have changed
+# in some way.
+#
+# It is assumed something already went through and removed all the solvedb.done stamp files
+# in advance.
+#
+# First argument - the rpm binary to use
+# Subsequent arguments - paths to process solution databases for
+#
+
+import sys, os
+import hashlib
+import stat
+
+if len(sys.argv) < 1:
+ print("Error, rpm command not specified")
+ sys.exit(1)
+
+if len(sys.argv) < 2:
+ print("Error, no paths specified")
+ sys.exit(1)
+
+paths = sys.argv[2:]
+
+for path in paths:
+ if os.path.exists(path + "/solvedb.done"):
+ continue
+ data = ""
+ manifest = []
+ for root, dirs, files in os.walk(path):
+ for file in files:
+ f = os.path.join(root, file)
+ if f.startswith(path + "/" + "solvedb"):
+ continue
+ data = data + str(os.stat(f)[stat.ST_MTIME])
+ manifest.append(f)
+ checksum = hashlib.md5(data).hexdigest()
+
+ if os.path.exists(path + "/solvedb.checksum") and open(path + "/solvedb.checksum", "r").read() == checksum:
+ open(path + "/solvedb.done", "w")
+ continue
+
+ if os.path.exists(path + "/solvedb"):
+ os.system("rm -rf %s" % (path + "/solvedb"))
+ os.mkdir(path + "/solvedb")
+ m = open(path + "/solvedb/manifest", "w")
+ m.write("# Dynamically generated solve manifest\n")
+ for f in manifest:
+ m.write(f + "\n")
+ m.close()
+
+ cmd = sys.argv[1] + ' -i --replacepkgs --replacefiles --oldpackage -D "_dbpath ' + path + '/solvedb" --justdb \
+ --noaid --nodeps --noorder --noscripts --notriggers --noparentdirs --nolinktos --stats \
+ --ignoresize --nosignature --nodigest -D "__dbi_txn create nofsync" \
+ ' + path + '/solvedb/manifest'
+ os.system(cmd)
+
+ open(path + "/solvedb.checksum", "w").write(checksum)
+ open(path + "/solvedb.done", "w")
+
^ permalink raw reply related [flat|nested] 2+ messages in thread
* Re: [PATCH] package_rpm: Only rebuild the indexes if the packages have changed
2012-04-27 15:10 [PATCH] package_rpm: Only rebuild the indexes if the packages have changed Richard Purdie
@ 2012-04-29 0:03 ` Khem Raj
0 siblings, 0 replies; 2+ messages in thread
From: Khem Raj @ 2012-04-29 0:03 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
> + if os.path.exists(path + "/solvedb.checksum") and open(path + "/solvedb.checksum", "r").read() == checksum:
> + open(path + "/solvedb.done", "w")
> + continue
> +
> + if os.path.exists(path + "/solvedb"):
> + os.system("rm -rf %s" % (path + "/solvedb"))
to be pythonic may be use shutils.rmtree() here
> + os.mkdir(path + "/solvedb")
> + m = open(path + "/solvedb/manifest", "w")
> + m.write("# Dynamically generated solve manifest\n")
> + for f in manifest:
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2012-04-29 0:13 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-04-27 15:10 [PATCH] package_rpm: Only rebuild the indexes if the packages have changed Richard Purdie
2012-04-29 0:03 ` Khem Raj
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.