All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 0/4] Fix the emenlow build failures and add optimizations
@ 2011-02-16  5:12 Mark Hatle
  2011-02-16  5:12 ` [PATCH 1/4] rpm: Add the ability to use the platform file during install Mark Hatle
                   ` (4 more replies)
  0 siblings, 5 replies; 6+ messages in thread
From: Mark Hatle @ 2011-02-16  5:12 UTC (permalink / raw)
  To: poky

We patch 1/4 fixes the emenlow build failure without adding any system optimizations.

This rootfs generation takes approx 11m42 on my machine.

Patch 2/4 adds a backport of head of tree RPM "nofsync" change.  This drops the
overall rootfs generation time to approx 6m38.

Patch 3/4 disables default shell debug logging during image creation.

Patch 4/4 fixes a problem where the solverdb's could be generated multiple times.
We now only generate the solvedb ONCE.  This drops the overall rootfs generation
time to 6m16.

Pull URL: git://git.pokylinux.org/poky-contrib.git
  Branch: mhatle/emenlow-fix
  Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=mhatle/emenlow-fix

Thanks,
    Mark Hatle <mark.hatle@windriver.com>
---


Mark Hatle (4):
  rpm: Add the ability to use the platform file during install
  rpm: Avoid fsync when writing files
  image.bbclass: Disable shell debug messages
  package_rpm: Fix solverdb generation

 meta/classes/image.bbclass                       |    2 +-
 meta/classes/package_rpm.bbclass                 |   47 +++++---
 meta/classes/populate_sdk_rpm.bbclass            |    7 +-
 meta/classes/rootfs_rpm.bbclass                  |    9 +-
 meta/recipes-devtools/rpm/rpm/rpm-nofsync.patch  |   34 ++++++
 meta/recipes-devtools/rpm/rpm/rpm-platform.patch |  132 ++++++++++++++++++++++
 meta/recipes-devtools/rpm/rpm/rpm-showrc.patch   |   23 ++++
 meta/recipes-devtools/rpm/rpm_5.4.0.bb           |    9 +-
 8 files changed, 234 insertions(+), 29 deletions(-)
 create mode 100644 meta/recipes-devtools/rpm/rpm/rpm-nofsync.patch
 create mode 100644 meta/recipes-devtools/rpm/rpm/rpm-platform.patch
 create mode 100644 meta/recipes-devtools/rpm/rpm/rpm-showrc.patch

-- 
1.7.3.4



^ permalink raw reply	[flat|nested] 6+ messages in thread
* [PATCH 1/4] rpm: Add the ability to use the platform file during install
  2011-02-16  5:12 [PATCH 0/4] Fix the emenlow build failures and add optimizations Mark Hatle
@ 2011-02-16  5:12 ` Mark Hatle
  2011-02-16  5:12 ` [PATCH 2/4] rpm: Avoid fsync when writing files Mark Hatle
                   ` (3 subsequent siblings)
  4 siblings, 0 replies; 6+ messages in thread
From: Mark Hatle @ 2011-02-16  5:12 UTC (permalink / raw)
  To: poky

Add a new rpm macro, rpmrc_platform_path to specify an alternative platform
file.  This is required to allow the dep resolver to identify compatible
packages.

Also workaround a minor problem with the --showrc command in RPM.  A bug
has been reported upstream on this.

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
 meta/classes/package_rpm.bbclass                 |   15 ++-
 meta/recipes-devtools/rpm/rpm/rpm-platform.patch |  132 ++++++++++++++++++++++
 meta/recipes-devtools/rpm/rpm/rpm-showrc.patch   |   23 ++++
 meta/recipes-devtools/rpm/rpm_5.4.0.bb           |    8 +-
 4 files changed, 170 insertions(+), 8 deletions(-)
 create mode 100644 meta/recipes-devtools/rpm/rpm/rpm-platform.patch
 create mode 100644 meta/recipes-devtools/rpm/rpm/rpm-showrc.patch

diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index cc262dc..215be8d 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -219,7 +219,8 @@ package_install_internal_rpm () {
 
 	# Generate an install solution by doing a --justdb install, then recreate it with
 	# an actual package install!
-	${RPM} -D "_rpmds_sysinfo_path ${target_rootfs}/etc/rpm/sysinfo" \
+	${RPM} --predefine "_rpmds_sysinfo_path ${target_rootfs}/etc/rpm/sysinfo" \
+		--predefine "_rpmrc_platform_path ${target_rootfs}/etc/rpm/platform" \
 		-D "_dbpath ${target_rootfs}/install" -D "`cat ${confbase}.macro`" \
 		-D "__dbi_txn create nofsync" \
 		-U --justdb --noscripts --notriggers --noparentdirs --nolinktos --ignoresize \
@@ -235,7 +236,8 @@ package_install_internal_rpm () {
 				exit 1
 			fi
 			echo "Attempting $pkg_name..." >> "${WORKDIR}/temp/log.do_${task}_attemptonly.${PID}"
-			${RPM} -D "_rpmds_sysinfo_path ${target_rootfs}/etc/rpm/sysinfo" \
+			${RPM} --predefine "_rpmds_sysinfo_path ${target_rootfs}/etc/rpm/sysinfo" \
+				--predefine "_rpmrc_platform_path ${target_rootfs}/etc/rpm/platform" \
 				-D "_dbpath ${target_rootfs}/install" -D "`cat ${confbase}.macro`" \
 				-D "__dbi_txn create nofsync private" \
 				-U --justdb --noscripts --notriggers --noparentdirs --nolinktos --ignoresize \
@@ -254,7 +256,8 @@ package_install_internal_rpm () {
 	 cat /dev/null >  ${target_rootfs}/install/recommend.list
 	 while [ $loop -eq 1 ]; do
 		# Dump the full set of recommends...
-		${RPM} -D "_rpmds_sysinfo_path ${target_rootfs}/etc/rpm/sysinfo" \
+		${RPM} --predefine "_rpmds_sysinfo_path ${target_rootfs}/etc/rpm/sysinfo" \
+			--predefine "_rpmrc_platform_path ${target_rootfs}/etc/rpm/platform" \
 			-D "_dbpath ${IMAGE_ROOTFS}/install" -D "`cat ${confbase}.macro`" \
 			-D "__dbi_txn create nofsync private" \
 			-qa --qf "[%{RECOMMENDS}\n]" | sort -u > ${target_rootfs}/install/recommend
@@ -273,7 +276,8 @@ package_install_internal_rpm () {
 				continue
 			fi
 			echo "Attempting $pkg_name..." >> "${WORKDIR}/temp/log.do_{task}_recommend.${PID}"
-			${RPM} -D "_rpmds_sysinfo_path ${target_rootfs}/etc/rpm/sysinfo" \
+			${RPM} --predefine "_rpmds_sysinfo_path ${target_rootfs}/etc/rpm/sysinfo" \
+				--predefine "_rpmrc_platform_path ${target_rootfs}/etc/rpm/platform" \
 				-D "_dbpath ${target_rootfs}/install" -D "`cat ${confbase}.macro`" \
 				-D "__dbi_txn create nofsync private" \
 				-U --justdb --noscripts --notriggers --noparentdirs --nolinktos --ignoresize \
@@ -293,7 +297,8 @@ package_install_internal_rpm () {
 
 	# Attempt install
 	${RPM} --root ${target_rootfs} \
-		-D "_rpmds_sysinfo_path ${target_rootfs}/etc/rpm/sysinfo" \
+		--predefine "_rpmds_sysinfo_path ${target_rootfs}/etc/rpm/sysinfo" \
+		--predefine "_rpmrc_platform_path ${target_rootfs}/etc/rpm/platform" \
 		-D "_dbpath ${rpmlibdir}" \
 		--noscripts --notriggers --noparentdirs --nolinktos \
 		-D "__dbi_txn create nofsync private" \
diff --git a/meta/recipes-devtools/rpm/rpm/rpm-platform.patch b/meta/recipes-devtools/rpm/rpm/rpm-platform.patch
new file mode 100644
index 0000000..09a1c13
--- /dev/null
+++ b/meta/recipes-devtools/rpm/rpm/rpm-platform.patch
@@ -0,0 +1,132 @@
+Fix up platform and related sysinfo file loading.
+
+This ensures that RPM knows the compatible set of package types at all times.
+
+Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
+
+diff -ur rpm-5.4.0.orig/lib/depends.c rpm-5.4.0/lib/depends.c
+--- rpm-5.4.0.orig/lib/depends.c	2011-02-15 20:40:13.002849708 -0600
++++ rpm-5.4.0/lib/depends.c	2011-02-15 20:47:05.838981632 -0600
+@@ -248,7 +248,7 @@
+ 	    he->p.ptr = _free(he->p.ptr);
+ 	}
+ 
+-#if defined(RPM_VENDOR_WINDRIVER)
++#if defined(RPM_VENDOR_WINDRIVER) && !defined(RPM_VENDOR_POKY)
+ 	/*
+ 	 * If we're capable of installing multiple colors
+ 	 * but at least one of the packages are white (0), we
+@@ -505,7 +505,7 @@
+     return 0;
+ }
+ 
+-#if defined(RPM_VENDOR_WINDRIVER)
++#if defined(RPM_VENDOR_WINDRIVER) && !defined(RPM_VENDOR_POKY)
+ /* Is "compat" compatible w/ arch? */
+ int _isCompatibleArch(const char * arch, const char * compat)
+ {
+@@ -649,7 +649,7 @@
+ 
+ 	    if (arch == NULL || (parch = rpmteA(p)) == NULL)
+ 		continue;
+-#if defined(RPM_VENDOR_WINDRIVER)
++#if defined(RPM_VENDOR_WINDRIVER) && !defined(RPM_VENDOR_POKY)
+ 	    /* XXX hackery for alias matching. */
+ 	    if (!_isCompatibleArch(arch, parch))
+ 		continue;
+@@ -815,6 +815,12 @@
+     return rc;
+ }
+ 
++#if defined(RPM_VENDOR_WINDRIVER)
++#define _ETC_RPM_SYSINFO        "%{_etcrpm}/sysinfo"
++#else
++#define _ETC_RPM_SYSINFO        SYSCONFIGDIR "/sysinfo"
++#endif
++
+ /*@only@*/ /*@null@*/ /*@unchecked@*/
+ static char *sysinfo_path = NULL;
+ 
+@@ -1296,7 +1302,7 @@
+ 	sysinfo_path = rpmExpand("%{?_rpmds_sysinfo_path}", NULL);
+ 	if (!(sysinfo_path != NULL && *sysinfo_path == '/')) {
+ 	    sysinfo_path = _free(sysinfo_path);
+-	    sysinfo_path = xstrdup(SYSCONFIGDIR "/sysinfo");
++	    sysinfo_path = rpmExpand(_ETC_RPM_SYSINFO, NULL);
+ 	}
+     }
+ 
+diff -ur rpm-5.4.0.orig/lib/rpmds.c rpm-5.4.0/lib/rpmds.c
+--- rpm-5.4.0.orig/lib/rpmds.c	2011-02-15 20:40:13.004855352 -0600
++++ rpm-5.4.0/lib/rpmds.c	2011-02-15 20:41:55.598846670 -0600
+@@ -1737,7 +1737,7 @@
+ /*@-observertrans @*/
+ 	    _sysinfo_path = _free(_sysinfo_path);
+ /*@=observertrans @*/
+-	    _sysinfo_path = xstrdup(_ETC_RPM_SYSINFO);
++	    _sysinfo_path = rpmExpand(_ETC_RPM_SYSINFO, NULL);
+ 	}
+     }
+ /*@=modobserver@*/
+diff -ur rpm-5.4.0.orig/lib/rpmrc.c rpm-5.4.0/lib/rpmrc.c
+--- rpm-5.4.0.orig/lib/rpmrc.c	2011-02-15 20:40:13.006853913 -0600
++++ rpm-5.4.0/lib/rpmrc.c	2011-02-15 20:44:39.708972391 -0600
+@@ -38,7 +38,13 @@
+ static const char * configTarget = NULL;
+ 
+ /*@observer@*/ /*@unchecked@*/
+-static const char * platform = SYSCONFIGDIR "/platform";
++#if defined(RPM_VENDOR_WINDRIVER)
++#define _ETC_RPM_PLATFORM        "%{_etcrpm}/platform"
++#else
++#define _ETC_RPM_PLATFORM        SYSCONFIGDIR "/platform"
++#endif
++
++static const char * _platform = NULL;
+ 
+ /*@only@*/ /*@relnull@*/ /*@unchecked@*/
+ void * platpat = NULL;
+@@ -685,16 +691,17 @@
+     int rc;
+ 
+     while (!gotDefaults) {
+-#if defined(RPM_VENDOR_WINDRIVER)
+-	const char * _platform = rpmGetPath(__etcrpm, "/platform", NULL);
+-#else
+-	const char * _platform = platform;
+-#endif
++	if (_platform == NULL) {
++	    _platform = rpmExpand("%{?_rpmrc_platform_path}", NULL);
++	    /* XXX may need to validate path existence somewhen. */
++	    if (!(_platform != NULL && *_platform == '/')) {
++		_platform = _free(_platform);
++		_platform = rpmExpand(_ETC_RPM_PLATFORM, NULL);
++	    }
++	}
+ 	CVOG_t cvog = NULL;
+ #if defined(RPM_VENDOR_OPENPKG) /* larger-utsname */
+ 	const char *cp;
+-#endif
+-#if defined(RPM_VENDOR_OPENPKG) /* larger-utsname */
+ 	/* utsname fields on some platforms (like HP-UX) are very small
+ 	   (just about 8 characters). This is too small for OpenPKG, so cheat! */
+ 	rc = uname(&un_real);
+@@ -771,9 +778,7 @@
+ 	if (cp != NULL && cp != _platform)
+ 	    cp = _free(cp);
+ #endif
+-#if defined(RPM_VENDOR_WINDRIVER)
+ 	_platform = _free(_platform);
+-#endif
+ 
+ 	if (configTarget && !parseCVOG(configTarget, &cvog) && cvog != NULL) {
+ 	    gotDefaults = 1;
+@@ -1096,6 +1101,8 @@
+ 
+ #ifdef PREMACROFILES
+     if (rpmReadRC(PREMACROFILES)) return -1;
++#else
++    if (rpmReadRC(NULL)) return -1;
+ #endif
+ 
+     /* Reset umask to its default umask(2) value. */
diff --git a/meta/recipes-devtools/rpm/rpm/rpm-showrc.patch b/meta/recipes-devtools/rpm/rpm/rpm-showrc.patch
new file mode 100644
index 0000000..5051ce6
--- /dev/null
+++ b/meta/recipes-devtools/rpm/rpm/rpm-showrc.patch
@@ -0,0 +1,23 @@
+Workaround for a memory leak in --showrc.
+
+Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
+
+diff -ur rpm-5.4.0.orig/lib/rpmrc.c rpm-5.4.0/lib/rpmrc.c
+--- rpm-5.4.0.orig/lib/rpmrc.c	2011-02-15 20:40:13.006853913 -0600
++++ rpm-5.4.0/lib/rpmrc.c	2011-02-15 20:44:39.708972391 -0600
+@@ -1216,11 +1223,15 @@
+ 		if (DNEVR != NULL)
+ 		    fprintf(fp, "    %s\n", DNEVR+2);
+ 	    }
++#if 0
+ 	    (void)rpmdsFree(ds);
+ 	    ds = NULL;
++#endif
+ 	    fprintf(fp, "\n");
+ 	}
++#if 0
+ 	PRCO = rpmdsFreePRCO(PRCO);
++#endif
+     }
+ 
+     if (rpmIsVerbose()) {
diff --git a/meta/recipes-devtools/rpm/rpm_5.4.0.bb b/meta/recipes-devtools/rpm/rpm_5.4.0.bb
index 56fcd9c..93951d2 100644
--- a/meta/recipes-devtools/rpm/rpm_5.4.0.bb
+++ b/meta/recipes-devtools/rpm/rpm_5.4.0.bb
@@ -43,7 +43,7 @@ LICENSE = "LGPL 2.1"
 LIC_FILES_CHKSUM = "file://COPYING.LIB;md5=2d5025d4aa3495befef8f17206a5b0a1"
 
 DEPENDS = "bzip2 zlib python perl db openssl elfutils expat libpcre attr acl popt"
-PR = "r11"
+PR = "r12"
 
 # rpm2cpio is a shell script, which is part of the rpm src.rpm.  It is needed
 # in order to extract the distribution SRPM into a format we can extract...
@@ -51,7 +51,9 @@ SRC_URI = "http://www.rpm5.org/files/rpm/rpm-5.4/rpm-5.4.0-0.20101229.src.rpm;ex
 	   file://perfile_rpmdeps.sh \
 	   file://rpm-autogen.patch \
 	   file://rpm-libsql-fix.patch \
-           file://header-include-fix.patch \
+	   file://header-include-fix.patch \
+	   file://rpm-platform.patch \
+	   file://rpm-showrc.patch \
 	  "
 
 #           file://hdraddorappend.patch \
@@ -155,7 +157,7 @@ EXTRA_OECONF = "--verbose \
 		--with-path-macros=${rpm_macros} \
 		--with-bugreport=http://bugzilla.pokylinux.org"
 
-CFLAGS_append = " -DRPM_VENDOR_WINDRIVER"
+CFLAGS_append = " -DRPM_VENDOR_WINDRIVER -DRPM_VENDOR_POKY"
 
 PACKAGES = "${PN}-dbg ${PN} ${PN}-doc ${PN}-libs ${PN}-dev ${PN}-common ${PN}-build python-rpm-dbg python-rpm perl-module-rpm perl-module-rpm-dev ${PN}-locale"
 
-- 
1.7.3.4



^ permalink raw reply related	[flat|nested] 6+ messages in thread
* [PATCH 2/4] rpm: Avoid fsync when writing files
  2011-02-16  5:12 [PATCH 0/4] Fix the emenlow build failures and add optimizations Mark Hatle
  2011-02-16  5:12 ` [PATCH 1/4] rpm: Add the ability to use the platform file during install Mark Hatle
@ 2011-02-16  5:12 ` Mark Hatle
  2011-02-16  5:12 ` [PATCH 3/4] image.bbclass: Disable shell debug messages Mark Hatle
                   ` (2 subsequent siblings)
  4 siblings, 0 replies; 6+ messages in thread
From: Mark Hatle @ 2011-02-16  5:12 UTC (permalink / raw)
  To: poky

By default RPM strives for the safest possible file creation and
enforces fsync on each file.  This changes that default by backporting
the fsync disabled from the latest CVS snapshot.

This will cut a few minutes off the rootfs generation using RPM.

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
 meta/recipes-devtools/rpm/rpm/rpm-nofsync.patch |   34 +++++++++++++++++++++++
 meta/recipes-devtools/rpm/rpm_5.4.0.bb          |    1 +
 2 files changed, 35 insertions(+), 0 deletions(-)
 create mode 100644 meta/recipes-devtools/rpm/rpm/rpm-nofsync.patch

diff --git a/meta/recipes-devtools/rpm/rpm/rpm-nofsync.patch b/meta/recipes-devtools/rpm/rpm/rpm-nofsync.patch
new file mode 100644
index 0000000..33039ad
--- /dev/null
+++ b/meta/recipes-devtools/rpm/rpm/rpm-nofsync.patch
@@ -0,0 +1,34 @@
+Backport from RPM 5.4.1 CVS, disabled fsync.
+
+Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
+
+diff -ur rpm-5.4.0.orig/CHANGES rpm-5.4.0/CHANGES
+--- rpm-5.4.0.orig/CHANGES	2010-12-29 07:51:38.000000000 -0600
++++ rpm-5.4.0/CHANGES	2011-02-15 20:54:27.021926840 -0600
+@@ -1,4 +1,5 @@
+ 5.4.0 -> 5.4.1:
++    - jbj: remove fsync(2) after file install, its intolerably slow.
+ 
+ 5.3.7 -> 5.4.0:
+     - jbj: remove markReplacedFiles.
+diff -ur rpm-5.4.0.orig/lib/fsm.c rpm-5.4.0/lib/fsm.c
+--- rpm-5.4.0.orig/lib/fsm.c	2009-12-10 12:52:18.000000000 -0600
++++ rpm-5.4.0/lib/fsm.c	2011-02-15 20:55:04.372938221 -0600
+@@ -1017,7 +1017,17 @@
+ 	    (void) fsmNext(fsm, IOSM_NOTIFY);
+     }
+ 
++#ifdef DYING
++/* Measurements from installing kernel-source package:
++ * +fsync
++ *     total:               1      0.000000 MB    640.854524 secs
++ * +fdatasync
++ *     total:               1      0.000000 MB    419.983200 secs
++ * w/o fsync/fdsatasync:
++ *     total:               1      0.000000 MB     12.492918 secs
++ */
+     xx = fsync(Fileno(fsm->wfd));
++#endif
+ 
+     if (st->st_size > 0 && (fsm->fdigest || fsm->digest)) {
+ 	void * digest = NULL;
diff --git a/meta/recipes-devtools/rpm/rpm_5.4.0.bb b/meta/recipes-devtools/rpm/rpm_5.4.0.bb
index 93951d2..45dd4e1 100644
--- a/meta/recipes-devtools/rpm/rpm_5.4.0.bb
+++ b/meta/recipes-devtools/rpm/rpm_5.4.0.bb
@@ -54,6 +54,7 @@ SRC_URI = "http://www.rpm5.org/files/rpm/rpm-5.4/rpm-5.4.0-0.20101229.src.rpm;ex
 	   file://header-include-fix.patch \
 	   file://rpm-platform.patch \
 	   file://rpm-showrc.patch \
+	   file://rpm-nofsync.patch \
 	  "
 
 #           file://hdraddorappend.patch \
-- 
1.7.3.4



^ permalink raw reply related	[flat|nested] 6+ messages in thread
* [PATCH 3/4] image.bbclass: Disable shell debug messages
  2011-02-16  5:12 [PATCH 0/4] Fix the emenlow build failures and add optimizations Mark Hatle
  2011-02-16  5:12 ` [PATCH 1/4] rpm: Add the ability to use the platform file during install Mark Hatle
  2011-02-16  5:12 ` [PATCH 2/4] rpm: Avoid fsync when writing files Mark Hatle
@ 2011-02-16  5:12 ` Mark Hatle
  2011-02-16  5:12 ` [PATCH 4/4] package_rpm: Fix solverdb generation Mark Hatle
  2011-02-22 15:51 ` [PATCH 0/4] Fix the emenlow build failures and add optimizations Saul Wold
  4 siblings, 0 replies; 6+ messages in thread
From: Mark Hatle @ 2011-02-16  5:12 UTC (permalink / raw)
  To: poky

By default image had a "set -x" to enable shell debug messages,
this adds a lot of overhead to the system due to logging.

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
 meta/classes/image.bbclass |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 14de3e1..2cc17e1 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -96,7 +96,7 @@ do_build[nostamp] = "1"
 # Must call real_do_rootfs() from inside here, rather than as a separate
 # task, so that we have a single fakeroot context for the whole process.
 fakeroot do_rootfs () {
-	set -x
+	#set -x
 	rm -rf ${IMAGE_ROOTFS}
 	mkdir -p ${IMAGE_ROOTFS}
 	mkdir -p ${DEPLOY_DIR_IMAGE}
-- 
1.7.3.4



^ permalink raw reply related	[flat|nested] 6+ messages in thread
* [PATCH 4/4] package_rpm: Fix solverdb generation
  2011-02-16  5:12 [PATCH 0/4] Fix the emenlow build failures and add optimizations Mark Hatle
                   ` (2 preceding siblings ...)
  2011-02-16  5:12 ` [PATCH 3/4] image.bbclass: Disable shell debug messages Mark Hatle
@ 2011-02-16  5:12 ` Mark Hatle
  2011-02-22 15:51 ` [PATCH 0/4] Fix the emenlow build failures and add optimizations Saul Wold
  4 siblings, 0 replies; 6+ messages in thread
From: Mark Hatle @ 2011-02-16  5:12 UTC (permalink / raw)
  To: poky

The RPM solverdb was potentially being generated multiple times.
Fix this by ensuring we only process each directory once.

Also correct an issue where the solution did not necessarily follow
the preferred architecture ordering, reverse the default Poky ordering
so that preferred is listed first.

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
 meta/classes/package_rpm.bbclass      |   32 +++++++++++++++++++-------------
 meta/classes/populate_sdk_rpm.bbclass |    7 ++-----
 meta/classes/rootfs_rpm.bbclass       |    9 +++++++--
 3 files changed, 28 insertions(+), 20 deletions(-)

diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index 215be8d..7cbdae6 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -35,15 +35,22 @@ package_update_index_rpm () {
 		if [ "$sdkarch" = "all" -o "$sdkarch" = "any" -o "$sdkarch" = "noarch" ]; then
 		    extension=""
 		fi
-		packagedirs="$packagedirs ${DEPLOY_DIR_RPM}/$arch"
-		packagedirs_sdk="$packagedirs_sdk ${DEPLOY_DIR_RPM}/$sdkarch$extension"
+		packagedirs="${DEPLOY_DIR_RPM}/$arch $packagedirs"
+		packagedirs_sdk="${DEPLOY_DIR_RPM}/$sdkarch$extension $packagedirs_sdk"
+
+		rm -rf ${DEPLOY_DIR_RPM}/$arch/solvedb
+		rm -rf ${DEPLOY_DIR_RPM}/$sdkarch$extension/solvedb
 	done
 
 	cat /dev/null > ${RPMCONF_TARGET_BASE}.conf
 	for pkgdir in $packagedirs; do
 		if [ -e $pkgdir/ ]; then
 			echo "Generating solve db for $pkgdir..."
-			rm -rf $pkgdir/solvedb
+			echo $pkgdir/solvedb >> ${RPMCONF_TARGET_BASE}.conf
+			if [ -d $pkgdir/solvedb ]; then
+				# We've already processed this and it's a duplicate
+				continue
+			fi
 			mkdir -p $pkgdir/solvedb
 			echo "# Dynamically generated solve manifest" >> $pkgdir/solvedb/manifest
 			find $pkgdir -maxdepth 1 -type f >> $pkgdir/solvedb/manifest
@@ -53,7 +60,6 @@ package_update_index_rpm () {
 				--ignoresize --nosignature --nodigest \
 				-D "__dbi_txn create nofsync" \
 				$pkgdir/solvedb/manifest
-			echo $pkgdir/solvedb >> ${RPMCONF_TARGET_BASE}.conf
 		fi
 	done
 
@@ -61,7 +67,11 @@ package_update_index_rpm () {
 	for pkgdir in $packagedirs_sdk; do
 		if [ -e $pkgdir/ ]; then
 			echo "Generating solve db for $pkgdir..."
-			rm -rf $pkgdir/solvedb
+			echo $pkgdir/solvedb >> ${RPMCONF_HOST_BASE}.conf
+			if [ -d $pkgdir/solvedb ]; then
+				# We've already processed this and it's a duplicate
+				continue
+			fi	
 			mkdir -p $pkgdir/solvedb
 			echo "# Dynamically generated solve manifest" >> $pkgdir/solvedb/manifest
 			find $pkgdir -maxdepth 1 -type f >> $pkgdir/solvedb/manifest
@@ -71,7 +81,6 @@ package_update_index_rpm () {
 				--ignoresize --nosignature --nodigest \
 				-D "__dbi_txn create nofsync" \
 				$pkgdir/solvedb/manifest
-			echo $pkgdir/solvedb >> ${RPMCONF_HOST_BASE}.conf
 		fi
 	done
 }
@@ -146,8 +155,7 @@ resolve_package_rpm () {
 # install a bunch of packages using rpm
 # the following shell variables needs to be set before calling this func:
 # INSTALL_ROOTFS_RPM - install root dir
-# INSTALL_PLATFORM_RPM - main platform
-# INSTALL_PLATFORM_EXTRA_RPM - extra platform
+# INSTALL_PLATFORM_RPM - extra platform
 # INSTALL_CONFBASE_RPM - configuration file base name
 # INSTALL_PACKAGES_NORMAL_RPM - packages to be installed
 # INSTALL_PACKAGES_ATTEMPTONLY_RPM - packages attemped to be installed only
@@ -158,8 +166,7 @@ resolve_package_rpm () {
 package_install_internal_rpm () {
 
 	local target_rootfs="${INSTALL_ROOTFS_RPM}"
-	local platform="${INSTALL_PLATFORM_RPM}"
-	local platform_extra="${INSTALL_PLATFORM_EXTRA_RPM}"
+	local platforms="${INSTALL_PLATFORM_RPM}"
 	local confbase="${INSTALL_CONFBASE_RPM}"
 	local package_to_install="${INSTALL_PACKAGES_NORMAL_RPM}"
 	local package_attemptonly="${INSTALL_PACKAGES_ATTEMPTONLY_RPM}"
@@ -169,9 +176,8 @@ package_install_internal_rpm () {
 
 	# Setup base system configuration
 	mkdir -p ${target_rootfs}/etc/rpm/
-	echo "${platform}-unknown-linux" >${target_rootfs}/etc/rpm/platform
-	if [ ! -z "$platform_extra" ]; then
-		for pt in $platform_extra ; do
+	if [ ! -z "$platforms" ]; then
+		for pt in $platforms ; do
 			echo "$pt-unknown-linux" >> ${target_rootfs}/etc/rpm/platform
 		done
 	fi
diff --git a/meta/classes/populate_sdk_rpm.bbclass b/meta/classes/populate_sdk_rpm.bbclass
index e1abbad..d179780 100644
--- a/meta/classes/populate_sdk_rpm.bbclass
+++ b/meta/classes/populate_sdk_rpm.bbclass
@@ -33,8 +33,7 @@ fakeroot populate_sdk_rpm () {
 
 	#install target
 	export INSTALL_ROOTFS_RPM="${SDK_OUTPUT}/${SDKTARGETSYSROOT}"
-	export INSTALL_PLATFORM_RPM="${TARGET_ARCH}"
-	export INSTALL_PLATFORM_EXTRA_RPM="${PACKAGE_ARCHS}"
+	export INSTALL_PLATFORM_RPM="${PACKAGE_ARCHS}"
 	export INSTALL_CONFBASE_RPM="${RPMCONF_TARGET_BASE}"
 	export INSTALL_PACKAGES_NORMAL_RPM="${TOOLCHAIN_TARGET_TASK}"
 
@@ -46,7 +45,6 @@ fakeroot populate_sdk_rpm () {
 	export INSTALL_PLATFORM_RPM="${SDK_ARCH}"
 	export INSTALL_CONFBASE_RPM="${RPMCONF_HOST_BASE}"
 	export INSTALL_PACKAGES_NORMAL_RPM="${TOOLCHAIN_HOST_TASK}"
-	INSTALL_PLATFORM_EXTRA_RPM=""
 	for arch in ${PACKAGE_ARCHS}; do
 		sdkarch=`echo $arch | sed -e 's/${HOST_ARCH}/${SDK_ARCH}/'`
 		extension="-nativesdk"
@@ -54,10 +52,9 @@ fakeroot populate_sdk_rpm () {
 		    extension=""
 		fi
 		if [ -e ${DEPLOY_DIR_RPM}/$sdkarch$extension ]; then
-			INSTALL_PLATFORM_EXTRA_RPM="$INSTALL_PLATFORM_EXTRA_RPM $sdkarch"
+			INSTALL_PLATFORM_RPM="$INSTALL_PLATFORM_RPM $sdkarch"
 		fi
 	done
-	export INSTALL_PLATFORM_EXTRA_RPM
 
 	package_install_internal_rpm
 	populate_sdk_post_rpm ${INSTALL_ROOTFS_RPM}
diff --git a/meta/classes/rootfs_rpm.bbclass b/meta/classes/rootfs_rpm.bbclass
index 6fe11f7..a722e68 100644
--- a/meta/classes/rootfs_rpm.bbclass
+++ b/meta/classes/rootfs_rpm.bbclass
@@ -48,8 +48,6 @@ fakeroot rootfs_rpm_do_rootfs () {
 
 	#install pacakges
 	export INSTALL_ROOTFS_RPM="${IMAGE_ROOTFS}"
-	export INSTALL_PLATFORM_RPM="${TARGET_ARCH}"
-	export INSTALL_PLATFORM_EXTRA_RPM="${PACKAGE_ARCHS}"
 	export INSTALL_CONFBASE_RPM="${RPMCONF_TARGET_BASE}"
 	export INSTALL_PACKAGES_NORMAL_RPM="${PACKAGE_INSTALL}"
 	export INSTALL_PACKAGES_ATTEMPTONLY_RPM="${PACKAGE_INSTALL_ATTEMPTONLY}"
@@ -57,6 +55,13 @@ fakeroot rootfs_rpm_do_rootfs () {
 	export INSTALL_PROVIDENAME_RPM=""
 	export INSTALL_TASK_RPM="populate_sdk"
 
+	# List must be prefered to least preferred order
+	INSTALL_PLATFORM_RPM=""
+	for each_arch in ${PACKAGE_ARCHS} ; do
+		INSTALL_PLATFORM_RPM="$each_arch $INSTALL_PLATFORM_RPM"
+	done
+	export INSTALL_PLATFORM_RPM
+
 	package_install_internal_rpm
 
 	export D=${IMAGE_ROOTFS}
-- 
1.7.3.4



^ permalink raw reply related	[flat|nested] 6+ messages in thread
* Re: [PATCH 0/4] Fix the emenlow build failures and add optimizations
  2011-02-16  5:12 [PATCH 0/4] Fix the emenlow build failures and add optimizations Mark Hatle
                   ` (3 preceding siblings ...)
  2011-02-16  5:12 ` [PATCH 4/4] package_rpm: Fix solverdb generation Mark Hatle
@ 2011-02-22 15:51 ` Saul Wold
  4 siblings, 0 replies; 6+ messages in thread
From: Saul Wold @ 2011-02-22 15:51 UTC (permalink / raw)
  To: Mark Hatle; +Cc: poky

On 02/15/2011 09:12 PM, Mark Hatle wrote:
> We patch 1/4 fixes the emenlow build failure without adding any system optimizations.
>
> This rootfs generation takes approx 11m42 on my machine.
>
> Patch 2/4 adds a backport of head of tree RPM "nofsync" change.  This drops the
> overall rootfs generation time to approx 6m38.
>
> Patch 3/4 disables default shell debug logging during image creation.
>
> Patch 4/4 fixes a problem where the solverdb's could be generated multiple times.
> We now only generate the solvedb ONCE.  This drops the overall rootfs generation
> time to 6m16.
>
> Pull URL: git://git.pokylinux.org/poky-contrib.git
>    Branch: mhatle/emenlow-fix
>    Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=mhatle/emenlow-fix
>
> Thanks,
>      Mark Hatle<mark.hatle@windriver.com>
> ---
>
>
> Mark Hatle (4):
>    rpm: Add the ability to use the platform file during install
>    rpm: Avoid fsync when writing files
>    image.bbclass: Disable shell debug messages
>    package_rpm: Fix solverdb generation
>
>   meta/classes/image.bbclass                       |    2 +-
>   meta/classes/package_rpm.bbclass                 |   47 +++++---
>   meta/classes/populate_sdk_rpm.bbclass            |    7 +-
>   meta/classes/rootfs_rpm.bbclass                  |    9 +-
>   meta/recipes-devtools/rpm/rpm/rpm-nofsync.patch  |   34 ++++++
>   meta/recipes-devtools/rpm/rpm/rpm-platform.patch |  132 ++++++++++++++++++++++
>   meta/recipes-devtools/rpm/rpm/rpm-showrc.patch   |   23 ++++
>   meta/recipes-devtools/rpm/rpm_5.4.0.bb           |    9 +-
>   8 files changed, 234 insertions(+), 29 deletions(-)
>   create mode 100644 meta/recipes-devtools/rpm/rpm/rpm-nofsync.patch
>   create mode 100644 meta/recipes-devtools/rpm/rpm/rpm-platform.patch
>   create mode 100644 meta/recipes-devtools/rpm/rpm/rpm-showrc.patch
>
Pulled into Master

Thanks
	Sau!



^ permalink raw reply	[flat|nested] 6+ messages in thread
end of thread, other threads:[~2011-02-22 15:51 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-02-16  5:12 [PATCH 0/4] Fix the emenlow build failures and add optimizations Mark Hatle
2011-02-16  5:12 ` [PATCH 1/4] rpm: Add the ability to use the platform file during install Mark Hatle
2011-02-16  5:12 ` [PATCH 2/4] rpm: Avoid fsync when writing files Mark Hatle
2011-02-16  5:12 ` [PATCH 3/4] image.bbclass: Disable shell debug messages Mark Hatle
2011-02-16  5:12 ` [PATCH 4/4] package_rpm: Fix solverdb generation Mark Hatle
2011-02-22 15:51 ` [PATCH 0/4] Fix the emenlow build failures and add optimizations Saul Wold

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.