* [PATCH 0/9 V2] packagefeed-stability: add class to help reduce package feed churn
@ 2016-07-18 3:32 Robert Yang
2016-07-18 3:32 ` [PATCH 1/9] rpm: make --nosignature work Robert Yang
` (8 more replies)
0 siblings, 9 replies; 10+ messages in thread
From: Robert Yang @ 2016-07-18 3:32 UTC (permalink / raw)
To: openembedded-core
* V2
- Make ipk and deb work, now all of rpm, ipk and deb work well.
- World rebuild testing:
1) bitbake world
2) Edit base_do_configure()
3) bitbake world
There are 667 recipes *rebuilt*:
a) For rpm: 61 recipes' PR have been bumped (less than 10%)
b) For ipk and deb: 55 recipes' PR have been bumped (less than 10%)
The result is better than I expected. I've check these packages, some of
the PR bump are caused by pseudo's permission issues. For the others,
they are caused by dbg, ptest of elf different.
* V1
- This work is based on Paul's packagefeed-stability, thanks Paul. It is
aimed for reducing PR bumps for binary packages when use PRServer.
It works well for rpm:
1) Afer bumped rpm-native's PR and rebuild world, less than 20 recipes'
Packages bumped PR, they are because of pseudo, I had sent email
before:
http://lists.openembedded.org/pipermail/openembedded-core/2016-July/123511.html
2) When elf files are rebuilt, it will bump the PR.
- Sent build-compare' patches to upstream.
// Robert
The following changes since commit da7a2c7b00b40a8759dbe9f4ab6df3e337e3d6b6:
useradd-staticids: use map() instead of imap() (2016-07-12 23:11:57 +0100)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib rbt/feed
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/feed
Paul Eggleton (1):
packagefeed-stability: add class to help reduce package feed churn
Robert Yang (8):
rpm: make --nosignature work
build-compare: remove space at head
build-compare: make pkg-diff.sh 75% faster
build-compare: fix checking for named pipe and others
build-compare: improve deb and ipk checking
package_deb.bbclass/package_ipk.bbclass: sort RPROVIDES
packagefeed-stability.bbclass: minor fixes
packagefeed-stability.bbclass: copy all packages of a recipe
meta/classes/package_deb.bbclass | 2 +
meta/classes/package_ipk.bbclass | 2 +
meta/classes/packagefeed-stability.bbclass | 239 ++++++++++++++
.../build-compare/build-compare_git.bb | 6 +
...001-Add-support-for-deb-and-ipk-packaging.patch | 4 +-
...functions.sh-improve-deb-and-ipk-checking.patch | 117 +++++++
.../files/functions.sh-remove-space-at-head.patch | 41 +++
...nctions.sh-run-rpm-once-to-make-it-faster.patch | 361 +++++++++++++++++++++
.../pkg-diff.sh-check-for-fifo-named-pipe.patch | 35 ++
...-check_single_file-return-at-once-when-sa.patch | 37 +++
...diff.sh-remove-space-in-the-end-for-ftype.patch | 32 ++
...0001-system.h-query.c-support-nosignature.patch | 64 ++++
meta/recipes-devtools/rpm/rpm_5.4.16.bb | 1 +
13 files changed, 939 insertions(+), 2 deletions(-)
create mode 100644 meta/classes/packagefeed-stability.bbclass
create mode 100644 meta/recipes-devtools/build-compare/files/functions.sh-improve-deb-and-ipk-checking.patch
create mode 100644 meta/recipes-devtools/build-compare/files/functions.sh-remove-space-at-head.patch
create mode 100644 meta/recipes-devtools/build-compare/files/functions.sh-run-rpm-once-to-make-it-faster.patch
create mode 100644 meta/recipes-devtools/build-compare/files/pkg-diff.sh-check-for-fifo-named-pipe.patch
create mode 100644 meta/recipes-devtools/build-compare/files/pkg-diff.sh-check_single_file-return-at-once-when-sa.patch
create mode 100644 meta/recipes-devtools/build-compare/files/pkg-diff.sh-remove-space-in-the-end-for-ftype.patch
create mode 100644 meta/recipes-devtools/rpm/rpm/0001-system.h-query.c-support-nosignature.patch
--
2.9.0
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH 1/9] rpm: make --nosignature work
2016-07-18 3:32 [PATCH 0/9 V2] packagefeed-stability: add class to help reduce package feed churn Robert Yang
@ 2016-07-18 3:32 ` Robert Yang
2016-07-18 3:32 ` [PATCH 2/9] build-compare: remove space at head Robert Yang
` (7 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Robert Yang @ 2016-07-18 3:32 UTC (permalink / raw)
To: openembedded-core
OE-core uses rpm's --nosignature, but it never worked:
self._invoke_smart('config --set rpm-check-signatures=false')
Now fix it with:
* Define SUPPORT_NOSIGNATURES to 1 in system.h
* !QVA_ISSET(qva->qva_flags, SIGNATURE) -> QVA_ISSET(qva->qva_flags, SIGNATURE),
otherwise, when use --nosignature would read database and verify
signature, this is not expected.
This can fix some race issues, for example, when more than one process
are querying rpm file with "rpm -qp --nosignature", they may hang up
because of race issues (the processes are trying to get RW/RD lock on
the database, but they shouldn't read the database at all since -qp and
--nosignature are used).
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
...0001-system.h-query.c-support-nosignature.patch | 64 ++++++++++++++++++++++
meta/recipes-devtools/rpm/rpm_5.4.16.bb | 1 +
2 files changed, 65 insertions(+)
create mode 100644 meta/recipes-devtools/rpm/rpm/0001-system.h-query.c-support-nosignature.patch
diff --git a/meta/recipes-devtools/rpm/rpm/0001-system.h-query.c-support-nosignature.patch b/meta/recipes-devtools/rpm/rpm/0001-system.h-query.c-support-nosignature.patch
new file mode 100644
index 0000000..410ae78
--- /dev/null
+++ b/meta/recipes-devtools/rpm/rpm/0001-system.h-query.c-support-nosignature.patch
@@ -0,0 +1,64 @@
+From d8fcc3cbeec4defb5babde5bd09ea294dd3fd08b Mon Sep 17 00:00:00 2001
+From: Robert Yang <liezhi.yang@windriver.com>
+Date: Sun, 3 Jul 2016 19:01:59 -0700
+Subject: [PATCH] system.h/query.c: support nosignature
+
+* Define SUPPORT_NOSIGNATURES to 1 in system.h
+* !QVA_ISSET(qva->qva_flags, SIGNATURE) -> QVA_ISSET(qva->qva_flags, SIGNATURE),
+ otherwise, when use --nosignature would read database and verify
+ signature, this is not expected.
+
+Upstream-Status: Submitting [Sent email to rpm-devel@rpm5.org]
+
+Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
+---
+ lib/query.c | 6 +++---
+ system.h | 4 ++--
+ 2 files changed, 5 insertions(+), 5 deletions(-)
+
+diff --git a/lib/query.c b/lib/query.c
+index 50a7453..b761d76 100644
+--- a/lib/query.c
++++ b/lib/query.c
+@@ -954,19 +954,19 @@ JBJDEBUG((stderr, "--> %s(%p,%p,%p)\n", __FUNCTION__, ts, qva, argv));
+ vsflags = (rpmVSFlags) rpmExpandNumeric("%{?_vsflags_query}");
+ vsflags = (rpmVSFlags) 0; /* XXX FIXME: ignore default disablers. */
+ #if defined(SUPPORT_NOSIGNATURES)
+- if (!QVA_ISSET(qva->qva_flags, DIGEST)) {
++ if (QVA_ISSET(qva->qva_flags, DIGEST)) {
+ VSF_SET(vsflags, NOSHA1HEADER);
+ VSF_SET(vsflags, NOMD5HEADER);
+ VSF_SET(vsflags, NOSHA1);
+ VSF_SET(vsflags, NOMD5);
+ }
+- if (!QVA_ISSET(qva->qva_flags, SIGNATURE)) {
++ if (QVA_ISSET(qva->qva_flags, SIGNATURE)) {
+ VSF_SET(vsflags, NODSAHEADER);
+ VSF_SET(vsflags, NORSAHEADER);
+ VSF_SET(vsflags, NODSA);
+ VSF_SET(vsflags, NORSA);
+ }
+- if (!QVA_ISSET(qva->qva_flags, HDRCHK)) {
++ if (QVA_ISSET(qva->qva_flags, HDRCHK)) {
+ VSF_SET(vsflags, NOHDRCHK);
+ }
+ VSF_CLR(vsflags, NEEDPAYLOAD); /* XXX needed? */
+diff --git a/system.h b/system.h
+index 2ff8906..ad4619a 100644
+--- a/system.h
++++ b/system.h
+@@ -787,9 +787,9 @@ static inline const char *rcsid(const char *p) { \
+ #define SUPPORT_I18NSTRING_TYPE 1
+
+ /**
+- * Eliminate signature/digest disablers.
++ * Signature/digest disablers.
+ */
+-#undef SUPPORT_NOSIGNATURES
++#define SUPPORT_NOSIGNATURES 1
+
+ /**
+ * Permit ar(1) payloads. Disabled while rpmio/iosm.c is under development.
+--
+2.9.0
+
diff --git a/meta/recipes-devtools/rpm/rpm_5.4.16.bb b/meta/recipes-devtools/rpm/rpm_5.4.16.bb
index 3adb29c..d264d50 100644
--- a/meta/recipes-devtools/rpm/rpm_5.4.16.bb
+++ b/meta/recipes-devtools/rpm/rpm_5.4.16.bb
@@ -117,6 +117,7 @@ SRC_URI += " \
file://rpmqv.c-check-_gpg_passphrase-before-ask-for-input.patch \
file://0001-Disable-__sync_add_and_fetch_8-on-nios2.patch \
file://gcc6-stdlib.patch \
+ file://0001-system.h-query.c-support-nosignature.patch \
"
# OE specific changes
--
2.9.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH 2/9] build-compare: remove space at head
2016-07-18 3:32 [PATCH 0/9 V2] packagefeed-stability: add class to help reduce package feed churn Robert Yang
2016-07-18 3:32 ` [PATCH 1/9] rpm: make --nosignature work Robert Yang
@ 2016-07-18 3:32 ` Robert Yang
2016-07-18 3:32 ` [PATCH 3/9] build-compare: make pkg-diff.sh 75% faster Robert Yang
` (6 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Robert Yang @ 2016-07-18 3:32 UTC (permalink / raw)
To: openembedded-core
The command like:
rpm -qp --nodigest --nosignature --qf '<foo> [%{REQUIRENAME}\n]\n'
^^space
The space will be printed, and will impact the check result, so remove it.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
.../build-compare/build-compare_git.bb | 1 +
.../files/functions.sh-remove-space-at-head.patch | 41 ++++++++++++++++++++++
2 files changed, 42 insertions(+)
create mode 100644 meta/recipes-devtools/build-compare/files/functions.sh-remove-space-at-head.patch
diff --git a/meta/recipes-devtools/build-compare/build-compare_git.bb b/meta/recipes-devtools/build-compare/build-compare_git.bb
index 676f11d..88996e2 100644
--- a/meta/recipes-devtools/build-compare/build-compare_git.bb
+++ b/meta/recipes-devtools/build-compare/build-compare_git.bb
@@ -9,6 +9,7 @@ SRC_URI = "git://github.com/openSUSE/build-compare.git \
file://Rename-rpm-check.sh-to-pkg-diff.sh.patch;striplevel=1 \
file://Ignore-DWARF-sections.patch;striplevel=1 \
file://0001-Add-support-for-deb-and-ipk-packaging.patch \
+ file://functions.sh-remove-space-at-head.patch \
"
SRCREV = "c5352c054c6ef15735da31b76d6d88620f4aff0a"
diff --git a/meta/recipes-devtools/build-compare/files/functions.sh-remove-space-at-head.patch b/meta/recipes-devtools/build-compare/files/functions.sh-remove-space-at-head.patch
new file mode 100644
index 0000000..02e96ba
--- /dev/null
+++ b/meta/recipes-devtools/build-compare/files/functions.sh-remove-space-at-head.patch
@@ -0,0 +1,41 @@
+From 402a821e604f280e62c34bf4d40c6d1b4c6c892d Mon Sep 17 00:00:00 2001
+From: Robert Yang <liezhi.yang@windriver.com>
+Date: Mon, 4 Jul 2016 20:11:33 -0700
+Subject: [PATCH] functions.sh: remove space at head
+
+The command like:
+rpm -qp --nodigest --nosignature --qf '<foo> [%{REQUIRENAME}\n]\n'
+ ^^space
+
+The space will be printed, and will impact the check result, so remove it.
+
+Upstream-Status: Submitted [https://github.com/openSUSE/build-compare/pull/9]
+
+Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
+---
+ functions.sh | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/functions.sh b/functions.sh
+index 85c9003..b1069d2 100644
+--- a/functions.sh
++++ b/functions.sh
+@@ -26,6 +26,7 @@ function trim_release_old()
+ sed -e "
+ /\(\/boot\|\/lib\/modules\|\/lib\/firmware\|\/usr\/src\|$version_release_old_regex_l\$\)/{s,$version_release_old_regex_l,@VERSION@-@RELEASE_LONG@,g;s,$version_release_old_regex_s,@VERSION@-@RELEASE_SHORT@,g}
+ s/\(\/var\/adm\/update-scripts\/\)${name_ver_rel_old_regex_l}\([^[:blank:]]\+\)/\1@NAME_VER_REL@\2/g
++ s/^ *//g
+ "
+ }
+ function trim_release_new()
+@@ -33,6 +34,7 @@ function trim_release_new()
+ sed -e "
+ /\(\/boot\|\/lib\/modules\|\/lib\/firmware\|\/usr\/src\|$version_release_new_regex_l\$\)/{s,$version_release_new_regex_l,@VERSION@-@RELEASE_LONG@,g;s,$version_release_new_regex_s,@VERSION@-@RELEASE_SHORT@,g}
+ s/\(\/var\/adm\/update-scripts\/\)${name_ver_rel_new_regex_l}\([^[:blank:]]\+\)/\1@NAME_VER_REL@\2/g
++ s/^ *//g
+ "
+ }
+ # Get single directory or filename with long or short release string
+--
+2.9.0
+
--
2.9.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH 3/9] build-compare: make pkg-diff.sh 75% faster
2016-07-18 3:32 [PATCH 0/9 V2] packagefeed-stability: add class to help reduce package feed churn Robert Yang
2016-07-18 3:32 ` [PATCH 1/9] rpm: make --nosignature work Robert Yang
2016-07-18 3:32 ` [PATCH 2/9] build-compare: remove space at head Robert Yang
@ 2016-07-18 3:32 ` Robert Yang
2016-07-18 3:32 ` [PATCH 4/9] build-compare: fix checking for named pipe and others Robert Yang
` (5 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Robert Yang @ 2016-07-18 3:32 UTC (permalink / raw)
To: openembedded-core
The rpm tool is a heavy process, pkg-diff.sh ran 16 (or 17 for kernel)
"rpm -qp" times when the pkgs are identical, now we only run
"rpm -qp --qf <all we need>" twice (one is for old pkg, and one is for
new), save the results to spec_old and spec_new, then use sed command to
get what we need later, this can make it 75% faster when the pkgs are
identical. Here is the rough data on my host Ubuntu 14.04.4, 32 cores
CPU and 128G mem:
* When the pkgs are identical:
- Before the patch: 1s
- After the patch: 0.26s
I compare the whole spec firstly, and return 0 if they are the same,
or go on checking one by one if not, without this, it would be 0.46s,
the gain is great when there are lot of packages, usually, we have
more than 10,000 rpms to compare.
* When the pkgs are different:
That depends on where is the different, if the different is at the
comparing rpmtags stage:
- Before the patch: 0.26s
- After the patch: 0.29s
Increased 0.03s, but if the different is happend later than comparing
rpmtags, it will save time.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
.../build-compare/build-compare_git.bb | 1 +
...nctions.sh-run-rpm-once-to-make-it-faster.patch | 361 +++++++++++++++++++++
2 files changed, 362 insertions(+)
create mode 100644 meta/recipes-devtools/build-compare/files/functions.sh-run-rpm-once-to-make-it-faster.patch
diff --git a/meta/recipes-devtools/build-compare/build-compare_git.bb b/meta/recipes-devtools/build-compare/build-compare_git.bb
index 88996e2..9ec8c23 100644
--- a/meta/recipes-devtools/build-compare/build-compare_git.bb
+++ b/meta/recipes-devtools/build-compare/build-compare_git.bb
@@ -10,6 +10,7 @@ SRC_URI = "git://github.com/openSUSE/build-compare.git \
file://Ignore-DWARF-sections.patch;striplevel=1 \
file://0001-Add-support-for-deb-and-ipk-packaging.patch \
file://functions.sh-remove-space-at-head.patch \
+ file://functions.sh-run-rpm-once-to-make-it-faster.patch \
"
SRCREV = "c5352c054c6ef15735da31b76d6d88620f4aff0a"
diff --git a/meta/recipes-devtools/build-compare/files/functions.sh-run-rpm-once-to-make-it-faster.patch b/meta/recipes-devtools/build-compare/files/functions.sh-run-rpm-once-to-make-it-faster.patch
new file mode 100644
index 0000000..f5b99d9
--- /dev/null
+++ b/meta/recipes-devtools/build-compare/files/functions.sh-run-rpm-once-to-make-it-faster.patch
@@ -0,0 +1,361 @@
+From f34d632c427b1e1570ef9136454fc01d8c9f10a6 Mon Sep 17 00:00:00 2001
+From: Robert Yang <liezhi.yang@windriver.com>
+Date: Thu, 7 Jul 2016 19:34:09 -0700
+Subject: [PATCH] functions.sh: run rpm once to make it faster
+
+The rpm tool is a heavy process, it ran 16 (or 17 for kernel)
+"rpm -qp" times when the pkgs are identical, now we only run
+"rpm -qp --qf <all we need>" twice (one is for old pkg, and one is for
+new), save the results to spec_old and spec_new, then use sed command to
+get what we need later, this can make it 75% faster when the pkgs are
+identical. Here is the rough data on my host Ubuntu 14.04.4, 32 cores
+CPU and 128G mem:
+* When the pkgs are identical:
+ - Before the patch: 1s
+ - After the patch: 0.26s
+ I compare the whole spec firstly, and return 0 if they are the same,
+ or go on checking one by one if not, without this, it would be 0.46s,
+ the gain is great when there are lot of packages, usually, we have
+ more than 10,000 rpms to compare.
+
+* When the pkgs are different:
+ That depends on where is the different, if the different is at the
+ comparing rpmtags stage:
+ - Before the patch: 0.26s
+ - After the patch: 0.29s
+ Increased 0.03s, but if the different is happend later than comparing
+ rpmtags, it will save time.
+
+Upstream-Status: Submitted [https://github.com/openSUSE/build-compare/pull/9]
+
+Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
+---
+ functions.sh | 245 ++++++++++++++++++++++++++++++++---------------------------
+ 1 file changed, 132 insertions(+), 113 deletions(-)
+ mode change 100644 => 100755 functions.sh
+
+diff --git a/functions.sh b/functions.sh
+old mode 100644
+new mode 100755
+index b1069d2..aa572f9
+--- a/functions.sh
++++ b/functions.sh
+@@ -10,9 +10,63 @@
+
+ RPM="rpm -qp --nodigest --nosignature"
+
+-check_header()
++# Name, Version, Release
++QF_NAME="%{NAME}"
++QF_VER_REL="%{VERSION}-%{RELEASE}"
++QF_NAME_VER_REL="%{NAME}-%{VERSION}-%{RELEASE}"
++
++# provides destroy this because at least the self-provide includes the
++# -buildnumber :-(
++QF_PROVIDES="[%{PROVIDENAME} %{PROVIDEFLAGS} %{PROVIDEVERSION}\\n]\\n"
++QF_PROVIDES="$QF_PROVIDES [%{REQUIRENAME} %{REQUIREFLAGS} %{REQUIREVERSION}\\n]\\n"
++QF_PROVIDES="$QF_PROVIDES [%{CONFLICTNAME} %{CONFLICTFLAGS} %{CONFLICTVERSION}\\n]\\n"
++QF_PROVIDES="$QF_PROVIDES [%{OBSOLETENAME} %{OBSOLETEFLAGS} %{OBSOLETEVERSION}\\n]\\n"
++
++# don't look at RELEASE, it contains our build number
++QF_TAGS="%{NAME} %{VERSION} %{EPOCH}\\n"
++QF_TAGS="$QF_TAGS %{SUMMARY}\\n%{DESCRIPTION}\\n"
++# the DISTURL tag can be used as checkin ID
++QF_TAGS="$QF_TAGS %{VENDOR} %{DISTRIBUTION} %{DISTURL}"
++QF_TAGS="$QF_TAGS %{LICENSE} %{LICENSE}\\n"
++QF_TAGS="$QF_TAGS %{GROUP} %{URL} %{EXCLUDEARCH} %{EXCLUDEOS} %{EXCLUSIVEARCH}\\n"
++QF_TAGS="$QF_TAGS %{EXCLUSIVEOS} %{RPMVERSION} %{PLATFORM}\\n"
++QF_TAGS="$QF_TAGS %{PAYLOADFORMAT} %{PAYLOADCOMPRESSOR} %{PAYLOADFLAGS}\\n"
++
++# XXX We also need to check the existence (but not the content (!))
++# of SIGGPG (and perhaps the other SIG*)
++# XXX We don't look at triggers
++QF_TAGS="$QF_TAGS [%{VERIFYSCRIPTPROG} %{VERIFYSCRIPT}]\\n"
++# Only the first ChangeLog entry; should be enough
++QF_TAGS="$QF_TAGS %{CHANGELOGTIME} %{CHANGELOGNAME} %{CHANGELOGTEXT}\\n"
++
++# scripts, might contain release number
++QF_SCRIPT="[%{PREINPROG} %{PREIN}\\n]\\n[%{POSTINPROG} %{POSTIN}\\n]\\n[%{PREUNPROG} %{PREUN}\\n]\\n[%{POSTUNPROG} %{POSTUN}\\n]\\n"
++
++# Now the files. We leave out mtime and size. For normal files
++# the size will influence the MD5 anyway. For directories the sizes can
++# differ, depending on which file system the package was built. To not
++# have to filter out directories we simply ignore all sizes.
++# Also leave out FILEDEVICES, FILEINODES (depends on the build host),
++# FILECOLORS, FILECLASS (normally useful but file output contains mtimes),
++# FILEDEPENDSX and FILEDEPENDSN.
++# Also FILELANGS (or?)
++QF_FILELIST="[%{FILENAMES} %{FILEFLAGS} %{FILESTATES} %{FILEMODES:octal} %{FILEUSERNAME} %{FILEGROUPNAME} %{FILERDEVS} %{FILEVERIFYFLAGS} %{FILELINKTOS}\n]\\n"
++# ??? what to do with FILEPROVIDE and FILEREQUIRE?
++
++QF_CHECKSUM="[%{FILENAMES} %{FILEMD5S} %{FILEFLAGS}\n]\\n"
++
++QF_ALL="\n___QF_NAME___\n${QF_NAME}\n___QF_NAME___\n"
++QF_ALL="$QF_ALL\n___QF_TAGS___\n${QF_TAGS}\n___QF_TAGS___\n"
++QF_ALL="$QF_ALL\n___QF_VER_REL___\n${QF_VER_REL}\n___QF_VER_REL___\n"
++QF_ALL="$QF_ALL\n___QF_NAME_VER_REL___\n${QF_NAME_VER_REL}\n___QF_NAME_VER_REL___\n"
++QF_ALL="$QF_ALL\n___QF_PROVIDES___\n${QF_PROVIDES}\n___QF_PROVIDES___\n"
++QF_ALL="$QF_ALL\n___QF_SCRIPT___\n${QF_SCRIPT}\n___QF_SCRIPT___\n"
++QF_ALL="$QF_ALL\n___QF_FILELIST___\n${QF_FILELIST}\n___QF_FILELIST___\n"
++QF_ALL="$QF_ALL\n___QF_CHECKSUM___\n${QF_CHECKSUM}\n___QF_CHECKSUM___\n"
++
++check_header()
+ {
+- $RPM --qf "$QF" "$1"
++ $RPM --qf "$1" "$2"
+ }
+
+ # Trim version-release string:
+@@ -47,18 +101,6 @@ function grep_release_new()
+ grep -E "(/boot|/lib/modules|/lib/firmware|/usr/src)/[^/]+(${version_release_new_regex_l}(\$|[^/]+\$)|${version_release_new_regex_s}(\$|[^/]+\$))"
+ }
+
+-function check_provides()
+-{
+- local pkg=$1
+- # provides destroy this because at least the self-provide includes the
+- # -buildnumber :-(
+- QF="[%{PROVIDENAME} %{PROVIDEFLAGS} %{PROVIDEVERSION}\\n]\\n"
+- QF="$QF [%{REQUIRENAME} %{REQUIREFLAGS} %{REQUIREVERSION}\\n]\\n"
+- QF="$QF [%{CONFLICTNAME} %{CONFLICTFLAGS} %{CONFLICTVERSION}\\n]\\n"
+- QF="$QF [%{OBSOLETENAME} %{OBSOLETEFLAGS} %{OBSOLETEVERSION}\\n]\\n"
+- check_header "$pkg"
+-}
+-
+ #usage unpackage <file> $dir
+ # Unpack files in directory $dir
+ # like /usr/bin/unpackage - just for one file and with no options
+@@ -98,6 +140,30 @@ function unpackage()
+ popd 1>/dev/null
+ }
+
++# Run diff command on the files
++# $1: printed info
++# $2: file1
++# $3: file2
++function comp_file()
++{
++ echo "comparing $1"
++ if ! diff -au $2 $3; then
++ if test -z "$check_all"; then
++ rm $2 $3 $spec_old $spec_new
++ return 1
++ fi
++ fi
++ return 0
++}
++
++# Get var's value from specfile.
++# $1: var name
++# $2: specfile
++function get_value()
++{
++ sed -n -e "/^___${1}___/,/^___${1}___/p" $2 | sed -e "/^___${1}___/d"
++}
++
+ # Compare just the rpm meta data of two rpms
+ # Returns:
+ # 0 in case of same content
+@@ -107,56 +173,29 @@ function unpackage()
+ function cmp_spec ()
+ {
+ local RES
+- local file1 file2
++ local file_old file_new
+ local f
+ local sh=$1
+ local oldrpm=$2
+ local newrpm=$3
+
+- QF="%{NAME}"
+-
+- # don't look at RELEASE, it contains our build number
+- QF="$QF %{VERSION} %{EPOCH}\\n"
+- QF="$QF %{SUMMARY}\\n%{DESCRIPTION}\\n"
+- QF="$QF %{VENDOR} %{DISTRIBUTION} %{DISTURL}"
+- QF="$QF %{LICENSE} %{LICENSE}\\n"
+- QF="$QF %{GROUP} %{URL} %{EXCLUDEARCH} %{EXCLUDEOS} %{EXCLUSIVEARCH}\\n"
+- QF="$QF %{EXCLUSIVEOS} %{RPMVERSION} %{PLATFORM}\\n"
+- QF="$QF %{PAYLOADFORMAT} %{PAYLOADCOMPRESSOR} %{PAYLOADFLAGS}\\n"
+-
+-
+- # XXX We also need to check the existence (but not the content (!))
+- # of SIGGPG (and perhaps the other SIG*)
+-
+- # XXX We don't look at triggers
+-
+- QF="$QF [%{VERIFYSCRIPTPROG} %{VERIFYSCRIPT}]\\n"
+-
+- # Only the first ChangeLog entry; should be enough
+- QF="$QF %{CHANGELOGTIME} %{CHANGELOGNAME} %{CHANGELOGTEXT}\\n"
+-
+- file1=`mktemp`
+- file2=`mktemp`
+-
+- check_header $oldrpm > $file1
+- check_header $newrpm > $file2
+-
+- # the DISTURL tag can be used as checkin ID
+- #echo "$QF"
+- echo "comparing rpmtags"
+- if ! diff -au $file1 $file2; then
+- if test -z "$check_all"; then
+- rm $file1 $file2
+- return 1
+- fi
+- fi
+-
++ file_old=`mktemp`
++ file_new=`mktemp`
++ spec_old=`mktemp`
++ spec_new=`mktemp`
++
++ check_header "$QF_ALL" $oldrpm > $spec_old
++ check_header "$QF_ALL" $newrpm > $spec_new
++
++ name_new="$(get_value QF_NAME $spec_new)"
++ version_release_new="$(get_value QF_VER_REL $spec_new)"
++ name_ver_rel_new="$(get_value QF_NAME_VER_REL $spec_new)"
++
++ version_release_old="$(get_value QF_VER_REL $spec_old)"
++ name_ver_rel_old="$(get_value QF_NAME_VER_REL $spec_old)"
++
+ # Remember to quote the . which is in release
+- version_release_old=$($RPM --qf "%{VERSION}-%{RELEASE}" "$oldrpm")
+- version_release_new=$($RPM --qf "%{VERSION}-%{RELEASE}" "$newrpm")
+- name_ver_rel_old=$($RPM --qf "%{NAME}-%{VERSION}-%{RELEASE}" "$oldrpm")
+- name_ver_rel_new=$($RPM --qf "%{NAME}-%{VERSION}-%{RELEASE}" "$newrpm")
+- # Short version without B_CNT
++ # Short version without B_CN
+ version_release_old_regex_s=${version_release_old%.*}
+ version_release_old_regex_s=${version_release_old_regex_s//./\\.}
+ version_release_new_regex_s=${version_release_new%.*}
+@@ -166,10 +205,27 @@ function cmp_spec ()
+ version_release_new_regex_l=${version_release_new//./\\.}
+ name_ver_rel_old_regex_l=${name_ver_rel_old//./\\.}
+ name_ver_rel_new_regex_l=${name_ver_rel_new//./\\.}
++
++ # Check the whole spec file at first, return 0 immediately if the
++ # are the same.
++ cat $spec_old | trim_release_old > $file_old
++ cat $spec_new | trim_release_new > $file_new
++ echo "comparing the whole specfile"
++ if diff -au $spec_old $spec_new; then
++ if test -z "$check_all"; then
++ rm $file_old $file_new $spec_old $spec_new
++ return 0
++ fi
++ fi
++
++ get_value QF_TAGS $spec_old > $file_old
++ get_value QF_TAGS $spec_new > $file_new
++ comp_file rpmtags $file_old $file_new || return 1
++
+ # This might happen when?!
+ echo "comparing RELEASE"
+ if [ "${version_release_old%.*}" != "${version_release_new%.*}" ] ; then
+- case $($RPM --qf '%{NAME}' "$newrpm") in
++ case $name_new in
+ kernel-*)
+ # Make sure all kernel packages have the same %RELEASE
+ echo "release prefix mismatch"
+@@ -181,71 +237,34 @@ function cmp_spec ()
+ *) ;;
+ esac
+ fi
+-
+- check_provides $oldrpm | trim_release_old | sort > $file1
+- check_provides $newrpm | trim_release_new | sort > $file2
+-
+- echo "comparing PROVIDES"
+- if ! diff -au $file1 $file2; then
+- if test -z "$check_all"; then
+- rm $file1 $file2
+- return 1
+- fi
+- fi
+
+- # scripts, might contain release number
+- QF="[%{PREINPROG} %{PREIN}\\n]\\n[%{POSTINPROG} %{POSTIN}\\n]\\n[%{PREUNPROG} %{PREUN}\\n]\\n[%{POSTUNPROG} %{POSTUN}\\n]\\n"
+- check_header $oldrpm | trim_release_old > $file1
+- check_header $newrpm | trim_release_new > $file2
++ get_value QF_PROVIDES $spec_old | trim_release_old | sort > $file_old
++ get_value QF_PROVIDES $spec_new | trim_release_new | sort > $file_new
++ comp_file PROVIDES $file_old $file_new || return 1
++
++ get_value QF_SCRIPT $spec_old | trim_release_old > $file_old
++ get_value QF_SCRIPT $spec_new | trim_release_new > $file_new
++ comp_file scripts $file_old $file_new || return 1
+
+- echo "comparing scripts"
+- if ! diff -au $file1 $file2; then
+- if test -z "$check_all"; then
+- rm $file1 $file2
+- return 1
+- fi
+- fi
+-
+ # First check the file attributes and later the md5s
+-
+- # Now the files. We leave out mtime and size. For normal files
+- # the size will influence the MD5 anyway. For directories the sizes can
+- # differ, depending on which file system the package was built. To not
+- # have to filter out directories we simply ignore all sizes.
+- # Also leave out FILEDEVICES, FILEINODES (depends on the build host),
+- # FILECOLORS, FILECLASS (normally useful but file output contains mtimes),
+- # FILEDEPENDSX and FILEDEPENDSN.
+- # Also FILELANGS (or?)
+- QF="[%{FILENAMES} %{FILEFLAGS} %{FILESTATES} %{FILEMODES:octal} %{FILEUSERNAME} %{FILEGROUPNAME} %{FILERDEVS} %{FILEVERIFYFLAGS} %{FILELINKTOS}\n]\\n"
+- # ??? what to do with FILEPROVIDE and FILEREQUIRE?
+-
+- check_header $oldrpm | trim_release_old > $file1
+- check_header $newrpm | trim_release_new > $file2
+-
+- echo "comparing filelist"
+- if ! diff -au $file1 $file2; then
+- if test -z "$check_all"; then
+- rm $file1 $file2
+- return 1
+- fi
+- fi
+-
++ get_value QF_FILELIST $spec_old | trim_release_old > $file_old
++ get_value QF_FILELIST $spec_new | trim_release_new > $file_new
++ comp_file filelist $file_old $file_new || return 1
++
+ # now the md5sums. if they are different, we check more detailed
+ # if there are different filenames, we will already have aborted before
+ # file flag 64 means "ghost", filter those out.
+- QF="[%{FILENAMES} %{FILEMD5S} %{FILEFLAGS}\n]\\n"
+- check_header $oldrpm |grep -v " 64$"| trim_release_old > $file1
+- check_header $newrpm |grep -v " 64$"| trim_release_new > $file2
+-
++ get_value QF_CHECKSUM $spec_old | grep -v " 64$" | trim_release_old > $file_old
++ get_value QF_CHECKSUM $spec_new | grep -v " 64$" | trim_release_new > $file_new
+ RES=2
+ # done if the same
+ echo "comparing file checksum"
+- if cmp -s $file1 $file2; then
++ if cmp -s $file_old $file_new; then
+ RES=0
+ fi
+-
++
+ # Get only files with different MD5sums
+- files=`diff -U0 $file1 $file2 | fgrep -v +++ | grep ^+ | cut -b2- | awk '{print $1}'`
++ files=`diff -U0 $file_old $file_new | fgrep -v +++ | grep ^+ | cut -b2- | awk '{print $1}'`
+
+ if test -f "$sh"; then
+ echo "creating rename script"
+@@ -261,7 +280,7 @@ function cmp_spec ()
+ done >> "${sh}"
+ fi
+ #
+- rm $file1 $file2
++ rm $file_old $file_new
+ return $RES
+ }
+
+--
+2.9.0
+
--
2.9.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH 4/9] build-compare: fix checking for named pipe and others
2016-07-18 3:32 [PATCH 0/9 V2] packagefeed-stability: add class to help reduce package feed churn Robert Yang
` (2 preceding siblings ...)
2016-07-18 3:32 ` [PATCH 3/9] build-compare: make pkg-diff.sh 75% faster Robert Yang
@ 2016-07-18 3:32 ` Robert Yang
2016-07-18 3:32 ` [PATCH 5/9] build-compare: improve deb and ipk checking Robert Yang
` (4 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Robert Yang @ 2016-07-18 3:32 UTC (permalink / raw)
To: openembedded-core
* Fixed checking for named pipe
* Return at once when archives are the same
* Fix for type "directory"
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
.../build-compare/build-compare_git.bb | 3 ++
.../pkg-diff.sh-check-for-fifo-named-pipe.patch | 35 ++++++++++++++++++++
...-check_single_file-return-at-once-when-sa.patch | 37 ++++++++++++++++++++++
...diff.sh-remove-space-in-the-end-for-ftype.patch | 32 +++++++++++++++++++
4 files changed, 107 insertions(+)
create mode 100644 meta/recipes-devtools/build-compare/files/pkg-diff.sh-check-for-fifo-named-pipe.patch
create mode 100644 meta/recipes-devtools/build-compare/files/pkg-diff.sh-check_single_file-return-at-once-when-sa.patch
create mode 100644 meta/recipes-devtools/build-compare/files/pkg-diff.sh-remove-space-in-the-end-for-ftype.patch
diff --git a/meta/recipes-devtools/build-compare/build-compare_git.bb b/meta/recipes-devtools/build-compare/build-compare_git.bb
index 9ec8c23..09740d4 100644
--- a/meta/recipes-devtools/build-compare/build-compare_git.bb
+++ b/meta/recipes-devtools/build-compare/build-compare_git.bb
@@ -11,6 +11,9 @@ SRC_URI = "git://github.com/openSUSE/build-compare.git \
file://0001-Add-support-for-deb-and-ipk-packaging.patch \
file://functions.sh-remove-space-at-head.patch \
file://functions.sh-run-rpm-once-to-make-it-faster.patch \
+ file://pkg-diff.sh-check-for-fifo-named-pipe.patch \
+ file://pkg-diff.sh-check_single_file-return-at-once-when-sa.patch \
+ file://pkg-diff.sh-remove-space-in-the-end-for-ftype.patch \
"
SRCREV = "c5352c054c6ef15735da31b76d6d88620f4aff0a"
diff --git a/meta/recipes-devtools/build-compare/files/pkg-diff.sh-check-for-fifo-named-pipe.patch b/meta/recipes-devtools/build-compare/files/pkg-diff.sh-check-for-fifo-named-pipe.patch
new file mode 100644
index 0000000..e4f0c54
--- /dev/null
+++ b/meta/recipes-devtools/build-compare/files/pkg-diff.sh-check-for-fifo-named-pipe.patch
@@ -0,0 +1,35 @@
+From a78fe4f792a9ac9f4d364e836c8855f48561d6f2 Mon Sep 17 00:00:00 2001
+From: Robert Yang <liezhi.yang@windriver.com>
+Date: Thu, 14 Jul 2016 19:52:18 -0700
+Subject: [PATCH 3/4] pkg-diff.sh: check for fifo(named pipe)
+
+Otherwise "cmp -s fifo1 fifo2" will wait for inputing forever.
+
+Upstream-Status: Submitted [https://github.com/openSUSE/build-compare/pull/10]
+
+Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
+---
+ pkg-diff.sh | 7 +++++++
+ 1 file changed, 7 insertions(+)
+
+diff --git a/pkg-diff.sh b/pkg-diff.sh
+index 5dd3a38..1f353aa 100644
+--- a/pkg-diff.sh
++++ b/pkg-diff.sh
+@@ -735,6 +735,13 @@ check_single_file()
+ return 1
+ fi
+ ;;
++ fifo*pipe*)
++ ftype_new="`/usr/bin/file new/$file | sed -e 's@^[^:]\+:[[:blank:]]*@@' -e 's@[[:blank:]]*$@@'`"
++ if [ "$ftype_new" = "$ftype" ]; then
++ return 0
++ fi
++ return 1
++ ;;
+ *)
+ if ! diff_two_files; then
+ return 1
+--
+2.9.0
+
diff --git a/meta/recipes-devtools/build-compare/files/pkg-diff.sh-check_single_file-return-at-once-when-sa.patch b/meta/recipes-devtools/build-compare/files/pkg-diff.sh-check_single_file-return-at-once-when-sa.patch
new file mode 100644
index 0000000..b42af25
--- /dev/null
+++ b/meta/recipes-devtools/build-compare/files/pkg-diff.sh-check_single_file-return-at-once-when-sa.patch
@@ -0,0 +1,37 @@
+From 657983ef9ca8f8354172682e17408c4f6b5bc667 Mon Sep 17 00:00:00 2001
+From: Robert Yang <liezhi.yang@windriver.com>
+Date: Thu, 14 Jul 2016 19:46:08 -0700
+Subject: [PATCH 1/4] pkg-diff.sh: check_single_file(): return at once when
+ same
+
+If the two files are the same, return at once, this can save a lot of
+time when there are archives inside archives.
+
+Upstream-Status: Submitted [https://github.com/openSUSE/build-compare/pull/10]
+
+Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
+---
+ pkg-diff.sh | 7 +++++++
+ 1 file changed, 7 insertions(+)
+
+diff --git a/pkg-diff.sh b/pkg-diff.sh
+index 3cf10aa..402d4a4 100644
+--- a/pkg-diff.sh
++++ b/pkg-diff.sh
+@@ -293,6 +293,13 @@ check_compressed_file()
+ check_single_file()
+ {
+ local file="$1"
++
++ # If the two files are the same, return at once.
++ if [ -f old/$file -a -f new/$file ]; then
++ if cmp -s old/$file new/$file; then
++ return 0
++ fi
++ fi
+ case $file in
+ *.spec)
+ sed -i -e "s,Release:.*$release1,Release: @RELEASE@," old/$file
+--
+2.9.0
+
diff --git a/meta/recipes-devtools/build-compare/files/pkg-diff.sh-remove-space-in-the-end-for-ftype.patch b/meta/recipes-devtools/build-compare/files/pkg-diff.sh-remove-space-in-the-end-for-ftype.patch
new file mode 100644
index 0000000..8077172
--- /dev/null
+++ b/meta/recipes-devtools/build-compare/files/pkg-diff.sh-remove-space-in-the-end-for-ftype.patch
@@ -0,0 +1,32 @@
+From 836a6783df9c582a834fca239f227063a5687715 Mon Sep 17 00:00:00 2001
+From: Robert Yang <liezhi.yang@windriver.com>
+Date: Thu, 14 Jul 2016 19:49:12 -0700
+Subject: [PATCH 2/4] pkg-diff.sh: remove space in the end for ftype
+
+Versions of file like 5.14 returns a " " in the end, for example:
+ftype="directory ", but we need ftype="directory", remove the space to
+fix the problem.
+
+Upstream-Status: Submitted [https://github.com/openSUSE/build-compare/pull/10]
+
+Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
+---
+ pkg-diff.sh | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/pkg-diff.sh b/pkg-diff.sh
+index 402d4a4..5dd3a38 100644
+--- a/pkg-diff.sh
++++ b/pkg-diff.sh
+@@ -633,7 +633,7 @@ check_single_file()
+ ;;
+ esac
+
+- ftype=`/usr/bin/file old/$file | sed 's@^[^:]\+:[[:blank:]]*@@'`
++ ftype=`/usr/bin/file old/$file | sed -e 's@^[^:]\+:[[:blank:]]*@@' -e 's@[[:blank:]]*$@@'`
+ case $ftype in
+ PE32\ executable*Mono\/\.Net\ assembly*)
+ echo "PE32 Mono/.Net assembly: $file"
+--
+2.9.0
+
--
2.9.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH 5/9] build-compare: improve deb and ipk checking
2016-07-18 3:32 [PATCH 0/9 V2] packagefeed-stability: add class to help reduce package feed churn Robert Yang
` (3 preceding siblings ...)
2016-07-18 3:32 ` [PATCH 4/9] build-compare: fix checking for named pipe and others Robert Yang
@ 2016-07-18 3:32 ` Robert Yang
2016-07-18 3:32 ` [PATCH 6/9] package_deb.bbclass/package_ipk.bbclass: sort RPROVIDES Robert Yang
` (3 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Robert Yang @ 2016-07-18 3:32 UTC (permalink / raw)
To: openembedded-core
* The deb and ipk's depends version string is like:
Depends: libc6 (>= 2.24)
Update trim_release_old and trim_release_new to match the bracket in
the end ")".
* The deb's data tarball now is .tar.xz, and ipk's is .tar.gz.
* Update adjust_controlfile() to make ituse trim_release_old and
trim_release_new.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
.../build-compare/build-compare_git.bb | 1 +
...001-Add-support-for-deb-and-ipk-packaging.patch | 4 +-
...functions.sh-improve-deb-and-ipk-checking.patch | 117 +++++++++++++++++++++
3 files changed, 120 insertions(+), 2 deletions(-)
create mode 100644 meta/recipes-devtools/build-compare/files/functions.sh-improve-deb-and-ipk-checking.patch
diff --git a/meta/recipes-devtools/build-compare/build-compare_git.bb b/meta/recipes-devtools/build-compare/build-compare_git.bb
index 09740d4..fa9c6b4 100644
--- a/meta/recipes-devtools/build-compare/build-compare_git.bb
+++ b/meta/recipes-devtools/build-compare/build-compare_git.bb
@@ -14,6 +14,7 @@ SRC_URI = "git://github.com/openSUSE/build-compare.git \
file://pkg-diff.sh-check-for-fifo-named-pipe.patch \
file://pkg-diff.sh-check_single_file-return-at-once-when-sa.patch \
file://pkg-diff.sh-remove-space-in-the-end-for-ftype.patch \
+ file://functions.sh-improve-deb-and-ipk-checking.patch \
"
SRCREV = "c5352c054c6ef15735da31b76d6d88620f4aff0a"
diff --git a/meta/recipes-devtools/build-compare/files/0001-Add-support-for-deb-and-ipk-packaging.patch b/meta/recipes-devtools/build-compare/files/0001-Add-support-for-deb-and-ipk-packaging.patch
index 5c15218..82fd816 100644
--- a/meta/recipes-devtools/build-compare/files/0001-Add-support-for-deb-and-ipk-packaging.patch
+++ b/meta/recipes-devtools/build-compare/files/0001-Add-support-for-deb-and-ipk-packaging.patch
@@ -3,7 +3,7 @@ From: Paul Eggleton <paul.eggleton@linux.intel.com>
Date: Tue, 1 Sep 2015 12:04:33 +0100
Subject: [PATCH] Add support for deb and ipk packaging
-Upstream-Status: Pending
+Upstream-Status: Submitted [https://github.com/openSUSE/build-compare/pull/10]
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
@@ -34,7 +34,7 @@ index 06079df..85c9003 100644
return $RES
}
+
-+function adjust_controlfile () {
++function adjust_controlfile() {
+ cat $1/control | sed '/^Version: /d' > $1/control.fixed
+ mv $1/control.fixed $1/control
+ cat $2/control | sed '/^Version: /d' > $2/control.fixed
diff --git a/meta/recipes-devtools/build-compare/files/functions.sh-improve-deb-and-ipk-checking.patch b/meta/recipes-devtools/build-compare/files/functions.sh-improve-deb-and-ipk-checking.patch
new file mode 100644
index 0000000..05420f8
--- /dev/null
+++ b/meta/recipes-devtools/build-compare/files/functions.sh-improve-deb-and-ipk-checking.patch
@@ -0,0 +1,117 @@
+From b4eb7fde10318162bd2d02b9017ef51746752a91 Mon Sep 17 00:00:00 2001
+From: Robert Yang <liezhi.yang@windriver.com>
+Date: Thu, 14 Jul 2016 23:14:21 -0700
+Subject: [PATCH 4/4] functions.sh: improve deb and ipk checking
+
+* The deb and ipk's depends version string is like:
+ Depends: libc6 (>= 2.24)
+ Update trim_release_old and trim_release_new to match the bracket in
+ the end ")".
+
+* The deb's data tarball now is .tar.xz, and ipk's is .tar.gz.
+
+* Update adjust_controlfile() to make ituse trim_release_old and
+ trim_release_new.
+
+Upstream-Status: Submitted [https://github.com/openSUSE/build-compare/pull/10]
+
+Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
+---
+ functions.sh | 45 ++++++++++++++++++++++++++++-----------------
+ 1 file changed, 28 insertions(+), 17 deletions(-)
+
+diff --git a/functions.sh b/functions.sh
+index aa572f9..fe493ba 100755
+--- a/functions.sh
++++ b/functions.sh
+@@ -78,7 +78,7 @@ check_header()
+ function trim_release_old()
+ {
+ sed -e "
+- /\(\/boot\|\/lib\/modules\|\/lib\/firmware\|\/usr\/src\|$version_release_old_regex_l\$\)/{s,$version_release_old_regex_l,@VERSION@-@RELEASE_LONG@,g;s,$version_release_old_regex_s,@VERSION@-@RELEASE_SHORT@,g}
++ /\(\/boot\|\/lib\/modules\|\/lib\/firmware\|\/usr\/src\|$version_release_old_regex_l\$\|$version_release_old_regex_l)\)/{s,$version_release_old_regex_l,@VERSION@-@RELEASE_LONG@,g;s,$version_release_old_regex_s,@VERSION@-@RELEASE_SHORT@,g}
+ s/\(\/var\/adm\/update-scripts\/\)${name_ver_rel_old_regex_l}\([^[:blank:]]\+\)/\1@NAME_VER_REL@\2/g
+ s/^ *//g
+ "
+@@ -86,7 +86,7 @@ function trim_release_old()
+ function trim_release_new()
+ {
+ sed -e "
+- /\(\/boot\|\/lib\/modules\|\/lib\/firmware\|\/usr\/src\|$version_release_new_regex_l\$\)/{s,$version_release_new_regex_l,@VERSION@-@RELEASE_LONG@,g;s,$version_release_new_regex_s,@VERSION@-@RELEASE_SHORT@,g}
++ /\(\/boot\|\/lib\/modules\|\/lib\/firmware\|\/usr\/src\|$version_release_new_regex_l\$\|$version_release_new_regex_l)\)/{s,$version_release_new_regex_l,@VERSION@-@RELEASE_LONG@,g;s,$version_release_new_regex_s,@VERSION@-@RELEASE_SHORT@,g}
+ s/\(\/var\/adm\/update-scripts\/\)${name_ver_rel_new_regex_l}\([^[:blank:]]\+\)/\1@NAME_VER_REL@\2/g
+ s/^ *//g
+ "
+@@ -133,8 +133,8 @@ function unpackage()
+ ar x $file
+ tar xf control.tar.gz
+ rm control.tar.gz
+- tar xf data.tar.gz
+- rm data.tar.gz
++ tar xf data.tar.[xg]z
++ rm data.tar.[xg]z
+ ;;
+ esac
+ popd 1>/dev/null
+@@ -164,6 +164,22 @@ function get_value()
+ sed -n -e "/^___${1}___/,/^___${1}___/p" $2 | sed -e "/^___${1}___/d"
+ }
+
++# Set version_release_old_regex_s, version_release_old_regex_l and
++# name_ver_rel_old_regex_l, also the new ones.
++function set_regex() {
++ # Remember to quote the . which is in release
++ # Short version without B_CN
++ version_release_old_regex_s=${version_release_old%.*}
++ version_release_old_regex_s=${version_release_old_regex_s//./\\.}
++ version_release_new_regex_s=${version_release_new%.*}
++ version_release_new_regex_s=${version_release_new_regex_s//./\\.}
++ # Long version with B_CNT
++ version_release_old_regex_l=${version_release_old//./\\.}
++ version_release_new_regex_l=${version_release_new//./\\.}
++ name_ver_rel_old_regex_l=${name_ver_rel_old//./\\.}
++ name_ver_rel_new_regex_l=${name_ver_rel_new//./\\.}
++}
++
+ # Compare just the rpm meta data of two rpms
+ # Returns:
+ # 0 in case of same content
+@@ -194,17 +210,7 @@ function cmp_spec ()
+ version_release_old="$(get_value QF_VER_REL $spec_old)"
+ name_ver_rel_old="$(get_value QF_NAME_VER_REL $spec_old)"
+
+- # Remember to quote the . which is in release
+- # Short version without B_CN
+- version_release_old_regex_s=${version_release_old%.*}
+- version_release_old_regex_s=${version_release_old_regex_s//./\\.}
+- version_release_new_regex_s=${version_release_new%.*}
+- version_release_new_regex_s=${version_release_new_regex_s//./\\.}
+- # Long version with B_CNT
+- version_release_old_regex_l=${version_release_old//./\\.}
+- version_release_new_regex_l=${version_release_new//./\\.}
+- name_ver_rel_old_regex_l=${name_ver_rel_old//./\\.}
+- name_ver_rel_new_regex_l=${name_ver_rel_new//./\\.}
++ set_regex
+
+ # Check the whole spec file at first, return 0 immediately if the
+ # are the same.
+@@ -285,9 +291,14 @@ function cmp_spec ()
+ }
+
+ function adjust_controlfile() {
+- cat $1/control | sed '/^Version: /d' > $1/control.fixed
++ version_release_old="`sed -ne 's/^Version: \(.*\)/\1/p' $1/control`"
++ name_ver_rel_old="`sed -n -e 's/^Package: \(.*\)/\1/p' $1/control`-`sed -n -e 's/^Version: \(.*\)/\1/p' $1/control`"
++ version_release_new="`sed -ne 's/^Version: \(.*\)/\1/p' $2/control`"
++ name_ver_rel_new="`sed -n -e 's/^Package: \(.*\)/\1/p' $2/control`-`sed -n -e 's/^Version: \(.*\)/\1/p' $2/control`"
++ set_regex
++ cat $1/control | trim_release_old > $1/control.fixed
+ mv $1/control.fixed $1/control
+- cat $2/control | sed '/^Version: /d' > $2/control.fixed
++ cat $2/control | trim_release_new > $2/control.fixed
+ mv $2/control.fixed $2/control
+ }
+
+--
+2.9.0
+
--
2.9.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH 6/9] package_deb.bbclass/package_ipk.bbclass: sort RPROVIDES
2016-07-18 3:32 [PATCH 0/9 V2] packagefeed-stability: add class to help reduce package feed churn Robert Yang
` (4 preceding siblings ...)
2016-07-18 3:32 ` [PATCH 5/9] build-compare: improve deb and ipk checking Robert Yang
@ 2016-07-18 3:32 ` Robert Yang
2016-07-18 3:32 ` [PATCH 7/9] packagefeed-stability: add class to help reduce package feed churn Robert Yang
` (2 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Robert Yang @ 2016-07-18 3:32 UTC (permalink / raw)
To: openembedded-core
The dict.fromkeys() creates a dict without order, there might be a
problem when build the same recipe again, for example:
- First build of make:
Provides: es-translation, make-locale
- Second build of acl:
Provides: make-locale, es-translation
They are exactly the same Provides, but tools like "diff" doesn't think
so. Sort RPROVIDES will fix the problem.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
meta/classes/package_deb.bbclass | 2 ++
meta/classes/package_ipk.bbclass | 2 ++
2 files changed, 4 insertions(+)
diff --git a/meta/classes/package_deb.bbclass b/meta/classes/package_deb.bbclass
index 8f9b09a..8fbd383 100644
--- a/meta/classes/package_deb.bbclass
+++ b/meta/classes/package_deb.bbclass
@@ -52,6 +52,7 @@ python do_package_deb () {
import re, copy
import textwrap
import subprocess
+ import collections
oldcwd = os.getcwd()
@@ -247,6 +248,7 @@ python do_package_deb () {
debian_cmp_remap(rsuggests)
# Deliberately drop version information here, not wanted/supported by deb
rprovides = dict.fromkeys(bb.utils.explode_dep_versions2(localdata.getVar("RPROVIDES", True) or ""), [])
+ rprovides = collections.OrderedDict(sorted(rprovides.items(), key=lambda x: x[0]))
debian_cmp_remap(rprovides)
rreplaces = bb.utils.explode_dep_versions2(localdata.getVar("RREPLACES", True) or "")
debian_cmp_remap(rreplaces)
diff --git a/meta/classes/package_ipk.bbclass b/meta/classes/package_ipk.bbclass
index 1b5f4f5..e32b9cb 100644
--- a/meta/classes/package_ipk.bbclass
+++ b/meta/classes/package_ipk.bbclass
@@ -20,6 +20,7 @@ python do_package_ipk () {
import re, copy
import textwrap
import subprocess
+ import collections
oldcwd = os.getcwd()
@@ -192,6 +193,7 @@ python do_package_ipk () {
debian_cmp_remap(rsuggests)
# Deliberately drop version information here, not wanted/supported by ipk
rprovides = dict.fromkeys(bb.utils.explode_dep_versions2(localdata.getVar("RPROVIDES", True) or ""), [])
+ rprovides = collections.OrderedDict(sorted(rprovides.items(), key=lambda x: x[0]))
debian_cmp_remap(rprovides)
rreplaces = bb.utils.explode_dep_versions2(localdata.getVar("RREPLACES", True) or "")
debian_cmp_remap(rreplaces)
--
2.9.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH 7/9] packagefeed-stability: add class to help reduce package feed churn
2016-07-18 3:32 [PATCH 0/9 V2] packagefeed-stability: add class to help reduce package feed churn Robert Yang
` (5 preceding siblings ...)
2016-07-18 3:32 ` [PATCH 6/9] package_deb.bbclass/package_ipk.bbclass: sort RPROVIDES Robert Yang
@ 2016-07-18 3:32 ` Robert Yang
2016-07-18 3:32 ` [PATCH 8/9] packagefeed-stability.bbclass: minor fixes Robert Yang
2016-07-18 3:32 ` [PATCH 9/9] packagefeed-stability.bbclass: copy all packages of a recipe Robert Yang
8 siblings, 0 replies; 10+ messages in thread
From: Robert Yang @ 2016-07-18 3:32 UTC (permalink / raw)
To: openembedded-core
From: Paul Eggleton <paul.eggleton@linux.intel.com>
When a dependency causes a recipe to effectively be rebuilt, its output
may in fact not change; but new packages (with an increased PR value, if
using the PR server) will be generated nonetheless. There's no practical
way for us to predict whether or not this is going to be the case based
solely on the inputs, but we can compare the package output and see if
that is materially different and based upon that decide to replace the
old package with the new one.
This class effectively intercepts packages as they are written out by
do_package_write_*, causing them to be written into a different
directory where we can compare them to whatever older packages might
be in the "real" package feed directory, and avoid copying the new
package to the feed if it has not materially changed. We use
build-compare to do the package comparison.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/packagefeed-stability.bbclass | 270 +++++++++++++++++++++++++++++
1 file changed, 270 insertions(+)
create mode 100644 meta/classes/packagefeed-stability.bbclass
diff --git a/meta/classes/packagefeed-stability.bbclass b/meta/classes/packagefeed-stability.bbclass
new file mode 100644
index 0000000..b5207d9
--- /dev/null
+++ b/meta/classes/packagefeed-stability.bbclass
@@ -0,0 +1,270 @@
+# Class to avoid copying packages into the feed if they haven't materially changed
+#
+# Copyright (C) 2015 Intel Corporation
+# Released under the MIT license (see COPYING.MIT for details)
+#
+# This class effectively intercepts packages as they are written out by
+# do_package_write_*, causing them to be written into a different
+# directory where we can compare them to whatever older packages might
+# be in the "real" package feed directory, and avoid copying the new
+# package to the feed if it has not materially changed. The idea is to
+# avoid unnecessary churn in the packages when dependencies trigger task
+# reexecution (and thus repackaging). Enabling the class is simple:
+#
+# INHERIT += "packagefeed-stability"
+#
+# Caveats:
+# 1) Latest PR values in the build system may not match those in packages
+# seen on the target (naturally)
+# 2) If you rebuild from sstate without the existing package feed present,
+# you will lose the "state" of the package feed i.e. the preserved old
+# package versions. Not the end of the world, but would negate the
+# entire purpose of this class.
+#
+# Note that running -c cleanall on a recipe will purposely delete the old
+# package files so they will definitely be copied the next time.
+
+python() {
+ # Package backend agnostic intercept
+ # This assumes that the package_write task is called package_write_<pkgtype>
+ # and that the directory in which packages should be written is
+ # pointed to by the variable DEPLOY_DIR_<PKGTYPE>
+ for pkgclass in (d.getVar('PACKAGE_CLASSES', True) or '').split():
+ if pkgclass.startswith('package_'):
+ pkgtype = pkgclass.split('_', 1)[1]
+ pkgwritefunc = 'do_package_write_%s' % pkgtype
+ sstate_outputdirs = d.getVarFlag(pkgwritefunc, 'sstate-outputdirs', False)
+ deploydirvar = 'DEPLOY_DIR_%s' % pkgtype.upper()
+ deploydirvarref = '${' + deploydirvar + '}'
+ pkgcomparefunc = 'do_package_compare_%s' % pkgtype
+
+ if bb.data.inherits_class('image', d):
+ d.appendVarFlag('do_rootfs', 'recrdeptask', ' ' + pkgcomparefunc)
+
+ if bb.data.inherits_class('populate_sdk_base', d):
+ d.appendVarFlag('do_populate_sdk', 'recrdeptask', ' ' + pkgcomparefunc)
+
+ if bb.data.inherits_class('populate_sdk_ext', d):
+ d.appendVarFlag('do_populate_sdk_ext', 'recrdeptask', ' ' + pkgcomparefunc)
+
+ d.appendVarFlag('do_build', 'recrdeptask', ' ' + pkgcomparefunc)
+
+ if d.getVarFlag(pkgwritefunc, 'noexec', True) or (not d.getVarFlag(pkgwritefunc, 'task', True)) or pkgwritefunc in (d.getVar('__BBDELTASKS', True) or []):
+ # Packaging is disabled for this recipe, we shouldn't do anything
+ continue
+
+ if deploydirvarref in sstate_outputdirs:
+ # Set intermediate output directory
+ d.setVarFlag(pkgwritefunc, 'sstate-outputdirs', sstate_outputdirs.replace(deploydirvarref, deploydirvarref + '-prediff'))
+
+ d.setVar(pkgcomparefunc, d.getVar('do_package_compare', False))
+ d.setVarFlags(pkgcomparefunc, d.getVarFlags('do_package_compare', False))
+ d.appendVarFlag(pkgcomparefunc, 'depends', ' build-compare-native:do_populate_sysroot')
+ bb.build.addtask(pkgcomparefunc, 'do_build', 'do_packagedata ' + pkgwritefunc, d)
+}
+
+# This isn't the real task function - it's a template that we use in the
+# anonymous python code above
+fakeroot python do_package_compare () {
+ currenttask = d.getVar('BB_CURRENTTASK', True)
+ pkgtype = currenttask.rsplit('_', 1)[1]
+ package_compare_impl(pkgtype, d)
+}
+
+def package_compare_impl(pkgtype, d):
+ import errno
+ import fnmatch
+ import glob
+ import subprocess
+ import oe.sstatesig
+
+ pn = d.getVar('PN', True)
+ deploydir = d.getVar('DEPLOY_DIR_%s' % pkgtype.upper(), True)
+ prepath = deploydir + '-prediff/'
+
+ # Find out PKGR values are
+ pkgdatadir = d.getVar('PKGDATA_DIR', True)
+ packages = []
+ try:
+ with open(os.path.join(pkgdatadir, pn), 'r') as f:
+ for line in f:
+ if line.startswith('PACKAGES:'):
+ packages = line.split(':', 1)[1].split()
+ break
+ except IOError as e:
+ if e.errno == errno.ENOENT:
+ pass
+
+ if not packages:
+ bb.debug(2, '%s: no packages, nothing to do' % pn)
+ return
+
+ pkgrvalues = {}
+ rpkgnames = {}
+ rdepends = {}
+ pkgvvalues = {}
+ for pkg in packages:
+ with open(os.path.join(pkgdatadir, 'runtime', pkg), 'r') as f:
+ for line in f:
+ if line.startswith('PKGR:'):
+ pkgrvalues[pkg] = line.split(':', 1)[1].strip()
+ if line.startswith('PKGV:'):
+ pkgvvalues[pkg] = line.split(':', 1)[1].strip()
+ elif line.startswith('PKG_%s:' % pkg):
+ rpkgnames[pkg] = line.split(':', 1)[1].strip()
+ elif line.startswith('RDEPENDS_%s:' % pkg):
+ rdepends[pkg] = line.split(':', 1)[1].strip()
+
+ # Prepare a list of the runtime package names for packages that were
+ # actually produced
+ rpkglist = []
+ for pkg, rpkg in rpkgnames.iteritems():
+ if os.path.exists(os.path.join(pkgdatadir, 'runtime', pkg + '.packaged')):
+ rpkglist.append((rpkg, pkg))
+ rpkglist.sort(key=lambda x: len(x[0]), reverse=True)
+
+ pvu = d.getVar('PV', False)
+ if '$' + '{SRCPV}' in pvu:
+ pvprefix = pvu.split('$' + '{SRCPV}', 1)[0]
+ else:
+ pvprefix = None
+
+ pkgwritetask = 'package_write_%s' % pkgtype
+ files = []
+ copypkgs = []
+ manifest, _ = oe.sstatesig.sstate_get_manifest_filename(pkgwritetask, d)
+ with open(manifest, 'r') as f:
+ for line in f:
+ if line.startswith(prepath):
+ srcpath = line.rstrip()
+ if os.path.isfile(srcpath):
+ destpath = os.path.join(deploydir, os.path.relpath(srcpath, prepath))
+
+ # This is crude but should work assuming the output
+ # package file name starts with the package name
+ # and rpkglist is sorted by length (descending)
+ pkgbasename = os.path.basename(destpath)
+ pkgname = None
+ for rpkg, pkg in rpkglist:
+ if pkgbasename.startswith(rpkg):
+ pkgr = pkgrvalues[pkg]
+ destpathspec = destpath.replace(pkgr, '*')
+ if pvprefix:
+ pkgv = pkgvvalues[pkg]
+ if pkgv.startswith(pvprefix):
+ pkgvsuffix = pkgv[len(pvprefix):]
+ if '+' in pkgvsuffix:
+ newpkgv = pvprefix + '*+' + pkgvsuffix.split('+', 1)[1]
+ destpathspec = destpathspec.replace(pkgv, newpkgv)
+ pkgname = pkg
+ break
+ else:
+ bb.warn('Unable to map %s back to package' % pkgbasename)
+ destpathspec = destpath
+
+ oldfiles = glob.glob(destpathspec)
+ oldfile = None
+ docopy = True
+ if oldfiles:
+ oldfile = oldfiles[-1]
+ result = subprocess.call(['pkg-diff.sh', oldfile, srcpath])
+ if result == 0:
+ docopy = False
+
+ files.append((pkgname, pkgbasename, srcpath, oldfile, destpath))
+ bb.debug(2, '%s: package %s %s' % (pn, files[-1], docopy))
+ if docopy:
+ copypkgs.append(pkgname)
+
+ # Ensure that dependencies on specific versions (such as -dev on the
+ # main package) are copied in lock-step
+ changed = True
+ while changed:
+ rpkgdict = {x[0]: x[1] for x in rpkglist}
+ changed = False
+ for pkgname, pkgbasename, srcpath, oldfile, destpath in files:
+ rdeps = rdepends.get(pkgname, None)
+ if not rdeps:
+ continue
+ rdepvers = bb.utils.explode_dep_versions2(rdeps)
+ for rdep, versions in rdepvers.iteritems():
+ dep = rpkgdict.get(rdep, None)
+ for version in versions:
+ if version and version.startswith('= '):
+ if dep in copypkgs and not pkgname in copypkgs:
+ bb.debug(2, '%s: copying %s because it has a fixed version dependency on %s and that package is going to be copied' % (pn, pkgname, dep))
+ changed = True
+ copypkgs.append(pkgname)
+ elif pkgname in copypkgs and not dep in copypkgs:
+ bb.debug(2, '%s: copying %s because %s has a fixed version dependency on it and that package is going to be copied' % (pn, dep, pkgname))
+ changed = True
+ copypkgs.append(dep)
+
+ # Read in old manifest so we can delete any packages we aren't going to replace or preserve
+ pcmanifest = os.path.join(prepath, d.expand('pkg-compare-manifest-${MULTIMACH_TARGET_SYS}-${PN}'))
+ try:
+ with open(pcmanifest, 'r') as f:
+ knownfiles = [x[3] for x in files if x[3]]
+ for line in f:
+ fn = line.rstrip()
+ if fn:
+ if fn in knownfiles:
+ knownfiles.remove(fn)
+ else:
+ try:
+ os.remove(fn)
+ bb.warn('Removed old package %s' % fn)
+ except OSError as e:
+ if e.errno == errno.ENOENT:
+ pass
+ except IOError as e:
+ if e.errno == errno.ENOENT:
+ pass
+
+ # Create new manifest
+ with open(pcmanifest, 'w') as f:
+ for pkgname, pkgbasename, srcpath, oldfile, destpath in files:
+ if pkgname in copypkgs:
+ bb.warn('Copying %s' % pkgbasename)
+ destdir = os.path.dirname(destpath)
+ bb.utils.mkdirhier(destdir)
+ if oldfile:
+ try:
+ os.remove(oldfile)
+ except OSError as e:
+ if e.errno == errno.ENOENT:
+ pass
+ if (os.stat(srcpath).st_dev == os.stat(destdir).st_dev):
+ # Use a hard link to save space
+ os.link(srcpath, destpath)
+ else:
+ shutil.copyfile(srcpath, destpath)
+ f.write('%s\n' % destpath)
+ else:
+ bb.warn('Not copying %s' % pkgbasename)
+ f.write('%s\n' % oldfile)
+
+
+do_cleanall_append() {
+ import errno
+ for pkgclass in (d.getVar('PACKAGE_CLASSES', True) or '').split():
+ if pkgclass.startswith('package_'):
+ pkgtype = pkgclass.split('_', 1)[1]
+ deploydir = d.getVar('DEPLOY_DIR_%s' % pkgtype.upper(), True)
+ prepath = deploydir + '-prediff'
+ pcmanifest = os.path.join(prepath, d.expand('pkg-compare-manifest-${MULTIMACH_TARGET_SYS}-${PN}'))
+ try:
+ with open(pcmanifest, 'r') as f:
+ for line in f:
+ fn = line.rstrip()
+ if fn:
+ try:
+ os.remove(fn)
+ except OSError as e:
+ if e.errno == errno.ENOENT:
+ pass
+ os.remove(pcmanifest)
+ except IOError as e:
+ if e.errno == errno.ENOENT:
+ pass
+}
--
2.9.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH 8/9] packagefeed-stability.bbclass: minor fixes
2016-07-18 3:32 [PATCH 0/9 V2] packagefeed-stability: add class to help reduce package feed churn Robert Yang
` (6 preceding siblings ...)
2016-07-18 3:32 ` [PATCH 7/9] packagefeed-stability: add class to help reduce package feed churn Robert Yang
@ 2016-07-18 3:32 ` Robert Yang
2016-07-18 3:32 ` [PATCH 9/9] packagefeed-stability.bbclass: copy all packages of a recipe Robert Yang
8 siblings, 0 replies; 10+ messages in thread
From: Robert Yang @ 2016-07-18 3:32 UTC (permalink / raw)
To: openembedded-core
* fix for python3
iteritems() -> items()
* Return immediately for native and cross.
* Remove the usage of __BBDELTASKS, there is no such var in bitbake.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
meta/classes/packagefeed-stability.bbclass | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/meta/classes/packagefeed-stability.bbclass b/meta/classes/packagefeed-stability.bbclass
index b5207d9..808a18f 100644
--- a/meta/classes/packagefeed-stability.bbclass
+++ b/meta/classes/packagefeed-stability.bbclass
@@ -25,6 +25,8 @@
# package files so they will definitely be copied the next time.
python() {
+ if bb.data.inherits_class('native', d) or bb.data.inherits_class('cross', d):
+ return
# Package backend agnostic intercept
# This assumes that the package_write task is called package_write_<pkgtype>
# and that the directory in which packages should be written is
@@ -49,7 +51,7 @@ python() {
d.appendVarFlag('do_build', 'recrdeptask', ' ' + pkgcomparefunc)
- if d.getVarFlag(pkgwritefunc, 'noexec', True) or (not d.getVarFlag(pkgwritefunc, 'task', True)) or pkgwritefunc in (d.getVar('__BBDELTASKS', True) or []):
+ if d.getVarFlag(pkgwritefunc, 'noexec', True) or not d.getVarFlag(pkgwritefunc, 'task', True):
# Packaging is disabled for this recipe, we shouldn't do anything
continue
@@ -118,7 +120,7 @@ def package_compare_impl(pkgtype, d):
# Prepare a list of the runtime package names for packages that were
# actually produced
rpkglist = []
- for pkg, rpkg in rpkgnames.iteritems():
+ for pkg, rpkg in rpkgnames.items():
if os.path.exists(os.path.join(pkgdatadir, 'runtime', pkg + '.packaged')):
rpkglist.append((rpkg, pkg))
rpkglist.sort(key=lambda x: len(x[0]), reverse=True)
@@ -187,7 +189,7 @@ def package_compare_impl(pkgtype, d):
if not rdeps:
continue
rdepvers = bb.utils.explode_dep_versions2(rdeps)
- for rdep, versions in rdepvers.iteritems():
+ for rdep, versions in rdepvers.items():
dep = rpkgdict.get(rdep, None)
for version in versions:
if version and version.startswith('= '):
--
2.9.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH 9/9] packagefeed-stability.bbclass: copy all packages of a recipe
2016-07-18 3:32 [PATCH 0/9 V2] packagefeed-stability: add class to help reduce package feed churn Robert Yang
` (7 preceding siblings ...)
2016-07-18 3:32 ` [PATCH 8/9] packagefeed-stability.bbclass: minor fixes Robert Yang
@ 2016-07-18 3:32 ` Robert Yang
8 siblings, 0 replies; 10+ messages in thread
From: Robert Yang @ 2016-07-18 3:32 UTC (permalink / raw)
To: openembedded-core
A recipes can generate several rpms such as a.rpm, a-dev.rpm, a-dbg.rpm,
when update one of them in the repo, we'd better update all of them,
otherwise, there might be a-dev.r0.1.rpm and a-dbg.r0.3.rpm in the repo,
which looks strange.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
meta/classes/packagefeed-stability.bbclass | 103 ++++++++++-------------------
1 file changed, 35 insertions(+), 68 deletions(-)
diff --git a/meta/classes/packagefeed-stability.bbclass b/meta/classes/packagefeed-stability.bbclass
index 808a18f..df4c2ba 100644
--- a/meta/classes/packagefeed-stability.bbclass
+++ b/meta/classes/packagefeed-stability.bbclass
@@ -133,8 +133,10 @@ def package_compare_impl(pkgtype, d):
pkgwritetask = 'package_write_%s' % pkgtype
files = []
- copypkgs = []
+ docopy = False
manifest, _ = oe.sstatesig.sstate_get_manifest_filename(pkgwritetask, d)
+ # Copy recipe's all packages if one of the packages are different to make
+ # they have the same PR.
with open(manifest, 'r') as f:
for line in f:
if line.startswith(prepath):
@@ -164,88 +166,53 @@ def package_compare_impl(pkgtype, d):
bb.warn('Unable to map %s back to package' % pkgbasename)
destpathspec = destpath
- oldfiles = glob.glob(destpathspec)
oldfile = None
- docopy = True
- if oldfiles:
- oldfile = oldfiles[-1]
- result = subprocess.call(['pkg-diff.sh', oldfile, srcpath])
- if result == 0:
- docopy = False
-
- files.append((pkgname, pkgbasename, srcpath, oldfile, destpath))
- bb.debug(2, '%s: package %s %s' % (pn, files[-1], docopy))
- if docopy:
- copypkgs.append(pkgname)
-
- # Ensure that dependencies on specific versions (such as -dev on the
- # main package) are copied in lock-step
- changed = True
- while changed:
- rpkgdict = {x[0]: x[1] for x in rpkglist}
- changed = False
- for pkgname, pkgbasename, srcpath, oldfile, destpath in files:
- rdeps = rdepends.get(pkgname, None)
- if not rdeps:
- continue
- rdepvers = bb.utils.explode_dep_versions2(rdeps)
- for rdep, versions in rdepvers.items():
- dep = rpkgdict.get(rdep, None)
- for version in versions:
- if version and version.startswith('= '):
- if dep in copypkgs and not pkgname in copypkgs:
- bb.debug(2, '%s: copying %s because it has a fixed version dependency on %s and that package is going to be copied' % (pn, pkgname, dep))
- changed = True
- copypkgs.append(pkgname)
- elif pkgname in copypkgs and not dep in copypkgs:
- bb.debug(2, '%s: copying %s because %s has a fixed version dependency on it and that package is going to be copied' % (pn, dep, pkgname))
- changed = True
- copypkgs.append(dep)
-
- # Read in old manifest so we can delete any packages we aren't going to replace or preserve
- pcmanifest = os.path.join(prepath, d.expand('pkg-compare-manifest-${MULTIMACH_TARGET_SYS}-${PN}'))
- try:
- with open(pcmanifest, 'r') as f:
- knownfiles = [x[3] for x in files if x[3]]
- for line in f:
- fn = line.rstrip()
- if fn:
- if fn in knownfiles:
- knownfiles.remove(fn)
- else:
+ if not docopy:
+ oldfiles = glob.glob(destpathspec)
+ if oldfiles:
+ oldfile = oldfiles[-1]
+ result = subprocess.call(['pkg-diff.sh', oldfile, srcpath])
+ if result != 0:
+ docopy = True
+ bb.note("%s and %s are different, will copy packages" % (oldfile, srcpath))
+ else:
+ docopy = True
+ bb.note("No old packages found for %s, will copy packages" % pkgname)
+
+ files.append((pkgname, pkgbasename, srcpath, destpath))
+
+ # Remove all the old files and copy again if docopy
+ if docopy:
+ bb.plain('Copying packages for recipe %s' % pn)
+ pcmanifest = os.path.join(prepath, d.expand('pkg-compare-manifest-${MULTIMACH_TARGET_SYS}-${PN}'))
+ try:
+ with open(pcmanifest, 'r') as f:
+ for line in f:
+ fn = line.rstrip()
+ if fn:
try:
os.remove(fn)
- bb.warn('Removed old package %s' % fn)
+ bb.note('Removed old package %s' % fn)
except OSError as e:
if e.errno == errno.ENOENT:
pass
- except IOError as e:
- if e.errno == errno.ENOENT:
- pass
+ except IOError as e:
+ if e.errno == errno.ENOENT:
+ pass
- # Create new manifest
- with open(pcmanifest, 'w') as f:
- for pkgname, pkgbasename, srcpath, oldfile, destpath in files:
- if pkgname in copypkgs:
- bb.warn('Copying %s' % pkgbasename)
+ # Create new manifest
+ with open(pcmanifest, 'w') as f:
+ for pkgname, pkgbasename, srcpath, destpath in files:
destdir = os.path.dirname(destpath)
bb.utils.mkdirhier(destdir)
- if oldfile:
- try:
- os.remove(oldfile)
- except OSError as e:
- if e.errno == errno.ENOENT:
- pass
if (os.stat(srcpath).st_dev == os.stat(destdir).st_dev):
# Use a hard link to save space
os.link(srcpath, destpath)
else:
shutil.copyfile(srcpath, destpath)
f.write('%s\n' % destpath)
- else:
- bb.warn('Not copying %s' % pkgbasename)
- f.write('%s\n' % oldfile)
-
+ else:
+ bb.plain('Not copying packages for %s' % pn)
do_cleanall_append() {
import errno
--
2.9.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
end of thread, other threads:[~2016-07-18 3:33 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-07-18 3:32 [PATCH 0/9 V2] packagefeed-stability: add class to help reduce package feed churn Robert Yang
2016-07-18 3:32 ` [PATCH 1/9] rpm: make --nosignature work Robert Yang
2016-07-18 3:32 ` [PATCH 2/9] build-compare: remove space at head Robert Yang
2016-07-18 3:32 ` [PATCH 3/9] build-compare: make pkg-diff.sh 75% faster Robert Yang
2016-07-18 3:32 ` [PATCH 4/9] build-compare: fix checking for named pipe and others Robert Yang
2016-07-18 3:32 ` [PATCH 5/9] build-compare: improve deb and ipk checking Robert Yang
2016-07-18 3:32 ` [PATCH 6/9] package_deb.bbclass/package_ipk.bbclass: sort RPROVIDES Robert Yang
2016-07-18 3:32 ` [PATCH 7/9] packagefeed-stability: add class to help reduce package feed churn Robert Yang
2016-07-18 3:32 ` [PATCH 8/9] packagefeed-stability.bbclass: minor fixes Robert Yang
2016-07-18 3:32 ` [PATCH 9/9] packagefeed-stability.bbclass: copy all packages of a recipe Robert Yang
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.