* [PATCH 0/5] package_manager.py/ootfs.py: support ipk incremental image generation
@ 2014-02-18 9:42 Hongxu Jia
2014-02-18 9:42 ` [PATCH 1/5] package_manager.py: " Hongxu Jia
` (4 more replies)
0 siblings, 5 replies; 13+ messages in thread
From: Hongxu Jia @ 2014-02-18 9:42 UTC (permalink / raw)
To: openembedded-core; +Cc: saul.wold
Test Cases
---------------------------------------------
Case 1: Rebuild in place
1. Edit local.conf, enable multilib and ipk incremental image generation
...
46 MACHINE ?= "qemux86-64"
254 require conf/multilib.conf
255 MULTILIBS = "multilib:lib32"
256 DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
257
258 IMAGE_INSTALL_append = " lib32-gzip"
260 IMAGE_FEATURES_append = " package-management ssh-server-dropbear "
262
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal
2. Edit local.conf, add some spaces in IMAGE_INSTALL_append
vim local.conf
...
258 IMAGE_INSTALL_append = " lib32-gzip "
...
3. bb core-image-minimal, the image should not be rebuilt
vim log.do_rootfs:
...
16 NOTE: Package packagegroup-core-ssh-dropbear (1.0-r1) installed in root is up to date.
17 Package opkg-collateral (1.0-r2) installed in root is up to date.
18 Package run-postinsts (1.0-r9) installed in root is up to date.
19 Package packagegroup-core-boot (1.0-r11) installed in root is up to date.
20 Package opkg (1:0.2.0-r0) installed in root is up to date.
21 Package poky-feed-config-opkg (1.0-r2) installed in root is up to date.
...
53 NOTE: Package lib32-gzip (1.6-r7) installed in root is up to date.
...
vim installed_pkgs.txt
...
lib32-gzip x86
...
4. runqemu qemux86-64, ssh could work and lib32-gzip is installed
On host:
runqemu qemux86-64
ssh root@192.168.7.2 and run 'opkg-cl list-installed'
...
lib32-gzip - 1.6-r7
...
---------------------------------------------
Case 2: Decremental Rebuild
1. Edit local.conf, enable multilib and ipk incremental image generation
...
46 MACHINE ?= "qemux86-64"
254 require conf/multilib.conf
255 MULTILIBS = "multilib:lib32"
256 DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
257
258 IMAGE_INSTALL_append = " lib32-gzip openssl"
260 IMAGE_FEATURES_append = " package-management ssh-server-dropbear "
262
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal
2. Edit local.conf, remove 'lib32-gzip openssl'
vim local.conf
...
258 # IMAGE_INSTALL_append = " lib32-gzip openssl"
...
3. bb core-image-minimal, lib32-gzip and openssl should be removed.
vim log.do_rootfs:
...
20 NOTE: Removing package lib32-eglibc from root...
21 Removing package lib32-gzip from root...
22 Removing package lib32-update-alternatives-opkg from root...
23 Removing package libcrypto1.0.0 from root...
24 Removing package libssl1.0.0 from root...
25 Removing package openssl from root...
26 Removing package openssl-conf from root...
...
31 Package opkg-collateral (1.0-r2) installed in root is up to date.
32 Package run-postinsts (1.0-r9) installed in root is up to date.
33 Package packagegroup-core-boot (1.0-r11) installed in root is up to date.
34 Package opkg (1:0.2.0-r0) installed in root is up to date.
35 Package poky-feed-config-opkg (1.0-r2) installed in root is up to date.
...
---------------------------------------------
Case 3: Incremental Rebuild
1. Edit local.conf, enable multilib and ipk incremental image generation
...
46 MACHINE ?= "qemux86-64"
254 require conf/multilib.conf
255 MULTILIBS = "multilib:lib32"
256 DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
257
258 # IMAGE_INSTALL_append = " lib32-gzip openssl"
260 IMAGE_FEATURES_append = " package-management ssh-server-dropbear "
262
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal
2. Edit local.conf, append 'lib32-gzip openssl' to IMAGE_INSTALL
vim local.conf
...
258 IMAGE_INSTALL_append = " lib32-gzip openssl"
...
3. bb core-image-minimal, lib32-gzip and openssl should be added.
vim log.do_rootfs:
...
20 NOTE: Package packagegroup-core-ssh-dropbear (1.0-r1) installed in root is up to date.
21 Package opkg-collateral (1.0-r2) installed in root is up to date.
22 Package run-postinsts (1.0-r9) installed in root is up to date.
23 Installing openssl (1.0.1e-r15.0) to root...
...
31 Package packagegroup-core-boot (1.0-r11) installed in root is up to date.
32 Package opkg (1:0.2.0-r0) installed in root is up to date.
33 Package poky-feed-config-opkg (1.0-r2) installed in root is up to date.
...
54 NOTE: Installing the following packages: lib32-gzip
...
---------------------------------------------
Case 4: Rebuild in place and Upgrade package
1. Edit local.conf, enable ipk incremental image generation
...
258 IMAGE_INSTALL_append = " gzip"
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal
2. Edit local.conf, upgrade gzip's PR
...
258 PR_pn-gzip = "r1"
...
3. bb core-image-minimal, gzip should be upgraded to r1
vim log.do_rootfs:
...
21 NOTE: Package packagegroup-core-ssh-dropbear (1.0-r1) installed in root is up to date.
22 Package opkg-collateral (1.0-r2) installed in root is up to date.
23 Package run-postinsts (1.0-r9) installed in root is up to date.
24 Package packagegroup-core-boot (1.0-r11) installed in root is up to date.
25 Package opkg (1:0.2.0-r0) installed in root is up to date.
26 Upgrading gzip on root from 1.6-r0 to 1.6-r1...
...
---------------------------------------------
Case 5: Test BAD_RECOMMENDATIONS
1. Edit local.conf, enable ipk incremental image generation
...
258 IMAGE_INSTALL_append = " util-linux"
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal
The util-linux-fdisk is in util-linux's RRECOMMENDS
vim installed_pkgs.txt
...
41 util-linux-fdisk core2-64
...
2. Edit local.conf, add some spaces to IMAGE_INSTALL and add
util-linux-fdisk to core-image-minimal's BAD_RECOMMENDATIONS
...
260 IMAGE_INSTALL_append = " util-linux "
265 BAD_RECOMMENDATIONS_pn-core-image-minimal += "util-linux-fdisk"
...
3. bb core-image-minimal, util-linux-fdisk should not be installed
vim log.do_rootfs:
...
34 NOTE: Requested ignored recommendation util-linux-fdisk is not a package
...
119 util-linux: ignoring recommendation for util-linux-fdisk at user request
120 util-linux: unsatisfied recommendation for util-linux-fdisk
...
The util-linux-fdisk doesn't exist in installed_pkgs.txt any more.
---------------------------------------------
Case 5: Test NO_RECOMMENDATIONS
1. Edit local.conf, enable ipk incremental image generation
...
258 IMAGE_INSTALL_append = " util-linux"
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal
The util-linux-fdisk util-linux-cfdisk util-linux-sfdisk
util-linux-mount util-linux-readprofile util-linux-mkfs are
in util-linux's RRECOMMENDS
vim installed_pkgs.txt
...
4 util-linux-mkfs core2-64
18 util-linux-sfdisk core2-64
20 util-linux-mount core2-64
38 util-linux-cfdisk core2-64
41 util-linux-fdisk core2-64
46 util-linux-readprofile core2-64
...
2. Edit local.conf, add some spaces to IMAGE_INSTALL and
assign core-image-minimal's NO_RECOMMENDATIONS with 1
...
260 IMAGE_INSTALL_append = " util-linux "
267 NO_RECOMMENDATIONS_pn-core-image-minimal = "1"
...
3. bb core-image-minimal, The six packages should not be installed
vim log.do_rootfs:
...
34 NOTE: Requested ignored recommendation util-linux-fdisk is not a package
...
113 util-linux: ignoring recommendation for util-linux-sfdisk at user request
114 util-linux: unsatisfied recommendation for util-linux-sfdisk
115 util-linux: ignoring recommendation for util-linux-cfdisk at user request
116 util-linux: unsatisfied recommendation for util-linux-cfdisk
117 util-linux: ignoring recommendation for util-linux-mount at user request
118 util-linux: unsatisfied recommendation for util-linux-mount
119 util-linux: ignoring recommendation for util-linux-fdisk at user request
120 util-linux: unsatisfied recommendation for util-linux-fdisk
121 util-linux: ignoring recommendation for util-linux-mkfs at user request
122 util-linux: unsatisfied recommendation for util-linux-mkfs
123 util-linux: ignoring recommendation for util-linux-readprofile at user request
124 util-linux: unsatisfied recommendation for util-linux-readprofile
...
They don't exist in installed_pkgs.txt any more.
//Hongxu
The following changes since commit ac499f0b9f8a421a4e63d2de2a0b89f20c43d81c:
dev-manual: Fixed code block example for ARCHIVER. (2014-02-17 15:39:46 +0000)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib hongxu/ipk-inc-gen
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/ipk-inc-gen
Hongxu Jia (5):
package_manager.py: support ipk incremental image generation
package_manager.py: tweak handle_bad_recommendations for ipk
incremental image generation
rootfs.py: support ipk incremental image generation
rootfs.py: fix BAD_RECOMMENDATIONS for ipk incremental image
generation
rootfs.py: tweak _multilib_sanity_test for ipk incremental image
generation
meta/lib/oe/package_manager.py | 124 +++++++++++++++++++++++++++++++++++---
meta/lib/oe/rootfs.py | 134 ++++++++++++++++++++++++++++++++++++++---
2 files changed, 242 insertions(+), 16 deletions(-)
--
1.8.1.2
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH 1/5] package_manager.py: support ipk incremental image generation
2014-02-18 9:42 [PATCH 0/5] package_manager.py/ootfs.py: support ipk incremental image generation Hongxu Jia
@ 2014-02-18 9:42 ` Hongxu Jia
2014-02-18 14:38 ` Laurentiu Palcu
2014-02-18 9:42 ` [PATCH 2/5] package_manager.py: tweak handle_bad_recommendations for " Hongxu Jia
` (3 subsequent siblings)
4 siblings, 1 reply; 13+ messages in thread
From: Hongxu Jia @ 2014-02-18 9:42 UTC (permalink / raw)
To: openembedded-core; +Cc: saul.wold
While incremental image generation enabled, 'load_old_install_solution'
is used to determine what we've got in the previous (existed) image and
'dump_install_solution' to determine what we need to install in the
current image.
The 'backup_packaging_data' is used to back up the current opkg database.
The 'recovery_packaging_data' is used to recover the opkg database which
backed up by the previous image creation.
Tweak 'remove' function in OpkgPM class, which the options for remove
with dependencies was incorrect.
[YOCTO #1894]
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
meta/lib/oe/package_manager.py | 106 +++++++++++++++++++++++++++++++++++++++--
1 file changed, 102 insertions(+), 4 deletions(-)
diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index 6dc8fbd..4ea9677 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -934,12 +934,13 @@ class RpmPM(PackageManager):
class OpkgPM(PackageManager):
- def __init__(self, d, target_rootfs, config_file, archs):
+ def __init__(self, d, target_rootfs, config_file, archs, task_name='target'):
super(OpkgPM, self).__init__(d)
self.target_rootfs = target_rootfs
self.config_file = config_file
self.pkg_archs = archs
+ self.task_name = task_name
self.deploy_dir = self.d.getVar("DEPLOY_DIR_IPK", True)
self.deploy_lock_file = os.path.join(self.deploy_dir, "deploy.lock")
@@ -956,6 +957,13 @@ class OpkgPM(PackageManager):
bb.utils.mkdirhier(self.opkg_dir)
+ self.solution_manifest = self.d.expand('${T}/saved/%s_solution' %
+ self.task_name)
+ self.saved_opkg_dir = self.d.expand('${T}/saved/%s' % self.task_name)
+ if not os.path.exists(self.d.expand('${T}/saved')):
+ bb.utils.mkdirhier(self.d.expand('${T}/saved'))
+
+
if (self.d.getVar('BUILD_IMAGES_FROM_FEEDS', True) or "") != "1":
self._create_config()
else:
@@ -1075,7 +1083,9 @@ class OpkgPM(PackageManager):
try:
bb.note("Installing the following packages: %s" % ' '.join(pkgs))
- subprocess.check_output(cmd.split())
+ bb.note(cmd)
+ output = subprocess.check_output(cmd.split())
+ bb.note(output)
except subprocess.CalledProcessError as e:
(bb.fatal, bb.note)[attempt_only]("Unable to install packages. "
"Command '%s' returned %d:\n%s" %
@@ -1083,14 +1093,16 @@ class OpkgPM(PackageManager):
def remove(self, pkgs, with_dependencies=True):
if with_dependencies:
- cmd = "%s %s remove %s" % \
+ cmd = "%s %s --force-depends --force-remove --force-removal-of-dependent-packages remove %s" % \
(self.opkg_cmd, self.opkg_args, ' '.join(pkgs))
else:
cmd = "%s %s --force-depends remove %s" % \
(self.opkg_cmd, self.opkg_args, ' '.join(pkgs))
try:
- subprocess.check_output(cmd.split())
+ bb.note(cmd)
+ output = subprocess.check_output(cmd.split())
+ bb.note(output)
except subprocess.CalledProcessError as e:
bb.fatal("Unable to remove packages. Command '%s' "
"returned %d:\n%s" % (e.cmd, e.returncode, e.output))
@@ -1175,6 +1187,92 @@ class OpkgPM(PackageManager):
else:
status.write(line + "\n")
+ '''
+ If incremental install, we need to determine what we've got,
+ what we need to add, and what to remove...
+ The dump_install_solution will dump and save the new install
+ solution.
+ '''
+ def dump_install_solution(self, pkgs):
+ bb.note('creating new install solution for incremental install')
+ if len(pkgs) == 0:
+ return
+
+ install_pkgs = list()
+
+ # Create an temp dir as opkg root for simulating the installation
+ temp_rootfs = self.d.expand('${T}/opkg')
+ temp_opkg_dir = os.path.join(temp_rootfs, 'var/lib/opkg')
+ bb.utils.mkdirhier(temp_opkg_dir)
+
+ opkg_args = "-f %s -o %s " % (self.config_file, temp_rootfs)
+ opkg_args += self.d.getVar("OPKG_ARGS", True)
+
+ cmd = "%s %s update" % (self.opkg_cmd,
+ opkg_args)
+ try:
+ subprocess.check_output(cmd, shell=True)
+ except subprocess.CalledProcessError as e:
+ bb.note("Unable to dump install packages. Command '%s' "
+ "returned %d:\n%s" % (cmd, e.returncode, e.output))
+
+ # Simulate installation from zero
+ cmd = "%s %s --noaction install %s " % (self.opkg_cmd,
+ opkg_args,
+ ' '.join(pkgs))
+ cmd += "| awk '/^Installing/{print $2}' "
+ cmd += "| sort -u -o %s" % self.solution_manifest
+ try:
+ subprocess.check_output(cmd, shell=True)
+ with open(self.solution_manifest, 'r') as manifest:
+ for pkg in manifest.read().split('\n'):
+ install_pkgs.append(pkg)
+ except subprocess.CalledProcessError as e:
+ bb.note("Unable to dump install packages. Command '%s' "
+ "returned %d:\n%s" % (cmd, e.returncode, e.output))
+
+ bb.utils.remove(temp_rootfs, True)
+
+ return install_pkgs
+
+ '''
+ If incremental install, we need to determine what we've got,
+ what we need to add, and what to remove...
+ The load_old_install_solution will load the previous install
+ solution
+ '''
+ def load_old_install_solution(self):
+ bb.note('load old install solution for incremental install')
+ installed_pkgs = list()
+ if not os.path.exists(self.solution_manifest):
+ bb.note('old install solution not exist')
+ return installed_pkgs
+
+ with open(self.solution_manifest, 'r') as manifest:
+ for pkg in manifest.read().split('\n'):
+ installed_pkgs.append(pkg.strip())
+
+ return installed_pkgs
+
+ def backup_packaging_data(self):
+ # Save the opkglib for increment rpm image generation
+ if os.path.exists(self.saved_opkg_dir):
+ bb.utils.remove(self.saved_opkg_dir, True)
+ shutil.copytree(self.opkg_dir,
+ self.saved_opkg_dir,
+ symlinks=True)
+
+ def recovery_packaging_data(self):
+ # Move the opkglib back
+ if os.path.exists(self.saved_opkg_dir):
+ if os.path.exists(self.opkg_dir):
+ bb.utils.remove(self.opkg_dir, True)
+
+ bb.note('Recovery packaging data')
+ shutil.copytree(self.saved_opkg_dir,
+ self.opkg_dir,
+ symlinks=True)
+
class DpkgPM(PackageManager):
def __init__(self, d, target_rootfs, archs, base_archs, apt_conf_dir=None):
--
1.8.1.2
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH 2/5] package_manager.py: tweak handle_bad_recommendations for ipk incremental image generation
2014-02-18 9:42 [PATCH 0/5] package_manager.py/ootfs.py: support ipk incremental image generation Hongxu Jia
2014-02-18 9:42 ` [PATCH 1/5] package_manager.py: " Hongxu Jia
@ 2014-02-18 9:42 ` Hongxu Jia
2014-02-18 9:42 ` [PATCH 3/5] rootfs.py: support " Hongxu Jia
` (2 subsequent siblings)
4 siblings, 0 replies; 13+ messages in thread
From: Hongxu Jia @ 2014-02-18 9:42 UTC (permalink / raw)
To: openembedded-core; +Cc: saul.wold
Let opkg_dir be configurable in handle_bad_recommendations,
so it could work on other root fs.
Invoke handle_bad_recommendations in dump_install_solution
which simulates package install on other root fs.
[YOCTO #1894]
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
meta/lib/oe/package_manager.py | 18 +++++++++++++-----
1 file changed, 13 insertions(+), 5 deletions(-)
diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index 4ea9677..3cfd4af 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -1157,12 +1157,19 @@ class OpkgPM(PackageManager):
return output
- def handle_bad_recommendations(self):
- bad_recommendations = self.d.getVar("BAD_RECOMMENDATIONS", True)
- if bad_recommendations is None:
+ def handle_bad_recommendations(self, opkg_dir=""):
+ bad_recommendations = self.d.getVar("BAD_RECOMMENDATIONS", True) or ""
+ if bad_recommendations.strip() == "":
return
- status_file = os.path.join(self.opkg_dir, "status")
+ if opkg_dir == "":
+ opkg_dir = self.opkg_dir
+ status_file = os.path.join(opkg_dir, "status")
+
+ # If status file existed, it means the bad recommendations has already
+ # been handled
+ if os.path.exists(status_file):
+ return
cmd = "%s %s info " % (self.opkg_cmd, self.opkg_args)
@@ -1177,7 +1184,7 @@ class OpkgPM(PackageManager):
"returned %d:\n%s" % (pkg_info, e.returncode, e.output))
if output == "":
- bb.note("Requested ignored recommendation $i is "
+ bb.note("Requested ignored recommendation %s is "
"not a package" % pkg)
continue
@@ -1204,6 +1211,7 @@ class OpkgPM(PackageManager):
temp_rootfs = self.d.expand('${T}/opkg')
temp_opkg_dir = os.path.join(temp_rootfs, 'var/lib/opkg')
bb.utils.mkdirhier(temp_opkg_dir)
+ self.handle_bad_recommendations(temp_opkg_dir)
opkg_args = "-f %s -o %s " % (self.config_file, temp_rootfs)
opkg_args += self.d.getVar("OPKG_ARGS", True)
--
1.8.1.2
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH 3/5] rootfs.py: support ipk incremental image generation
2014-02-18 9:42 [PATCH 0/5] package_manager.py/ootfs.py: support ipk incremental image generation Hongxu Jia
2014-02-18 9:42 ` [PATCH 1/5] package_manager.py: " Hongxu Jia
2014-02-18 9:42 ` [PATCH 2/5] package_manager.py: tweak handle_bad_recommendations for " Hongxu Jia
@ 2014-02-18 9:42 ` Hongxu Jia
2014-02-18 9:42 ` [PATCH 4/5] rootfs.py: fix BAD_RECOMMENDATIONS for " Hongxu Jia
2014-02-18 9:42 ` [PATCH 5/5] rootfs.py: tweak _multilib_sanity_test " Hongxu Jia
4 siblings, 0 replies; 13+ messages in thread
From: Hongxu Jia @ 2014-02-18 9:42 UTC (permalink / raw)
To: openembedded-core; +Cc: saul.wold
While incremental image generation enabled, if previous image existed,
it restores the opkg database. Based on the previous image, it gets what
need to remove for the current and remove them.
The newly added and upgraded packages will be done since opkg install
invoked.
[YOCTO #1894]
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
meta/lib/oe/rootfs.py | 50 +++++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 47 insertions(+), 3 deletions(-)
diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
index d149ca3..5561bb9 100644
--- a/meta/lib/oe/rootfs.py
+++ b/meta/lib/oe/rootfs.py
@@ -438,13 +438,25 @@ class OpkgRootfs(Rootfs):
def __init__(self, d, manifest_dir):
super(OpkgRootfs, self).__init__(d)
- bb.utils.remove(self.image_rootfs, True)
- bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True)
self.manifest = OpkgManifest(d, manifest_dir)
self.opkg_conf = self.d.getVar("IPKGCONF_TARGET", True)
self.pkg_archs = self.d.getVar("ALL_MULTILIB_PACKAGE_ARCHS", True)
- self.pm = OpkgPM(d, self.image_rootfs, self.opkg_conf, self.pkg_archs)
+ self.inc_opkg_image_gen = self.d.getVar('INC_IPK_IMAGE_GEN', True)
+ if self.inc_opkg_image_gen != "1":
+ bb.utils.remove(self.image_rootfs, True)
+ self.pm = OpkgPM(d,
+ self.image_rootfs,
+ self.opkg_conf,
+ self.pkg_archs)
+ else:
+ self.pm = OpkgPM(d,
+ self.image_rootfs,
+ self.opkg_conf,
+ self.pkg_archs)
+ self.pm.recovery_packaging_data()
+
+ bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True)
"""
This function was reused from the old implementation.
@@ -508,6 +520,32 @@ class OpkgRootfs(Rootfs):
self._multilib_sanity_test(dirs)
+ '''
+ While ipk incremental image generation is enabled, it will remove the
+ unneeded pkgs by comparing the new install solution manifest and the
+ old installed manifest.
+ '''
+ def _create_incremental(self, pkgs_initial_install):
+ if self.inc_opkg_image_gen == "1":
+
+ pkgs_to_install = list()
+ for pkg_type in pkgs_initial_install:
+ pkgs_to_install += pkgs_initial_install[pkg_type]
+
+ installed_manifest = self.pm.load_old_install_solution()
+ solution_manifest = self.pm.dump_install_solution(pkgs_to_install)
+
+ pkg_to_remove = list()
+ for pkg in installed_manifest:
+ if pkg not in solution_manifest:
+ pkg_to_remove.append(pkg)
+
+ self.pm.update()
+
+ if pkg_to_remove != []:
+ bb.note('incremental removed: %s' % ' '.join(pkg_to_remove))
+ self.pm.remove(pkg_to_remove)
+
def _create(self):
pkgs_to_install = self.manifest.parse_initial_manifest()
opkg_pre_process_cmds = self.d.getVar('OPKG_PREPROCESS_COMMANDS', True)
@@ -518,6 +556,9 @@ class OpkgRootfs(Rootfs):
if (self.d.getVar('BUILD_IMAGES_FROM_FEEDS', True) or "") != "1":
self.pm.write_index()
+ if self.inc_opkg_image_gen == "1":
+ self._create_incremental(pkgs_to_install)
+
execute_pre_post_process(self.d, opkg_pre_process_cmds)
self.pm.update()
@@ -540,6 +581,9 @@ class OpkgRootfs(Rootfs):
execute_pre_post_process(self.d, opkg_post_process_cmds)
execute_pre_post_process(self.d, rootfs_post_install_cmds)
+ if self.inc_opkg_image_gen == "1":
+ self.pm.backup_packaging_data()
+
def _get_delayed_postinsts(self):
pkg_list = []
status_file = os.path.join(self.image_rootfs,
--
1.8.1.2
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH 4/5] rootfs.py: fix BAD_RECOMMENDATIONS for ipk incremental image generation
2014-02-18 9:42 [PATCH 0/5] package_manager.py/ootfs.py: support ipk incremental image generation Hongxu Jia
` (2 preceding siblings ...)
2014-02-18 9:42 ` [PATCH 3/5] rootfs.py: support " Hongxu Jia
@ 2014-02-18 9:42 ` Hongxu Jia
2014-02-18 15:03 ` Laurentiu Palcu
2014-02-18 9:42 ` [PATCH 5/5] rootfs.py: tweak _multilib_sanity_test " Hongxu Jia
4 siblings, 1 reply; 13+ messages in thread
From: Hongxu Jia @ 2014-02-18 9:42 UTC (permalink / raw)
To: openembedded-core; +Cc: saul.wold
While incremental image generation enabled and the previous image existed,
if BAD_RECOMMENDATIONS is changed, the operation on the existed image is
complicated, so remove the existed image in this situation.
The same with PACKAGE_EXCLUDE and NO_RECOMMENDATIONS.
[YOCTO #1894]
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
meta/lib/oe/rootfs.py | 30 +++++++++++++++++++++++++++++-
1 file changed, 29 insertions(+), 1 deletion(-)
diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
index 5561bb9..3d7adf9 100644
--- a/meta/lib/oe/rootfs.py
+++ b/meta/lib/oe/rootfs.py
@@ -443,7 +443,7 @@ class OpkgRootfs(Rootfs):
self.pkg_archs = self.d.getVar("ALL_MULTILIB_PACKAGE_ARCHS", True)
self.inc_opkg_image_gen = self.d.getVar('INC_IPK_IMAGE_GEN', True)
- if self.inc_opkg_image_gen != "1":
+ if self._remove_existed_image():
bb.utils.remove(self.image_rootfs, True)
self.pm = OpkgPM(d,
self.image_rootfs,
@@ -546,6 +546,34 @@ class OpkgRootfs(Rootfs):
bb.note('incremental removed: %s' % ' '.join(pkg_to_remove))
self.pm.remove(pkg_to_remove)
+ '''
+ Compare with previous (existed) image creation, if some conditions
+ triggered, the previous (existed) image should be removed.
+ The conditions included any of 'PACKAGE_EXCLUDE, NO_RECOMMENDATIONS
+ and BAD_RECOMMENDATIONS' has been changed.
+ '''
+ def _remove_existed_image(self):
+ # Incremental image creation is not enable
+ if self.inc_opkg_image_gen != "1":
+ return True
+
+ vars_list_dir = self.d.expand('${T}/vars_list')
+
+ old_vars_list = ""
+ if os.path.exists(vars_list_dir):
+ old_vars_list = open(vars_list_dir, 'r+').read()
+
+ new_vars_list = '%s:%s:%s\n' % \
+ ((self.d.getVar('BAD_RECOMMENDATIONS', True) or '').strip(),
+ (self.d.getVar('NO_RECOMMENDATIONS', True) or '').strip(),
+ (self.d.getVar('PACKAGE_EXCLUDE', True) or '').strip())
+ open(vars_list_dir, 'w+').write(new_vars_list)
+
+ if old_vars_list != new_vars_list:
+ return True
+
+ return False
+
def _create(self):
pkgs_to_install = self.manifest.parse_initial_manifest()
opkg_pre_process_cmds = self.d.getVar('OPKG_PREPROCESS_COMMANDS', True)
--
1.8.1.2
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH 5/5] rootfs.py: tweak _multilib_sanity_test for ipk incremental image generation
2014-02-18 9:42 [PATCH 0/5] package_manager.py/ootfs.py: support ipk incremental image generation Hongxu Jia
` (3 preceding siblings ...)
2014-02-18 9:42 ` [PATCH 4/5] rootfs.py: fix BAD_RECOMMENDATIONS for " Hongxu Jia
@ 2014-02-18 9:42 ` Hongxu Jia
2014-02-18 15:36 ` Laurentiu Palcu
4 siblings, 1 reply; 13+ messages in thread
From: Hongxu Jia @ 2014-02-18 9:42 UTC (permalink / raw)
To: openembedded-core; +Cc: saul.wold
The _multilib_sanity_test installs multilib packages in a temporary
root fs, and compare with the current image to figure out duplicated
file that comes from different packages.
While incremental image generation enabled and the previous image
existed, there was a Multilib check error:
...
ERROR: Multilib check error: duplicate files tmp/work/qemux86_64-poky-
linux/core-image-minimal/1.0-r0/multilib/lib32/lib/libc.so.6 tmp/work/
qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/lib/libc.so.6
is not the same
...
The reason is the file in current image has been prelinked in previous
image generation and the file in a temporary root fs is not prelinked,
even though the files come from the same package, the Multilib check
considers they are different.
[YOCTO #1894]
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
meta/lib/oe/rootfs.py | 56 +++++++++++++++++++++++++++++++++++++++++++++++----
1 file changed, 52 insertions(+), 4 deletions(-)
diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
index 3d7adf9..5054d1e 100644
--- a/meta/lib/oe/rootfs.py
+++ b/meta/lib/oe/rootfs.py
@@ -3,6 +3,8 @@ from oe.utils import execute_pre_post_process
from oe.utils import contains as base_contains
from oe.package_manager import *
from oe.manifest import *
+import oe.path
+import filecmp
import shutil
import os
import subprocess
@@ -458,13 +460,61 @@ class OpkgRootfs(Rootfs):
bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True)
+ def _prelink_file(self, root_dir, filename):
+ bb.note('prelink %s in %s' % (filename, root_dir))
+ prelink_cfg = oe.path.join(root_dir,
+ self.d.expand('${sysconfdir}/prelink.conf'))
+ if not os.path.exists(prelink_cfg):
+ shutil.copy(self.d.expand('${STAGING_DIR_NATIVE}${sysconfdir_native}/prelink.conf'),
+ prelink_cfg)
+
+ cmd_prelink = self.d.expand('${STAGING_DIR_NATIVE}${sbindir_native}/prelink')
+ self._exec_shell_cmd([cmd_prelink,
+ '--root',
+ root_dir,
+ '-amR',
+ '-N',
+ '-c',
+ self.d.expand('${sysconfdir}/prelink.conf')])
+
+ '''
+ Compare two files with the same key twice to see if they came
+ from the same package. If they are not same, they are duplicated
+ and come from different packages.
+ 1st: Comapre them directly;
+ 2nd: While incremental image creation is enabled, one of the
+ files could be probaly prelinked in the previous image
+ creation and the file has been changed, so we need to
+ prelink the other one and compare them.
+ '''
+ def _file_duplicate(self, key, f1, f2):
+
+ if not os.path.exists(f1) or not os.path.exists(f2):
+ return False
+
+ # f1 is the same with f2, both of them were not prelinked
+ if filecmp.cmp(f1, f2):
+ return False
+
+ if self.image_rootfs not in f1:
+ self._prelink_file(f1.replace(key, ''), f1)
+
+ if self.image_rootfs not in f2:
+ self._prelink_file(f2.replace(key, ''), f2)
+
+ # f1 is the same with f2, both of them were prelinked
+ if filecmp.cmp(f1, f2):
+ return False
+
+ # Duplicated
+ return True
+
"""
This function was reused from the old implementation.
See commit: "image.bbclass: Added variables for multilib support." by
Lianhao Lu.
"""
def _multilib_sanity_test(self, dirs):
- import filecmp
allow_replace = self.d.getVar("MULTILIBRE_ALLOW_REP", True)
if allow_replace is None:
@@ -486,9 +536,7 @@ class OpkgRootfs(Rootfs):
if allow_rep.match(key):
valid = True
else:
- if os.path.exists(files[key]) and \
- os.path.exists(item) and \
- not filecmp.cmp(files[key], item):
+ if self._file_duplicate(key, files[key], item):
valid = False
bb.fatal("%s duplicate files %s %s is not the same\n" %
(error_prompt, item, files[key]))
--
1.8.1.2
^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PATCH 1/5] package_manager.py: support ipk incremental image generation
2014-02-18 9:42 ` [PATCH 1/5] package_manager.py: " Hongxu Jia
@ 2014-02-18 14:38 ` Laurentiu Palcu
2014-02-19 9:39 ` Hongxu Jia
0 siblings, 1 reply; 13+ messages in thread
From: Laurentiu Palcu @ 2014-02-18 14:38 UTC (permalink / raw)
To: Hongxu Jia; +Cc: saul.wold, openembedded-core
On Tue, Feb 18, 2014 at 05:42:24PM +0800, Hongxu Jia wrote:
> While incremental image generation enabled, 'load_old_install_solution'
> is used to determine what we've got in the previous (existed) image and
> 'dump_install_solution' to determine what we need to install in the
> current image.
>
> The 'backup_packaging_data' is used to back up the current opkg database.
>
> The 'recovery_packaging_data' is used to recover the opkg database which
> backed up by the previous image creation.
>
> Tweak 'remove' function in OpkgPM class, which the options for remove
> with dependencies was incorrect.
>
> [YOCTO #1894]
>
> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
> ---
> meta/lib/oe/package_manager.py | 106 +++++++++++++++++++++++++++++++++++++++--
> 1 file changed, 102 insertions(+), 4 deletions(-)
>
> diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
> index 6dc8fbd..4ea9677 100644
> --- a/meta/lib/oe/package_manager.py
> +++ b/meta/lib/oe/package_manager.py
> @@ -934,12 +934,13 @@ class RpmPM(PackageManager):
>
>
> class OpkgPM(PackageManager):
> - def __init__(self, d, target_rootfs, config_file, archs):
> + def __init__(self, d, target_rootfs, config_file, archs, task_name='target'):
> super(OpkgPM, self).__init__(d)
>
> self.target_rootfs = target_rootfs
> self.config_file = config_file
> self.pkg_archs = archs
> + self.task_name = task_name
>
> self.deploy_dir = self.d.getVar("DEPLOY_DIR_IPK", True)
> self.deploy_lock_file = os.path.join(self.deploy_dir, "deploy.lock")
> @@ -956,6 +957,13 @@ class OpkgPM(PackageManager):
>
> bb.utils.mkdirhier(self.opkg_dir)
>
> + self.solution_manifest = self.d.expand('${T}/saved/%s_solution' %
> + self.task_name)
> + self.saved_opkg_dir = self.d.expand('${T}/saved/%s' % self.task_name)
> + if not os.path.exists(self.d.expand('${T}/saved')):
> + bb.utils.mkdirhier(self.d.expand('${T}/saved'))
> +
> +
> if (self.d.getVar('BUILD_IMAGES_FROM_FEEDS', True) or "") != "1":
> self._create_config()
> else:
> @@ -1075,7 +1083,9 @@ class OpkgPM(PackageManager):
>
> try:
> bb.note("Installing the following packages: %s" % ' '.join(pkgs))
> - subprocess.check_output(cmd.split())
> + bb.note(cmd)
> + output = subprocess.check_output(cmd.split())
> + bb.note(output)
> except subprocess.CalledProcessError as e:
> (bb.fatal, bb.note)[attempt_only]("Unable to install packages. "
> "Command '%s' returned %d:\n%s" %
> @@ -1083,14 +1093,16 @@ class OpkgPM(PackageManager):
>
> def remove(self, pkgs, with_dependencies=True):
> if with_dependencies:
> - cmd = "%s %s remove %s" % \
> + cmd = "%s %s --force-depends --force-remove --force-removal-of-dependent-packages remove %s" % \
> (self.opkg_cmd, self.opkg_args, ' '.join(pkgs))
> else:
> cmd = "%s %s --force-depends remove %s" % \
> (self.opkg_cmd, self.opkg_args, ' '.join(pkgs))
>
> try:
> - subprocess.check_output(cmd.split())
> + bb.note(cmd)
> + output = subprocess.check_output(cmd.split())
> + bb.note(output)
> except subprocess.CalledProcessError as e:
> bb.fatal("Unable to remove packages. Command '%s' "
> "returned %d:\n%s" % (e.cmd, e.returncode, e.output))
> @@ -1175,6 +1187,92 @@ class OpkgPM(PackageManager):
> else:
> status.write(line + "\n")
>
> + '''
> + If incremental install, we need to determine what we've got,
> + what we need to add, and what to remove...
> + The dump_install_solution will dump and save the new install
> + solution.
> + '''
> + def dump_install_solution(self, pkgs):
Why not have this function in the Manifest class? We have an API in
place: Manifest.create_final() that should probably take care of
anything related to manifest(s) creation after all packages are installed.
> + bb.note('creating new install solution for incremental install')
> + if len(pkgs) == 0:
> + return
> +
> + install_pkgs = list()
> +
> + # Create an temp dir as opkg root for simulating the installation
> + temp_rootfs = self.d.expand('${T}/opkg')
> + temp_opkg_dir = os.path.join(temp_rootfs, 'var/lib/opkg')
> + bb.utils.mkdirhier(temp_opkg_dir)
> +
> + opkg_args = "-f %s -o %s " % (self.config_file, temp_rootfs)
> + opkg_args += self.d.getVar("OPKG_ARGS", True)
> +
> + cmd = "%s %s update" % (self.opkg_cmd,
> + opkg_args)
> + try:
> + subprocess.check_output(cmd, shell=True)
> + except subprocess.CalledProcessError as e:
> + bb.note("Unable to dump install packages. Command '%s' "
> + "returned %d:\n%s" % (cmd, e.returncode, e.output))
> +
> + # Simulate installation from zero
> + cmd = "%s %s --noaction install %s " % (self.opkg_cmd,
> + opkg_args,
> + ' '.join(pkgs))
> + cmd += "| awk '/^Installing/{print $2}' "
> + cmd += "| sort -u -o %s" % self.solution_manifest
Do we need to preserve this bash one-liner? Why not handle the command
output using Python?
> + try:
> + subprocess.check_output(cmd, shell=True)
> + with open(self.solution_manifest, 'r') as manifest:
> + for pkg in manifest.read().split('\n'):
> + install_pkgs.append(pkg)
> + except subprocess.CalledProcessError as e:
> + bb.note("Unable to dump install packages. Command '%s' "
> + "returned %d:\n%s" % (cmd, e.returncode, e.output))
> +
> + bb.utils.remove(temp_rootfs, True)
> +
> + return install_pkgs
> +
> + '''
> + If incremental install, we need to determine what we've got,
> + what we need to add, and what to remove...
> + The load_old_install_solution will load the previous install
> + solution
> + '''
> + def load_old_install_solution(self):
Same here: why not put it in the Manifest class? As we already have the
Manifest.parse_initial_manifest(), we can have something similar for the
final_manifest (or install_solution, etc) parsing.
> + bb.note('load old install solution for incremental install')
> + installed_pkgs = list()
> + if not os.path.exists(self.solution_manifest):
> + bb.note('old install solution not exist')
> + return installed_pkgs
> +
> + with open(self.solution_manifest, 'r') as manifest:
> + for pkg in manifest.read().split('\n'):
> + installed_pkgs.append(pkg.strip())
> +
> + return installed_pkgs
> +
> + def backup_packaging_data(self):
> + # Save the opkglib for increment rpm image generation
s/rpm/opkg/
> + if os.path.exists(self.saved_opkg_dir):
> + bb.utils.remove(self.saved_opkg_dir, True)
> + shutil.copytree(self.opkg_dir,
> + self.saved_opkg_dir,
> + symlinks=True)
> +
> + def recovery_packaging_data(self):
s/recovery/recover/ ?
> + # Move the opkglib back
> + if os.path.exists(self.saved_opkg_dir):
> + if os.path.exists(self.opkg_dir):
> + bb.utils.remove(self.opkg_dir, True)
> +
> + bb.note('Recovery packaging data')
> + shutil.copytree(self.saved_opkg_dir,
> + self.opkg_dir,
> + symlinks=True)
> +
>
> class DpkgPM(PackageManager):
> def __init__(self, d, target_rootfs, archs, base_archs, apt_conf_dir=None):
> --
> 1.8.1.2
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 4/5] rootfs.py: fix BAD_RECOMMENDATIONS for ipk incremental image generation
2014-02-18 9:42 ` [PATCH 4/5] rootfs.py: fix BAD_RECOMMENDATIONS for " Hongxu Jia
@ 2014-02-18 15:03 ` Laurentiu Palcu
2014-02-19 9:44 ` Hongxu Jia
0 siblings, 1 reply; 13+ messages in thread
From: Laurentiu Palcu @ 2014-02-18 15:03 UTC (permalink / raw)
To: Hongxu Jia; +Cc: saul.wold, openembedded-core
On Tue, Feb 18, 2014 at 05:42:27PM +0800, Hongxu Jia wrote:
> While incremental image generation enabled and the previous image existed,
> if BAD_RECOMMENDATIONS is changed, the operation on the existed image is
> complicated, so remove the existed image in this situation.
>
> The same with PACKAGE_EXCLUDE and NO_RECOMMENDATIONS.
>
> [YOCTO #1894]
> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
> ---
> meta/lib/oe/rootfs.py | 30 +++++++++++++++++++++++++++++-
> 1 file changed, 29 insertions(+), 1 deletion(-)
>
> diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
> index 5561bb9..3d7adf9 100644
> --- a/meta/lib/oe/rootfs.py
> +++ b/meta/lib/oe/rootfs.py
> @@ -443,7 +443,7 @@ class OpkgRootfs(Rootfs):
> self.pkg_archs = self.d.getVar("ALL_MULTILIB_PACKAGE_ARCHS", True)
>
> self.inc_opkg_image_gen = self.d.getVar('INC_IPK_IMAGE_GEN', True)
> - if self.inc_opkg_image_gen != "1":
> + if self._remove_existed_image():
> bb.utils.remove(self.image_rootfs, True)
> self.pm = OpkgPM(d,
> self.image_rootfs,
> @@ -546,6 +546,34 @@ class OpkgRootfs(Rootfs):
> bb.note('incremental removed: %s' % ' '.join(pkg_to_remove))
> self.pm.remove(pkg_to_remove)
>
> + '''
> + Compare with previous (existed) image creation, if some conditions
> + triggered, the previous (existed) image should be removed.
> + The conditions included any of 'PACKAGE_EXCLUDE, NO_RECOMMENDATIONS
> + and BAD_RECOMMENDATIONS' has been changed.
> + '''
> + def _remove_existed_image(self):
s/existed/existing/
Also in the comment above.
> + # Incremental image creation is not enable
> + if self.inc_opkg_image_gen != "1":
> + return True
> +
> + vars_list_dir = self.d.expand('${T}/vars_list')
s/vars_list_dir/vars_list_file/ in the line above and the code below.
Leaving this variable with a 'dir' suffix while we're opening a file for
reading/writing, is a little misleading.
> +
> + old_vars_list = ""
> + if os.path.exists(vars_list_dir):
> + old_vars_list = open(vars_list_dir, 'r+').read()
> +
> + new_vars_list = '%s:%s:%s\n' % \
> + ((self.d.getVar('BAD_RECOMMENDATIONS', True) or '').strip(),
> + (self.d.getVar('NO_RECOMMENDATIONS', True) or '').strip(),
> + (self.d.getVar('PACKAGE_EXCLUDE', True) or '').strip())
> + open(vars_list_dir, 'w+').write(new_vars_list)
> +
> + if old_vars_list != new_vars_list:
> + return True
> +
> + return False
> +
> def _create(self):
> pkgs_to_install = self.manifest.parse_initial_manifest()
> opkg_pre_process_cmds = self.d.getVar('OPKG_PREPROCESS_COMMANDS', True)
> --
> 1.8.1.2
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 5/5] rootfs.py: tweak _multilib_sanity_test for ipk incremental image generation
2014-02-18 9:42 ` [PATCH 5/5] rootfs.py: tweak _multilib_sanity_test " Hongxu Jia
@ 2014-02-18 15:36 ` Laurentiu Palcu
2014-02-19 9:51 ` Hongxu Jia
0 siblings, 1 reply; 13+ messages in thread
From: Laurentiu Palcu @ 2014-02-18 15:36 UTC (permalink / raw)
To: Hongxu Jia; +Cc: saul.wold, openembedded-core
On Tue, Feb 18, 2014 at 05:42:28PM +0800, Hongxu Jia wrote:
> The _multilib_sanity_test installs multilib packages in a temporary
> root fs, and compare with the current image to figure out duplicated
> file that comes from different packages.
>
> While incremental image generation enabled and the previous image
> existed, there was a Multilib check error:
> ...
> ERROR: Multilib check error: duplicate files tmp/work/qemux86_64-poky-
> linux/core-image-minimal/1.0-r0/multilib/lib32/lib/libc.so.6 tmp/work/
> qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/lib/libc.so.6
> is not the same
> ...
>
> The reason is the file in current image has been prelinked in previous
> image generation and the file in a temporary root fs is not prelinked,
> even though the files come from the same package, the Multilib check
> considers they are different.
>
> [YOCTO #1894]
> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
> ---
> meta/lib/oe/rootfs.py | 56 +++++++++++++++++++++++++++++++++++++++++++++++----
> 1 file changed, 52 insertions(+), 4 deletions(-)
>
> diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
> index 3d7adf9..5054d1e 100644
> --- a/meta/lib/oe/rootfs.py
> +++ b/meta/lib/oe/rootfs.py
> @@ -3,6 +3,8 @@ from oe.utils import execute_pre_post_process
> from oe.utils import contains as base_contains
> from oe.package_manager import *
> from oe.manifest import *
> +import oe.path
> +import filecmp
> import shutil
> import os
> import subprocess
> @@ -458,13 +460,61 @@ class OpkgRootfs(Rootfs):
>
> bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True)
>
> + def _prelink_file(self, root_dir, filename):
> + bb.note('prelink %s in %s' % (filename, root_dir))
> + prelink_cfg = oe.path.join(root_dir,
> + self.d.expand('${sysconfdir}/prelink.conf'))
> + if not os.path.exists(prelink_cfg):
> + shutil.copy(self.d.expand('${STAGING_DIR_NATIVE}${sysconfdir_native}/prelink.conf'),
> + prelink_cfg)
> +
> + cmd_prelink = self.d.expand('${STAGING_DIR_NATIVE}${sbindir_native}/prelink')
> + self._exec_shell_cmd([cmd_prelink,
> + '--root',
> + root_dir,
> + '-amR',
> + '-N',
> + '-c',
> + self.d.expand('${sysconfdir}/prelink.conf')])
> +
> + '''
> + Compare two files with the same key twice to see if they came
> + from the same package. If they are not same, they are duplicated
> + and come from different packages.
I'm kind of confused by this comment. Doesn't same = duplicate? There
might be a small confusion of terms here because the function's behavior
is not as the name implies.
> + 1st: Comapre them directly;
> + 2nd: While incremental image creation is enabled, one of the
> + files could be probaly prelinked in the previous image
> + creation and the file has been changed, so we need to
> + prelink the other one and compare them.
> + '''
> + def _file_duplicate(self, key, f1, f2):
Shouldn't be better to rename this function to something else in order
to avoid confusion? Let's say: _files_are_equal() ?
> +
> + if not os.path.exists(f1) or not os.path.exists(f2):
> + return False
> +
> + # f1 is the same with f2, both of them were not prelinked
> + if filecmp.cmp(f1, f2):
> + return False
filecmp.cmp() returns True if files are equal. Hence the confusion:
_file_duplicate() returns False here, if files are equal... I think the logic
is a little bit the other way around! :)
> +
> + if self.image_rootfs not in f1:
> + self._prelink_file(f1.replace(key, ''), f1)
> +
> + if self.image_rootfs not in f2:
> + self._prelink_file(f2.replace(key, ''), f2)
> +
> + # f1 is the same with f2, both of them were prelinked
> + if filecmp.cmp(f1, f2):
> + return False
> +
> + # Duplicated
> + return True
here the return value matches the comment and the function name! :)
> +
> """
> This function was reused from the old implementation.
> See commit: "image.bbclass: Added variables for multilib support." by
> Lianhao Lu.
> """
> def _multilib_sanity_test(self, dirs):
> - import filecmp
>
> allow_replace = self.d.getVar("MULTILIBRE_ALLOW_REP", True)
> if allow_replace is None:
> @@ -486,9 +536,7 @@ class OpkgRootfs(Rootfs):
> if allow_rep.match(key):
> valid = True
> else:
> - if os.path.exists(files[key]) and \
> - os.path.exists(item) and \
> - not filecmp.cmp(files[key], item):
> + if self._file_duplicate(key, files[key], item):
> valid = False
> bb.fatal("%s duplicate files %s %s is not the same\n" %
> (error_prompt, item, files[key]))
> --
> 1.8.1.2
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 1/5] package_manager.py: support ipk incremental image generation
2014-02-18 14:38 ` Laurentiu Palcu
@ 2014-02-19 9:39 ` Hongxu Jia
0 siblings, 0 replies; 13+ messages in thread
From: Hongxu Jia @ 2014-02-19 9:39 UTC (permalink / raw)
To: Laurentiu Palcu; +Cc: saul.wold, openembedded-core
On 02/18/2014 10:38 PM, Laurentiu Palcu wrote:
> On Tue, Feb 18, 2014 at 05:42:24PM +0800, Hongxu Jia wrote:
>> While incremental image generation enabled, 'load_old_install_solution'
>> is used to determine what we've got in the previous (existed) image and
>> 'dump_install_solution' to determine what we need to install in the
>> current image.
>>
>> The 'backup_packaging_data' is used to back up the current opkg database.
>>
>> The 'recovery_packaging_data' is used to recover the opkg database which
>> backed up by the previous image creation.
>>
>> Tweak 'remove' function in OpkgPM class, which the options for remove
>> with dependencies was incorrect.
>>
>> [YOCTO #1894]
>>
>> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
>> ---
>> meta/lib/oe/package_manager.py | 106 +++++++++++++++++++++++++++++++++++++++--
>> 1 file changed, 102 insertions(+), 4 deletions(-)
>>
>> diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
>> index 6dc8fbd..4ea9677 100644
>> --- a/meta/lib/oe/package_manager.py
>> +++ b/meta/lib/oe/package_manager.py
>> @@ -934,12 +934,13 @@ class RpmPM(PackageManager):
>>
>>
>> class OpkgPM(PackageManager):
>> - def __init__(self, d, target_rootfs, config_file, archs):
>> + def __init__(self, d, target_rootfs, config_file, archs, task_name='target'):
>> super(OpkgPM, self).__init__(d)
>>
>> self.target_rootfs = target_rootfs
>> self.config_file = config_file
>> self.pkg_archs = archs
>> + self.task_name = task_name
>>
>> self.deploy_dir = self.d.getVar("DEPLOY_DIR_IPK", True)
>> self.deploy_lock_file = os.path.join(self.deploy_dir, "deploy.lock")
>> @@ -956,6 +957,13 @@ class OpkgPM(PackageManager):
>>
>> bb.utils.mkdirhier(self.opkg_dir)
>>
>> + self.solution_manifest = self.d.expand('${T}/saved/%s_solution' %
>> + self.task_name)
>> + self.saved_opkg_dir = self.d.expand('${T}/saved/%s' % self.task_name)
>> + if not os.path.exists(self.d.expand('${T}/saved')):
>> + bb.utils.mkdirhier(self.d.expand('${T}/saved'))
>> +
>> +
>> if (self.d.getVar('BUILD_IMAGES_FROM_FEEDS', True) or "") != "1":
>> self._create_config()
>> else:
>> @@ -1075,7 +1083,9 @@ class OpkgPM(PackageManager):
>>
>> try:
>> bb.note("Installing the following packages: %s" % ' '.join(pkgs))
>> - subprocess.check_output(cmd.split())
>> + bb.note(cmd)
>> + output = subprocess.check_output(cmd.split())
>> + bb.note(output)
>> except subprocess.CalledProcessError as e:
>> (bb.fatal, bb.note)[attempt_only]("Unable to install packages. "
>> "Command '%s' returned %d:\n%s" %
>> @@ -1083,14 +1093,16 @@ class OpkgPM(PackageManager):
>>
>> def remove(self, pkgs, with_dependencies=True):
>> if with_dependencies:
>> - cmd = "%s %s remove %s" % \
>> + cmd = "%s %s --force-depends --force-remove --force-removal-of-dependent-packages remove %s" % \
>> (self.opkg_cmd, self.opkg_args, ' '.join(pkgs))
>> else:
>> cmd = "%s %s --force-depends remove %s" % \
>> (self.opkg_cmd, self.opkg_args, ' '.join(pkgs))
>>
>> try:
>> - subprocess.check_output(cmd.split())
>> + bb.note(cmd)
>> + output = subprocess.check_output(cmd.split())
>> + bb.note(output)
>> except subprocess.CalledProcessError as e:
>> bb.fatal("Unable to remove packages. Command '%s' "
>> "returned %d:\n%s" % (e.cmd, e.returncode, e.output))
>> @@ -1175,6 +1187,92 @@ class OpkgPM(PackageManager):
>> else:
>> status.write(line + "\n")
>>
>> + '''
>> + If incremental install, we need to determine what we've got,
>> + what we need to add, and what to remove...
>> + The dump_install_solution will dump and save the new install
>> + solution.
>> + '''
>> + def dump_install_solution(self, pkgs):
> Why not have this function in the Manifest class? We have an API in
> place: Manifest.create_final() that should probably take care of
> anything related to manifest(s) creation after all packages are installed.
Got it,
The dump_install_solution creates a manifest with the
packages that *will be* installed into the image, it is a
dummy installation.
I will move 'dump_install_solution' to Manifest class as
create_full.
>> + bb.note('creating new install solution for incremental install')
>> + if len(pkgs) == 0:
>> + return
>> +
>> + install_pkgs = list()
>> +
>> + # Create an temp dir as opkg root for simulating the installation
>> + temp_rootfs = self.d.expand('${T}/opkg')
>> + temp_opkg_dir = os.path.join(temp_rootfs, 'var/lib/opkg')
>> + bb.utils.mkdirhier(temp_opkg_dir)
>> +
>> + opkg_args = "-f %s -o %s " % (self.config_file, temp_rootfs)
>> + opkg_args += self.d.getVar("OPKG_ARGS", True)
>> +
>> + cmd = "%s %s update" % (self.opkg_cmd,
>> + opkg_args)
>> + try:
>> + subprocess.check_output(cmd, shell=True)
>> + except subprocess.CalledProcessError as e:
>> + bb.note("Unable to dump install packages. Command '%s' "
>> + "returned %d:\n%s" % (cmd, e.returncode, e.output))
>> +
>> + # Simulate installation from zero
>> + cmd = "%s %s --noaction install %s " % (self.opkg_cmd,
>> + opkg_args,
>> + ' '.join(pkgs))
>> + cmd += "| awk '/^Installing/{print $2}' "
>> + cmd += "| sort -u -o %s" % self.solution_manifest
> Do we need to preserve this bash one-liner? Why not handle the command
> output using Python?
Agree, I will handle this in Python.
>> + try:
>> + subprocess.check_output(cmd, shell=True)
>> + with open(self.solution_manifest, 'r') as manifest:
>> + for pkg in manifest.read().split('\n'):
>> + install_pkgs.append(pkg)
>> + except subprocess.CalledProcessError as e:
>> + bb.note("Unable to dump install packages. Command '%s' "
>> + "returned %d:\n%s" % (cmd, e.returncode, e.output))
>> +
>> + bb.utils.remove(temp_rootfs, True)
>> +
>> + return install_pkgs
>> +
>> + '''
>> + If incremental install, we need to determine what we've got,
>> + what we need to add, and what to remove...
>> + The load_old_install_solution will load the previous install
>> + solution
>> + '''
>> + def load_old_install_solution(self):
> Same here: why not put it in the Manifest class? As we already have the
> Manifest.parse_initial_manifest(), we can have something similar for the
> final_manifest (or install_solution, etc) parsing.
Agree, Manifest.parse_full_manifest() will be added
>> + bb.note('load old install solution for incremental install')
>> + installed_pkgs = list()
>> + if not os.path.exists(self.solution_manifest):
>> + bb.note('old install solution not exist')
>> + return installed_pkgs
>> +
>> + with open(self.solution_manifest, 'r') as manifest:
>> + for pkg in manifest.read().split('\n'):
>> + installed_pkgs.append(pkg.strip())
>> +
>> + return installed_pkgs
>> +
>> + def backup_packaging_data(self):
>> + # Save the opkglib for increment rpm image generation
> s/rpm/opkg/
Sorry for the typo
>> + if os.path.exists(self.saved_opkg_dir):
>> + bb.utils.remove(self.saved_opkg_dir, True)
>> + shutil.copytree(self.opkg_dir,
>> + self.saved_opkg_dir,
>> + symlinks=True)
>> +
>> + def recovery_packaging_data(self):
> s/recovery/recover/ ?
Sorry for the typo
V2 incoming
//Hongxu
>> + # Move the opkglib back
>> + if os.path.exists(self.saved_opkg_dir):
>> + if os.path.exists(self.opkg_dir):
>> + bb.utils.remove(self.opkg_dir, True)
>> +
>> + bb.note('Recovery packaging data')
>> + shutil.copytree(self.saved_opkg_dir,
>> + self.opkg_dir,
>> + symlinks=True)
>> +
>>
>> class DpkgPM(PackageManager):
>> def __init__(self, d, target_rootfs, archs, base_archs, apt_conf_dir=None):
>> --
>> 1.8.1.2
>>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 4/5] rootfs.py: fix BAD_RECOMMENDATIONS for ipk incremental image generation
2014-02-18 15:03 ` Laurentiu Palcu
@ 2014-02-19 9:44 ` Hongxu Jia
0 siblings, 0 replies; 13+ messages in thread
From: Hongxu Jia @ 2014-02-19 9:44 UTC (permalink / raw)
To: Laurentiu Palcu; +Cc: saul.wold, openembedded-core
On 02/18/2014 11:03 PM, Laurentiu Palcu wrote:
> On Tue, Feb 18, 2014 at 05:42:27PM +0800, Hongxu Jia wrote:
>> While incremental image generation enabled and the previous image existed,
>> if BAD_RECOMMENDATIONS is changed, the operation on the existed image is
>> complicated, so remove the existed image in this situation.
>>
>> The same with PACKAGE_EXCLUDE and NO_RECOMMENDATIONS.
>>
>> [YOCTO #1894]
>> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
>> ---
>> meta/lib/oe/rootfs.py | 30 +++++++++++++++++++++++++++++-
>> 1 file changed, 29 insertions(+), 1 deletion(-)
>>
>> diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
>> index 5561bb9..3d7adf9 100644
>> --- a/meta/lib/oe/rootfs.py
>> +++ b/meta/lib/oe/rootfs.py
>> @@ -443,7 +443,7 @@ class OpkgRootfs(Rootfs):
>> self.pkg_archs = self.d.getVar("ALL_MULTILIB_PACKAGE_ARCHS", True)
>>
>> self.inc_opkg_image_gen = self.d.getVar('INC_IPK_IMAGE_GEN', True)
>> - if self.inc_opkg_image_gen != "1":
>> + if self._remove_existed_image():
>> bb.utils.remove(self.image_rootfs, True)
>> self.pm = OpkgPM(d,
>> self.image_rootfs,
>> @@ -546,6 +546,34 @@ class OpkgRootfs(Rootfs):
>> bb.note('incremental removed: %s' % ' '.join(pkg_to_remove))
>> self.pm.remove(pkg_to_remove)
>>
>> + '''
>> + Compare with previous (existed) image creation, if some conditions
>> + triggered, the previous (existed) image should be removed.
>> + The conditions included any of 'PACKAGE_EXCLUDE, NO_RECOMMENDATIONS
>> + and BAD_RECOMMENDATIONS' has been changed.
>> + '''
>> + def _remove_existed_image(self):
> s/existed/existing/
>
> Also in the comment above.
Sorry for the typo
>
>> + # Incremental image creation is not enable
>> + if self.inc_opkg_image_gen != "1":
>> + return True
>> +
>> + vars_list_dir = self.d.expand('${T}/vars_list')
> s/vars_list_dir/vars_list_file/ in the line above and the code below.
> Leaving this variable with a 'dir' suffix while we're opening a file for
> reading/writing, is a little misleading.
Sorry for the misleading
V2 incoming
//Hongxu
>> +
>> + old_vars_list = ""
>> + if os.path.exists(vars_list_dir):
>> + old_vars_list = open(vars_list_dir, 'r+').read()
>> +
>> + new_vars_list = '%s:%s:%s\n' % \
>> + ((self.d.getVar('BAD_RECOMMENDATIONS', True) or '').strip(),
>> + (self.d.getVar('NO_RECOMMENDATIONS', True) or '').strip(),
>> + (self.d.getVar('PACKAGE_EXCLUDE', True) or '').strip())
>> + open(vars_list_dir, 'w+').write(new_vars_list)
>> +
>> + if old_vars_list != new_vars_list:
>> + return True
>> +
>> + return False
>> +
>> def _create(self):
>> pkgs_to_install = self.manifest.parse_initial_manifest()
>> opkg_pre_process_cmds = self.d.getVar('OPKG_PREPROCESS_COMMANDS', True)
>> --
>> 1.8.1.2
>>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 5/5] rootfs.py: tweak _multilib_sanity_test for ipk incremental image generation
2014-02-18 15:36 ` Laurentiu Palcu
@ 2014-02-19 9:51 ` Hongxu Jia
0 siblings, 0 replies; 13+ messages in thread
From: Hongxu Jia @ 2014-02-19 9:51 UTC (permalink / raw)
To: Laurentiu Palcu; +Cc: saul.wold, openembedded-core
I will rename _file_duplicate with _file_equal, and do
the necessary ajustment to avoid confusions.
Thank you for pointing out
V2 incoming
//Hongxu
On 02/18/2014 11:36 PM, Laurentiu Palcu wrote:
> On Tue, Feb 18, 2014 at 05:42:28PM +0800, Hongxu Jia wrote:
>> The _multilib_sanity_test installs multilib packages in a temporary
>> root fs, and compare with the current image to figure out duplicated
>> file that comes from different packages.
>>
>> While incremental image generation enabled and the previous image
>> existed, there was a Multilib check error:
>> ...
>> ERROR: Multilib check error: duplicate files tmp/work/qemux86_64-poky-
>> linux/core-image-minimal/1.0-r0/multilib/lib32/lib/libc.so.6 tmp/work/
>> qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/lib/libc.so.6
>> is not the same
>> ...
>>
>> The reason is the file in current image has been prelinked in previous
>> image generation and the file in a temporary root fs is not prelinked,
>> even though the files come from the same package, the Multilib check
>> considers they are different.
>>
>> [YOCTO #1894]
>> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
>> ---
>> meta/lib/oe/rootfs.py | 56 +++++++++++++++++++++++++++++++++++++++++++++++----
>> 1 file changed, 52 insertions(+), 4 deletions(-)
>>
>> diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
>> index 3d7adf9..5054d1e 100644
>> --- a/meta/lib/oe/rootfs.py
>> +++ b/meta/lib/oe/rootfs.py
>> @@ -3,6 +3,8 @@ from oe.utils import execute_pre_post_process
>> from oe.utils import contains as base_contains
>> from oe.package_manager import *
>> from oe.manifest import *
>> +import oe.path
>> +import filecmp
>> import shutil
>> import os
>> import subprocess
>> @@ -458,13 +460,61 @@ class OpkgRootfs(Rootfs):
>>
>> bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True)
>>
>> + def _prelink_file(self, root_dir, filename):
>> + bb.note('prelink %s in %s' % (filename, root_dir))
>> + prelink_cfg = oe.path.join(root_dir,
>> + self.d.expand('${sysconfdir}/prelink.conf'))
>> + if not os.path.exists(prelink_cfg):
>> + shutil.copy(self.d.expand('${STAGING_DIR_NATIVE}${sysconfdir_native}/prelink.conf'),
>> + prelink_cfg)
>> +
>> + cmd_prelink = self.d.expand('${STAGING_DIR_NATIVE}${sbindir_native}/prelink')
>> + self._exec_shell_cmd([cmd_prelink,
>> + '--root',
>> + root_dir,
>> + '-amR',
>> + '-N',
>> + '-c',
>> + self.d.expand('${sysconfdir}/prelink.conf')])
>> +
>> + '''
>> + Compare two files with the same key twice to see if they came
>> + from the same package. If they are not same, they are duplicated
>> + and come from different packages.
> I'm kind of confused by this comment. Doesn't same = duplicate? There
> might be a small confusion of terms here because the function's behavior
> is not as the name implies.
>
>> + 1st: Comapre them directly;
>> + 2nd: While incremental image creation is enabled, one of the
>> + files could be probaly prelinked in the previous image
>> + creation and the file has been changed, so we need to
>> + prelink the other one and compare them.
>> + '''
>> + def _file_duplicate(self, key, f1, f2):
> Shouldn't be better to rename this function to something else in order
> to avoid confusion? Let's say: _files_are_equal() ?
>
>> +
>> + if not os.path.exists(f1) or not os.path.exists(f2):
>> + return False
>> +
>> + # f1 is the same with f2, both of them were not prelinked
>> + if filecmp.cmp(f1, f2):
>> + return False
> filecmp.cmp() returns True if files are equal. Hence the confusion:
> _file_duplicate() returns False here, if files are equal... I think the logic
> is a little bit the other way around! :)
>
>> +
>> + if self.image_rootfs not in f1:
>> + self._prelink_file(f1.replace(key, ''), f1)
>> +
>> + if self.image_rootfs not in f2:
>> + self._prelink_file(f2.replace(key, ''), f2)
>> +
>> + # f1 is the same with f2, both of them were prelinked
>> + if filecmp.cmp(f1, f2):
>> + return False
>> +
>> + # Duplicated
>> + return True
> here the return value matches the comment and the function name! :)
>> +
>> """
>> This function was reused from the old implementation.
>> See commit: "image.bbclass: Added variables for multilib support." by
>> Lianhao Lu.
>> """
>> def _multilib_sanity_test(self, dirs):
>> - import filecmp
>>
>> allow_replace = self.d.getVar("MULTILIBRE_ALLOW_REP", True)
>> if allow_replace is None:
>> @@ -486,9 +536,7 @@ class OpkgRootfs(Rootfs):
>> if allow_rep.match(key):
>> valid = True
>> else:
>> - if os.path.exists(files[key]) and \
>> - os.path.exists(item) and \
>> - not filecmp.cmp(files[key], item):
>> + if self._file_duplicate(key, files[key], item):
>> valid = False
>> bb.fatal("%s duplicate files %s %s is not the same\n" %
>> (error_prompt, item, files[key]))
>> --
>> 1.8.1.2
>>
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH 5/5] rootfs.py: tweak _multilib_sanity_test for ipk incremental image generation
2014-02-21 6:38 [PATCH V3 0/5] manifest.py/package_manager.py/rootfs.py: support " Hongxu Jia
@ 2014-02-21 6:38 ` Hongxu Jia
0 siblings, 0 replies; 13+ messages in thread
From: Hongxu Jia @ 2014-02-21 6:38 UTC (permalink / raw)
To: openembedded-core; +Cc: saul.wold
The _multilib_sanity_test installs multilib packages in a temporary
root fs, and compare with the current image to figure out duplicated
files that come from different packages.
While incremental image generation enabled and the previous image
was existed, there was an Multilib check error:
...
ERROR: Multilib check error: duplicate files tmp/work/qemux86_64-poky-
linux/core-image-minimal/1.0-r0/multilib/lib32/lib/libc.so.6 tmp/work/
qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/lib/libc.so.6
is not the same
...
The reason is the file in the existing image has been prelinked by
previous image generation and the file in a temporary root fs is not
prelinked, even though both of them came from the same package, the
Multilib check failed.
[YOCTO #1894]
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
meta/lib/oe/rootfs.py | 51 +++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 49 insertions(+), 2 deletions(-)
diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
index 78e9627..2b2915e 100644
--- a/meta/lib/oe/rootfs.py
+++ b/meta/lib/oe/rootfs.py
@@ -3,6 +3,8 @@ from oe.utils import execute_pre_post_process
from oe.utils import contains as base_contains
from oe.package_manager import *
from oe.manifest import *
+import oe.path
+import filecmp
import shutil
import os
import subprocess
@@ -458,13 +460,58 @@ class OpkgRootfs(Rootfs):
bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True)
+ def _prelink_file(self, root_dir, filename):
+ bb.note('prelink %s in %s' % (filename, root_dir))
+ prelink_cfg = oe.path.join(root_dir,
+ self.d.expand('${sysconfdir}/prelink.conf'))
+ if not os.path.exists(prelink_cfg):
+ shutil.copy(self.d.expand('${STAGING_DIR_NATIVE}${sysconfdir_native}/prelink.conf'),
+ prelink_cfg)
+
+ cmd_prelink = self.d.expand('${STAGING_DIR_NATIVE}${sbindir_native}/prelink')
+ self._exec_shell_cmd([cmd_prelink,
+ '--root',
+ root_dir,
+ '-amR',
+ '-N',
+ '-c',
+ self.d.expand('${sysconfdir}/prelink.conf')])
+
+ '''
+ Compare two files with the same key twice to see if they are equal.
+ If they are not equal, it means they are duplicated and come from
+ different packages.
+ 1st: Comapre them directly;
+ 2nd: While incremental image creation is enabled, one of the
+ files could be probaly prelinked in the previous image
+ creation and the file has been changed, so we need to
+ prelink the other one and compare them.
+ '''
+ def _file_equal(self, key, f1, f2):
+
+ # Both of them are not prelinked
+ if filecmp.cmp(f1, f2):
+ return True
+
+ if self.image_rootfs not in f1:
+ self._prelink_file(f1.replace(key, ''), f1)
+
+ if self.image_rootfs not in f2:
+ self._prelink_file(f2.replace(key, ''), f2)
+
+ # Both of them are prelinked
+ if filecmp.cmp(f1, f2):
+ return True
+
+ # Not equal
+ return False
+
"""
This function was reused from the old implementation.
See commit: "image.bbclass: Added variables for multilib support." by
Lianhao Lu.
"""
def _multilib_sanity_test(self, dirs):
- import filecmp
allow_replace = self.d.getVar("MULTILIBRE_ALLOW_REP", True)
if allow_replace is None:
@@ -488,7 +535,7 @@ class OpkgRootfs(Rootfs):
else:
if os.path.exists(files[key]) and \
os.path.exists(item) and \
- not filecmp.cmp(files[key], item):
+ not self._file_equal(key, files[key], item):
valid = False
bb.fatal("%s duplicate files %s %s is not the same\n" %
(error_prompt, item, files[key]))
--
1.8.1.2
^ permalink raw reply related [flat|nested] 13+ messages in thread
end of thread, other threads:[~2014-02-21 6:38 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-02-18 9:42 [PATCH 0/5] package_manager.py/ootfs.py: support ipk incremental image generation Hongxu Jia
2014-02-18 9:42 ` [PATCH 1/5] package_manager.py: " Hongxu Jia
2014-02-18 14:38 ` Laurentiu Palcu
2014-02-19 9:39 ` Hongxu Jia
2014-02-18 9:42 ` [PATCH 2/5] package_manager.py: tweak handle_bad_recommendations for " Hongxu Jia
2014-02-18 9:42 ` [PATCH 3/5] rootfs.py: support " Hongxu Jia
2014-02-18 9:42 ` [PATCH 4/5] rootfs.py: fix BAD_RECOMMENDATIONS for " Hongxu Jia
2014-02-18 15:03 ` Laurentiu Palcu
2014-02-19 9:44 ` Hongxu Jia
2014-02-18 9:42 ` [PATCH 5/5] rootfs.py: tweak _multilib_sanity_test " Hongxu Jia
2014-02-18 15:36 ` Laurentiu Palcu
2014-02-19 9:51 ` Hongxu Jia
2014-02-21 6:38 [PATCH V3 0/5] manifest.py/package_manager.py/rootfs.py: support " Hongxu Jia
2014-02-21 6:38 ` [PATCH 5/5] rootfs.py: tweak _multilib_sanity_test for " Hongxu Jia
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.