* [RFC][PATCH 0/7] image creation improvements
@ 2016-05-06 14:58 Ed Bartosh
2016-05-06 14:58 ` [RFC][PATCH 1/7] image_types: add support for zip compression Ed Bartosh
` (7 more replies)
0 siblings, 8 replies; 9+ messages in thread
From: Ed Bartosh @ 2016-05-06 14:58 UTC (permalink / raw)
To: openembedded-core
Hi,
This patchset contains various improvements for the image creation functionality
made by Patchick Ohly during his work on bug #9076:
- added support for zip compression
- fixed dependency calculation for conversion chaining
- converted vmdk/vdi/qcow2 image types into conversion commands
- renamed COMPRESSION variables to CONVERSION as the term "compression"
is no longer accurate
- prioritized specialized image creation methods over using conversion chaining
The changes look reasonable to me. However, it would be good to hear other
people opinions as some changes are quite complex.
The following changes since commit 912e372bd3097b1475df2689f9aec88dd3198976:
scripts/lib/bsp/kernel.py: force patching when branch is machine branch is re-use (2016-05-06 10:32:24 +0100)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib ed/oe-core/image-type-9076
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/oe-core/image-type-9076
Patrick Ohly (7):
image_types: add support for zip compression
image.bbclass: support duplicate compression types
image.bbclass: fix dependency calculation when using conversion
chaining
image.bbclass: additional output in create_symlinks
image creation: support converting masked types
image.bbclass: rename COMPRESS(ION) to CONVERSION
image.bbclass: prefer specialized image creation methods over chaining
meta/classes/image-live.bbclass | 2 +-
meta/classes/image-vm.bbclass | 31 +---
meta/classes/image.bbclass | 186 +++++++++++++++------
meta/classes/image_types.bbclass | 104 ++++++------
meta/classes/image_types_uboot.bbclass | 18 +-
meta/conf/documentation.conf | 1 -
.../images/build-appliance-image_14.0.0.bb | 6 +-
7 files changed, 204 insertions(+), 144 deletions(-)
--
Regards,
Ed
^ permalink raw reply [flat|nested] 9+ messages in thread
* [RFC][PATCH 1/7] image_types: add support for zip compression
2016-05-06 14:58 [RFC][PATCH 0/7] image creation improvements Ed Bartosh
@ 2016-05-06 14:58 ` Ed Bartosh
2016-05-06 14:58 ` [RFC][PATCH 2/7] image.bbclass: support duplicate compression types Ed Bartosh
` (6 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Ed Bartosh @ 2016-05-06 14:58 UTC (permalink / raw)
To: openembedded-core
From: Patrick Ohly <patrick.ohly@intel.com>
Support for the other compression format is not always readily
available on all OSes. Using zip instead of, say, xz is less
efficient, but perhaps more user-friendly for users on such OSes.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
meta/classes/image_types.bbclass | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 53af7ca..d13bdf5 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -42,6 +42,8 @@ XZ_COMPRESSION_LEVEL ?= "-e -6"
XZ_INTEGRITY_CHECK ?= "crc32"
XZ_THREADS ?= "-T 0"
+ZIP_COMPRESSION_LEVEL ?= "-9"
+
JFFS2_SUM_EXTRA_ARGS ?= ""
IMAGE_CMD_jffs2 = "mkfs.jffs2 --root=${IMAGE_ROOTFS} --faketime --output=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.jffs2 ${EXTRA_IMAGECMD}"
@@ -262,12 +264,13 @@ IMAGE_TYPES = " \
wic wic.gz wic.bz2 wic.lzma \
"
-COMPRESSIONTYPES = "gz bz2 lzma xz lz4 sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum"
+COMPRESSIONTYPES = "gz bz2 lzma xz lz4 zip sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum"
COMPRESS_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
COMPRESS_CMD_gz = "gzip -f -9 -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
COMPRESS_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
COMPRESS_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS} --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"
COMPRESS_CMD_lz4 = "lz4c -9 -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
+COMPRESS_CMD_zip = "zip ${ZIP_COMPRESSION_LEVEL} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zip ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
COMPRESS_CMD_sum = "sumtool -i ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} -o ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sum ${JFFS2_SUM_EXTRA_ARGS}"
COMPRESS_CMD_md5sum = "md5sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.md5sum"
COMPRESS_CMD_sha1sum = "sha1sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha1sum"
@@ -280,6 +283,7 @@ COMPRESS_DEPENDS_gz = ""
COMPRESS_DEPENDS_bz2 = "pbzip2-native"
COMPRESS_DEPENDS_xz = "xz-native"
COMPRESS_DEPENDS_lz4 = "lz4-native"
+COMPRESS_DEPENDS_zip = "zip-native"
COMPRESS_DEPENDS_sum = "mtd-utils-native"
RUNNABLE_IMAGE_TYPES ?= "ext2 ext3 ext4"
--
2.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [RFC][PATCH 2/7] image.bbclass: support duplicate compression types
2016-05-06 14:58 [RFC][PATCH 0/7] image creation improvements Ed Bartosh
2016-05-06 14:58 ` [RFC][PATCH 1/7] image_types: add support for zip compression Ed Bartosh
@ 2016-05-06 14:58 ` Ed Bartosh
2016-05-06 14:58 ` [RFC][PATCH 3/7] image.bbclass: fix dependency calculation when using conversion chaining Ed Bartosh
` (5 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Ed Bartosh @ 2016-05-06 14:58 UTC (permalink / raw)
To: openembedded-core
From: Patrick Ohly <patrick.ohly@intel.com>
When a derived distro adds a certain type, say zip, to
COMPRESSIONTYPES and later OE-core does the same, we end up with the
type being listed twice, and that would have undesired effects
(commands generated twice).
So to support such loosely coupled extension, we de-duplicated the
list of types first.
Alternatively, such a situation could also be treated as error. But that
seems unnecessary because typically commands for the same type will also
do the same thing.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
meta/classes/image.bbclass | 10 +++++++++-
meta/classes/image_types.bbclass | 2 +-
2 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 4542e95..56a4ddd 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -297,7 +297,15 @@ python setup_debugfs () {
python () {
vardeps = set()
- ctypes = d.getVar('COMPRESSIONTYPES', True).split()
+ # We allow COMPRESSIONTYPES to have duplicates. That avoids breaking
+ # derived distros when OE-core or some other layer independently adds
+ # the same type. There is still only one command for each type, but
+ # presumably the commands will do the same when the type is the same,
+ # even when added in different places.
+ #
+ # Without de-duplication, gen_conversion_cmds() below
+ # would create the same compression command multiple times.
+ ctypes = set(d.getVar('COMPRESSIONTYPES', True).split())
old_overrides = d.getVar('OVERRIDES', 0)
def _image_base_type(type):
diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index d13bdf5..40a4a1b 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -16,7 +16,7 @@ def imagetypes_getdepends(d):
deps.append(i)
deps = []
- ctypes = d.getVar('COMPRESSIONTYPES', True).split()
+ ctypes = set(d.getVar('COMPRESSIONTYPES', True).split())
fstypes = set((d.getVar('IMAGE_FSTYPES', True) or "").split())
fstypes |= set((d.getVar('IMAGE_FSTYPES_DEBUGFS', True) or "").split())
for type in fstypes:
--
2.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [RFC][PATCH 3/7] image.bbclass: fix dependency calculation when using conversion chaining
2016-05-06 14:58 [RFC][PATCH 0/7] image creation improvements Ed Bartosh
2016-05-06 14:58 ` [RFC][PATCH 1/7] image_types: add support for zip compression Ed Bartosh
2016-05-06 14:58 ` [RFC][PATCH 2/7] image.bbclass: support duplicate compression types Ed Bartosh
@ 2016-05-06 14:58 ` Ed Bartosh
2016-05-06 14:58 ` [RFC][PATCH 4/7] image.bbclass: additional output in create_symlinks Ed Bartosh
` (4 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Ed Bartosh @ 2016-05-06 14:58 UTC (permalink / raw)
To: openembedded-core
From: Patrick Ohly <patrick.ohly@intel.com>
When using conversion chaining (for example example: .dsk.vdi.xz),
the imagetypes_getdepends() did not properly detect all compression
commands (thus skipping the corresponding COMPRESS_DEPENDS) and
the base type, leading to missing dependencies of the image's do_rootfs
task.
This is not a big problem in practice because in those cases where
conversion chaining is useful (as in the example above), the missing
dependency is qemu-native, which typically gets built anyway.
The solution moves imagetypes_getdepends() to image.bbclass because that
is the only place where it gets used and rewrites it so that it and
generating the image commands use a new common utility method, image_split_type().
imagetypes_getdepends() had hard-coded special treatment for certain
image types. This gets replaced with setting the normal IMAGE_DEPENDS
variables for these types.
[YOCTO #9076]
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
meta/classes/image.bbclass | 61 +++++++++++++++++++++++++++++-----------
meta/classes/image_types.bbclass | 36 +++++-------------------
2 files changed, 52 insertions(+), 45 deletions(-)
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 56a4ddd..7b79e3a 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -142,8 +142,50 @@ inherit ${IMAGE_TYPE_live}
IMAGE_TYPE_vm = '${@bb.utils.contains_any("IMAGE_FSTYPES", ["vmdk", "vdi", "qcow2", "hdddirect"], "image-vm", "", d)}'
inherit ${IMAGE_TYPE_vm}
+def image_split_type(type, allctypes):
+ '''Returns (basetype, set of compression types in use).'''
+ basetype = type
+ compressiontypes = set()
+ for ctype in allctypes:
+ if type.endswith("." + ctype):
+ basetype = type[:-len("." + ctype)]
+ compressiontypes.add(ctype)
+ break
+
+ if basetype != type:
+ # New base type itself might be generated by a conversion command.
+ basetype, newctypes = image_split_type(basetype, allctypes)
+ compressiontypes.update(newctypes)
+
+ return (basetype, compressiontypes)
+
+
+def image_getdepends(d):
+ def adddep(depstr, deps):
+ # It is not an error if the dependency was not set,
+ # simply do nothing in that case.
+ for i in (depstr or "").split():
+ if i not in deps:
+ deps.append(i)
+
+ deps = []
+ ctypes = set(d.getVar('COMPRESSIONTYPES', True).split())
+ for type in (d.getVar('IMAGE_FSTYPES', True) or "").split():
+ basetype, compressiontypes = image_split_type(type, ctypes)
+ for ctype in compressiontypes:
+ adddep(d.getVar("COMPRESS_DEPENDS_%s" % ctype, True), deps)
+ for typedepends in (d.getVar("IMAGE_TYPEDEP_%s" % basetype, True) or "").split():
+ adddep(d.getVar('IMAGE_DEPENDS_%s' % typedepends, True) , deps)
+ adddep(d.getVar('IMAGE_DEPENDS_%s' % basetype, True) , deps)
+
+ depstr = ""
+ for dep in deps:
+ depstr += " " + dep + ":do_populate_sysroot"
+ return depstr
+
+
python () {
- deps = " " + imagetypes_getdepends(d)
+ deps = " " + image_getdepends(d)
d.appendVarFlag('do_rootfs', 'depends', deps)
deps = ""
@@ -308,19 +350,6 @@ python () {
ctypes = set(d.getVar('COMPRESSIONTYPES', True).split())
old_overrides = d.getVar('OVERRIDES', 0)
- def _image_base_type(type):
- basetype = type
- for ctype in ctypes:
- if type.endswith("." + ctype):
- basetype = type[:-len("." + ctype)]
- break
-
- if basetype != type:
- # New base type itself might be generated by a conversion command.
- basetype = _image_base_type(basetype)
-
- return basetype
-
basetypes = {}
alltypes = d.getVar('IMAGE_FSTYPES', True).split()
typedeps = {}
@@ -331,7 +360,7 @@ python () {
alltypes.append("debugfs_" + t)
def _add_type(t):
- baset = _image_base_type(t)
+ baset = image_split_type(t, ctypes)[0]
input_t = t
if baset not in basetypes:
basetypes[baset]= []
@@ -350,7 +379,7 @@ python () {
if dep not in alltypes:
alltypes.append(dep)
_add_type(dep)
- basedep = _image_base_type(dep)
+ basedep = image_split_type(dep, ctypes)[0]
typedeps[baset].add(basedep)
if baset != input_t:
diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 40a4a1b..e2b7ebb 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -9,35 +9,6 @@ IMAGE_NAME_SUFFIX ??= ".rootfs"
# set this value to 2048 (2MiB alignment).
IMAGE_ROOTFS_ALIGNMENT ?= "1"
-def imagetypes_getdepends(d):
- def adddep(depstr, deps):
- for i in (depstr or "").split():
- if i not in deps:
- deps.append(i)
-
- deps = []
- ctypes = set(d.getVar('COMPRESSIONTYPES', True).split())
- fstypes = set((d.getVar('IMAGE_FSTYPES', True) or "").split())
- fstypes |= set((d.getVar('IMAGE_FSTYPES_DEBUGFS', True) or "").split())
- for type in fstypes:
- if type in ["vmdk", "vdi", "qcow2", "hdddirect", "live", "iso", "hddimg"]:
- type = "ext4"
- basetype = type
- for ctype in ctypes:
- if type.endswith("." + ctype):
- basetype = type[:-len("." + ctype)]
- adddep(d.getVar("COMPRESS_DEPENDS_%s" % ctype, True), deps)
- break
- for typedepends in (d.getVar("IMAGE_TYPEDEP_%s" % basetype, True) or "").split():
- adddep(d.getVar('IMAGE_DEPENDS_%s' % typedepends, True) , deps)
- adddep(d.getVar('IMAGE_DEPENDS_%s' % basetype, True) , deps)
-
- depstr = ""
- for dep in deps:
- depstr += " " + dep + ":do_populate_sysroot"
- return depstr
-
-
XZ_COMPRESSION_LEVEL ?= "-e -6"
XZ_INTEGRITY_CHECK ?= "crc32"
XZ_THREADS ?= "-T 0"
@@ -242,6 +213,13 @@ IMAGE_DEPENDS_ubifs = "mtd-utils-native"
IMAGE_DEPENDS_multiubi = "mtd-utils-native"
IMAGE_DEPENDS_wic = "parted-native"
+# Same dependencies as in ext4. image_getdepends() shouldn't
+# have to hard-code this, so just define it normally in
+# variables.
+IMAGE_DEPENDS_live = "${IMAGE_DEPENDS_ext4}"
+IMAGE_DEPENDS_iso = "${IMAGE_DEPENDS_ext4}"
+IMAGE_DEPENDS_hddimg = "${IMAGE_DEPENDS_ext4}"
+
# This variable is available to request which values are suitable for IMAGE_FSTYPES
IMAGE_TYPES = " \
jffs2 jffs2.sum \
--
2.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [RFC][PATCH 4/7] image.bbclass: additional output in create_symlinks
2016-05-06 14:58 [RFC][PATCH 0/7] image creation improvements Ed Bartosh
` (2 preceding siblings ...)
2016-05-06 14:58 ` [RFC][PATCH 3/7] image.bbclass: fix dependency calculation when using conversion chaining Ed Bartosh
@ 2016-05-06 14:58 ` Ed Bartosh
2016-05-06 14:58 ` [RFC][PATCH 5/7] image creation: support converting masked types Ed Bartosh
` (3 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Ed Bartosh @ 2016-05-06 14:58 UTC (permalink / raw)
To: openembedded-core
From: Patrick Ohly <patrick.ohly@intel.com>
When a symlink does not get created, it is useful for debugging to log
what would have been created and why it was skipped.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
meta/classes/image.bbclass | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 7b79e3a..37fb33c 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -552,9 +552,9 @@ python create_symlinks() {
if not link_name:
return
for type in subimages:
- if os.path.exists(img_name + imgsuffix + type):
- dst = deploy_dir + "/" + link_name + "." + type
- src = img_name + imgsuffix + type
+ dst = deploy_dir + "/" + link_name + "." + type
+ src = img_name + imgsuffix + type
+ if os.path.exists(src):
bb.note("Creating symlink: %s -> %s" % (dst, src))
if os.path.islink(dst):
if d.getVar('RM_OLD_IMAGE', True) == "1" and \
@@ -562,6 +562,8 @@ python create_symlinks() {
os.remove(os.path.realpath(dst))
os.remove(dst)
os.symlink(src, dst)
+ else:
+ bb.note("Skipping symlink, source does not exist: %s -> %s" % (dst, src))
}
MULTILIBRE_ALLOW_REP =. "${base_bindir}|${base_sbindir}|${bindir}|${sbindir}|${libexecdir}|${sysconfdir}|${nonarch_base_libdir}/udev|/lib/modules/[^/]*/modules.*|"
--
2.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [RFC][PATCH 5/7] image creation: support converting masked types
2016-05-06 14:58 [RFC][PATCH 0/7] image creation improvements Ed Bartosh
` (3 preceding siblings ...)
2016-05-06 14:58 ` [RFC][PATCH 4/7] image.bbclass: additional output in create_symlinks Ed Bartosh
@ 2016-05-06 14:58 ` Ed Bartosh
2016-05-06 14:58 ` [RFC][PATCH 6/7] image.bbclass: rename COMPRESS(ION) to CONVERSION Ed Bartosh
` (2 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Ed Bartosh @ 2016-05-06 14:58 UTC (permalink / raw)
To: openembedded-core
From: Patrick Ohly <patrick.ohly@intel.com>
Conversion to vmdk/vdi/qcow2 is also useful for other base images
types, not just for .hdddirect. This can be achieved by definining
them as conversion commands and relying on the conversion chaining
to convert arbitrary base images.
For this to work when the base image gets created by a masked image
type, the additional conversion commands now get executed in a
do_image_complete prefunc.
With all of that in place it becomes possible to remove the special
purpose code for vmdk/vdi/qcow2 types from image-vm.bbclass and
several other classes. This has (intentional!) implications on the
valid IMAGE_FSTYPES and the file suffices: now
"hdddirect.vmdk/vdi/qcow2" must be used as IMAGE_FSTYPES to select the
former special-case types "vmdk/vdi/qcow2", and the image files and
links will also have the extra .hdddirect suffix.
This is intentional because it makes it makes it possible to
distinguish between virtual machine images created from .hdddirect and
those created from other base images.
The new support for virtual machine images can also be combined with
compression, thus making it possible to create image files for
publication in compressed format, for example with:
IMAGE_FSTYPES = "hdddirect.vdi.xz"
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
meta/classes/image-live.bbclass | 2 +-
meta/classes/image-vm.bbclass | 31 +--------
meta/classes/image.bbclass | 73 +++++++++++++++-------
meta/classes/image_types.bbclass | 20 ++++--
meta/conf/documentation.conf | 1 -
.../images/build-appliance-image_14.0.0.bb | 6 +-
6 files changed, 74 insertions(+), 59 deletions(-)
diff --git a/meta/classes/image-live.bbclass b/meta/classes/image-live.bbclass
index c8a8610..dfc548d 100644
--- a/meta/classes/image-live.bbclass
+++ b/meta/classes/image-live.bbclass
@@ -43,7 +43,7 @@ ROOT_LIVE ?= "root=/dev/ram0"
INITRD_IMAGE_LIVE ?= "core-image-minimal-initramfs"
INITRD_LIVE ?= "${DEPLOY_DIR_IMAGE}/${INITRD_IMAGE_LIVE}-${MACHINE}.cpio.gz"
-ROOTFS ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.ext4"
+ROOTFS ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.ext4"
IMAGE_TYPEDEP_live = "ext4"
IMAGE_TYPEDEP_iso = "ext4"
diff --git a/meta/classes/image-vm.bbclass b/meta/classes/image-vm.bbclass
index 47f7326..fbe2924 100644
--- a/meta/classes/image-vm.bbclass
+++ b/meta/classes/image-vm.bbclass
@@ -26,13 +26,10 @@ do_bootdirectdisk[depends] += "dosfstools-native:do_populate_sysroot \
${PN}:do_image_ext4 \
"
-IMAGE_TYPEDEP_vmdk = "ext4"
-IMAGE_TYPEDEP_vdi = "ext4"
-IMAGE_TYPEDEP_qcow2 = "ext4"
IMAGE_TYPEDEP_hdddirect = "ext4"
-IMAGE_TYPES_MASKED += "vmdk vdi qcow2 hdddirect"
+IMAGE_TYPES_MASKED += "hdddirect"
-ROOTFS ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.ext4"
+ROOTFS ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.ext4"
# Used by bootloader
LABELS_VM ?= "boot"
@@ -149,27 +146,5 @@ run_qemu_img (){
qemu-img convert -O $type ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.hdddirect ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.$type
ln -sf ${IMAGE_NAME}.$type ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.$type
}
-create_vmdk_image () {
- run_qemu_img vmdk
-}
-
-create_vdi_image () {
- run_qemu_img vdi
-}
-
-create_qcow2_image () {
- run_qemu_img qcow2
-}
-
-python do_vmimg() {
- if 'vmdk' in d.getVar('IMAGE_FSTYPES', True):
- bb.build.exec_func('create_vmdk_image', d)
- if 'vdi' in d.getVar('IMAGE_FSTYPES', True):
- bb.build.exec_func('create_vdi_image', d)
- if 'qcow2' in d.getVar('IMAGE_FSTYPES', True):
- bb.build.exec_func('create_qcow2_image', d)
-}
-addtask bootdirectdisk before do_vmimg
-addtask vmimg after do_bootdirectdisk before do_image_complete
-do_vmimg[depends] += "qemu-native:do_populate_sysroot"
+addtask bootdirectdisk before do_image_complete
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 37fb33c..958c0e9 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -127,6 +127,13 @@ do_rootfs[vardeps] += "${@rootfs_variables(d)}"
do_build[depends] += "virtual/kernel:do_deploy"
+def image_type_active(type, d):
+ '''True if any entry in IMAGE_FSTYPES is type or depends on it.'''
+ for entry in d.getVar("IMAGE_FSTYPES", True).split():
+ if entry == type or entry.startswith(type + "."):
+ return True
+ return False
+
def build_live(d):
if bb.utils.contains("IMAGE_FSTYPES", "live", "live", "0", d) == "0": # live is not set but hob might set iso or hddimg
d.setVar('NOISO', bb.utils.contains('IMAGE_FSTYPES', "iso", "0", "1", d))
@@ -139,7 +146,7 @@ def build_live(d):
IMAGE_TYPE_live = "${@build_live(d)}"
inherit ${IMAGE_TYPE_live}
-IMAGE_TYPE_vm = '${@bb.utils.contains_any("IMAGE_FSTYPES", ["vmdk", "vdi", "qcow2", "hdddirect"], "image-vm", "", d)}'
+IMAGE_TYPE_vm = '${@ "image-vm" if image_type_active("hdddirect", d) else ""}'
inherit ${IMAGE_TYPE_vm}
def image_split_type(type, allctypes):
@@ -397,9 +404,7 @@ python () {
cmds = []
subimages = []
realt = t
-
- if t in maskedtypes:
- continue
+ ismasked = t in maskedtypes
localdata = bb.data.createCopy(d)
debug = ""
@@ -417,12 +422,14 @@ python () {
localdata.delVar('DATETIME')
localdata.delVar('TMPDIR')
- image_cmd = localdata.getVar("IMAGE_CMD", True)
- vardeps.add('IMAGE_CMD_' + realt)
- if image_cmd:
- cmds.append("\t" + image_cmd)
- else:
- bb.fatal("No IMAGE_CMD defined for IMAGE_FSTYPES entry '%s' - possibly invalid type name or missing support class" % t)
+ if not ismasked:
+ image_cmd = localdata.getVar("IMAGE_CMD", True)
+ vardeps.add('IMAGE_CMD_' + realt)
+ if image_cmd:
+ cmds.append("\t" + image_cmd)
+ else:
+ bb.fatal("No IMAGE_CMD defined for IMAGE_FSTYPES entry '%s' - possibly invalid type name or missing support class" % t)
+
cmds.append(localdata.expand("\tcd ${DEPLOY_DIR_IMAGE}"))
# Since a copy of IMAGE_CMD_xxx will be inlined within do_image_xxx,
@@ -470,17 +477,41 @@ python () {
t = t.replace("-", "_").replace(".", "_")
- d.setVar('do_image_%s' % t, '\n'.join(cmds))
- d.setVarFlag('do_image_%s' % t, 'func', '1')
- d.setVarFlag('do_image_%s' % t, 'fakeroot', '1')
- d.setVarFlag('do_image_%s' % t, 'prefuncs', debug + 'set_image_size')
- d.setVarFlag('do_image_%s' % t, 'postfuncs', 'create_symlinks')
- d.setVarFlag('do_image_%s' % t, 'subimages', ' '.join(subimages))
- d.appendVarFlag('do_image_%s' % t, 'vardeps', ' '.join(vardeps))
- d.appendVarFlag('do_image_%s' % t, 'vardepsexclude', 'DATETIME')
-
- bb.debug(2, "Adding type %s before %s, after %s" % (t, 'do_image_complete', after))
- bb.build.addtask('do_image_%s' % t, 'do_image_complete', after, d)
+ if ismasked:
+ # If the type is masked, then some unknown task (for example,
+ # do_bootdirectdisk in boot-directdisk.bbclass for IMAGE_FSTYPES hdddirect)
+ # will create the actual base image. All we know is that the files will
+ # be there right before do_image_complete. So in that case we put the
+ # conversion commands into a do_image_complete prefunc. The 'after'
+ # dependencies can be ignored because we are guaranteed to run after
+ # all of them, and the conversion dependencies are dealt with
+ # by making do_rootfs depend on them.
+ d.setVar('image_%s_conversion' % t, '\n'.join(cmds))
+ d.setVarFlag('image_%s_conversion' % t, 'func', '1')
+ d.setVarFlag('image_%s_conversion' % t, 'fakeroot', '1')
+ d.appendVarFlag('image_%s_conversion' % t, 'vardeps', ' '.join(vardeps))
+ d.appendVarFlag('image_%s_conversion' % t, 'vardepsexclude', 'DATETIME')
+ prefuncs = set((d.getVarFlag('do_image_complete', 'prefuncs', True) or '').split())
+ prefuncs.add('image_%s_conversion' % t)
+ # create_symlinks must run after the commands which create the real files
+ # because create_symlinks checks for them.
+ prefuncs.discard('create_symlinks')
+ d.setVarFlag('do_image_complete', 'prefuncs', ' '.join(sorted(prefuncs)) + ' create_symlinks')
+ d.appendVarFlag('do_image_complete', 'subimages', ' ' + ' '.join(subimages))
+ else:
+ # If not masked, we generate a new task which executes the image
+ # creation and the conversion commands.
+ d.setVar('do_image_%s' % t, '\n'.join(cmds))
+ d.setVarFlag('do_image_%s' % t, 'func', '1')
+ d.setVarFlag('do_image_%s' % t, 'fakeroot', '1')
+ d.setVarFlag('do_image_%s' % t, 'prefuncs', debug + 'set_image_size')
+ d.setVarFlag('do_image_%s' % t, 'postfuncs', 'create_symlinks')
+ d.setVarFlag('do_image_%s' % t, 'subimages', ' '.join(subimages))
+ d.appendVarFlag('do_image_%s' % t, 'vardeps', ' '.join(vardeps))
+ d.appendVarFlag('do_image_%s' % t, 'vardepsexclude', 'DATETIME')
+
+ bb.debug(2, "Adding type %s before %s, after %s" % (t, 'do_image_complete', after))
+ bb.build.addtask('do_image_%s' % t, 'do_image_complete', after, d)
}
#
diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index e2b7ebb..42db267 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -203,6 +203,7 @@ IMAGE_DEPENDS_cramfs = "util-linux-native"
IMAGE_DEPENDS_ext2 = "e2fsprogs-native"
IMAGE_DEPENDS_ext3 = "e2fsprogs-native"
IMAGE_DEPENDS_ext4 = "e2fsprogs-native"
+IMAGE_DEPENDS_hdddirect = "${IMAGE_DEPENDS_ext4}"
IMAGE_DEPENDS_btrfs = "btrfs-tools-native"
IMAGE_DEPENDS_squashfs = "squashfs-tools-native"
IMAGE_DEPENDS_squashfs-xz = "squashfs-tools-native"
@@ -221,6 +222,9 @@ IMAGE_DEPENDS_iso = "${IMAGE_DEPENDS_ext4}"
IMAGE_DEPENDS_hddimg = "${IMAGE_DEPENDS_ext4}"
# This variable is available to request which values are suitable for IMAGE_FSTYPES
+# TODO: several other variations are now also possible, for example ext4.zip,
+# hdddirect.xz, hdddirect.vdi.xz. Which variations are supposed to be listed here
+# and which not? Generate all possible variations dynamically?
IMAGE_TYPES = " \
jffs2 jffs2.sum \
cramfs \
@@ -234,15 +238,15 @@ IMAGE_TYPES = " \
ubi ubifs multiubi \
tar tar.gz tar.bz2 tar.xz tar.lz4 \
cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 \
- vmdk \
- vdi \
- qcow2 \
+ hdddirect.vmdk \
+ hdddirect.vdi \
+ hdddirect.qcow2 \
hdddirect \
elf \
wic wic.gz wic.bz2 wic.lzma \
"
-COMPRESSIONTYPES = "gz bz2 lzma xz lz4 zip sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum"
+COMPRESSIONTYPES = "gz bz2 lzma xz lz4 zip sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum vmdk vdi qcow2"
COMPRESS_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
COMPRESS_CMD_gz = "gzip -f -9 -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
COMPRESS_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
@@ -250,6 +254,9 @@ COMPRESS_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS} --check=${X
COMPRESS_CMD_lz4 = "lz4c -9 -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
COMPRESS_CMD_zip = "zip ${ZIP_COMPRESSION_LEVEL} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zip ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
COMPRESS_CMD_sum = "sumtool -i ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} -o ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sum ${JFFS2_SUM_EXTRA_ARGS}"
+COMPRESS_CMD_vmdk = "qemu-img convert -O vmdk ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.vmdk"
+COMPRESS_CMD_vdi = "qemu-img convert -O vdi ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.vdi"
+COMPRESS_CMD_qcow2 = "qemu-img convert -O qcow2 ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.qcow2"
COMPRESS_CMD_md5sum = "md5sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.md5sum"
COMPRESS_CMD_sha1sum = "sha1sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha1sum"
COMPRESS_CMD_sha224sum = "sha224sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha224sum"
@@ -263,6 +270,9 @@ COMPRESS_DEPENDS_xz = "xz-native"
COMPRESS_DEPENDS_lz4 = "lz4-native"
COMPRESS_DEPENDS_zip = "zip-native"
COMPRESS_DEPENDS_sum = "mtd-utils-native"
+COMPRESS_DEPENDS_vmdk = "qemu-native"
+COMPRESS_DEPENDS_vdi = "qemu-native"
+COMPRESS_DEPENDS_qcow2 = "qemu-native"
RUNNABLE_IMAGE_TYPES ?= "ext2 ext3 ext4"
RUNNABLE_MACHINE_PATTERNS ?= "qemu"
@@ -273,7 +283,7 @@ DEPLOYABLE_IMAGE_TYPES ?= "hddimg iso"
IMAGE_EXTENSION_live = "hddimg iso"
# The IMAGE_TYPES_MASKED variable will be used to mask out from the IMAGE_FSTYPES,
-# images that will not be built at do_rootfs time: vmdk, vdi, qcow2, hdddirect, hddimg, iso, etc.
+# images that will not be built at do_rootfs time: hdddirect, hddimg, iso, etc.
IMAGE_TYPES_MASKED ?= ""
# The WICVARS variable is used to define list of bitbake variables used in wic code
diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
index 4234d75..7543362 100644
--- a/meta/conf/documentation.conf
+++ b/meta/conf/documentation.conf
@@ -56,7 +56,6 @@ do_testsdk[doc] = "Installs an SDK and performs runtime tests on the tools insta
do_uboot_mkimage[doc] = "Creates a uImage file from the kernel for the U-Boot bootloader"
do_unpack[doc] = "Unpacks the source code into a working directory"
do_validate_branches[doc] = "Ensures that the source/meta branches are on the locations specified by their SRCREV values for a linux-yocto style kernel"
-do_vmimg[doc] = "Creates an image for use with VMware or VirtualBox compatible virtual machine hosts (based on IMAGE_FSTYPES either .vmdk or .vdi)"
# DESCRIPTIONS FOR VARIABLES #
diff --git a/meta/recipes-core/images/build-appliance-image_14.0.0.bb b/meta/recipes-core/images/build-appliance-image_14.0.0.bb
index 569fb3a..48f8282 100644
--- a/meta/recipes-core/images/build-appliance-image_14.0.0.bb
+++ b/meta/recipes-core/images/build-appliance-image_14.0.0.bb
@@ -17,7 +17,7 @@ IMAGE_ROOTFS_EXTRA_SPACE = "41943040"
APPEND += "rootfstype=ext4 quiet"
DEPENDS = "zip-native"
-IMAGE_FSTYPES = "vmdk"
+IMAGE_FSTYPES = "hdddirect.vmdk"
inherit core-image module-base
@@ -106,7 +106,7 @@ create_bundle_files () {
cd ${WORKDIR}
mkdir -p Yocto_Build_Appliance
cp *.vmx* Yocto_Build_Appliance
- ln -sf ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.vmdk Yocto_Build_Appliance/Yocto_Build_Appliance.vmdk
+ ln -sf ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hdddirect.vmdk Yocto_Build_Appliance/Yocto_Build_Appliance.vmdk
zip -r ${DEPLOY_DIR_IMAGE}/Yocto_Build_Appliance-${DATETIME}.zip Yocto_Build_Appliance
ln -sf Yocto_Build_Appliance-${DATETIME}.zip ${DEPLOY_DIR_IMAGE}/Yocto_Build_Appliance.zip
}
@@ -116,5 +116,5 @@ python do_bundle_files() {
bb.build.exec_func('create_bundle_files', d)
}
-addtask bundle_files after do_vmimg before do_build
+addtask bundle_files after do_image_complete before do_build
do_bundle_files[nostamp] = "1"
--
2.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [RFC][PATCH 6/7] image.bbclass: rename COMPRESS(ION) to CONVERSION
2016-05-06 14:58 [RFC][PATCH 0/7] image creation improvements Ed Bartosh
` (4 preceding siblings ...)
2016-05-06 14:58 ` [RFC][PATCH 5/7] image creation: support converting masked types Ed Bartosh
@ 2016-05-06 14:58 ` Ed Bartosh
2016-05-06 14:58 ` [RFC][PATCH 7/7] image.bbclass: prefer specialized image creation methods over chaining Ed Bartosh
2016-05-13 17:06 ` [RFC][PATCH 0/7] image creation improvements Richard Purdie
7 siblings, 0 replies; 9+ messages in thread
From: Ed Bartosh @ 2016-05-06 14:58 UTC (permalink / raw)
To: openembedded-core
From: Patrick Ohly <patrick.ohly@intel.com>
With the enhanced functionality, the term "compression" is no longer
accurate, because the mechanism also gets used for conversion
operations that do not actually compress data.
It is possible to remove this naming problem in a backward-compatible
manner by including COMPRESSIONTYPES in CONVERSIONTYPES and checking for
the old COMPRESS_CMD/DEPENDS as fallbacks.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
meta/classes/image.bbclass | 15 ++++----
meta/classes/image_types.bbclass | 62 +++++++++++++++++++---------------
meta/classes/image_types_uboot.bbclass | 18 +++++-----
3 files changed, 52 insertions(+), 43 deletions(-)
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 958c0e9..5c68b01 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -118,7 +118,7 @@ def rootfs_variables(d):
'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','RM_OLD_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
- 'COMPRESSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED']
+ 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED']
variables.extend(rootfs_command_variables(d))
variables.extend(variable_depends(d))
return " ".join(variables)
@@ -176,11 +176,13 @@ def image_getdepends(d):
deps.append(i)
deps = []
- ctypes = set(d.getVar('COMPRESSIONTYPES', True).split())
+ ctypes = set(d.getVar('CONVERSIONTYPES', True).split())
for type in (d.getVar('IMAGE_FSTYPES', True) or "").split():
basetype, compressiontypes = image_split_type(type, ctypes)
for ctype in compressiontypes:
- adddep(d.getVar("COMPRESS_DEPENDS_%s" % ctype, True), deps)
+ adddep(d.getVar("CONVERSION_DEPENDS_%s" % ctype, True) if d.getVar("CONVERSION_CMD_%s" % ctype, True)
+ else d.getVar("COMPRESSION_DEPENDS_%s" % ctype, True),
+ deps)
for typedepends in (d.getVar("IMAGE_TYPEDEP_%s" % basetype, True) or "").split():
adddep(d.getVar('IMAGE_DEPENDS_%s' % typedepends, True) , deps)
adddep(d.getVar('IMAGE_DEPENDS_%s' % basetype, True) , deps)
@@ -346,7 +348,7 @@ python setup_debugfs () {
python () {
vardeps = set()
- # We allow COMPRESSIONTYPES to have duplicates. That avoids breaking
+ # We allow CONVERSIONTYPES to have duplicates. That avoids breaking
# derived distros when OE-core or some other layer independently adds
# the same type. There is still only one command for each type, but
# presumably the commands will do the same when the type is the same,
@@ -354,7 +356,7 @@ python () {
#
# Without de-duplication, gen_conversion_cmds() below
# would create the same compression command multiple times.
- ctypes = set(d.getVar('COMPRESSIONTYPES', True).split())
+ ctypes = set(d.getVar('CONVERSIONTYPES', True).split())
old_overrides = d.getVar('OVERRIDES', 0)
basetypes = {}
@@ -446,9 +448,10 @@ python () {
# Create input image first.
gen_conversion_cmds(type)
localdata.setVar('type', type)
- cmd = "\t" + localdata.getVar("COMPRESS_CMD_" + ctype, True)
+ cmd = "\t" + (localdata.getVar("CONVERSION_CMD_" + ctype, True) or localdata.getVar("COMPRESS_CMD_" + ctype, True))
if cmd not in cmds:
cmds.append(cmd)
+ vardeps.add('CONVERSION_CMD_' + ctype)
vardeps.add('COMPRESS_CMD_' + ctype)
subimage = type + "." + ctype
if subimage not in subimages:
diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 42db267..ca08970 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -180,7 +180,7 @@ IMAGE_CMD_wic () {
IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES"
# Rebuild when the wks file or vars in WICVARS change
-USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${COMPRESSIONTYPES}'.split()), '1', '', d)}"
+USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${CONVERSIONTYPES}'.split()), '1', '', d)}"
do_image_wic[file-checksums] += "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
EXTRA_IMAGECMD = ""
@@ -246,33 +246,39 @@ IMAGE_TYPES = " \
wic wic.gz wic.bz2 wic.lzma \
"
-COMPRESSIONTYPES = "gz bz2 lzma xz lz4 zip sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum vmdk vdi qcow2"
-COMPRESS_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
-COMPRESS_CMD_gz = "gzip -f -9 -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
-COMPRESS_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
-COMPRESS_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS} --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"
-COMPRESS_CMD_lz4 = "lz4c -9 -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
-COMPRESS_CMD_zip = "zip ${ZIP_COMPRESSION_LEVEL} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zip ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
-COMPRESS_CMD_sum = "sumtool -i ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} -o ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sum ${JFFS2_SUM_EXTRA_ARGS}"
-COMPRESS_CMD_vmdk = "qemu-img convert -O vmdk ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.vmdk"
-COMPRESS_CMD_vdi = "qemu-img convert -O vdi ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.vdi"
-COMPRESS_CMD_qcow2 = "qemu-img convert -O qcow2 ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.qcow2"
-COMPRESS_CMD_md5sum = "md5sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.md5sum"
-COMPRESS_CMD_sha1sum = "sha1sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha1sum"
-COMPRESS_CMD_sha224sum = "sha224sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha224sum"
-COMPRESS_CMD_sha256sum = "sha256sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha256sum"
-COMPRESS_CMD_sha384sum = "sha384sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha384sum"
-COMPRESS_CMD_sha512sum = "sha512sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha512sum"
-COMPRESS_DEPENDS_lzma = "xz-native"
-COMPRESS_DEPENDS_gz = ""
-COMPRESS_DEPENDS_bz2 = "pbzip2-native"
-COMPRESS_DEPENDS_xz = "xz-native"
-COMPRESS_DEPENDS_lz4 = "lz4-native"
-COMPRESS_DEPENDS_zip = "zip-native"
-COMPRESS_DEPENDS_sum = "mtd-utils-native"
-COMPRESS_DEPENDS_vmdk = "qemu-native"
-COMPRESS_DEPENDS_vdi = "qemu-native"
-COMPRESS_DEPENDS_qcow2 = "qemu-native"
+# Compression is a special case of conversion. The old variable
+# names are still supported for backward-compatibility. When defining
+# new compression or conversion commands, use CONVERSIONTYPES and
+# CONVERSION_CMD/DEPENDS.
+COMPRESSIONTYPES = ""
+
+CONVERSIONTYPES = "gz bz2 lzma xz lz4 zip sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum vmdk vdi qcow2 ${COMPRESSIONTYPES}"
+CONVERSION_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
+CONVERSION_CMD_gz = "gzip -f -9 -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
+CONVERSION_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
+CONVERSION_CMD_xz = "xz -f -k -c ${XZ_CONVERSIONION_LEVEL} ${XZ_THREADS} --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"
+CONVERSION_CMD_lz4 = "lz4c -9 -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
+CONVERSION_CMD_zip = "zip ${ZIP_CONVERSIONION_LEVEL} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zip ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
+CONVERSION_CMD_sum = "sumtool -i ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} -o ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sum ${JFFS2_SUM_EXTRA_ARGS}"
+CONVERSION_CMD_md5sum = "md5sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.md5sum"
+CONVERSION_CMD_sha1sum = "sha1sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha1sum"
+CONVERSION_CMD_sha224sum = "sha224sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha224sum"
+CONVERSION_CMD_sha256sum = "sha256sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha256sum"
+CONVERSION_CMD_sha384sum = "sha384sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha384sum"
+CONVERSION_CMD_sha512sum = "sha512sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha512sum"
+CONVERSION_CMD_vmdk = "qemu-img convert -O vmdk ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.vmdk"
+CONVERSION_CMD_vdi = "qemu-img convert -O vdi ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.vdi"
+CONVERSION_CMD_qcow2 = "qemu-img convert -O qcow2 ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.qcow2"
+CONVERSION_DEPENDS_lzma = "xz-native"
+CONVERSION_DEPENDS_gz = ""
+CONVERSION_DEPENDS_bz2 = "pbzip2-native"
+CONVERSION_DEPENDS_xz = "xz-native"
+CONVERSION_DEPENDS_lz4 = "lz4-native"
+CONVERSION_DEPENDS_zip = "zip-native"
+CONVERSION_DEPENDS_sum = "mtd-utils-native"
+CONVERSION_DEPENDS_vmdk = "qemu-native"
+CONVERSION_DEPENDS_vdi = "qemu-native"
+CONVERSION_DEPENDS_qcow2 = "qemu-native"
RUNNABLE_IMAGE_TYPES ?= "ext2 ext3 ext4"
RUNNABLE_MACHINE_PATTERNS ?= "qemu"
diff --git a/meta/classes/image_types_uboot.bbclass b/meta/classes/image_types_uboot.bbclass
index 19e4aa2..f72d9b2 100644
--- a/meta/classes/image_types_uboot.bbclass
+++ b/meta/classes/image_types_uboot.bbclass
@@ -8,19 +8,19 @@ oe_mkimage () {
fi
}
-COMPRESSIONTYPES += "gz.u-boot bz2.u-boot lzma.u-boot u-boot"
+CONVERSIONTYPES += "gz.u-boot bz2.u-boot lzma.u-boot u-boot"
-COMPRESS_DEPENDS_u-boot = "u-boot-mkimage-native"
-COMPRESS_CMD_u-boot = "oe_mkimage ${IMAGE_NAME}.rootfs.${type} none"
+CONVERSION_DEPENDS_u-boot = "u-boot-mkimage-native"
+CONVERSION_CMD_u-boot = "oe_mkimage ${IMAGE_NAME}.rootfs.${type} none"
-COMPRESS_DEPENDS_gz.u-boot = "u-boot-mkimage-native"
-COMPRESS_CMD_gz.u-boot = "${COMPRESS_CMD_gz}; oe_mkimage ${IMAGE_NAME}.rootfs.${type}.gz gzip clean"
+CONVERSION_DEPENDS_gz.u-boot = "u-boot-mkimage-native"
+CONVERSION_CMD_gz.u-boot = "${CONVERSION_CMD_gz}; oe_mkimage ${IMAGE_NAME}.rootfs.${type}.gz gzip clean"
-COMPRESS_DEPENDS_bz2.u-boot = "u-boot-mkimage-native"
-COMPRESS_CMD_bz2.u-boot = "${COMPRESS_CMD_bz2}; oe_mkimage ${IMAGE_NAME}.rootfs.${type}.bz2 bzip2 clean"
+CONVERSION_DEPENDS_bz2.u-boot = "u-boot-mkimage-native"
+CONVERSION_CMD_bz2.u-boot = "${CONVERSION_CMD_bz2}; oe_mkimage ${IMAGE_NAME}.rootfs.${type}.bz2 bzip2 clean"
-COMPRESS_DEPENDS_lzma.u-boot = "u-boot-mkimage-native"
-COMPRESS_CMD_lzma.u-boot = "${COMPRESS_CMD_lzma}; oe_mkimage ${IMAGE_NAME}.rootfs.${type}.lzma lzma clean"
+CONVERSION_DEPENDS_lzma.u-boot = "u-boot-mkimage-native"
+CONVERSION_CMD_lzma.u-boot = "${CONVERSION_CMD_lzma}; oe_mkimage ${IMAGE_NAME}.rootfs.${type}.lzma lzma clean"
IMAGE_TYPES += "ext2.u-boot ext2.gz.u-boot ext2.bz2.u-boot ext2.lzma.u-boot ext3.gz.u-boot ext4.gz.u-boot cpio.gz.u-boot"
--
2.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [RFC][PATCH 7/7] image.bbclass: prefer specialized image creation methods over chaining
2016-05-06 14:58 [RFC][PATCH 0/7] image creation improvements Ed Bartosh
` (5 preceding siblings ...)
2016-05-06 14:58 ` [RFC][PATCH 6/7] image.bbclass: rename COMPRESS(ION) to CONVERSION Ed Bartosh
@ 2016-05-06 14:58 ` Ed Bartosh
2016-05-13 17:06 ` [RFC][PATCH 0/7] image creation improvements Richard Purdie
7 siblings, 0 replies; 9+ messages in thread
From: Ed Bartosh @ 2016-05-06 14:58 UTC (permalink / raw)
To: openembedded-core
From: Patrick Ohly <patrick.ohly@intel.com>
If a certain image type can be created both by applying conversion
commands to a base type as well as via a specialized command (like a
fictious IMAGE_CMD_ext4.xz, or some dedicated task, as indicated by
IMAGE_TYPES_MASKED = ext4.xz), then pick the more specialized method
and skip generating the conversion commands.
That way developers can choose between using the builtin conversion
support or re-implementing certain types, for example more efficiently
without intermediate files.
Fixes: [YOCTO #9076]
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
meta/classes/image.bbclass | 37 ++++++++++++++++++++++++++-----------
1 file changed, 26 insertions(+), 11 deletions(-)
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 5c68b01..06c0918 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -149,10 +149,20 @@ inherit ${IMAGE_TYPE_live}
IMAGE_TYPE_vm = '${@ "image-vm" if image_type_active("hdddirect", d) else ""}'
inherit ${IMAGE_TYPE_vm}
-def image_split_type(type, allctypes):
+def image_split_type(type, allctypes, d):
'''Returns (basetype, set of compression types in use).'''
basetype = type
compressiontypes = set()
+
+ # Abort stripping the type further if "basetype" has a matching
+ # IMAGE_CMD or is a masked image type. For example, if there
+ # was a fictional IMAGE_CMD_ext4.xz which creates a compressed
+ # file directly, then we should use that instead of using
+ # IMAGE_CMD_ext4 + CONVERSION_CMD_xz.
+ if d.getVar('IMAGE_CMD_' + basetype, True) or \
+ basetype in d.getVar('IMAGE_TYPES_MASKED', True).split():
+ return (basetype, compressiontypes)
+
for ctype in allctypes:
if type.endswith("." + ctype):
basetype = type[:-len("." + ctype)]
@@ -161,7 +171,7 @@ def image_split_type(type, allctypes):
if basetype != type:
# New base type itself might be generated by a conversion command.
- basetype, newctypes = image_split_type(basetype, allctypes)
+ basetype, newctypes = image_split_type(basetype, allctypes, d)
compressiontypes.update(newctypes)
return (basetype, compressiontypes)
@@ -178,7 +188,7 @@ def image_getdepends(d):
deps = []
ctypes = set(d.getVar('CONVERSIONTYPES', True).split())
for type in (d.getVar('IMAGE_FSTYPES', True) or "").split():
- basetype, compressiontypes = image_split_type(type, ctypes)
+ basetype, compressiontypes = image_split_type(type, ctypes, d)
for ctype in compressiontypes:
adddep(d.getVar("CONVERSION_DEPENDS_%s" % ctype, True) if d.getVar("CONVERSION_CMD_%s" % ctype, True)
else d.getVar("COMPRESSION_DEPENDS_%s" % ctype, True),
@@ -369,7 +379,7 @@ python () {
alltypes.append("debugfs_" + t)
def _add_type(t):
- baset = image_split_type(t, ctypes)[0]
+ baset = image_split_type(t, ctypes, d)[0]
input_t = t
if baset not in basetypes:
basetypes[baset]= []
@@ -388,7 +398,7 @@ python () {
if dep not in alltypes:
alltypes.append(dep)
_add_type(dep)
- basedep = image_split_type(dep, ctypes)[0]
+ basedep = image_split_type(dep, ctypes, d)[0]
typedeps[baset].add(basedep)
if baset != input_t:
@@ -439,14 +449,17 @@ python () {
d.delVarFlag('IMAGE_CMD_' + realt, 'func')
rm_tmp_images = set()
- def gen_conversion_cmds(bt):
+ def gen_conversion_cmds(basetype, type):
+ if basetype == type:
+ # Done, no further conversions needed.
+ return
for ctype in ctypes:
- if bt.endswith("." + ctype):
- type = bt[0:-len(ctype) - 1]
+ if type.endswith("." + ctype):
+ type = type[0:-len(ctype) - 1]
if type.startswith("debugfs_"):
type = type[8:]
# Create input image first.
- gen_conversion_cmds(type)
+ gen_conversion_cmds(basetype, type)
localdata.setVar('type', type)
cmd = "\t" + (localdata.getVar("CONVERSION_CMD_" + ctype, True) or localdata.getVar("COMPRESS_CMD_" + ctype, True))
if cmd not in cmds:
@@ -459,8 +472,10 @@ python () {
if type not in alltypes:
rm_tmp_images.add(localdata.expand("${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"))
- for bt in basetypes[t]:
- gen_conversion_cmds(bt)
+ for type in basetypes[t]:
+ # Probably t == bt, but better check explicitly, perhaps that'll change.
+ bt = image_split_type(type, ctypes, d)[0]
+ gen_conversion_cmds(bt, type)
localdata.setVar('type', realt)
if t not in alltypes:
--
2.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [RFC][PATCH 0/7] image creation improvements
2016-05-06 14:58 [RFC][PATCH 0/7] image creation improvements Ed Bartosh
` (6 preceding siblings ...)
2016-05-06 14:58 ` [RFC][PATCH 7/7] image.bbclass: prefer specialized image creation methods over chaining Ed Bartosh
@ 2016-05-13 17:06 ` Richard Purdie
7 siblings, 0 replies; 9+ messages in thread
From: Richard Purdie @ 2016-05-13 17:06 UTC (permalink / raw)
To: Ed Bartosh, openembedded-core
On Fri, 2016-05-06 at 17:58 +0300, Ed Bartosh wrote:
> Hi,
>
> This patchset contains various improvements for the image creation
> functionality
> made by Patchick Ohly during his work on bug #9076:
> - added support for zip compression
> - fixed dependency calculation for conversion chaining
> - converted vmdk/vdi/qcow2 image types into conversion commands
> - renamed COMPRESSION variables to CONVERSION as the term
> "compression"
> is no longer accurate
> - prioritized specialized image creation methods over using
> conversion chaining
>
> The changes look reasonable to me. However, it would be good to hear
> other
> people opinions as some changes are quite complex.
Ross made some changes to the imagetypes code and I suspect this series
won't apply on top of them. I also don't like moving that dependency
code to image.bbclass, even if its the only caller. The idea is to keep
the "image types" code into its own class.
I'll queue the zip piece but the rest probably needs to be rebased...
Cheers,
Richard
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2016-05-13 17:06 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-05-06 14:58 [RFC][PATCH 0/7] image creation improvements Ed Bartosh
2016-05-06 14:58 ` [RFC][PATCH 1/7] image_types: add support for zip compression Ed Bartosh
2016-05-06 14:58 ` [RFC][PATCH 2/7] image.bbclass: support duplicate compression types Ed Bartosh
2016-05-06 14:58 ` [RFC][PATCH 3/7] image.bbclass: fix dependency calculation when using conversion chaining Ed Bartosh
2016-05-06 14:58 ` [RFC][PATCH 4/7] image.bbclass: additional output in create_symlinks Ed Bartosh
2016-05-06 14:58 ` [RFC][PATCH 5/7] image creation: support converting masked types Ed Bartosh
2016-05-06 14:58 ` [RFC][PATCH 6/7] image.bbclass: rename COMPRESS(ION) to CONVERSION Ed Bartosh
2016-05-06 14:58 ` [RFC][PATCH 7/7] image.bbclass: prefer specialized image creation methods over chaining Ed Bartosh
2016-05-13 17:06 ` [RFC][PATCH 0/7] image creation improvements Richard Purdie
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.