[PATCH 0/5] Switch to OE's implementation of IMAGE_FEATURES

[PATCH 0/5] Switch to OE's implementation of IMAGE_FEATURES

[Chris Larson]

From: Chris Larson <chris_larson@mentor.com>

This version uses variables to define the groups of packages rather than calls to base_contains.

Note: this applies on top of kergoth/oe-sync-base.

Example:

PACKAGE_GROUP_stuff_i_like = "openssh nano foo"
IMAGE_FEATURES += "stuff_i_like"

Pull URL: git://git.openembedded.org/openembedded-core-contrib
  Branch: kergoth/oe-sync-image-features
  Browse: http://git.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=kergoth/oe-sync-image-features

Thanks,
    Chris Larson <chris_larson@mentor.com>
---


Chris Larson (5):
  oe.packagegroup: add code for package groups (sync from OE)
  Move packagedata code into oe.packagedata (sync from OE)
  packagedata: don't choke on empty PACKAGES
  image.bbclass: switch to OE's IMAGE_FEATURES
  Use oe.data for IMAGE_FEATURES

 meta/classes/base.bbclass        |    2 +-
 meta/classes/core-image.bbclass  |  112 +++++++++++++-------------------------
 meta/classes/image.bbclass       |   41 +++++++++++++-
 meta/classes/package.bbclass     |    4 +-
 meta/classes/package_rpm.bbclass |    3 +-
 meta/classes/packagedata.bbclass |   68 ++----------------------
 meta/conf/bitbake.conf           |    1 +
 meta/lib/oe/packagedata.py       |  107 ++++++++++++++++++++++++++++++++++++
 meta/lib/oe/packagegroup.py      |   29 ++++++++++
 9 files changed, 225 insertions(+), 142 deletions(-)
 create mode 100644 meta/lib/oe/packagedata.py
 create mode 100644 meta/lib/oe/packagegroup.py

[PATCH 1/5] oe.packagegroup: add code for package groups (sync from OE)

[Chris Larson]

From: Chris Larson <chris_larson@mentor.com>

This includes some utility functions for dealing with groups of packages
defined in the metadata.  Metadata syntax:

    PACKAGE_GROUP_<group> = "<list of packages>"

If the packages in the group are optional:

    PACKAGE_GROUP_<group>[optional] = "1"

Signed-off-by: Chris Larson <chris_larson@mentor.com>
---
 meta/classes/base.bbclass   |    2 +-
 meta/lib/oe/packagegroup.py |   29 +++++++++++++++++++++++++++++
 2 files changed, 30 insertions(+), 1 deletions(-)
 create mode 100644 meta/lib/oe/packagegroup.py

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 7950bc3..8f4ef1e 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -10,7 +10,7 @@ inherit metadata_scm
 inherit buildstats
 inherit logging
 
-OE_IMPORTS += "os sys time oe.path oe.utils oe.data"
+OE_IMPORTS += "os sys time oe.path oe.utils oe.data oe.packagegroup"
 OE_IMPORTS[type] = "list"
 
 def oe_import(d):
diff --git a/meta/lib/oe/packagegroup.py b/meta/lib/oe/packagegroup.py
new file mode 100644
index 0000000..b04c45a
--- /dev/null
+++ b/meta/lib/oe/packagegroup.py
@@ -0,0 +1,29 @@
+import itertools
+
+def is_optional(group, d):
+    return bool(d.getVarFlag("PACKAGE_GROUP_%s" % group, "optional"))
+
+def packages(groups, d):
+    for group in groups:
+        for pkg in (d.getVar("PACKAGE_GROUP_%s" % group, True) or "").split():
+            yield pkg
+
+def required_packages(groups, d):
+    req = filter(lambda group: not is_optional(group, d), groups)
+    return packages(req, d)
+
+def optional_packages(groups, d):
+    opt = filter(lambda group: is_optional(group, d), groups)
+    return packages(opt, d)
+
+def active_packages(features, d):
+    return itertools.chain(required_packages(features, d),
+                           optional_packages(features, d))
+
+def active_recipes(features, d):
+    import oe.packagedata
+
+    for pkg in active_packages(features, d):
+        recipe = oe.packagedata.recipename(pkg, d)
+        if recipe:
+            yield recipe
-- 
1.7.1

[PATCH 2/5] Move packagedata code into oe.packagedata (sync from OE)

[Chris Larson]

From: Chris Larson <chris_larson@mentor.com>

Signed-off-by: Chris Larson <chris_larson@mentor.com>
---
 meta/classes/package.bbclass     |    4 +-
 meta/classes/package_rpm.bbclass |    3 +-
 meta/classes/packagedata.bbclass |   68 ++-----------------------
 meta/lib/oe/packagedata.py       |  106 ++++++++++++++++++++++++++++++++++++++
 4 files changed, 115 insertions(+), 66 deletions(-)
 create mode 100644 meta/lib/oe/packagedata.py

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 4eb349d..2c6d30c 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -295,7 +295,9 @@ def runstrip(file, elftype, d):
 #
 
 def get_package_mapping (pkg, d):
-	data = read_subpkgdata(pkg, d)
+	import oe.packagedata
+
+	data = oe.packagedata.read_subpkgdata(pkg, d)
 	key = "PKG_%s" % pkg
 
 	if key in data:
diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index 1cf9f79..6d96719 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -316,6 +316,7 @@ package_install_internal_rpm () {
 
 python write_specfile () {
 	import textwrap
+	import oe.packagedata
 
 	# We need to change '-' in a version field to '+'
 	# This needs to be done BEFORE the mapping_rename_hook
@@ -328,7 +329,7 @@ python write_specfile () {
 				ver = depends_dict[dep]
 				if dep and ver:
 					if '-' in ver:
-						subd = read_subpkgdata_dict(dep, d)
+						subd = oe.packagedata.read_subpkgdata_dict(dep, d)
 						pv = subd['PV']
 						reppv = pv.replace('-', '+')
 						ver = ver.replace(pv, reppv)
diff --git a/meta/classes/packagedata.bbclass b/meta/classes/packagedata.bbclass
index 86f18a9..bf051fe 100644
--- a/meta/classes/packagedata.bbclass
+++ b/meta/classes/packagedata.bbclass
@@ -1,73 +1,13 @@
-def packaged(pkg, d):
-	return os.access(get_subpkgedata_fn(pkg, d) + '.packaged', os.R_OK)
-
-def read_pkgdatafile(fn):
-	pkgdata = {}
-
-	def decode(str):
-		import codecs
-		c = codecs.getdecoder("string_escape")
-		return c(str)[0]
-
-	if os.access(fn, os.R_OK):
-		import re
-		f = file(fn, 'r')
-		lines = f.readlines()
-		f.close()
-		r = re.compile("([^:]+):\s*(.*)")
-		for l in lines:
-			m = r.match(l)
-			if m:
-				pkgdata[m.group(1)] = decode(m.group(2))
-
-	return pkgdata
-
-def get_subpkgedata_fn(pkg, d):
-	archs = bb.data.expand("${PACKAGE_ARCHS}", d).split(" ")
-	archs.reverse()
-	pkgdata = bb.data.expand('${TMPDIR}/pkgdata/', d)
-	targetdir = bb.data.expand('${TARGET_VENDOR}-${TARGET_OS}/runtime/', d)
-	for arch in archs:
-		fn = pkgdata + arch + targetdir + pkg
-		if os.path.exists(fn):
-			return fn
-	return bb.data.expand('${PKGDATA_DIR}/runtime/%s' % pkg, d)
-
-def has_subpkgdata(pkg, d):
-	return os.access(get_subpkgedata_fn(pkg, d), os.R_OK)
-
-def read_subpkgdata(pkg, d):
-	return read_pkgdatafile(get_subpkgedata_fn(pkg, d))
-
-def has_pkgdata(pn, d):
-	fn = bb.data.expand('${PKGDATA_DIR}/%s' % pn, d)
-	return os.access(fn, os.R_OK)
-
-def read_pkgdata(pn, d):
-	fn = bb.data.expand('${PKGDATA_DIR}/%s' % pn, d)
-	return read_pkgdatafile(fn)
-
 python read_subpackage_metadata () {
-	data = read_pkgdata(bb.data.getVar('PN', d, 1), d)
+	import oe.packagedata
+
+	data = oe.packagedata.read_pkgdata(bb.data.getVar('PN', d, 1), d)
 
 	for key in data.keys():
 		bb.data.setVar(key, data[key], d)
 
 	for pkg in bb.data.getVar('PACKAGES', d, 1).split():
-		sdata = read_subpkgdata(pkg, d)
+		sdata = oe.packagedata.read_subpkgdata(pkg, d)
 		for key in sdata.keys():
 			bb.data.setVar(key, sdata[key], d)
 }
-
-
-#
-# Collapse FOO_pkg variables into FOO
-#
-def read_subpkgdata_dict(pkg, d):
-	ret = {}
-	subd = read_pkgdatafile(get_subpkgedata_fn(pkg, d))
-	for var in subd:
-		newvar = var.replace("_" + pkg, "")
-		ret[newvar] = subd[var]
-	return ret
-
diff --git a/meta/lib/oe/packagedata.py b/meta/lib/oe/packagedata.py
new file mode 100644
index 0000000..7f0a89d
--- /dev/null
+++ b/meta/lib/oe/packagedata.py
@@ -0,0 +1,106 @@
+import os
+import bb.data
+import codecs
+
+def packaged(pkg, d):
+    return os.access(get_subpkgedata_fn(pkg, d) + '.packaged', os.R_OK)
+
+def read_pkgdatafile(fn):
+    pkgdata = {}
+
+    def decode(str):
+        c = codecs.getdecoder("string_escape")
+        return c(str)[0]
+
+    if os.access(fn, os.R_OK):
+        import re
+        f = file(fn, 'r')
+        lines = f.readlines()
+        f.close()
+        r = re.compile("([^:]+):\s*(.*)")
+        for l in lines:
+            m = r.match(l)
+            if m:
+                pkgdata[m.group(1)] = decode(m.group(2))
+
+    return pkgdata
+
+def get_subpkgedata_fn(pkg, d):
+    archs = bb.data.expand("${PACKAGE_ARCHS}", d).split(" ")
+    archs.reverse()
+    pkgdata = bb.data.expand('${TMPDIR}/pkgdata/', d)
+    targetdir = bb.data.expand('${TARGET_VENDOR}-${TARGET_OS}/runtime/', d)
+    for arch in archs:
+        fn = pkgdata + arch + targetdir + pkg
+        if os.path.exists(fn):
+            return fn
+    return bb.data.expand('${PKGDATA_DIR}/runtime/%s' % pkg, d)
+
+def has_subpkgdata(pkg, d):
+    return os.access(get_subpkgedata_fn(pkg, d), os.R_OK)
+
+def read_subpkgdata(pkg, d):
+    return read_pkgdatafile(get_subpkgedata_fn(pkg, d))
+
+def has_pkgdata(pn, d):
+    fn = bb.data.expand('${PKGDATA_DIR}/%s' % pn, d)
+    return os.access(fn, os.R_OK)
+
+def read_pkgdata(pn, d):
+    fn = bb.data.expand('${PKGDATA_DIR}/%s' % pn, d)
+    return read_pkgdatafile(fn)
+
+#
+# Collapse FOO_pkg variables into FOO
+#
+def read_subpkgdata_dict(pkg, d):
+    ret = {}
+    subd = read_pkgdatafile(get_subpkgedata_fn(pkg, d))
+    for var in subd:
+        newvar = var.replace("_" + pkg, "")
+        ret[newvar] = subd[var]
+    return ret
+
+def _pkgmap(d):
+    """Return a dictionary mapping package to recipe name."""
+
+    target_os = d.getVar("TARGET_OS", True)
+    target_vendor = d.getVar("TARGET_VENDOR", True)
+    basedir = os.path.dirname(d.getVar("PKGDATA_DIR", True))
+
+    dirs = ("%s%s-%s" % (arch, target_vendor, target_os)
+            for arch in d.getVar("PACKAGE_ARCHS", True).split())
+
+    pkgmap = {}
+    for pkgdatadir in (os.path.join(basedir, sys) for sys in dirs):
+        try:
+            files = os.listdir(pkgdatadir)
+        except OSError:
+            continue
+
+        for pn in filter(lambda f: not os.path.isdir(os.path.join(pkgdatadir, f)), files):
+            try:
+                pkgdata = read_pkgdatafile(os.path.join(pkgdatadir, pn))
+            except OSError:
+                continue
+
+            for pkg in pkgdata["PACKAGES"].split():
+                pkgmap[pkg] = pn
+
+    return pkgmap
+
+def pkgmap(d):
+    """Return a dictionary mapping package to recipe name.
+    Cache the mapping in the metadata"""
+
+    pkgmap_data = d.getVar("__pkgmap_data", False)
+    if pkgmap_data is None:
+        pkgmap_data = _pkgmap(d)
+        d.setVar("__pkgmap_data", pkgmap_data)
+
+    return pkgmap_data
+
+def recipename(pkg, d):
+    """Return the recipe name for the given binary package name."""
+
+    return pkgmap(d).get(pkg)
-- 
1.7.1

[PATCH 3/5] packagedata: don't choke on empty PACKAGES

[Chris Larson]

From: Chris Larson <chris_larson@mentor.com>

Signed-off-by: Chris Larson <chris_larson@mentor.com>
---
 meta/lib/oe/packagedata.py |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/meta/lib/oe/packagedata.py b/meta/lib/oe/packagedata.py
index 7f0a89d..ee10a23 100644
--- a/meta/lib/oe/packagedata.py
+++ b/meta/lib/oe/packagedata.py
@@ -84,7 +84,8 @@ def _pkgmap(d):
             except OSError:
                 continue
 
-            for pkg in pkgdata["PACKAGES"].split():
+            packages = pkgdata.get("PACKAGES") or ""
+            for pkg in packages.split():
                 pkgmap[pkg] = pn
 
     return pkgmap
-- 
1.7.1

[PATCH 4/5] image.bbclass: switch to OE's IMAGE_FEATURES

[Chris Larson]

From: Chris Larson <chris_larson@mentor.com>

Currently, all image features are assumed to be package groups defined with
oe.packagegroup (PACKAGE_GROUP_<group> = "<list of packages>").

Signed-off-by: Chris Larson <chris_larson@mentor.com>
---
 meta/classes/core-image.bbclass |  112 ++++++++++++++-------------------------
 meta/classes/image.bbclass      |   41 ++++++++++++++-
 meta/conf/bitbake.conf          |    1 +
 3 files changed, 79 insertions(+), 75 deletions(-)

diff --git a/meta/classes/core-image.bbclass b/meta/classes/core-image.bbclass
index c8c6a57..fcfce71 100644
--- a/meta/classes/core-image.bbclass
+++ b/meta/classes/core-image.bbclass
@@ -5,18 +5,16 @@
 LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=3f40d7994397109285ec7b81fdeb3b58 \
                     file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
 
-# IMAGE_FEATURES control content of the core reference images
-# 
 # By default we install task-core-boot and task-base packages - this gives us
 # working (console only) rootfs.
 #
-# Available IMAGE_FEATURES:
+# Additional IMAGE_FEATURES:
 #
 # - apps-console-core
-# - x11-base            - X11 server + minimal desktop	
-# - x11-sato            - OpenedHand Sato environment
-# - x11-netbook         - Metacity based environment for netbooks
-# - apps-x11-core       - X Terminal, file manager, file editor
+# - x11-base          - X11 server + minimal desktop
+# - x11-sato          - OpenedHand Sato environment
+# - x11-netbook       - Metacity based environment for netbooks
+# - apps-x11-core     - X Terminal, file manager, file editor
 # - apps-x11-games
 # - apps-x11-pimlico    - OpenedHand Pimlico apps
 # - tools-sdk           - SDK
@@ -26,75 +24,43 @@ LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=3f40d7994397109285ec7b81fdeb3
 # - nfs-server          - NFS server (exports / over NFS to everybody)
 # - ssh-server-dropbear - SSH server (dropbear)
 # - ssh-server-openssh  - SSH server (openssh)
-# - dev-pkgs            - development packages
-# - dbg-pkgs            - debug packages
 #
 
+PACKAGE_GROUP_apps-console-core = "task-core-apps-console"
+PACKAGE_GROUP_x11-base = "task-core-x11-base"
+PACKAGE_GROUP_x11-sato = "task-core-x11-sato"
+PACKAGE_GROUP_x11-netbook = "task-core-x11-netbook"
+PACKAGE_GROUP_apps-x11-core = "task-core-apps-x11-core"
+PACKAGE_GROUP_apps-x11-games = "task-core-apps-x11-games"
+PACKAGE_GROUP_apps-x11-pimlico = "task-core-apps-x11-pimlico"
+PACKAGE_GROUP_tools-debug = "task-core-tools-debug"
+PACKAGE_GROUP_tools-profile = "task-core-tools-profile"
+PACKAGE_GROUP_tools-testapps = "task-core-tools-testapps"
+PACKAGE_GROUP_tools-sdk = "task-core-sdk task-core-standalone-sdk-target"
+PACKAGE_GROUP_nfs-server = "task-core-nfs-server"
+PACKAGE_GROUP_package-management = "${ROOTFS_PKGMANAGE}"
+PACKAGE_GROUP_package-management-bootstrap = "${ROOTFS_PKGMANAGE_BOOTSTRAP}"
+PACKAGE_GROUP_qt4-pkgs = "task-core-qt-demos"
+PACKAGE_GROUP_ssh-server-dropbear = "task-core-ssh-dropbear"
+PACKAGE_GROUP_ssh-server-openssh = "task-core-ssh-openssh"
+
+python handle_rootfs_pkgmanage_bootstrap() {
+    if not isinstance(e, bb.event.RecipeParsed):
+        return
+
+    d = e.data
+    features = d.getVar('IMAGE_FEATURES', True).split()
+    if 'package-management' not in features:
+        features.append('package-management-bootstrap')
+        d.setVar('IMAGE_FEATURES', ' '.join(features))
+}
+addhandler handle_rootfs_pkgmanage_bootstrap
+
 POKY_BASE_INSTALL = '\
-    task-core-boot \
-    task-base-extended \
-    ${@base_contains("IMAGE_FEATURES", "dbg-pkgs", "task-core-boot-dbg task-base-dbg", "",d)} \
-    ${@base_contains("IMAGE_FEATURES", "dev-pkgs", "task-core-boot-dev task-base-dev", "",d)} \
-    \
-    ${@base_contains("IMAGE_FEATURES", "apps-console-core", "task-core-apps-console", "",d)} \
-    ${@base_contains("IMAGE_FEATURES", ["apps-console-core", "dbg-pkgs"], "task-core-apps-console-dbg", "",d)} \
-    ${@base_contains("IMAGE_FEATURES", ["apps-console-core", "dev-pkgs"], "task-core-apps-console-dev", "",d)} \
-    \
-    ${@base_contains("IMAGE_FEATURES", "x11-base", "task-core-x11-base", "",d)} \
-    ${@base_contains("IMAGE_FEATURES", ["x11-base", "dbg-pkgs"], "task-core-x11-base-dbg", "",d)} \
-    ${@base_contains("IMAGE_FEATURES", ["x11-base", "dev-pkgs"], "task-core-x11-base-dev", "",d)} \
-    \
-    ${@base_contains("IMAGE_FEATURES", "x11-sato", "task-core-x11-sato", "",d)} \
-    ${@base_contains("IMAGE_FEATURES", ["x11-sato", "dbg-pkgs"], "task-core-x11-sato-dbg", "",d)} \
-    ${@base_contains("IMAGE_FEATURES", ["x11-sato", "dev-pkgs"], "task-core-x11-sato-dev", "",d)} \
-    \
-    ${@base_contains("IMAGE_FEATURES", "x11-netbook", "task-core-x11-netbook", "", d)} \
-    ${@base_contains("IMAGE_FEATURES", ["x11-netbook", "dbg-pkgs"], "task-core-x11-netbook-dbg", "", d)} \
-    ${@base_contains("IMAGE_FEATURES", ["x11-netbook", "dev-pkgs"], "task-core-x11-netbook-dev", "", d)} \
-    ${@base_contains("IMAGE_FEATURES", "apps-x11-core", "task-core-apps-x11-core", "",d)} \
-    ${@base_contains("IMAGE_FEATURES", ["apps-x11-core", "dbg-pkgs"], "task-core-apps-x11-core-dbg", "",d)} \
-    ${@base_contains("IMAGE_FEATURES", ["apps-x11-core", "dev-pkgs"], "task-core-apps-x11-core-dev", "",d)} \
-    \
-    ${@base_contains("IMAGE_FEATURES", "apps-x11-games", "task-core-apps-x11-games", "",d)} \
-    ${@base_contains("IMAGE_FEATURES", ["apps-x11-games", "dbg-pkgs"], "task-core-apps-x11-games-dbg", "",d)} \
-    ${@base_contains("IMAGE_FEATURES", ["apps-x11-games", "dev-pkgs"], "task-core-apps-x11-games-dev", "",d)} \
-    \
-    ${@base_contains("IMAGE_FEATURES", "apps-x11-pimlico", "task-core-apps-x11-pimlico", "",d)} \
-    ${@base_contains("IMAGE_FEATURES", ["apps-x11-pimlico", "dbg-pkgs"], "task-core-apps-x11-pimlico-dbg", "",d)} \
-    ${@base_contains("IMAGE_FEATURES", ["apps-x11-pimlico", "dev-pkgs"], "task-core-apps-x11-pimlico-dev", "",d)} \
-    \
-    ${@base_contains("IMAGE_FEATURES", "tools-debug", "task-core-tools-debug", "",d)} \
-    ${@base_contains("IMAGE_FEATURES", ["tools-debug", "dbg-pkgs"], "task-core-tools-debug-dbg", "",d)} \
-    ${@base_contains("IMAGE_FEATURES", ["tools-debug", "dev-pkgs"], "task-core-tools-debug-dev", "",d)} \
-    \
-    ${@base_contains("IMAGE_FEATURES", "tools-profile", "task-core-tools-profile", "",d)} \
-    ${@base_contains("IMAGE_FEATURES", ["tools-profile", "dbg-pkgs"], "task-core-tools-profile-dbg", "",d)} \
-    ${@base_contains("IMAGE_FEATURES", ["tools-profile", "dev-pkgs"], "task-core-tools-profile-dev", "",d)} \
-    \
-    ${@base_contains("IMAGE_FEATURES", "tools-testapps", "task-core-tools-testapps", "",d)} \
-    ${@base_contains("IMAGE_FEATURES", ["tools-testapps", "dbg-pkgs"], "task-core-tools-testapps-dbg", "",d)} \
-    ${@base_contains("IMAGE_FEATURES", ["tools-testapps", "dev-pkgs"], "task-core-tools-testapps-dev", "",d)} \
-    \
-    ${@base_contains("IMAGE_FEATURES", "tools-sdk", "task-core-sdk task-core-standalone-sdk-target", "",d)} \    
-    ${@base_contains("IMAGE_FEATURES", ["tools-sdk", "dbg-pkgs"], "task-core-sdk-dbg", "",d)} \
-    ${@base_contains("IMAGE_FEATURES", ["tools-sdk", "dev-pkgs"], "task-core-sdk-dev", "",d)} \
-    \
-    ${@base_contains("IMAGE_FEATURES", "nfs-server", "task-core-nfs-server", "",d)} \
-    ${@base_contains("IMAGE_FEATURES", ["nfs-server", "dbg-pkgs"], "task-core-nfs-server-dbg", "",d)} \
-    ${@base_contains("IMAGE_FEATURES", ["nfs-server", "dev-pkgs"], "task-core-nfs-server-dev", "",d)} \
-    \
-    ${@base_contains("IMAGE_FEATURES", "ssh-server-dropbear", "task-core-ssh-dropbear", "",d)} \
-    ${@base_contains("IMAGE_FEATURES", ["ssh-server-dropbear", "dbg-pkgs"], "task-core-ssh-dropbear-dbg", "",d)} \
-    ${@base_contains("IMAGE_FEATURES", ["ssh-server-dropbear", "dev-pkgs"], "task-core-ssh-dropbear-dev", "",d)} \
-    \
-    ${@base_contains("IMAGE_FEATURES", "ssh-server-openssh", "task-core-ssh-openssh", "",d)} \
-    ${@base_contains("IMAGE_FEATURES", ["ssh-server-openssh", "dbg-pkgs"], "task-core-ssh-openssh-dbg", "",d)} \
-    ${@base_contains("IMAGE_FEATURES", ["ssh-server-openssh", "dev-pkgs"], "task-core-ssh-openssh-dev", "",d)} \
-    \
-    ${@base_contains("IMAGE_FEATURES", "package-management", "${ROOTFS_PKGMANAGE}", "${ROOTFS_PKGMANAGE_BOOTSTRAP}",d)} \
-    ${@base_contains("IMAGE_FEATURES", "qt4-pkgs", "task-core-qt-demos", "",d)} \
-    ${POKY_EXTRA_INSTALL} \
-    '
+     task-core-boot \
+     task-base-extended \
+     ${POKY_EXTRA_INSTALL} \
+     '
 
 POKY_EXTRA_INSTALL ?= ""
 
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 2469442..3c62688 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -11,8 +11,45 @@ INHIBIT_DEFAULT_DEPS = "1"
 
 # "export IMAGE_BASENAME" not supported at this time
 IMAGE_BASENAME[export] = "1"
-export PACKAGE_INSTALL ?= "${IMAGE_INSTALL}"
-PACKAGE_INSTALL_ATTEMPTONLY ?= ""
+
+PACKAGE_INSTALL = "${@' '.join(oe.packagegroup.required_packages('${IMAGE_FEATURES}'.split(), d))}"
+PACKAGE_INSTALL_ATTEMPTONLY = "${@' '.join(oe.packagegroup.optional_packages('${IMAGE_FEATURES}'.split(), d))}"
+RDEPENDS += "${@' '.join(oe.packagegroup.active_packages('${IMAGE_FEATURES}'.split(), d))}"
+
+
+IMAGE_FEATURES ?= ""
+IMAGE_FEATURES[type] = "list"
+IMAGE_FEATURES_prepend = "image_base "
+
+# Define our always included package group
+PACKAGE_GROUP_image_base = "${IMAGE_INSTALL}"
+
+# The following package groups allow one to add debugging, development, and
+# documentation files for all packages installed in the image.
+
+def string_set(iterable):
+    return ' '.join(set(iterable))
+
+def image_features_noextras(d):
+    for f in d.getVar("IMAGE_FEATURES", True).split():
+        if not f in ('dbg', 'dev', 'doc'):
+            yield f
+
+def dbg_packages(d):
+    from itertools import chain
+
+    features = image_features_noextras(d)
+    return string_set("%s-dbg" % pkg
+                      for pkg in chain(oe.packagegroup.active_packages(features, d),
+                                       oe.packagegroup.active_recipes(features, d)))
+
+PACKAGE_GROUP_dbg = "${@dbg_packages(d)}"
+PACKAGE_GROUP_dbg[optional] = "1"
+PACKAGE_GROUP_dev = "${@string_set('%s-dev' % pn for pn in oe.packagegroup.active_recipes(image_features_noextras(d), d))}"
+PACKAGE_GROUP_dev[optional] = "1"
+PACKAGE_GROUP_doc = "${@string_set('%s-doc' % pn for pn in oe.packagegroup.active_recipes(image_features_noextras(d), d))}"
+PACKAGE_GROUP_doc[optional] = "1"
+
 
 # Images are generally built explicitly, do not need to be part of world.
 EXCLUDE_FROM_WORLD = "1"
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index a0af672..f282960 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -675,6 +675,7 @@ MACHINE_EXTRA_RDEPENDS ?= ""
 MACHINE_EXTRA_RRECOMMENDS ?= ""
 MACHINE_ESSENTIAL_EXTRA_RDEPENDS ?= ""
 MACHINE_ESSENTIAL_EXTRA_RRECOMMENDS ?= ""
+EXTRA_IMAGE_FEATURES ?= ""
 IMAGE_FEATURES += "${EXTRA_IMAGE_FEATURES}"
 
 COMBINED_FEATURES = "\
-- 
1.7.1

[PATCH 5/5] Use oe.data for IMAGE_FEATURES

[Chris Larson]

From: Chris Larson <chris_larson@mentor.com>

Signed-off-by: Chris Larson <chris_larson@mentor.com>
---
 meta/classes/image.bbclass |   12 ++++++------
 1 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 3c62688..6b4e01e 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -12,9 +12,9 @@ INHIBIT_DEFAULT_DEPS = "1"
 # "export IMAGE_BASENAME" not supported at this time
 IMAGE_BASENAME[export] = "1"
 
-PACKAGE_INSTALL = "${@' '.join(oe.packagegroup.required_packages('${IMAGE_FEATURES}'.split(), d))}"
-PACKAGE_INSTALL_ATTEMPTONLY = "${@' '.join(oe.packagegroup.optional_packages('${IMAGE_FEATURES}'.split(), d))}"
-RDEPENDS += "${@' '.join(oe.packagegroup.active_packages('${IMAGE_FEATURES}'.split(), d))}"
+PACKAGE_INSTALL = "${@' '.join(oe.packagegroup.required_packages(oe.data.typed_value('IMAGE_FEATURES', d), d))}"
+PACKAGE_INSTALL_ATTEMPTONLY = "${@' '.join(oe.packagegroup.optional_packages(oe.data.typed_value('IMAGE_FEATURES', d), d))}"
+RDEPENDS += "${@' '.join(oe.packagegroup.active_packages(oe.data.typed_value('IMAGE_FEATURES', d), d))}"
 
 
 IMAGE_FEATURES ?= ""
@@ -31,9 +31,9 @@ def string_set(iterable):
     return ' '.join(set(iterable))
 
 def image_features_noextras(d):
-    for f in d.getVar("IMAGE_FEATURES", True).split():
-        if not f in ('dbg', 'dev', 'doc'):
-            yield f
+    for feature in oe.data.typed_value('IMAGE_FEATURES', d):
+        if not feature in ('dbg', 'dev', 'doc'):
+            yield feature
 
 def dbg_packages(d):
     from itertools import chain
-- 
1.7.1

Re: [PATCH 4/5] image.bbclass: switch to OE's IMAGE_FEATURES

[Richard Purdie]

On Wed, 2011-05-18 at 14:06 -0700, Chris Larson wrote:
> From: Chris Larson <chris_larson@mentor.com>
> 
> Currently, all image features are assumed to be package groups defined with
> oe.packagegroup (PACKAGE_GROUP_<group> = "<list of packages>").
> 
> Signed-off-by: Chris Larson <chris_larson@mentor.com>
> ---
>  meta/classes/core-image.bbclass |  112 ++++++++++++++-------------------------
>  meta/classes/image.bbclass      |   41 ++++++++++++++-
>  meta/conf/bitbake.conf          |    1 +
>  3 files changed, 79 insertions(+), 75 deletions(-)
>          - SDK
> @@ -26,75 +24,43 @@ LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=3f40d7994397109285ec7b81fdeb3
>  # - nfs-server          - NFS server (exports / over NFS to everybody)
>  # - ssh-server-dropbear - SSH server (dropbear)
>  # - ssh-server-openssh  - SSH server (openssh)
> -# - dev-pkgs            - development packages
> -# - dbg-pkgs            - debug packages
>  #

I like the patch, there are just two things which bother me a little.
Firstly, if I understand correctly and haven't missed anything,
"dev-pkgs" becomes "dev" with this change?

A quick grep shows users of this such as:

recipes-core/images/core-image-minimal-dev.bb:IMAGE_FEATURES += "dev-pkgs"
recipes-extended/images/core-image-lsb-sdk.bb:IMAGE_FEATURES += "apps-console-core tools-debug tools-profile tools-sdk dev-pkgs ssh-server-openssh"
recipes-extended/images/core-image-lsb-dev.bb:IMAGE_FEATURES += "apps-console-core dev-pkgs ssh-server-openssh"
recipes-sato/images/core-image-sato-dev.bb:IMAGE_FEATURES += "apps-console-core ${SATO_IMAGE_FEATURES} dev-pkgs"
recipes-sato/images/core-image-sato-sdk.bb:IMAGE_FEATURES += "apps-console-core ${SATO_IMAGE_FEATURES} tools-debug tools-profile tools-sdk dev-pkgs qt4-pkgs"

and I'd expect the changelog to at least mention it and correct this.

I'm also thinking IMAGE_FEATURES = "dev" doesn't really indicate what it
means very clearly which is why dev-pkgs was originally used...

> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -11,8 +11,45 @@ INHIBIT_DEFAULT_DEPS = "1"
>  
>  # "export IMAGE_BASENAME" not supported at this time
>  IMAGE_BASENAME[export] = "1"
> -export PACKAGE_INSTALL ?= "${IMAGE_INSTALL}"
> -PACKAGE_INSTALL_ATTEMPTONLY ?= ""
> +
> +PACKAGE_INSTALL = "${@' '.join(oe.packagegroup.required_packages('${IMAGE_FEATURES}'.split(), d))}"
> +PACKAGE_INSTALL_ATTEMPTONLY = "${@' '.join(oe.packagegroup.optional_packages('${IMAGE_FEATURES}'.split(), d))}"
> +RDEPENDS += "${@' '.join(oe.packagegroup.active_packages('${IMAGE_FEATURES}'.split(), d))}"

I also noticed this patch changes things so PACKAGE_INSTALL_ATTEMPTONLY
is used for the dev/doc/dbg packages. I'm not sure its a major issue but
it is a change in behaviour and I'd have expected it in the commit
message.

Cheers,

Richard

Re: [PATCH 4/5] image.bbclass: switch to OE's IMAGE_FEATURES

[Chris Larson]

On Fri, May 20, 2011 at 11:05 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Wed, 2011-05-18 at 14:06 -0700, Chris Larson wrote:
>> From: Chris Larson <chris_larson@mentor.com>
>>
>> Currently, all image features are assumed to be package groups defined with
>> oe.packagegroup (PACKAGE_GROUP_<group> = "<list of packages>").
>>
>> Signed-off-by: Chris Larson <chris_larson@mentor.com>
>> ---
>>  meta/classes/core-image.bbclass |  112 ++++++++++++++-------------------------
>>  meta/classes/image.bbclass      |   41 ++++++++++++++-
>>  meta/conf/bitbake.conf          |    1 +
>>  3 files changed, 79 insertions(+), 75 deletions(-)
>>          - SDK
>> @@ -26,75 +24,43 @@ LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=3f40d7994397109285ec7b81fdeb3
>>  # - nfs-server          - NFS server (exports / over NFS to everybody)
>>  # - ssh-server-dropbear - SSH server (dropbear)
>>  # - ssh-server-openssh  - SSH server (openssh)
>> -# - dev-pkgs            - development packages
>> -# - dbg-pkgs            - debug packages
>>  #
>
> I like the patch, there are just two things which bother me a little.
> Firstly, if I understand correctly and haven't missed anything,
> "dev-pkgs" becomes "dev" with this change?
>
> A quick grep shows users of this such as:
>
> recipes-core/images/core-image-minimal-dev.bb:IMAGE_FEATURES += "dev-pkgs"
> recipes-extended/images/core-image-lsb-sdk.bb:IMAGE_FEATURES += "apps-console-core tools-debug tools-profile tools-sdk dev-pkgs ssh-server-openssh"
> recipes-extended/images/core-image-lsb-dev.bb:IMAGE_FEATURES += "apps-console-core dev-pkgs ssh-server-openssh"
> recipes-sato/images/core-image-sato-dev.bb:IMAGE_FEATURES += "apps-console-core ${SATO_IMAGE_FEATURES} dev-pkgs"
> recipes-sato/images/core-image-sato-sdk.bb:IMAGE_FEATURES += "apps-console-core ${SATO_IMAGE_FEATURES} tools-debug tools-profile tools-sdk dev-pkgs qt4-pkgs"
>
> and I'd expect the changelog to at least mention it and correct this.
>
> I'm also thinking IMAGE_FEATURES = "dev" doesn't really indicate what it
> means very clearly which is why dev-pkgs was originally used...

Good points, that's fair enough. I only went with 'dev' for
compatibility with upstream OE. I'm not opposed to switching to
dev-pkgs. Would you like me to do that and resubmit, with the other
issues addressed?

>> --- a/meta/classes/image.bbclass
>> +++ b/meta/classes/image.bbclass
>> @@ -11,8 +11,45 @@ INHIBIT_DEFAULT_DEPS = "1"
>>
>>  # "export IMAGE_BASENAME" not supported at this time
>>  IMAGE_BASENAME[export] = "1"
>> -export PACKAGE_INSTALL ?= "${IMAGE_INSTALL}"
>> -PACKAGE_INSTALL_ATTEMPTONLY ?= ""
>> +
>> +PACKAGE_INSTALL = "${@' '.join(oe.packagegroup.required_packages('${IMAGE_FEATURES}'.split(), d))}"
>> +PACKAGE_INSTALL_ATTEMPTONLY = "${@' '.join(oe.packagegroup.optional_packages('${IMAGE_FEATURES}'.split(), d))}"
>> +RDEPENDS += "${@' '.join(oe.packagegroup.active_packages('${IMAGE_FEATURES}'.split(), d))}"
>
> I also noticed this patch changes things so PACKAGE_INSTALL_ATTEMPTONLY
> is used for the dev/doc/dbg packages. I'm not sure its a major issue but
> it is a change in behaviour and I'd have expected it in the commit
> message.

Ah, that's my mistake then, I didn't realize the behavior was
different in the current implementation. I just figured some packages
might not have dev/doc/dbg, so it should be nonfatal to miss them. We
may also need to make sure rpm/deb both handle attemptonly properly,
as upstream's did not. I'll add this to the commit message, unless you
think they shouldn't be optional? I'm inclined to prefer it this way,
as, iirc, the depchain stuff uses recommends rather than depends.
(though i may be remembering wrong?)
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: [PATCH 4/5] image.bbclass: switch to OE's IMAGE_FEATURES

[Richard Purdie]

On Fri, 2011-05-20 at 11:09 -0700, Chris Larson wrote:
> On Fri, May 20, 2011 at 11:05 AM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > On Wed, 2011-05-18 at 14:06 -0700, Chris Larson wrote:
> >> From: Chris Larson <chris_larson@mentor.com>
> >>
> >> Currently, all image features are assumed to be package groups defined with
> >> oe.packagegroup (PACKAGE_GROUP_<group> = "<list of packages>").
> >>
> >> Signed-off-by: Chris Larson <chris_larson@mentor.com>
> >> ---
> >>  meta/classes/core-image.bbclass |  112 ++++++++++++++-------------------------
> >>  meta/classes/image.bbclass      |   41 ++++++++++++++-
> >>  meta/conf/bitbake.conf          |    1 +
> >>  3 files changed, 79 insertions(+), 75 deletions(-)
> >>          - SDK
> >> @@ -26,75 +24,43 @@ LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=3f40d7994397109285ec7b81fdeb3
> >>  # - nfs-server          - NFS server (exports / over NFS to everybody)
> >>  # - ssh-server-dropbear - SSH server (dropbear)
> >>  # - ssh-server-openssh  - SSH server (openssh)
> >> -# - dev-pkgs            - development packages
> >> -# - dbg-pkgs            - debug packages
> >>  #
> >
> > I like the patch, there are just two things which bother me a little.
> > Firstly, if I understand correctly and haven't missed anything,
> > "dev-pkgs" becomes "dev" with this change?
> >
> > A quick grep shows users of this such as:
> >
> > recipes-core/images/core-image-minimal-dev.bb:IMAGE_FEATURES += "dev-pkgs"
> > recipes-extended/images/core-image-lsb-sdk.bb:IMAGE_FEATURES += "apps-console-core tools-debug tools-profile tools-sdk dev-pkgs ssh-server-openssh"
> > recipes-extended/images/core-image-lsb-dev.bb:IMAGE_FEATURES += "apps-console-core dev-pkgs ssh-server-openssh"
> > recipes-sato/images/core-image-sato-dev.bb:IMAGE_FEATURES += "apps-console-core ${SATO_IMAGE_FEATURES} dev-pkgs"
> > recipes-sato/images/core-image-sato-sdk.bb:IMAGE_FEATURES += "apps-console-core ${SATO_IMAGE_FEATURES} tools-debug tools-profile tools-sdk dev-pkgs qt4-pkgs"
> >
> > and I'd expect the changelog to at least mention it and correct this.
> >
> > I'm also thinking IMAGE_FEATURES = "dev" doesn't really indicate what it
> > means very clearly which is why dev-pkgs was originally used...
> 
> Good points, that's fair enough. I only went with 'dev' for
> compatibility with upstream OE. I'm not opposed to switching to
> dev-pkgs. Would you like me to do that and resubmit, with the other
> issues addressed?

Yes please :)

> >> --- a/meta/classes/image.bbclass
> >> +++ b/meta/classes/image.bbclass
> >> @@ -11,8 +11,45 @@ INHIBIT_DEFAULT_DEPS = "1"
> >>
> >>  # "export IMAGE_BASENAME" not supported at this time
> >>  IMAGE_BASENAME[export] = "1"
> >> -export PACKAGE_INSTALL ?= "${IMAGE_INSTALL}"
> >> -PACKAGE_INSTALL_ATTEMPTONLY ?= ""
> >> +
> >> +PACKAGE_INSTALL = "${@' '.join(oe.packagegroup.required_packages('${IMAGE_FEATURES}'.split(), d))}"
> >> +PACKAGE_INSTALL_ATTEMPTONLY = "${@' '.join(oe.packagegroup.optional_packages('${IMAGE_FEATURES}'.split(), d))}"
> >> +RDEPENDS += "${@' '.join(oe.packagegroup.active_packages('${IMAGE_FEATURES}'.split(), d))}"
> >
> > I also noticed this patch changes things so PACKAGE_INSTALL_ATTEMPTONLY
> > is used for the dev/doc/dbg packages. I'm not sure its a major issue but
> > it is a change in behaviour and I'd have expected it in the commit
> > message.
> 
> Ah, that's my mistake then, I didn't realize the behavior was
> different in the current implementation. I just figured some packages
> might not have dev/doc/dbg, so it should be nonfatal to miss them. We
> may also need to make sure rpm/deb both handle attemptonly properly,
> as upstream's did not. I'll add this to the commit message, unless you
> think they shouldn't be optional? I'm inclined to prefer it this way,
> as, iirc, the depchain stuff uses recommends rather than depends.
> (though i may be remembering wrong?)

I just checked and all the rootfs package backends have code which looks
like it makes that work. I'm fine with the behaviour change as long as
we document it.

FWIW, I merged the other patches in the series since they were not
directly related.

Cheers,

Richard

Re: [PATCH 4/5] image.bbclass: switch to OE's IMAGE_FEATURES

[Chris Larson]

On Fri, May 20, 2011 at 11:29 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> Yes please :)
>
>> >> --- a/meta/classes/image.bbclass
>> >> +++ b/meta/classes/image.bbclass
>> >> @@ -11,8 +11,45 @@ INHIBIT_DEFAULT_DEPS = "1"
>> >>
>> >>  # "export IMAGE_BASENAME" not supported at this time
>> >>  IMAGE_BASENAME[export] = "1"
>> >> -export PACKAGE_INSTALL ?= "${IMAGE_INSTALL}"
>> >> -PACKAGE_INSTALL_ATTEMPTONLY ?= ""
>> >> +
>> >> +PACKAGE_INSTALL = "${@' '.join(oe.packagegroup.required_packages('${IMAGE_FEATURES}'.split(), d))}"
>> >> +PACKAGE_INSTALL_ATTEMPTONLY = "${@' '.join(oe.packagegroup.optional_packages('${IMAGE_FEATURES}'.split(), d))}"
>> >> +RDEPENDS += "${@' '.join(oe.packagegroup.active_packages('${IMAGE_FEATURES}'.split(), d))}"
>> >
>> > I also noticed this patch changes things so PACKAGE_INSTALL_ATTEMPTONLY
>> > is used for the dev/doc/dbg packages. I'm not sure its a major issue but
>> > it is a change in behaviour and I'd have expected it in the commit
>> > message.
>>
>> Ah, that's my mistake then, I didn't realize the behavior was
>> different in the current implementation. I just figured some packages
>> might not have dev/doc/dbg, so it should be nonfatal to miss them. We
>> may also need to make sure rpm/deb both handle attemptonly properly,
>> as upstream's did not. I'll add this to the commit message, unless you
>> think they shouldn't be optional? I'm inclined to prefer it this way,
>> as, iirc, the depchain stuff uses recommends rather than depends.
>> (though i may be remembering wrong?)
>
> I just checked and all the rootfs package backends have code which looks
> like it makes that work. I'm fine with the behaviour change as long as
> we document it.
>
> FWIW, I merged the other patches in the series since they were not
> directly related.

Thanks, I'll resend this commit shortly.  Aside: can anyone think of a
better name than PACKAGE_GROUP for defining named groups of packages?
I rather dislike it, but can't think of anything better. PACKAGES is
clearly already used, and isn't sufficiently explicit.  PACKAGESET or
something could work, but I don't know that it's an improvement..
maybe this is best, but I thought I'd ask. I'm terrible at naming just
about everything, projects included (*cough* OE *cough*) :)
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics