[oe-core][PATCH v2 0/7] Add new packagefeed recipe class

[oe-core][PATCH v2 0/7] Add new packagefeed recipe class

[Charlie Johnston]

Currently, the only way to build a feed natively in OE is to build all the desired packages and then manually run bitbake package-index. This approach has a few drawbacks:
- The index creation methods ONLY work on the package deploy directory. If there are packages that are not meant to be in the feed in the deploy directory, they will be included in the package index. Also, multiple feeds cannot be built in one command due to this limitation.
- If a package feed depends on another package feed being side-by-side to it (that is, if packages in Feed A depend on packages in Feed B and users of Feed B are required to use Feed A) a user would have to manually remove duplicate packages from the deploy directory before making Feed B's index.

To address this, this patch set creates a new packagefeed.bbclass that enables the creation of feeds in a new deploy directory location for feeds. The patch set updates existing logic in the package_manager class that was previously used to create a temporary feed when building a rootfs. That logic is then used to create a feed and exclude any packages which might be included in any provided side-by-side feeds. 

Note that currently the class does not make use of sstate. Since the individual packages are cached, there did not seem to be anything to be gained from the extra effort required to cache the feed. Caching the whole feed was not space efficient, and reworking package index creation to be compatible with caching didn't seem worth the effort given that operation is fairly inexpensive.

I've tested this in oe-core by creating a few sample recipes and running with each combination of the PACKAGE_CLASSES variable. The testimage cases for dnf were also updated to use the new packagefeed creation mechanism and several images were run with testimage to confirm proper behavior.

Changes since v1:
- Added example of implementation to bbclass comments.
- Updated testimage to use do_packagefeed for dnf tests and ran testimage on several images with and without dnf tests.

Changes since RFC v2: 
- maps used to look up configurations for each package class have been updated to use nested dicts to clarify what each item is.
- DEPLOY_DIR_FEED_<pkg type> definitions now include the ${PN} in the paths instead of relying on adding it in the bbclass.

Regards,
Charlie Johnston

[oe-core][PATCH v2 1/7] bitbake.conf: Add new DEPLOY_DIR_FEED variables.

[Charlie Johnston]

This change adds a new variable that defines where
feeds should be created when building a packagefeed.
A feed location for each package type is also added
to allow multiple package type feeds to be created
in parallel.

The location for feeds is ${DEPLOY_DIR}/feeds/ and
each package type will further drill down to
${DEPLOY_DIR_FEED}/<pkg_type>/${PN} where PN is the
name of the feed.

Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
---
 meta/conf/bitbake.conf | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 8daaaad615..381ca7f3e8 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -450,6 +450,10 @@ DEPLOY_DIR_RPM = "${DEPLOY_DIR}/rpm"
 DEPLOY_DIR_DEB = "${DEPLOY_DIR}/deb"
 DEPLOY_DIR_IMAGE ?= "${DEPLOY_DIR}/images/${MACHINE}"
 DEPLOY_DIR_TOOLS = "${DEPLOY_DIR}/tools"
+DEPLOY_DIR_FEED ?= "${DEPLOY_DIR}/feeds/"
+DEPLOY_DIR_FEED_IPK = "${DEPLOY_DIR_FEED}/ipk/${PN}"
+DEPLOY_DIR_FEED_RPM = "${DEPLOY_DIR_FEED}/rpm/${PN}"
+DEPLOY_DIR_FEED_DEB = "${DEPLOY_DIR_FEED}/deb/${PN}"
 
 PKGDATA_DIR = "${TMPDIR}/pkgdata/${MACHINE}"
 PKGDATA_DIR_SDK = "${TMPDIR}/pkgdata/${SDK_SYS}"
-- 
2.41.0

[oe-core][PATCH v2 2/7] package_manager: Add feed support to generate_index_files.

[Charlie Johnston]

Currently, the generate_index_files function only handles
the creation of index files in the DEPLOY_DIR_<PKG_TYPE>
directories. This change adds an optional isFeed input
that will instead point the index generation at a package
specific feed directory. If no feedname is specified,
the original behavior persists and the index is created
in the DEPLOY_DIR_<PKG_TYPE> directory.

The directory for index creation when isFeed is true will
be DEPLOY_DIR_FEED_<PKG_TYPE>.

Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
---
 meta/lib/oe/package_manager/__init__.py | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)

diff --git a/meta/lib/oe/package_manager/__init__.py b/meta/lib/oe/package_manager/__init__.py
index 0c313190cf..af4254caf5 100644
--- a/meta/lib/oe/package_manager/__init__.py
+++ b/meta/lib/oe/package_manager/__init__.py
@@ -533,27 +533,29 @@ def create_packages_dir(d, subrepo_dir, deploydir, taskname, filterbydependencie
                         raise
 
 
-def generate_index_files(d):
+def generate_index_files(d, isFeed = False):
     from oe.package_manager.rpm import RpmSubdirIndexer
     from oe.package_manager.ipk import OpkgIndexer
     from oe.package_manager.deb import DpkgIndexer
 
     classes = d.getVar('PACKAGE_CLASSES').replace("package_", "").split()
 
-    indexer_map = {
-        "rpm": (RpmSubdirIndexer, d.getVar('DEPLOY_DIR_RPM')),
-        "ipk": (OpkgIndexer, d.getVar('DEPLOY_DIR_IPK')),
-        "deb": (DpkgIndexer, d.getVar('DEPLOY_DIR_DEB'))
+    pkg_class_map = {
+        "rpm": { 'indexer': RpmSubdirIndexer, 'pkgdir': d.getVar('DEPLOY_DIR_RPM'), 'feedir': d.getVar('DEPLOY_DIR_FEED_RPM')},
+        "ipk": { 'indexer': OpkgIndexer, 'pkgdir': d.getVar('DEPLOY_DIR_IPK'), 'feedir': d.getVar('DEPLOY_DIR_FEED_IPK')},
+        "deb": { 'indexer': DpkgIndexer, 'pkgdir': d.getVar('DEPLOY_DIR_DEB'), 'feedir': d.getVar('DEPLOY_DIR_FEED_DEB')}
     }
 
     result = None
 
     for pkg_class in classes:
-        if not pkg_class in indexer_map:
+        if not pkg_class in pkg_class_map:
             continue
 
-        if os.path.exists(indexer_map[pkg_class][1]):
-            result = indexer_map[pkg_class][0](d, indexer_map[pkg_class][1]).write_index()
+        pkgcfg = pkg_class_map[pkg_class]
+        feedpath = pkgcfg['feedir'] if isFeed else pkgcfg['pkgdir']
+        if os.path.exists(feedpath):
+            result = pkgcfg['indexer'](d, feedpath).write_index()
 
             if result is not None:
                 bb.fatal(result)
-- 
2.41.0

[oe-core][PATCH v2 3/7] package_manager: Add _find_task_pkg_deps helper method.

[Charlie Johnston]

To make the logic from create_feed_dir reusable, this
change splits the logic used to traverse the package
dependencies into a helper function.

Additionally, the logic used to find the initial task was
updated.

Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
---
 meta/lib/oe/package_manager/__init__.py | 51 +++++++++++++------------
 1 file changed, 26 insertions(+), 25 deletions(-)

diff --git a/meta/lib/oe/package_manager/__init__.py b/meta/lib/oe/package_manager/__init__.py
index af4254caf5..10376dd9cd 100644
--- a/meta/lib/oe/package_manager/__init__.py
+++ b/meta/lib/oe/package_manager/__init__.py
@@ -473,31 +473,7 @@ def create_packages_dir(d, subrepo_dir, deploydir, taskname, filterbydependencie
         oe.path.symlink(deploydir, subrepo_dir, True)
         return
 
-    start = None
-    for dep in taskdepdata:
-        data = taskdepdata[dep]
-        if data[1] == mytaskname and data[0] == pn:
-            start = dep
-            break
-    if start is None:
-        bb.fatal("Couldn't find ourself in BB_TASKDEPDATA?")
-    pkgdeps = set()
-    start = [start]
-    seen = set(start)
-    # Support direct dependencies (do_rootfs -> do_package_write_X)
-    # or indirect dependencies within PN (do_populate_sdk_ext -> do_rootfs -> do_package_write_X)
-    while start:
-        next = []
-        for dep2 in start:
-            for dep in taskdepdata[dep2][3]:
-                if taskdepdata[dep][0] != pn:
-                    if "do_" + taskname in dep:
-                        pkgdeps.add(dep)
-                elif dep not in seen:
-                    next.append(dep)
-                    seen.add(dep)
-        start = next
-
+    pkgdeps = _find_task_pkg_deps(pn, taskdepdata, mytaskname, taskname)
     for dep in pkgdeps:
         c = taskdepdata[dep][0]
         manifest, d2 = oe.sstatesig.find_sstate_manifest(c, taskdepdata[dep][2], taskname, d, multilibs)
@@ -533,6 +509,31 @@ def create_packages_dir(d, subrepo_dir, deploydir, taskname, filterbydependencie
                         raise
 
 
+def _find_task_pkg_deps(pn, taskdepdata, mytaskname, taskname):
+    start_task = next((dep for dep, data in taskdepdata.items()
+                  if data[1] == mytaskname and data[0] == pn), None)
+    if start_task is None:
+        bb.fatal("Couldn't find %s:%s in BB_TASKDEPDATA?" % (pn, mytaskname))
+    pkgdeps = set()
+    tasks = [start_task]
+    seen = set(start_task)
+    # Support direct dependencies (do_rootfs -> do_package_write_X)
+    # or indirect dependencies within PN (do_populate_sdk_ext -> do_rootfs -> do_package_write_X)
+    while tasks:
+        new_tasks = []
+        for task in tasks:
+            deps = taskdepdata[task][3]
+            for dep in deps:
+                if taskdepdata[dep][0] != pn:
+                    if "do_" + taskname in dep:
+                        pkgdeps.add(dep)
+                elif dep not in seen:
+                    new_tasks.append(dep)
+                    seen.add(dep)
+        tasks = new_tasks
+    return pkgdeps
+
+
 def generate_index_files(d, isFeed = False):
     from oe.package_manager.rpm import RpmSubdirIndexer
     from oe.package_manager.ipk import OpkgIndexer
-- 
2.41.0

[oe-core][PATCH v2 4/7] package_manager: Add generate_feed_dirs method.

[Charlie Johnston]

Add a generate_feed_dirs method that will call the
create_packages_dir method for each package class supported
in the build environment. The value of the PACKAGE_CLASSES
determines which feed types are built. For each package
type, the new method will determine the proper name for the
package_write_<type> task, the directory to pull finished
packages from, and the feed directory to copy them to.

To support side-by-side feeds or situations where one feed
is dependent on another, the create_packages_dir method has
been updated to support an input called assumeprovidedfeeds
which lists feeds whose packages and dependencies can be
assumed to be provided in a separate feed. Those packages
and dependencies will be excluded the feed directory for
the new feed.

Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
---
 meta/lib/oe/package_manager/__init__.py | 27 ++++++++++++++++++++++++-
 1 file changed, 26 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oe/package_manager/__init__.py b/meta/lib/oe/package_manager/__init__.py
index 10376dd9cd..e87ae6d8d7 100644
--- a/meta/lib/oe/package_manager/__init__.py
+++ b/meta/lib/oe/package_manager/__init__.py
@@ -449,7 +449,23 @@ class PackageManager(object, metaclass=ABCMeta):
             return res
         return _append(uris, base_paths)
 
-def create_packages_dir(d, subrepo_dir, deploydir, taskname, filterbydependencies):
+def generate_feed_dirs(d, assumeprovidedfeeds):
+    classes = d.getVar('PACKAGE_CLASSES').replace("package_", "").split()
+
+    pkg_class_map = {
+        "rpm": {"taskname": "package_write_rpm", "pkgdir": d.getVar('DEPLOY_DIR_RPM'), "feeddir": d.getVar('DEPLOY_DIR_FEED_RPM')},
+        "ipk": {"taskname": "package_write_ipk", "pkgdir": d.getVar('DEPLOY_DIR_IPK'), "feeddir": d.getVar('DEPLOY_DIR_FEED_IPK')},
+        "deb": {"taskname": "package_write_deb", "pkgdir": d.getVar('DEPLOY_DIR_DEB'), "feeddir": d.getVar('DEPLOY_DIR_FEED_DEB')}
+    }
+
+    for pkg_class in classes:
+        if not pkg_class in pkg_class_map:
+            continue
+
+        pkgcfg = pkg_class_map[pkg_class]
+        create_packages_dir(d, pkgcfg['feeddir'], pkgcfg['pkgdir'], pkgcfg['taskname'], True, assumeprovidedfeeds)
+
+def create_packages_dir(d, subrepo_dir, deploydir, taskname, filterbydependencies, assumeprovidedfeeds = None):
     """
     Go through our do_package_write_X dependencies and hardlink the packages we depend
     upon into the repo directory. This prevents us seeing other packages that may
@@ -474,6 +490,15 @@ def create_packages_dir(d, subrepo_dir, deploydir, taskname, filterbydependencie
         return
 
     pkgdeps = _find_task_pkg_deps(pn, taskdepdata, mytaskname, taskname)
+
+    # Find any packages which might already be provided in a separate feed or repo
+    # and remove them to avoid duplicates. This assumes any dependencies of the packages
+    # are already met as well.
+    if assumeprovidedfeeds is not None:
+        for pkg_pn in assumeprovidedfeeds.split():
+            provided_pkgdeps = _find_task_pkg_deps(pkg_pn, taskdepdata, mytaskname, taskname)
+            pkgdeps = pkgdeps.difference(provided_pkgdeps)
+
     for dep in pkgdeps:
         c = taskdepdata[dep][0]
         manifest, d2 = oe.sstatesig.find_sstate_manifest(c, taskdepdata[dep][2], taskname, d, multilibs)
-- 
2.41.0

[oe-core][PATCH v2 5/7] packagefeed.bbclass: Add new bbclass for building feeds.

[Charlie Johnston]

Add a new bbclass that allows building a feed using the
new oe.package_manager class. Additionally, there are
packagefeed_<type> bbclasses to define package type
specific configurations.

The do_packagefeed task currently does no use SSTATE
data and is set to always run via [nostamp] = "1".

The variable FEED_DEPENDS is used to specify feeds that
the packagefeed depends on and will be available
side-by-side. This prevents duplicate packages in the
two feeds.

Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
---
 meta/classes-recipe/packagefeed.bbclass     | 50 +++++++++++++++++++++
 meta/classes-recipe/packagefeed_deb.bbclass |  8 ++++
 meta/classes-recipe/packagefeed_ipk.bbclass |  8 ++++
 meta/classes-recipe/packagefeed_rpm.bbclass |  8 ++++
 4 files changed, 74 insertions(+)
 create mode 100644 meta/classes-recipe/packagefeed.bbclass
 create mode 100644 meta/classes-recipe/packagefeed_deb.bbclass
 create mode 100644 meta/classes-recipe/packagefeed_ipk.bbclass
 create mode 100644 meta/classes-recipe/packagefeed_rpm.bbclass

diff --git a/meta/classes-recipe/packagefeed.bbclass b/meta/classes-recipe/packagefeed.bbclass
new file mode 100644
index 0000000000..de07529464
--- /dev/null
+++ b/meta/classes-recipe/packagefeed.bbclass
@@ -0,0 +1,50 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+# Class for creating package feeds with indexes for a group of packages and their
+# dependencies. 
+#
+# When deploying a package feed or repo for an image such that deployed images can
+# use them, it's not always desireable to include all packages in the default 
+# DEPLOY_DIR_<PKG_TYPE> directory in the feed. This class allows specifying one or
+# more packages (or packagegroups) to define what should be included in a feed and
+# creates a feed in DEPLOY_DIR_FEED with only those packages and dependencies.
+# This allows for creating one or more feeds at once and removes the need to
+# remove unwanted packages manually before an index is created.
+#
+# Example:
+#  inherit packagefeed
+#
+#  RDEPENDS:${PN} += "\
+#    packagegroup-core-buildessential \
+#  "
+#
+# Optional variables:
+#  FEED_DEPENDS:
+#    Specifies one or more feeds that the feed depends on and excludes packages in
+#    those feeds from the feed being built. Used if a feed depends on packages in
+#    the feeds listed and it's safe to assume those feeds will always be present.
+
+PKGFEED_INHERITS = "${@' '.join(['packagefeed_' + x for x in d.getVar('PACKAGE_CLASSES').replace("package_", "").split()])}"
+inherit ${PKGFEED_INHERITS} nopackages
+
+LICENSE ?= "MIT"
+
+# Feeds listed in FEED_DEPENDS and their dependencies will be excluded from the feed.
+# This allows for side-by-side feeds without duplicate packages.
+FEED_DEPENDS ??= ""
+
+fakeroot python do_packagefeed() {
+    from oe.package_manager import generate_feed_dirs, generate_index_files
+
+    generate_feed_dirs(d, d.getVar("FEED_DEPENDS"))
+    generate_index_files(d, isFeed=True)
+}
+addtask packagefeed before do_build
+do_packagefeed[recrdeptask] += "do_package_qa"
+do_packagefeed[nostamp] = "1"
+do_packagefeed[rdepends] += "${@' '.join([x + ':do_packagefeed' for x in d.getVar('FEED_DEPENDS').split()])}"
+do_packagefeed[cleandirs] += "${DEPLOY_DIR_FEED_DEB}/${PN} ${DEPLOY_DIR_FEED_IPK}/${PN} ${DEPLOY_DIR_FEED_RPM}/${PN}"
diff --git a/meta/classes-recipe/packagefeed_deb.bbclass b/meta/classes-recipe/packagefeed_deb.bbclass
new file mode 100644
index 0000000000..2decc70a4f
--- /dev/null
+++ b/meta/classes-recipe/packagefeed_deb.bbclass
@@ -0,0 +1,8 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+do_packagefeed[depends] += "apt-native:do_populate_sysroot"
+do_packagefeed[recrdeptask] += "do_package_write_deb"
diff --git a/meta/classes-recipe/packagefeed_ipk.bbclass b/meta/classes-recipe/packagefeed_ipk.bbclass
new file mode 100644
index 0000000000..89d296200b
--- /dev/null
+++ b/meta/classes-recipe/packagefeed_ipk.bbclass
@@ -0,0 +1,8 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+do_packagefeed[depends] += "opkg-native:do_populate_sysroot opkg-utils-native:do_populate_sysroot"
+do_packagefeed[recrdeptask] += "do_package_write_ipk"
diff --git a/meta/classes-recipe/packagefeed_rpm.bbclass b/meta/classes-recipe/packagefeed_rpm.bbclass
new file mode 100644
index 0000000000..8ce37cc855
--- /dev/null
+++ b/meta/classes-recipe/packagefeed_rpm.bbclass
@@ -0,0 +1,8 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+do_packagefeed[depends] += "createrepo-c-native:do_populate_sysroot"
+do_packagefeed[recrdeptask] += "do_package_write_rpm"
-- 
2.41.0

[oe-core][PATCH v2 6/7] packagefeed.bbclass: Add cleanfunc for cleaning feeds.

[Charlie Johnston]

Since the packagefeed build logic does not use sstate,
the deploy directories will not be cleaned by a do_clean
or similar commands. This change adds a function to wipe
all feed deploy directories for the given feed when a
clean command is run. That is, regardless of the value
of PACKAGE_CLASSES, all DEPLOY_DIR_FEED_<PKG_TYPE>
directories will be cleaned.

Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
---
 meta/classes-recipe/packagefeed.bbclass | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/meta/classes-recipe/packagefeed.bbclass b/meta/classes-recipe/packagefeed.bbclass
index de07529464..0bd37790eb 100644
--- a/meta/classes-recipe/packagefeed.bbclass
+++ b/meta/classes-recipe/packagefeed.bbclass
@@ -47,4 +47,16 @@ addtask packagefeed before do_build
 do_packagefeed[recrdeptask] += "do_package_qa"
 do_packagefeed[nostamp] = "1"
 do_packagefeed[rdepends] += "${@' '.join([x + ':do_packagefeed' for x in d.getVar('FEED_DEPENDS').split()])}"
-do_packagefeed[cleandirs] += "${DEPLOY_DIR_FEED_DEB}/${PN} ${DEPLOY_DIR_FEED_IPK}/${PN} ${DEPLOY_DIR_FEED_RPM}/${PN}"
+
+CLEANFUNCS += "packagefeed_clean"
+
+python packagefeed_clean() {
+    bb.note("Cleaning feed directories for %s" % d.getVar('PN'))
+
+    deploy_dirs = [d.getVar("DEPLOY_DIR_FEED_DEB"),
+                   d.getVar("DEPLOY_DIR_FEED_IPK"),
+                   d.getVar("DEPLOY_DIR_FEED_RPM")]
+
+    for dir in deploy_dirs:
+        oe.path.remove(dir)
+}
-- 
2.41.0

[oe-core][PATCH v2 7/7] testimage.bbclass: Update rpm dnf tests to use do_packagefeed.

[Charlie Johnston]

To test the new packagefeed.bbclass, the testimage case for
testing on rpm images has been updated to use a newly added
packagefeed (packagefeed-core-rpmtest) instead
of manually moving the feeds from DEPLOY_DIR_RPM, removing
unwanted packages, and building the indexes manually. The
new logic creates the feed in DEPLOY_DIR_FEED_RPM via
building the packagefeed and links to it from the location
expected for the tests to run.

The resulting feed is technically bigger than before but is
still smaller than creating a feed with all packages present
in DEPLOY_DIR_RPM without the manual removal step.

These changes were tested against testimage runs of the
following images:
- core-image-minimal
- core-image-full-cmdline
- core-image-sato
- core-image-sato-sdk

Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
---
 meta/classes-recipe/testimage.bbclass         | 57 +++----------------
 .../packagefeeds/packagefeed-core-rpmtest.bb  | 14 +++++
 2 files changed, 22 insertions(+), 49 deletions(-)
 create mode 100644 meta/recipes-core/packagefeeds/packagefeed-core-rpmtest.bb

diff --git a/meta/classes-recipe/testimage.bbclass b/meta/classes-recipe/testimage.bbclass
index e3068348ff..d862e6d98e 100644
--- a/meta/classes-recipe/testimage.bbclass
+++ b/meta/classes-recipe/testimage.bbclass
@@ -101,6 +101,7 @@ TESTIMAGEDEPENDS += "${@bb.utils.contains('IMAGE_PKGTYPE', 'rpm', 'dnf-native:do
 TESTIMAGEDEPENDS += "${@bb.utils.contains('IMAGE_PKGTYPE', 'rpm', 'createrepo-c-native:do_populate_sysroot', '', d)}"
 TESTIMAGEDEPENDS += "${@bb.utils.contains('IMAGE_PKGTYPE', 'ipk', 'opkg-utils-native:do_populate_sysroot package-index:do_package_index', '', d)}"
 TESTIMAGEDEPENDS += "${@bb.utils.contains('IMAGE_PKGTYPE', 'deb', 'apt-native:do_populate_sysroot  package-index:do_package_index', '', d)}"
+TESTIMAGEDEPENDS += "${@oe.utils.ifelse(d.getVar('IMAGE_PKGTYPE') == 'rpm' and ('dnf' in d.getVar('TEST_SUITES') or 'auto' in d.getVar('TEST_SUITES')), 'packagefeed-core-rpmtest:do_packagefeed', '')}"
 
 TESTIMAGELOCK = "${TMPDIR}/testimage.lock"
 TESTIMAGELOCK:qemuall = ""
@@ -251,7 +252,7 @@ def testimage_main(d):
 
     if (d.getVar('IMAGE_PKGTYPE') == 'rpm'
        and ('dnf' in d.getVar('TEST_SUITES') or 'auto' in d.getVar('TEST_SUITES'))):
-        create_rpm_index(d)
+        copy_rpm_repos(d)
 
     logger = make_logger_bitbake_compatible(logging.getLogger("BitBake"))
     pn = d.getVar("PN")
@@ -459,55 +460,13 @@ def get_runtime_paths(d):
             paths.append(path)
     return paths
 
-def create_index(arg):
-    import subprocess
+def copy_rpm_repos(d):
+    rpm_dir = os.path.join(d.getVar('DEPLOY_DIR_FEED'), 'rpm', 'packagefeed-core-rpmtest')
+    idx_path = os.path.join(d.getVar('WORKDIR'), 'oe-testimage-repo')
 
-    index_cmd = arg
-    try:
-        bb.note("Executing '%s' ..." % index_cmd)
-        result = subprocess.check_output(index_cmd,
-                                        stderr=subprocess.STDOUT,
-                                        shell=True)
-        result = result.decode('utf-8')
-    except subprocess.CalledProcessError as e:
-        return("Index creation command '%s' failed with return code "
-               '%d:\n%s' % (e.cmd, e.returncode, e.output.decode("utf-8")))
-    if result:
-        bb.note(result)
-    return None
-
-def create_rpm_index(d):
-    import glob
-    # Index RPMs
-    rpm_createrepo = bb.utils.which(os.getenv('PATH'), "createrepo_c")
-    index_cmds = []
-    archs = (d.getVar('ALL_MULTILIB_PACKAGE_ARCHS') or '').replace('-', '_')
-
-    for arch in archs.split():
-        rpm_dir = os.path.join(d.getVar('DEPLOY_DIR_RPM'), arch)
-        idx_path = os.path.join(d.getVar('WORKDIR'), 'oe-testimage-repo', arch)
-
-        if not os.path.isdir(rpm_dir):
-            continue
-
-        lockfilename = os.path.join(d.getVar('DEPLOY_DIR_RPM'), 'rpm.lock')
-        lf = bb.utils.lockfile(lockfilename, False)
-        oe.path.copyhardlinktree(rpm_dir, idx_path)
-        # Full indexes overload a 256MB image so reduce the number of rpms
-        # in the feed by filtering to specific packages needed by the tests.
-        package_list = glob.glob(idx_path + "*/*.rpm")
-
-        for pkg in package_list:
-            if not os.path.basename(pkg).startswith(("dnf-test-", "busybox", "update-alternatives", "libc6", "musl")):
-                bb.utils.remove(pkg)
-
-        bb.utils.unlockfile(lf)
-        cmd = '%s --update -q %s' % (rpm_createrepo, idx_path)
-
-        # Create repodata
-        result = create_index(cmd)
-        if result:
-            bb.fatal('%s' % ('\n'.join(result)))
+    oe.path.copyhardlinktree(rpm_dir, idx_path)
+    # Test cases assume only arch directories and not a top-level repodata next to them.
+    oe.path.remove("%s/repodata" % idx_path)
 
 def package_extraction(d, test_suites):
     from oeqa.utils.package_manager import find_packages_to_extract
diff --git a/meta/recipes-core/packagefeeds/packagefeed-core-rpmtest.bb b/meta/recipes-core/packagefeeds/packagefeed-core-rpmtest.bb
new file mode 100644
index 0000000000..9cb0b933b9
--- /dev/null
+++ b/meta/recipes-core/packagefeeds/packagefeed-core-rpmtest.bb
@@ -0,0 +1,14 @@
+#
+# Package feed containing the packages needed for rpm based testimages.
+#
+
+inherit packagefeed
+
+RDEPENDS:${PN} += " \
+    busybox \
+    update-alternatives \
+    glibc \
+    dnf \
+"
+
+SKIP_RECIPE[packagefeed-core-rpmtest] ?= "${@bb.utils.contains('PACKAGE_CLASSES', 'package_rpm', '', 'is intended only for tests where PACKAGE_CLASSES includes package_rpm.', d)}"
-- 
2.41.0

Re: [oe-core][PATCH v2 7/7] testimage.bbclass: Update rpm dnf tests to use do_packagefeed.

[Alexandre Belloni]

Hello,

On 16/08/2023 15:08:37-0500, Charlie Johnston wrote:
> To test the new packagefeed.bbclass, the testimage case for
> testing on rpm images has been updated to use a newly added
> packagefeed (packagefeed-core-rpmtest) instead
> of manually moving the feeds from DEPLOY_DIR_RPM, removing
> unwanted packages, and building the indexes manually. The
> new logic creates the feed in DEPLOY_DIR_FEED_RPM via
> building the packagefeed and links to it from the location
> expected for the tests to run.
> 
> The resulting feed is technically bigger than before but is
> still smaller than creating a feed with all packages present
> in DEPLOY_DIR_RPM without the manual removal step.
> 
> These changes were tested against testimage runs of the
> following images:
> - core-image-minimal
> - core-image-full-cmdline
> - core-image-sato
> - core-image-sato-sdk
> 
> Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
> ---
>  meta/classes-recipe/testimage.bbclass         | 57 +++----------------
>  .../packagefeeds/packagefeed-core-rpmtest.bb  | 14 +++++
>  2 files changed, 22 insertions(+), 49 deletions(-)
>  create mode 100644 meta/recipes-core/packagefeeds/packagefeed-core-rpmtest.bb

This causes:

WARNING: packagefeed-core-rpmtest-1.0-r0 do_recipe_qa: QA Issue: Recipe packagefeed-core-rpmtest in /home/pokybuild/yocto-worker/a-full/build/meta/recipes-core/packagefeeds/packagefeed-core-rpmtest.bb does not contain a SUMMARY. Please add an entry. [missing-metadata]
WARNING: packagefeed-core-rpmtest-1.0-r0 do_recipe_qa: QA Issue: Recipe packagefeed-core-rpmtest in /home/pokybuild/yocto-worker/a-full/build/meta/recipes-core/packagefeeds/packagefeed-core-rpmtest.bb does not have an assigned maintainer. Please add an entry into meta/conf/distro/include/maintainers.inc. [missing-maintainer]


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [oe-core][PATCH v2 7/7] testimage.bbclass: Update rpm dnf tests to use do_packagefeed.

[Charlie Johnston]

On 8/18/23 03:40, Alexandre Belloni wrote:
> [You don't often get email from alexandre.belloni@bootlin.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> 
> Hello,
> 
> On 16/08/2023 15:08:37-0500, Charlie Johnston wrote:
>> To test the new packagefeed.bbclass, the testimage case for
>> testing on rpm images has been updated to use a newly added
>> packagefeed (packagefeed-core-rpmtest) instead
>> of manually moving the feeds from DEPLOY_DIR_RPM, removing
>> unwanted packages, and building the indexes manually. The
>> new logic creates the feed in DEPLOY_DIR_FEED_RPM via
>> building the packagefeed and links to it from the location
>> expected for the tests to run.
>>
>> The resulting feed is technically bigger than before but is
>> still smaller than creating a feed with all packages present
>> in DEPLOY_DIR_RPM without the manual removal step.
>>
>> These changes were tested against testimage runs of the
>> following images:
>> - core-image-minimal
>> - core-image-full-cmdline
>> - core-image-sato
>> - core-image-sato-sdk
>>
>> Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
>> ---
>>  meta/classes-recipe/testimage.bbclass         | 57 +++----------------
>>  .../packagefeeds/packagefeed-core-rpmtest.bb  | 14 +++++
>>  2 files changed, 22 insertions(+), 49 deletions(-)
>>  create mode 100644 meta/recipes-core/packagefeeds/packagefeed-core-rpmtest.bb
> 
> This causes:
> 
> WARNING: packagefeed-core-rpmtest-1.0-r0 do_recipe_qa: QA Issue: Recipe packagefeed-core-rpmtest in /home/pokybuild/yocto-worker/a-full/build/meta/recipes-core/packagefeeds/packagefeed-core-rpmtest.bb does not contain a SUMMARY. Please add an entry. [missing-metadata]
> WARNING: packagefeed-core-rpmtest-1.0-r0 do_recipe_qa: QA Issue: Recipe packagefeed-core-rpmtest in /home/pokybuild/yocto-worker/a-full/build/meta/recipes-core/packagefeeds/packagefeed-core-rpmtest.bb does not have an assigned maintainer. Please add an entry into meta/conf/distro/include/maintainers.inc. [missing-maintainer]
> 
> 
> --
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

Ah, thanks for catching that. Hadn't rebased since the do_recipe_qa task was added.
Fixed and resubmitted.

Thanks,
Charlie Johnston