[oe-core][PATCH 0/6] Add new packagefeed recipe class.

[oe-core][PATCH 0/6] Add new packagefeed recipe class.

[Charlie Johnston]

Currently, the only way to build a feed natively in OE is to build all
the desired packages and then manually run bitbake package-index. This
approach has a few drawbacks:
- The index creation methods ONLY work on the package deploy directory.
If there are packages that are not meant to be in the feed in the
deploy directory, they will be included in the package index. Also,
multiple feeds cannot be built in one command due to this limitation.
- If a package feed depends on another package feed being side-by-side
to it (that is, if packages in Feed A depend on packages in Feed B and
users of Feed B are required to use Feed A) a user would have to
manually remove duplicate packages from the deploy directory before
making Feed B's index.

To address this, this patch set creates a new packagefeed.bbclass that
enables the creation of feeds in a new deploy directory location for
feeds. The patch set updates existing logic in the package_manager
class that was previously used to create a temporary feed when building
a rootfs. That logic is then used to create a feed and exclude any
packages which might be included in any provided side-by-side feeds. 

Note that currently the class does not make use of sstate. Since the
individual packages are cached, there did not seem to be anything to be
gained from the extra effort required to cache the feed. Caching the
whole feed was not space efficient, and reworking package index creation
to be compatible with caching didn't seem worth the effort given that
operation is fairly inexpensive.

I've tested this in oe-core by creating a few sample recipes and running
with each combination of the PACKAGE_CLASSES variable.

Changes since RFC v2: 
- maps used to look up configurations for each package class have been
updated to use nested dicts to clarify what each item is.
- DEPLOY_DIR_FEED_<pkg type> definitions now include the ${PN} in the
paths instead of relying on adding it in the bbclass.

Regards,
Charlie Johnston

[oe-core][PATCH 1/6] bitbake.conf: Add new DEPLOY_DIR_FEED variables.

[Charlie Johnston]

This change adds a new variable that defines where
feeds should be created when building a packagefeed.
A feed location for each package type is also added
to allow multiple package type feeds to be created
in parallel.

The location for feeds is ${DEPLOY_DIR}/feeds/ and
each package type will further drill down to
${DEPLOY_DIR_FEED}/<pkg_type>/${PN} where PN is the
name of the feed.

Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
---
 meta/conf/bitbake.conf | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 8daaaad615..381ca7f3e8 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -450,6 +450,10 @@ DEPLOY_DIR_RPM = "${DEPLOY_DIR}/rpm"
 DEPLOY_DIR_DEB = "${DEPLOY_DIR}/deb"
 DEPLOY_DIR_IMAGE ?= "${DEPLOY_DIR}/images/${MACHINE}"
 DEPLOY_DIR_TOOLS = "${DEPLOY_DIR}/tools"
+DEPLOY_DIR_FEED ?= "${DEPLOY_DIR}/feeds/"
+DEPLOY_DIR_FEED_IPK = "${DEPLOY_DIR_FEED}/ipk/${PN}"
+DEPLOY_DIR_FEED_RPM = "${DEPLOY_DIR_FEED}/rpm/${PN}"
+DEPLOY_DIR_FEED_DEB = "${DEPLOY_DIR_FEED}/deb/${PN}"
 
 PKGDATA_DIR = "${TMPDIR}/pkgdata/${MACHINE}"
 PKGDATA_DIR_SDK = "${TMPDIR}/pkgdata/${SDK_SYS}"
-- 
2.41.0

[oe-core][PATCH 2/6] package_manager: Add feed support to generate_index_files.

[Charlie Johnston]

Currently, the generate_index_files function only handles
the creation of index files in the DEPLOY_DIR_<PKG_TYPE>
directories. This change adds an optional isFeed input
that will instead point the index generation at a package
specific feed directory. If no feedname is specified,
the original behavior persists and the index is created
in the DEPLOY_DIR_<PKG_TYPE> directory.

The directory for index creation when isFeed is true will
be DEPLOY_DIR_FEED_<PKG_TYPE>.

Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
---
 meta/lib/oe/package_manager/__init__.py | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)

diff --git a/meta/lib/oe/package_manager/__init__.py b/meta/lib/oe/package_manager/__init__.py
index 0c313190cf..af4254caf5 100644
--- a/meta/lib/oe/package_manager/__init__.py
+++ b/meta/lib/oe/package_manager/__init__.py
@@ -533,27 +533,29 @@ def create_packages_dir(d, subrepo_dir, deploydir, taskname, filterbydependencie
                         raise
 
 
-def generate_index_files(d):
+def generate_index_files(d, isFeed = False):
     from oe.package_manager.rpm import RpmSubdirIndexer
     from oe.package_manager.ipk import OpkgIndexer
     from oe.package_manager.deb import DpkgIndexer
 
     classes = d.getVar('PACKAGE_CLASSES').replace("package_", "").split()
 
-    indexer_map = {
-        "rpm": (RpmSubdirIndexer, d.getVar('DEPLOY_DIR_RPM')),
-        "ipk": (OpkgIndexer, d.getVar('DEPLOY_DIR_IPK')),
-        "deb": (DpkgIndexer, d.getVar('DEPLOY_DIR_DEB'))
+    pkg_class_map = {
+        "rpm": { 'indexer': RpmSubdirIndexer, 'pkgdir': d.getVar('DEPLOY_DIR_RPM'), 'feedir': d.getVar('DEPLOY_DIR_FEED_RPM')},
+        "ipk": { 'indexer': OpkgIndexer, 'pkgdir': d.getVar('DEPLOY_DIR_IPK'), 'feedir': d.getVar('DEPLOY_DIR_FEED_IPK')},
+        "deb": { 'indexer': DpkgIndexer, 'pkgdir': d.getVar('DEPLOY_DIR_DEB'), 'feedir': d.getVar('DEPLOY_DIR_FEED_DEB')}
     }
 
     result = None
 
     for pkg_class in classes:
-        if not pkg_class in indexer_map:
+        if not pkg_class in pkg_class_map:
             continue
 
-        if os.path.exists(indexer_map[pkg_class][1]):
-            result = indexer_map[pkg_class][0](d, indexer_map[pkg_class][1]).write_index()
+        pkgcfg = pkg_class_map[pkg_class]
+        feedpath = pkgcfg['feedir'] if isFeed else pkgcfg['pkgdir']
+        if os.path.exists(feedpath):
+            result = pkgcfg['indexer'](d, feedpath).write_index()
 
             if result is not None:
                 bb.fatal(result)
-- 
2.41.0

[oe-core][PATCH 3/6] package_manager: Add _find_task_pkg_deps helper method.

[Charlie Johnston]

To make the logic from create_feed_dir reusable, this
change splits the logic used to traverse the package
dependencies into a helper function.

Additionally, the logic used to find the initial task was
updated.

Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
---
 meta/lib/oe/package_manager/__init__.py | 51 +++++++++++++------------
 1 file changed, 26 insertions(+), 25 deletions(-)

diff --git a/meta/lib/oe/package_manager/__init__.py b/meta/lib/oe/package_manager/__init__.py
index af4254caf5..10376dd9cd 100644
--- a/meta/lib/oe/package_manager/__init__.py
+++ b/meta/lib/oe/package_manager/__init__.py
@@ -473,31 +473,7 @@ def create_packages_dir(d, subrepo_dir, deploydir, taskname, filterbydependencie
         oe.path.symlink(deploydir, subrepo_dir, True)
         return
 
-    start = None
-    for dep in taskdepdata:
-        data = taskdepdata[dep]
-        if data[1] == mytaskname and data[0] == pn:
-            start = dep
-            break
-    if start is None:
-        bb.fatal("Couldn't find ourself in BB_TASKDEPDATA?")
-    pkgdeps = set()
-    start = [start]
-    seen = set(start)
-    # Support direct dependencies (do_rootfs -> do_package_write_X)
-    # or indirect dependencies within PN (do_populate_sdk_ext -> do_rootfs -> do_package_write_X)
-    while start:
-        next = []
-        for dep2 in start:
-            for dep in taskdepdata[dep2][3]:
-                if taskdepdata[dep][0] != pn:
-                    if "do_" + taskname in dep:
-                        pkgdeps.add(dep)
-                elif dep not in seen:
-                    next.append(dep)
-                    seen.add(dep)
-        start = next
-
+    pkgdeps = _find_task_pkg_deps(pn, taskdepdata, mytaskname, taskname)
     for dep in pkgdeps:
         c = taskdepdata[dep][0]
         manifest, d2 = oe.sstatesig.find_sstate_manifest(c, taskdepdata[dep][2], taskname, d, multilibs)
@@ -533,6 +509,31 @@ def create_packages_dir(d, subrepo_dir, deploydir, taskname, filterbydependencie
                         raise
 
 
+def _find_task_pkg_deps(pn, taskdepdata, mytaskname, taskname):
+    start_task = next((dep for dep, data in taskdepdata.items()
+                  if data[1] == mytaskname and data[0] == pn), None)
+    if start_task is None:
+        bb.fatal("Couldn't find %s:%s in BB_TASKDEPDATA?" % (pn, mytaskname))
+    pkgdeps = set()
+    tasks = [start_task]
+    seen = set(start_task)
+    # Support direct dependencies (do_rootfs -> do_package_write_X)
+    # or indirect dependencies within PN (do_populate_sdk_ext -> do_rootfs -> do_package_write_X)
+    while tasks:
+        new_tasks = []
+        for task in tasks:
+            deps = taskdepdata[task][3]
+            for dep in deps:
+                if taskdepdata[dep][0] != pn:
+                    if "do_" + taskname in dep:
+                        pkgdeps.add(dep)
+                elif dep not in seen:
+                    new_tasks.append(dep)
+                    seen.add(dep)
+        tasks = new_tasks
+    return pkgdeps
+
+
 def generate_index_files(d, isFeed = False):
     from oe.package_manager.rpm import RpmSubdirIndexer
     from oe.package_manager.ipk import OpkgIndexer
-- 
2.41.0

[oe-core][PATCH 4/6] package_manager: Add generate_feed_dirs method.

[Charlie Johnston]

Add a generate_feed_dirs method that will call the
create_packages_dir method for each package class supported
in the build environment. The value of the PACKAGE_CLASSES
determines which feed types are built. For each package
type, the new method will determine the proper name for the
package_write_<type> task, the directory to pull finished
packages from, and the feed directory to copy them to.

To support side-by-side feeds or situations where one feed
is dependent on another, the create_packages_dir method has
been updated to support an input called assumeprovidedfeeds
which lists feeds whose packages and dependencies can be
assumed to be provided in a separate feed. Those packages
and dependencies will be excluded the feed directory for
the new feed.

Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
---
 meta/lib/oe/package_manager/__init__.py | 27 ++++++++++++++++++++++++-
 1 file changed, 26 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oe/package_manager/__init__.py b/meta/lib/oe/package_manager/__init__.py
index 10376dd9cd..e87ae6d8d7 100644
--- a/meta/lib/oe/package_manager/__init__.py
+++ b/meta/lib/oe/package_manager/__init__.py
@@ -449,7 +449,23 @@ class PackageManager(object, metaclass=ABCMeta):
             return res
         return _append(uris, base_paths)
 
-def create_packages_dir(d, subrepo_dir, deploydir, taskname, filterbydependencies):
+def generate_feed_dirs(d, assumeprovidedfeeds):
+    classes = d.getVar('PACKAGE_CLASSES').replace("package_", "").split()
+
+    pkg_class_map = {
+        "rpm": {"taskname": "package_write_rpm", "pkgdir": d.getVar('DEPLOY_DIR_RPM'), "feeddir": d.getVar('DEPLOY_DIR_FEED_RPM')},
+        "ipk": {"taskname": "package_write_ipk", "pkgdir": d.getVar('DEPLOY_DIR_IPK'), "feeddir": d.getVar('DEPLOY_DIR_FEED_IPK')},
+        "deb": {"taskname": "package_write_deb", "pkgdir": d.getVar('DEPLOY_DIR_DEB'), "feeddir": d.getVar('DEPLOY_DIR_FEED_DEB')}
+    }
+
+    for pkg_class in classes:
+        if not pkg_class in pkg_class_map:
+            continue
+
+        pkgcfg = pkg_class_map[pkg_class]
+        create_packages_dir(d, pkgcfg['feeddir'], pkgcfg['pkgdir'], pkgcfg['taskname'], True, assumeprovidedfeeds)
+
+def create_packages_dir(d, subrepo_dir, deploydir, taskname, filterbydependencies, assumeprovidedfeeds = None):
     """
     Go through our do_package_write_X dependencies and hardlink the packages we depend
     upon into the repo directory. This prevents us seeing other packages that may
@@ -474,6 +490,15 @@ def create_packages_dir(d, subrepo_dir, deploydir, taskname, filterbydependencie
         return
 
     pkgdeps = _find_task_pkg_deps(pn, taskdepdata, mytaskname, taskname)
+
+    # Find any packages which might already be provided in a separate feed or repo
+    # and remove them to avoid duplicates. This assumes any dependencies of the packages
+    # are already met as well.
+    if assumeprovidedfeeds is not None:
+        for pkg_pn in assumeprovidedfeeds.split():
+            provided_pkgdeps = _find_task_pkg_deps(pkg_pn, taskdepdata, mytaskname, taskname)
+            pkgdeps = pkgdeps.difference(provided_pkgdeps)
+
     for dep in pkgdeps:
         c = taskdepdata[dep][0]
         manifest, d2 = oe.sstatesig.find_sstate_manifest(c, taskdepdata[dep][2], taskname, d, multilibs)
-- 
2.41.0

[oe-core][PATCH 5/6] packagefeed.bbclass: Add new bbclass for building feeds.

[Charlie Johnston]

Add a new bbclass that allows building a feed using the
new oe.package_manager class. Additionally, there are
packagefeed_<type> bbclasses to define package type
specific configurations.

The do_packagefeed task currently does no use SSTATE
data and is set to always run via [nostamp] = "1".

The variable FEED_DEPENDS is used to specify feeds that
the packagefeed depends on and will be available
side-by-side. This prevents duplicate packages in the
two feeds.

Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
---
 meta/classes-recipe/packagefeed.bbclass     | 26 +++++++++++++++++++++
 meta/classes-recipe/packagefeed_deb.bbclass |  8 +++++++
 meta/classes-recipe/packagefeed_ipk.bbclass |  8 +++++++
 meta/classes-recipe/packagefeed_rpm.bbclass |  8 +++++++
 4 files changed, 50 insertions(+)
 create mode 100644 meta/classes-recipe/packagefeed.bbclass
 create mode 100644 meta/classes-recipe/packagefeed_deb.bbclass
 create mode 100644 meta/classes-recipe/packagefeed_ipk.bbclass
 create mode 100644 meta/classes-recipe/packagefeed_rpm.bbclass

diff --git a/meta/classes-recipe/packagefeed.bbclass b/meta/classes-recipe/packagefeed.bbclass
new file mode 100644
index 0000000000..08df96f662
--- /dev/null
+++ b/meta/classes-recipe/packagefeed.bbclass
@@ -0,0 +1,26 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+PKGFEED_INHERITS = "${@' '.join(['packagefeed_' + x for x in d.getVar('PACKAGE_CLASSES').replace("package_", "").split()])}"
+inherit ${PKGFEED_INHERITS} nopackages
+
+LICENSE ?= "MIT"
+
+# Feeds listed in FEED_DEPENDS and their dependencies will be excluded from the feed.
+# This allows for side-by-side feeds without duplicate packages.
+FEED_DEPENDS ??= ""
+
+fakeroot python do_packagefeed() {
+    from oe.package_manager import generate_feed_dirs, generate_index_files
+
+    generate_feed_dirs(d, d.getVar("FEED_DEPENDS"))
+    generate_index_files(d, isFeed=True)
+}
+addtask packagefeed before do_build
+do_packagefeed[recrdeptask] += "do_package_qa"
+do_packagefeed[nostamp] = "1"
+do_packagefeed[rdepends] += "${@' '.join([x + ':do_packagefeed' for x in d.getVar('FEED_DEPENDS').split()])}"
+do_packagefeed[cleandirs] += "${DEPLOY_DIR_FEED_DEB}/${PN} ${DEPLOY_DIR_FEED_IPK}/${PN} ${DEPLOY_DIR_FEED_RPM}/${PN}"
diff --git a/meta/classes-recipe/packagefeed_deb.bbclass b/meta/classes-recipe/packagefeed_deb.bbclass
new file mode 100644
index 0000000000..2decc70a4f
--- /dev/null
+++ b/meta/classes-recipe/packagefeed_deb.bbclass
@@ -0,0 +1,8 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+do_packagefeed[depends] += "apt-native:do_populate_sysroot"
+do_packagefeed[recrdeptask] += "do_package_write_deb"
diff --git a/meta/classes-recipe/packagefeed_ipk.bbclass b/meta/classes-recipe/packagefeed_ipk.bbclass
new file mode 100644
index 0000000000..89d296200b
--- /dev/null
+++ b/meta/classes-recipe/packagefeed_ipk.bbclass
@@ -0,0 +1,8 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+do_packagefeed[depends] += "opkg-native:do_populate_sysroot opkg-utils-native:do_populate_sysroot"
+do_packagefeed[recrdeptask] += "do_package_write_ipk"
diff --git a/meta/classes-recipe/packagefeed_rpm.bbclass b/meta/classes-recipe/packagefeed_rpm.bbclass
new file mode 100644
index 0000000000..8ce37cc855
--- /dev/null
+++ b/meta/classes-recipe/packagefeed_rpm.bbclass
@@ -0,0 +1,8 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+do_packagefeed[depends] += "createrepo-c-native:do_populate_sysroot"
+do_packagefeed[recrdeptask] += "do_package_write_rpm"
-- 
2.41.0

[oe-core][PATCH 6/6] packagefeed.bbclass: Add cleanfunc for cleaning feeds.

[Charlie Johnston]

Since the packagefeed build logic does not use sstate,
the deploy directories will not be cleaned by a do_clean
or similar commands. This change adds a function to wipe
all feed deploy directories for the given feed when a
clean command is run. That is, regardless of the value
of PACKAGE_CLASSES, all DEPLOY_DIR_FEED_<PKG_TYPE>
directories will be cleaned.

Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
---
 meta/classes-recipe/packagefeed.bbclass | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/meta/classes-recipe/packagefeed.bbclass b/meta/classes-recipe/packagefeed.bbclass
index 08df96f662..2eedce9f74 100644
--- a/meta/classes-recipe/packagefeed.bbclass
+++ b/meta/classes-recipe/packagefeed.bbclass
@@ -23,4 +23,16 @@ addtask packagefeed before do_build
 do_packagefeed[recrdeptask] += "do_package_qa"
 do_packagefeed[nostamp] = "1"
 do_packagefeed[rdepends] += "${@' '.join([x + ':do_packagefeed' for x in d.getVar('FEED_DEPENDS').split()])}"
-do_packagefeed[cleandirs] += "${DEPLOY_DIR_FEED_DEB}/${PN} ${DEPLOY_DIR_FEED_IPK}/${PN} ${DEPLOY_DIR_FEED_RPM}/${PN}"
+
+CLEANFUNCS += "packagefeed_clean"
+
+python packagefeed_clean() {
+    bb.note("Cleaning feed directories for %s" % d.getVar('PN'))
+
+    deploy_dirs = [d.getVar("DEPLOY_DIR_FEED_DEB"),
+                   d.getVar("DEPLOY_DIR_FEED_IPK"),
+                   d.getVar("DEPLOY_DIR_FEED_RPM")]
+
+    for dir in deploy_dirs:
+        oe.path.remove(dir)
+}
-- 
2.41.0

Re: [oe-core][PATCH 5/6] packagefeed.bbclass: Add new bbclass for building feeds.

[Alexander Kanavin]

Same comment as before:
This does need to be tested, and it does need examples. Can you rework
create_rpm_index() from meta/classes-recipe/testimage.bbclass to use
this class instead of package-index?

Alex

On Wed, 2 Aug 2023 at 23:40, Charlie Johnston <charlie.johnston@ni.com> wrote:
>
> Add a new bbclass that allows building a feed using the
> new oe.package_manager class. Additionally, there are
> packagefeed_<type> bbclasses to define package type
> specific configurations.
>
> The do_packagefeed task currently does no use SSTATE
> data and is set to always run via [nostamp] = "1".
>
> The variable FEED_DEPENDS is used to specify feeds that
> the packagefeed depends on and will be available
> side-by-side. This prevents duplicate packages in the
> two feeds.
>
> Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
> ---
>  meta/classes-recipe/packagefeed.bbclass     | 26 +++++++++++++++++++++
>  meta/classes-recipe/packagefeed_deb.bbclass |  8 +++++++
>  meta/classes-recipe/packagefeed_ipk.bbclass |  8 +++++++
>  meta/classes-recipe/packagefeed_rpm.bbclass |  8 +++++++
>  4 files changed, 50 insertions(+)
>  create mode 100644 meta/classes-recipe/packagefeed.bbclass
>  create mode 100644 meta/classes-recipe/packagefeed_deb.bbclass
>  create mode 100644 meta/classes-recipe/packagefeed_ipk.bbclass
>  create mode 100644 meta/classes-recipe/packagefeed_rpm.bbclass
>
> diff --git a/meta/classes-recipe/packagefeed.bbclass b/meta/classes-recipe/packagefeed.bbclass
> new file mode 100644
> index 0000000000..08df96f662
> --- /dev/null
> +++ b/meta/classes-recipe/packagefeed.bbclass
> @@ -0,0 +1,26 @@
> +#
> +# Copyright OpenEmbedded Contributors
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +
> +PKGFEED_INHERITS = "${@' '.join(['packagefeed_' + x for x in d.getVar('PACKAGE_CLASSES').replace("package_", "").split()])}"
> +inherit ${PKGFEED_INHERITS} nopackages
> +
> +LICENSE ?= "MIT"
> +
> +# Feeds listed in FEED_DEPENDS and their dependencies will be excluded from the feed.
> +# This allows for side-by-side feeds without duplicate packages.
> +FEED_DEPENDS ??= ""
> +
> +fakeroot python do_packagefeed() {
> +    from oe.package_manager import generate_feed_dirs, generate_index_files
> +
> +    generate_feed_dirs(d, d.getVar("FEED_DEPENDS"))
> +    generate_index_files(d, isFeed=True)
> +}
> +addtask packagefeed before do_build
> +do_packagefeed[recrdeptask] += "do_package_qa"
> +do_packagefeed[nostamp] = "1"
> +do_packagefeed[rdepends] += "${@' '.join([x + ':do_packagefeed' for x in d.getVar('FEED_DEPENDS').split()])}"
> +do_packagefeed[cleandirs] += "${DEPLOY_DIR_FEED_DEB}/${PN} ${DEPLOY_DIR_FEED_IPK}/${PN} ${DEPLOY_DIR_FEED_RPM}/${PN}"
> diff --git a/meta/classes-recipe/packagefeed_deb.bbclass b/meta/classes-recipe/packagefeed_deb.bbclass
> new file mode 100644
> index 0000000000..2decc70a4f
> --- /dev/null
> +++ b/meta/classes-recipe/packagefeed_deb.bbclass
> @@ -0,0 +1,8 @@
> +#
> +# Copyright OpenEmbedded Contributors
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +
> +do_packagefeed[depends] += "apt-native:do_populate_sysroot"
> +do_packagefeed[recrdeptask] += "do_package_write_deb"
> diff --git a/meta/classes-recipe/packagefeed_ipk.bbclass b/meta/classes-recipe/packagefeed_ipk.bbclass
> new file mode 100644
> index 0000000000..89d296200b
> --- /dev/null
> +++ b/meta/classes-recipe/packagefeed_ipk.bbclass
> @@ -0,0 +1,8 @@
> +#
> +# Copyright OpenEmbedded Contributors
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +
> +do_packagefeed[depends] += "opkg-native:do_populate_sysroot opkg-utils-native:do_populate_sysroot"
> +do_packagefeed[recrdeptask] += "do_package_write_ipk"
> diff --git a/meta/classes-recipe/packagefeed_rpm.bbclass b/meta/classes-recipe/packagefeed_rpm.bbclass
> new file mode 100644
> index 0000000000..8ce37cc855
> --- /dev/null
> +++ b/meta/classes-recipe/packagefeed_rpm.bbclass
> @@ -0,0 +1,8 @@
> +#
> +# Copyright OpenEmbedded Contributors
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +
> +do_packagefeed[depends] += "createrepo-c-native:do_populate_sysroot"
> +do_packagefeed[recrdeptask] += "do_package_write_rpm"
> --
> 2.41.0
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#185439): https://lists.openembedded.org/g/openembedded-core/message/185439
> Mute This Topic: https://lists.openembedded.org/mt/100515098/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [oe-core][PATCH 5/6] packagefeed.bbclass: Add new bbclass for building feeds.

[Ross Burton]

On 2 Aug 2023, at 22:35, Charlie Johnston via lists.openembedded.org <charlie.johnston=ni.com@lists.openembedded.org> wrote:
> +++ b/meta/classes-recipe/packagefeed.bbclass

So this is a recipe-scope class which adds a task to generate a package feed before do_build finishes.  I imagine the usecase would be that you add the inherit to a packagegroup or image that defines the scope of the feed?

Personally I’ve often doing `bitbake package-index` to use the local feed on my target, but this class wouldn’t be useful because I’m not just building a single image and if I want to use the feed it’s often because I want to add a new package that wasn’t in the image before.   All I want is a way to ensure that the deploy directory has an up to date feed in, am I right in thinking this class doesn’t solve my usecase?

Cheers,
Ross

Re: [oe-core][PATCH 5/6] packagefeed.bbclass: Add new bbclass for building feeds.

[Charlie Johnston]

On 8/3/23 13:32, Ross Burton wrote:
> On 2 Aug 2023, at 22:35, Charlie Johnston via lists.openembedded.org <charlie.johnston=ni.com@lists.openembedded.org> wrote:
>> +++ b/meta/classes-recipe/packagefeed.bbclass
> 
> So this is a recipe-scope class which adds a task to generate a package feed before do_build finishes.  I imagine the usecase would be that you add the inherit to a packagegroup or image that defines the scope of the feed?

Currently the use case would be a new recipe defining the feed. That feed could simply contain a packagegroup, or multiple packages or packagegroups. I did consider leaving it up to the user to determine if the recipe were
also a packagegroup or image or similar but I currently have packagefeed inherit nopackages. I could be convinced to go back on that.

> Personally I’ve often doing `bitbake package-index` to use the local feed on my target, but this class wouldn’t be useful because I’m not just building a single image and if I want to use the feed it’s often because I want to add a new package that wasn’t in the image before.   All I want is a way to ensure that the deploy directory has an up to date feed in, am I right in thinking this class doesn’t solve my usecase?
> 
> Cheers,
> Ross
I don't think the changes address your use case as-is. If you used a packagefeed recipe to build all your packages and used that packagefeed's directory as the local feed, it might work but you would have to add the missing package to the packagefeed recipe (or a packagegroup in that recipe).
I'm targeting the use cases where the feeds will be deployed publicly for use with images and (hopefully) for some future quality of life improvements to BUILD_IMAGES_FROM_FEEDS. I'm open to any ideas that might cover other uses. 

Thanks,
Charlie Johnston

Re: [oe-core][PATCH 5/6] packagefeed.bbclass: Add new bbclass for building feeds.

[Charlie Johnston]

Got it. Is running runtime_test.TestImage.test_testimage_dnf sufficient for testing the change? Or are there specific images I should try with testimage?

For the examples, how many sample recipes would be adequate? 1 or 2?

Thanks,
Charlie Johnston

On 8/3/23 02:18, Alexander Kanavin wrote:
> Same comment as before:
> This does need to be tested, and it does need examples. Can you rework
> create_rpm_index() from meta/classes-recipe/testimage.bbclass to use
> this class instead of package-index?
> 
> Alex
> 
> On Wed, 2 Aug 2023 at 23:40, Charlie Johnston <charlie.johnston@ni.com> wrote:
>>
>> Add a new bbclass that allows building a feed using the
>> new oe.package_manager class. Additionally, there are
>> packagefeed_<type> bbclasses to define package type
>> specific configurations.
>>
>> The do_packagefeed task currently does no use SSTATE
>> data and is set to always run via [nostamp] = "1".
>>
>> The variable FEED_DEPENDS is used to specify feeds that
>> the packagefeed depends on and will be available
>> side-by-side. This prevents duplicate packages in the
>> two feeds.
>>
>> Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
>> ---
>>  meta/classes-recipe/packagefeed.bbclass     | 26 +++++++++++++++++++++
>>  meta/classes-recipe/packagefeed_deb.bbclass |  8 +++++++
>>  meta/classes-recipe/packagefeed_ipk.bbclass |  8 +++++++
>>  meta/classes-recipe/packagefeed_rpm.bbclass |  8 +++++++
>>  4 files changed, 50 insertions(+)
>>  create mode 100644 meta/classes-recipe/packagefeed.bbclass
>>  create mode 100644 meta/classes-recipe/packagefeed_deb.bbclass
>>  create mode 100644 meta/classes-recipe/packagefeed_ipk.bbclass
>>  create mode 100644 meta/classes-recipe/packagefeed_rpm.bbclass
>>
>> diff --git a/meta/classes-recipe/packagefeed.bbclass b/meta/classes-recipe/packagefeed.bbclass
>> new file mode 100644
>> index 0000000000..08df96f662
>> --- /dev/null
>> +++ b/meta/classes-recipe/packagefeed.bbclass
>> @@ -0,0 +1,26 @@
>> +#
>> +# Copyright OpenEmbedded Contributors
>> +#
>> +# SPDX-License-Identifier: MIT
>> +#
>> +
>> +PKGFEED_INHERITS = "${@' '.join(['packagefeed_' + x for x in d.getVar('PACKAGE_CLASSES').replace("package_", "").split()])}"
>> +inherit ${PKGFEED_INHERITS} nopackages
>> +
>> +LICENSE ?= "MIT"
>> +
>> +# Feeds listed in FEED_DEPENDS and their dependencies will be excluded from the feed.
>> +# This allows for side-by-side feeds without duplicate packages.
>> +FEED_DEPENDS ??= ""
>> +
>> +fakeroot python do_packagefeed() {
>> +    from oe.package_manager import generate_feed_dirs, generate_index_files
>> +
>> +    generate_feed_dirs(d, d.getVar("FEED_DEPENDS"))
>> +    generate_index_files(d, isFeed=True)
>> +}
>> +addtask packagefeed before do_build
>> +do_packagefeed[recrdeptask] += "do_package_qa"
>> +do_packagefeed[nostamp] = "1"
>> +do_packagefeed[rdepends] += "${@' '.join([x + ':do_packagefeed' for x in d.getVar('FEED_DEPENDS').split()])}"
>> +do_packagefeed[cleandirs] += "${DEPLOY_DIR_FEED_DEB}/${PN} ${DEPLOY_DIR_FEED_IPK}/${PN} ${DEPLOY_DIR_FEED_RPM}/${PN}"
>> diff --git a/meta/classes-recipe/packagefeed_deb.bbclass b/meta/classes-recipe/packagefeed_deb.bbclass
>> new file mode 100644
>> index 0000000000..2decc70a4f
>> --- /dev/null
>> +++ b/meta/classes-recipe/packagefeed_deb.bbclass
>> @@ -0,0 +1,8 @@
>> +#
>> +# Copyright OpenEmbedded Contributors
>> +#
>> +# SPDX-License-Identifier: MIT
>> +#
>> +
>> +do_packagefeed[depends] += "apt-native:do_populate_sysroot"
>> +do_packagefeed[recrdeptask] += "do_package_write_deb"
>> diff --git a/meta/classes-recipe/packagefeed_ipk.bbclass b/meta/classes-recipe/packagefeed_ipk.bbclass
>> new file mode 100644
>> index 0000000000..89d296200b
>> --- /dev/null
>> +++ b/meta/classes-recipe/packagefeed_ipk.bbclass
>> @@ -0,0 +1,8 @@
>> +#
>> +# Copyright OpenEmbedded Contributors
>> +#
>> +# SPDX-License-Identifier: MIT
>> +#
>> +
>> +do_packagefeed[depends] += "opkg-native:do_populate_sysroot opkg-utils-native:do_populate_sysroot"
>> +do_packagefeed[recrdeptask] += "do_package_write_ipk"
>> diff --git a/meta/classes-recipe/packagefeed_rpm.bbclass b/meta/classes-recipe/packagefeed_rpm.bbclass
>> new file mode 100644
>> index 0000000000..8ce37cc855
>> --- /dev/null
>> +++ b/meta/classes-recipe/packagefeed_rpm.bbclass
>> @@ -0,0 +1,8 @@
>> +#
>> +# Copyright OpenEmbedded Contributors
>> +#
>> +# SPDX-License-Identifier: MIT
>> +#
>> +
>> +do_packagefeed[depends] += "createrepo-c-native:do_populate_sysroot"
>> +do_packagefeed[recrdeptask] += "do_package_write_rpm"
>> --
>> 2.41.0
>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#185439): https://lists.openembedded.org/g/openembedded-core/message/185439
>> Mute This Topic: https://lists.openembedded.org/mt/100515098/1686489
>> Group Owner: openembedded-core+owner@lists.openembedded.org
>> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>

Re: [oe-core][PATCH 5/6] packagefeed.bbclass: Add new bbclass for building feeds.

[Alexander Kanavin]

On Fri, 4 Aug 2023 at 01:53, Charlie Johnston <charlie.johnston@ni.com> wrote:

> Got it. Is running runtime_test.TestImage.test_testimage_dnf sufficient for testing the change? Or are there specific images I should try with testimage?

core-image-minimal, core-image-full-cmdline, core-image-sato,
core-image-sato-sdk should do.

> For the examples, how many sample recipes would be adequate? 1 or 2?

I think just writing an example or two into the comment at the top of
the class would be ok.

Also, update the documentation, everywhere where package-index is mentioned:
https://git.yoctoproject.org/yocto-docs/about/

Alex