[oe-core][RFC PATCH v2 0/6] Add new packagefeed recipe class.

[oe-core][RFC PATCH v2 0/6] Add new packagefeed recipe class.

[Charlie Johnston]

Currently, the only way to build a feed natively in OE is to build all
the desired packages and then manually run bitbake package-index. This
approach has a few drawbacks:
- The index creation methods ONLY work on the package deploy directory.
If there are packages that are not meant to be in the feed in the
deploy directory, they will be included in the package index. Also,
multiple feeds cannot be built in one command due to this limitation.
- If a package feed depends on another package feed being side-by-side
to it (that is, if packages in Feed A depend on packages in Feed B and
users of Feed B are required to use Feed A) a user would have to
manually remove duplicate packages from the deploy directory before
making Feed B's index.

To address this, this patch set creates a new packagefeed.bbclass
that enables the creation of feeds in a new deploy directory location
for feeds. The patch set updates existing logic in the package_manager
class that was previously used to create a temporary feed when building
a rootfs. That logic is then used to create a feed and exclude any
packages which might be included in any provided side-by-side feeds. 

Note that currently the class does not make use of sstate. Since the
individual packages are cached, there did not seem to be anything to be
gained from the extra effort required to cache the feed. Caching the
whole feed was not space efficient, and reworking package index creation
to be compatible with caching didn't seem worth the effort given that
operation is fairly inexpensive.

I've tested this in oe-core by creating a few sample recipes and running
with each combination of the PACKAGE_CLASSES variable.

Regards,
Charlie Johnston

[oe-core][RFC PATCH v2 1/6] bitbake.conf: Add new DEPLOY_DIR_FEED variables.

[Charlie Johnston]

This change adds a new variable that defines where
feeds should be created when building a packagefeed.
A feed location for each package type is also added
to allow multiple package type feeds to be created
in parallel.

The location is ${DEPLOY_DIR}/feeds/<pkg_type>

Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
---
 meta/conf/bitbake.conf | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 8daaaad615..fd1738069f 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -450,6 +450,10 @@ DEPLOY_DIR_RPM = "${DEPLOY_DIR}/rpm"
 DEPLOY_DIR_DEB = "${DEPLOY_DIR}/deb"
 DEPLOY_DIR_IMAGE ?= "${DEPLOY_DIR}/images/${MACHINE}"
 DEPLOY_DIR_TOOLS = "${DEPLOY_DIR}/tools"
+DEPLOY_DIR_FEED ?= "${DEPLOY_DIR}/feeds/"
+DEPLOY_DIR_FEED_IPK = "${DEPLOY_DIR_FEED}/ipk"
+DEPLOY_DIR_FEED_RPM = "${DEPLOY_DIR_FEED}/rpm"
+DEPLOY_DIR_FEED_DEB = "${DEPLOY_DIR_FEED}/deb"
 
 PKGDATA_DIR = "${TMPDIR}/pkgdata/${MACHINE}"
 PKGDATA_DIR_SDK = "${TMPDIR}/pkgdata/${SDK_SYS}"
-- 
2.41.0

[oe-core][RFC PATCH v2 2/6] package_manager: Add feed support to generate_index_files.

[Charlie Johnston]

Currently, the generate_index_files function only handles
the creation of index files in the DEPLOY_DIR_<PKG_TYPE>
directories. This change adds an optional feedname input
that will instead point the index generation at a package
specific feed directory. If no feedname is specified,
the original behavior persists.

The directory for index creation will be
${DEPLOY_DIR_FEED_<PKG_TYPE>}/feedname.

Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
---
 meta/lib/oe/package_manager/__init__.py | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/meta/lib/oe/package_manager/__init__.py b/meta/lib/oe/package_manager/__init__.py
index 0c313190cf..0934cda89d 100644
--- a/meta/lib/oe/package_manager/__init__.py
+++ b/meta/lib/oe/package_manager/__init__.py
@@ -533,7 +533,7 @@ def create_packages_dir(d, subrepo_dir, deploydir, taskname, filterbydependencie
                         raise
 
 
-def generate_index_files(d):
+def generate_index_files(d, feedname = None):
     from oe.package_manager.rpm import RpmSubdirIndexer
     from oe.package_manager.ipk import OpkgIndexer
     from oe.package_manager.deb import DpkgIndexer
@@ -541,9 +541,9 @@ def generate_index_files(d):
     classes = d.getVar('PACKAGE_CLASSES').replace("package_", "").split()
 
     indexer_map = {
-        "rpm": (RpmSubdirIndexer, d.getVar('DEPLOY_DIR_RPM')),
-        "ipk": (OpkgIndexer, d.getVar('DEPLOY_DIR_IPK')),
-        "deb": (DpkgIndexer, d.getVar('DEPLOY_DIR_DEB'))
+        "rpm": (RpmSubdirIndexer, d.getVar('DEPLOY_DIR_RPM'), d.expand('${DEPLOY_DIR_FEED_RPM}/%s' % feedname)),
+        "ipk": (OpkgIndexer, d.getVar('DEPLOY_DIR_IPK'), d.expand('${DEPLOY_DIR_FEED_IPK}/%s' % feedname)),
+        "deb": (DpkgIndexer, d.getVar('DEPLOY_DIR_DEB'), d.expand('${DEPLOY_DIR_FEED_DEB}/%s' % feedname))
     }
 
     result = None
@@ -552,8 +552,9 @@ def generate_index_files(d):
         if not pkg_class in indexer_map:
             continue
 
-        if os.path.exists(indexer_map[pkg_class][1]):
-            result = indexer_map[pkg_class][0](d, indexer_map[pkg_class][1]).write_index()
+        feedpath = indexer_map[pkg_class][1] if feedname is None else indexer_map[pkg_class][2]
+        if os.path.exists(feedpath):
+            result = indexer_map[pkg_class][0](d, feedpath).write_index()
 
             if result is not None:
                 bb.fatal(result)
-- 
2.41.0

[oe-core][RFC PATCH v2 3/6] package_manager: Add _find_task_pkg_deps helper method.

[Charlie Johnston]

To make the logic from create_feed_dir reusable, this
change splits the logic used to traverse the package
dependencies into a helper function.

Additionally, the logic used to find the initial
task was updated.

Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
---
 meta/lib/oe/package_manager/__init__.py | 60 ++++++++++++++-----------
 1 file changed, 35 insertions(+), 25 deletions(-)

diff --git a/meta/lib/oe/package_manager/__init__.py b/meta/lib/oe/package_manager/__init__.py
index 0934cda89d..7d040bcaf2 100644
--- a/meta/lib/oe/package_manager/__init__.py
+++ b/meta/lib/oe/package_manager/__init__.py
@@ -449,7 +449,7 @@ class PackageManager(object, metaclass=ABCMeta):
             return res
         return _append(uris, base_paths)
 
-def create_packages_dir(d, subrepo_dir, deploydir, taskname, filterbydependencies):
+def create_packages_dir(d, subrepo_dir, deploydir, taskname, filterbydependencies, assumeprovidedfeeds = None):
     """
     Go through our do_package_write_X dependencies and hardlink the packages we depend
     upon into the repo directory. This prevents us seeing other packages that may
@@ -473,30 +473,15 @@ def create_packages_dir(d, subrepo_dir, deploydir, taskname, filterbydependencie
         oe.path.symlink(deploydir, subrepo_dir, True)
         return
 
-    start = None
-    for dep in taskdepdata:
-        data = taskdepdata[dep]
-        if data[1] == mytaskname and data[0] == pn:
-            start = dep
-            break
-    if start is None:
-        bb.fatal("Couldn't find ourself in BB_TASKDEPDATA?")
-    pkgdeps = set()
-    start = [start]
-    seen = set(start)
-    # Support direct dependencies (do_rootfs -> do_package_write_X)
-    # or indirect dependencies within PN (do_populate_sdk_ext -> do_rootfs -> do_package_write_X)
-    while start:
-        next = []
-        for dep2 in start:
-            for dep in taskdepdata[dep2][3]:
-                if taskdepdata[dep][0] != pn:
-                    if "do_" + taskname in dep:
-                        pkgdeps.add(dep)
-                elif dep not in seen:
-                    next.append(dep)
-                    seen.add(dep)
-        start = next
+    pkgdeps = _find_task_pkg_deps(pn, taskdepdata, mytaskname, taskname)
+
+    # Find any packages which might already be provided in a separate feed or repo
+    # and remove them to avoid duplicates. This assumes any dependencies of the packages
+    # are already met as well.
+    if assumeprovidedfeeds is not None:
+        for pkg_pn in assumeprovidedfeeds.split():
+            provided_pkgdeps = _find_task_pkg_deps(pkg_pn, taskdepdata, mytaskname, taskname)
+            pkgdeps = pkgdeps.difference(provided_pkgdeps)
 
     for dep in pkgdeps:
         c = taskdepdata[dep][0]
@@ -533,6 +518,31 @@ def create_packages_dir(d, subrepo_dir, deploydir, taskname, filterbydependencie
                         raise
 
 
+def _find_task_pkg_deps(pn, taskdepdata, mytaskname, taskname):
+    start_task = next((dep for dep, data in taskdepdata.items()
+                  if data[1] == mytaskname and data[0] == pn), None)
+    if start_task is None:
+        bb.fatal("Couldn't find %s:%s in BB_TASKDEPDATA?" % (pn, mytaskname))
+    pkgdeps = set()
+    tasks = [start_task]
+    seen = set(start_task)
+    # Support direct dependencies (do_rootfs -> do_package_write_X)
+    # or indirect dependencies within PN (do_populate_sdk_ext -> do_rootfs -> do_package_write_X)
+    while tasks:
+        new_tasks = []
+        for task in tasks:
+            deps = taskdepdata[task][3]
+            for dep in deps:
+                if taskdepdata[dep][0] != pn:
+                    if "do_" + taskname in dep:
+                        pkgdeps.add(dep)
+                elif dep not in seen:
+                    new_tasks.append(dep)
+                    seen.add(dep)
+        tasks = new_tasks
+    return pkgdeps
+
+
 def generate_index_files(d, feedname = None):
     from oe.package_manager.rpm import RpmSubdirIndexer
     from oe.package_manager.ipk import OpkgIndexer
-- 
2.41.0

[oe-core][RFC PATCH v2 4/6] package_manager: Add generate_feed_dirs method.

[Charlie Johnston]

Add a generate_feed_dirs method that will call the
create_packages_dir method for each package class
currently supported in the build environment. The
value of the PACKAGE_CLASSES determines which feed
types are built.

This handles knowing which tasks to look for, which
deploy directory to pull packages from, and the
deploy path for the feed to be created at.

Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
---
 meta/lib/oe/package_manager/__init__.py | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/meta/lib/oe/package_manager/__init__.py b/meta/lib/oe/package_manager/__init__.py
index 7d040bcaf2..3daee3b0bc 100644
--- a/meta/lib/oe/package_manager/__init__.py
+++ b/meta/lib/oe/package_manager/__init__.py
@@ -449,6 +449,23 @@ class PackageManager(object, metaclass=ABCMeta):
             return res
         return _append(uris, base_paths)
 
+
+def generate_feed_dirs(d, feedname, assumeprovidedfeeds):
+    classes = d.getVar('PACKAGE_CLASSES').replace("package_", "").split()
+
+    indexer_map = {
+        "rpm": ("package_write_rpm", d.getVar('DEPLOY_DIR_RPM'), d.expand('${DEPLOY_DIR_FEED_RPM}/%s' % feedname)),
+        "ipk": ("package_write_ipk", d.getVar('DEPLOY_DIR_IPK'), d.expand('${DEPLOY_DIR_FEED_IPK}/%s' % feedname)),
+        "deb": ("package_write_deb", d.getVar('DEPLOY_DIR_DEB'), d.expand('${DEPLOY_DIR_FEED_DEB}/%s' % feedname))
+    }
+
+    for pkg_class in classes:
+        if not pkg_class in indexer_map:
+            continue
+
+        create_packages_dir(d, indexer_map[pkg_class][2], indexer_map[pkg_class][1], indexer_map[pkg_class][0], True, assumeprovidedfeeds)
+
+
 def create_packages_dir(d, subrepo_dir, deploydir, taskname, filterbydependencies, assumeprovidedfeeds = None):
     """
     Go through our do_package_write_X dependencies and hardlink the packages we depend
-- 
2.41.0

[oe-core][RFC PATCH v2 5/6] packagefeed.bbclass: Add new bbclass for building feeds.

[Charlie Johnston]

Add a new bbclass that allows building a feed using the
new oe.package_manager class. Additionally, there are
packagefeed_<type> bbclasses to define package type
specific configurations.

The do_packagefeed task currently does no use SSTATE
data and is set to always run via [nostamp] = "1".

The variable FEED_DEPENDS is used to specify feeds that
the packagefeed depends on and will be available
side-by-side. This prevents duplicate packages in the
two feeds.

Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
---
 meta/classes-recipe/packagefeed.bbclass     | 27 +++++++++++++++++++++
 meta/classes-recipe/packagefeed_deb.bbclass |  8 ++++++
 meta/classes-recipe/packagefeed_ipk.bbclass |  8 ++++++
 meta/classes-recipe/packagefeed_rpm.bbclass |  8 ++++++
 4 files changed, 51 insertions(+)
 create mode 100644 meta/classes-recipe/packagefeed.bbclass
 create mode 100644 meta/classes-recipe/packagefeed_deb.bbclass
 create mode 100644 meta/classes-recipe/packagefeed_ipk.bbclass
 create mode 100644 meta/classes-recipe/packagefeed_rpm.bbclass

diff --git a/meta/classes-recipe/packagefeed.bbclass b/meta/classes-recipe/packagefeed.bbclass
new file mode 100644
index 0000000000..04b45e7723
--- /dev/null
+++ b/meta/classes-recipe/packagefeed.bbclass
@@ -0,0 +1,27 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+PKGFEED_INHERITS = "${@' '.join(['packagefeed_' + x for x in d.getVar('PACKAGE_CLASSES').replace("package_", "").split()])}"
+inherit ${PKGFEED_INHERITS} nopackages
+
+LICENSE ?= "MIT"
+
+# Feeds listed in FEED_DEPENDS and their dependencies will be excluded from the feed.
+# This allows for side-by-side feeds without duplicate packages.
+FEED_DEPENDS ??= ""
+
+fakeroot python do_packagefeed() {
+    from oe.package_manager import generate_feed_dirs, generate_index_files
+
+    pn = d.getVar("PN")
+    generate_feed_dirs(d, pn, d.getVar("FEED_DEPENDS"))
+    generate_index_files(d, pn)
+}
+addtask packagefeed before do_build
+do_packagefeed[recrdeptask] += "do_package_qa"
+do_packagefeed[nostamp] = "1"
+do_packagefeed[rdepends] += "${@' '.join([x + ':do_packagefeed' for x in d.getVar('FEED_DEPENDS').split()])}"
+do_packagefeed[cleandirs] += "${DEPLOY_DIR_FEED_DEB}/${PN} ${DEPLOY_DIR_FEED_IPK}/${PN} ${DEPLOY_DIR_FEED_RPM}/${PN}"
diff --git a/meta/classes-recipe/packagefeed_deb.bbclass b/meta/classes-recipe/packagefeed_deb.bbclass
new file mode 100644
index 0000000000..2decc70a4f
--- /dev/null
+++ b/meta/classes-recipe/packagefeed_deb.bbclass
@@ -0,0 +1,8 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+do_packagefeed[depends] += "apt-native:do_populate_sysroot"
+do_packagefeed[recrdeptask] += "do_package_write_deb"
diff --git a/meta/classes-recipe/packagefeed_ipk.bbclass b/meta/classes-recipe/packagefeed_ipk.bbclass
new file mode 100644
index 0000000000..89d296200b
--- /dev/null
+++ b/meta/classes-recipe/packagefeed_ipk.bbclass
@@ -0,0 +1,8 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+do_packagefeed[depends] += "opkg-native:do_populate_sysroot opkg-utils-native:do_populate_sysroot"
+do_packagefeed[recrdeptask] += "do_package_write_ipk"
diff --git a/meta/classes-recipe/packagefeed_rpm.bbclass b/meta/classes-recipe/packagefeed_rpm.bbclass
new file mode 100644
index 0000000000..8ce37cc855
--- /dev/null
+++ b/meta/classes-recipe/packagefeed_rpm.bbclass
@@ -0,0 +1,8 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+do_packagefeed[depends] += "createrepo-c-native:do_populate_sysroot"
+do_packagefeed[recrdeptask] += "do_package_write_rpm"
-- 
2.41.0

[oe-core][RFC PATCH v2 6/6] packagefeed.bbclass: Add cleanfunc for cleaning feeds.

[Charlie Johnston]

Since the packagefeed build logic does not use sstate,
the deploy directories will not be cleaned by a do_clean
or similar commands. This change adds a function to wipe
all feed deploy directories for the given feed when a
clean command is run. That is, regardless of the value
of PACKAGE_CLASSES, all <tmp>/feeds/<pkg_type>/<pn>
directories will be cleaned.

Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
---
 meta/classes-recipe/packagefeed.bbclass | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/meta/classes-recipe/packagefeed.bbclass b/meta/classes-recipe/packagefeed.bbclass
index 04b45e7723..1df09b85c2 100644
--- a/meta/classes-recipe/packagefeed.bbclass
+++ b/meta/classes-recipe/packagefeed.bbclass
@@ -24,4 +24,16 @@ addtask packagefeed before do_build
 do_packagefeed[recrdeptask] += "do_package_qa"
 do_packagefeed[nostamp] = "1"
 do_packagefeed[rdepends] += "${@' '.join([x + ':do_packagefeed' for x in d.getVar('FEED_DEPENDS').split()])}"
-do_packagefeed[cleandirs] += "${DEPLOY_DIR_FEED_DEB}/${PN} ${DEPLOY_DIR_FEED_IPK}/${PN} ${DEPLOY_DIR_FEED_RPM}/${PN}"
+
+CLEANFUNCS += "packagefeed_clean"
+
+python packagefeed_clean() {
+    bb.note("Cleaning feed directories for %s" % d.getVar('PN'))
+
+    deploy_dirs = [d.expand("${DEPLOY_DIR_FEED_DEB}/${PN}"),
+                   d.expand("${DEPLOY_DIR_FEED_IPK}/${PN}"),
+                   d.expand("${DEPLOY_DIR_FEED_RPM}/${PN}")]
+
+    for dir in deploy_dirs:
+        oe.path.remove(dir)
+}
-- 
2.41.0

Re: [oe-core][RFC PATCH v2 2/6] package_manager: Add feed support to generate_index_files.

[Alexander Kanavin]

I think it's better to change indexer_map either to three different
dictionaries, or a dictionary of dictionaries. Putting more than two
things in a tuple (that aren't of the same kind) results in a code
that's hard to read. For example:

feedpath = indexer_map[pkg_class][1] if feedname is None else
indexer_map[pkg_class][2]
if os.path.exists(feedpath):
            result = indexer_map[pkg_class][0](d, feedpath).write_index()

What is '1' and what is '2' and what is '0'? They should be named
properly, and not numbered.

Same applies to patch 4/6.

Alex



On Mon, 31 Jul 2023 at 23:53, Charlie Johnston <charlie.johnston@ni.com> wrote:
>
> Currently, the generate_index_files function only handles
> the creation of index files in the DEPLOY_DIR_<PKG_TYPE>
> directories. This change adds an optional feedname input
> that will instead point the index generation at a package
> specific feed directory. If no feedname is specified,
> the original behavior persists.
>
> The directory for index creation will be
> ${DEPLOY_DIR_FEED_<PKG_TYPE>}/feedname.
>
> Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
> ---
>  meta/lib/oe/package_manager/__init__.py | 13 +++++++------
>  1 file changed, 7 insertions(+), 6 deletions(-)
>
> diff --git a/meta/lib/oe/package_manager/__init__.py b/meta/lib/oe/package_manager/__init__.py
> index 0c313190cf..0934cda89d 100644
> --- a/meta/lib/oe/package_manager/__init__.py
> +++ b/meta/lib/oe/package_manager/__init__.py
> @@ -533,7 +533,7 @@ def create_packages_dir(d, subrepo_dir, deploydir, taskname, filterbydependencie
>                          raise
>
>
> -def generate_index_files(d):
> +def generate_index_files(d, feedname = None):
>      from oe.package_manager.rpm import RpmSubdirIndexer
>      from oe.package_manager.ipk import OpkgIndexer
>      from oe.package_manager.deb import DpkgIndexer
> @@ -541,9 +541,9 @@ def generate_index_files(d):
>      classes = d.getVar('PACKAGE_CLASSES').replace("package_", "").split()
>
>      indexer_map = {
> -        "rpm": (RpmSubdirIndexer, d.getVar('DEPLOY_DIR_RPM')),
> -        "ipk": (OpkgIndexer, d.getVar('DEPLOY_DIR_IPK')),
> -        "deb": (DpkgIndexer, d.getVar('DEPLOY_DIR_DEB'))
> +        "rpm": (RpmSubdirIndexer, d.getVar('DEPLOY_DIR_RPM'), d.expand('${DEPLOY_DIR_FEED_RPM}/%s' % feedname)),
> +        "ipk": (OpkgIndexer, d.getVar('DEPLOY_DIR_IPK'), d.expand('${DEPLOY_DIR_FEED_IPK}/%s' % feedname)),
> +        "deb": (DpkgIndexer, d.getVar('DEPLOY_DIR_DEB'), d.expand('${DEPLOY_DIR_FEED_DEB}/%s' % feedname))
>      }
>
>      result = None
> @@ -552,8 +552,9 @@ def generate_index_files(d):
>          if not pkg_class in indexer_map:
>              continue
>
> -        if os.path.exists(indexer_map[pkg_class][1]):
> -            result = indexer_map[pkg_class][0](d, indexer_map[pkg_class][1]).write_index()
> +        feedpath = indexer_map[pkg_class][1] if feedname is None else indexer_map[pkg_class][2]
> +        if os.path.exists(feedpath):
> +            result = indexer_map[pkg_class][0](d, feedpath).write_index()
>
>              if result is not None:
>                  bb.fatal(result)
> --
> 2.41.0
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#185176): https://lists.openembedded.org/g/openembedded-core/message/185176
> Mute This Topic: https://lists.openembedded.org/mt/100471803/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [oe-core][RFC PATCH v2 5/6] packagefeed.bbclass: Add new bbclass for building feeds.

[Alexander Kanavin]

This does need to be tested, and it does need examples. Can you rework
create_rpm_index() from meta/classes-recipe/testimage.bbclass to use
this class instead of package-index?

Alex

On Mon, 31 Jul 2023 at 23:53, Charlie Johnston <charlie.johnston@ni.com> wrote:
>
> Add a new bbclass that allows building a feed using the
> new oe.package_manager class. Additionally, there are
> packagefeed_<type> bbclasses to define package type
> specific configurations.
>
> The do_packagefeed task currently does no use SSTATE
> data and is set to always run via [nostamp] = "1".
>
> The variable FEED_DEPENDS is used to specify feeds that
> the packagefeed depends on and will be available
> side-by-side. This prevents duplicate packages in the
> two feeds.
>
> Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
> ---
>  meta/classes-recipe/packagefeed.bbclass     | 27 +++++++++++++++++++++
>  meta/classes-recipe/packagefeed_deb.bbclass |  8 ++++++
>  meta/classes-recipe/packagefeed_ipk.bbclass |  8 ++++++
>  meta/classes-recipe/packagefeed_rpm.bbclass |  8 ++++++
>  4 files changed, 51 insertions(+)
>  create mode 100644 meta/classes-recipe/packagefeed.bbclass
>  create mode 100644 meta/classes-recipe/packagefeed_deb.bbclass
>  create mode 100644 meta/classes-recipe/packagefeed_ipk.bbclass
>  create mode 100644 meta/classes-recipe/packagefeed_rpm.bbclass
>
> diff --git a/meta/classes-recipe/packagefeed.bbclass b/meta/classes-recipe/packagefeed.bbclass
> new file mode 100644
> index 0000000000..04b45e7723
> --- /dev/null
> +++ b/meta/classes-recipe/packagefeed.bbclass
> @@ -0,0 +1,27 @@
> +#
> +# Copyright OpenEmbedded Contributors
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +
> +PKGFEED_INHERITS = "${@' '.join(['packagefeed_' + x for x in d.getVar('PACKAGE_CLASSES').replace("package_", "").split()])}"
> +inherit ${PKGFEED_INHERITS} nopackages
> +
> +LICENSE ?= "MIT"
> +
> +# Feeds listed in FEED_DEPENDS and their dependencies will be excluded from the feed.
> +# This allows for side-by-side feeds without duplicate packages.
> +FEED_DEPENDS ??= ""
> +
> +fakeroot python do_packagefeed() {
> +    from oe.package_manager import generate_feed_dirs, generate_index_files
> +
> +    pn = d.getVar("PN")
> +    generate_feed_dirs(d, pn, d.getVar("FEED_DEPENDS"))
> +    generate_index_files(d, pn)
> +}
> +addtask packagefeed before do_build
> +do_packagefeed[recrdeptask] += "do_package_qa"
> +do_packagefeed[nostamp] = "1"
> +do_packagefeed[rdepends] += "${@' '.join([x + ':do_packagefeed' for x in d.getVar('FEED_DEPENDS').split()])}"
> +do_packagefeed[cleandirs] += "${DEPLOY_DIR_FEED_DEB}/${PN} ${DEPLOY_DIR_FEED_IPK}/${PN} ${DEPLOY_DIR_FEED_RPM}/${PN}"
> diff --git a/meta/classes-recipe/packagefeed_deb.bbclass b/meta/classes-recipe/packagefeed_deb.bbclass
> new file mode 100644
> index 0000000000..2decc70a4f
> --- /dev/null
> +++ b/meta/classes-recipe/packagefeed_deb.bbclass
> @@ -0,0 +1,8 @@
> +#
> +# Copyright OpenEmbedded Contributors
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +
> +do_packagefeed[depends] += "apt-native:do_populate_sysroot"
> +do_packagefeed[recrdeptask] += "do_package_write_deb"
> diff --git a/meta/classes-recipe/packagefeed_ipk.bbclass b/meta/classes-recipe/packagefeed_ipk.bbclass
> new file mode 100644
> index 0000000000..89d296200b
> --- /dev/null
> +++ b/meta/classes-recipe/packagefeed_ipk.bbclass
> @@ -0,0 +1,8 @@
> +#
> +# Copyright OpenEmbedded Contributors
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +
> +do_packagefeed[depends] += "opkg-native:do_populate_sysroot opkg-utils-native:do_populate_sysroot"
> +do_packagefeed[recrdeptask] += "do_package_write_ipk"
> diff --git a/meta/classes-recipe/packagefeed_rpm.bbclass b/meta/classes-recipe/packagefeed_rpm.bbclass
> new file mode 100644
> index 0000000000..8ce37cc855
> --- /dev/null
> +++ b/meta/classes-recipe/packagefeed_rpm.bbclass
> @@ -0,0 +1,8 @@
> +#
> +# Copyright OpenEmbedded Contributors
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +
> +do_packagefeed[depends] += "createrepo-c-native:do_populate_sysroot"
> +do_packagefeed[recrdeptask] += "do_package_write_rpm"
> --
> 2.41.0
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#185179): https://lists.openembedded.org/g/openembedded-core/message/185179
> Mute This Topic: https://lists.openembedded.org/mt/100471807/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [oe-core][RFC PATCH v2 1/6] bitbake.conf: Add new DEPLOY_DIR_FEED variables.

[Alexander Kanavin]

As these directories will be written into by multiple recipes that use
the packagefeed class, won't they race and step on each other? Should
there be a ${PN} in these definitions?

Alex

On Mon, 31 Jul 2023 at 23:53, Charlie Johnston <charlie.johnston@ni.com> wrote:
>
> This change adds a new variable that defines where
> feeds should be created when building a packagefeed.
> A feed location for each package type is also added
> to allow multiple package type feeds to be created
> in parallel.
>
> The location is ${DEPLOY_DIR}/feeds/<pkg_type>
>
> Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
> ---
>  meta/conf/bitbake.conf | 4 ++++
>  1 file changed, 4 insertions(+)
>
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 8daaaad615..fd1738069f 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -450,6 +450,10 @@ DEPLOY_DIR_RPM = "${DEPLOY_DIR}/rpm"
>  DEPLOY_DIR_DEB = "${DEPLOY_DIR}/deb"
>  DEPLOY_DIR_IMAGE ?= "${DEPLOY_DIR}/images/${MACHINE}"
>  DEPLOY_DIR_TOOLS = "${DEPLOY_DIR}/tools"
> +DEPLOY_DIR_FEED ?= "${DEPLOY_DIR}/feeds/"
> +DEPLOY_DIR_FEED_IPK = "${DEPLOY_DIR_FEED}/ipk"
> +DEPLOY_DIR_FEED_RPM = "${DEPLOY_DIR_FEED}/rpm"
> +DEPLOY_DIR_FEED_DEB = "${DEPLOY_DIR_FEED}/deb"
>
>  PKGDATA_DIR = "${TMPDIR}/pkgdata/${MACHINE}"
>  PKGDATA_DIR_SDK = "${TMPDIR}/pkgdata/${SDK_SYS}"
> --
> 2.41.0
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#185175): https://lists.openembedded.org/g/openembedded-core/message/185175
> Mute This Topic: https://lists.openembedded.org/mt/100471799/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [oe-core][RFC PATCH v2 1/6] bitbake.conf: Add new DEPLOY_DIR_FEED variables.

[Charlie Johnston]

Ah, good idea. I had the bbclass adding the ${PN} but it definitely
makes more sense here.

Is it worth it to add [lockfiles] for each package type's directory?
Or maybe one at the top level "feeds" location?

Thanks!
Charlie Johnston

On 8/1/23 05:25, Alexander Kanavin wrote:
> As these directories will be written into by multiple recipes that use
> the packagefeed class, won't they race and step on each other? Should
> there be a ${PN} in these definitions?
> 
> Alex
> 
> On Mon, 31 Jul 2023 at 23:53, Charlie Johnston <charlie.johnston@ni.com> wrote:
>>
>> This change adds a new variable that defines where
>> feeds should be created when building a packagefeed.
>> A feed location for each package type is also added
>> to allow multiple package type feeds to be created
>> in parallel.
>>
>> The location is ${DEPLOY_DIR}/feeds/<pkg_type>
>>
>> Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
>> ---
>>  meta/conf/bitbake.conf | 4 ++++
>>  1 file changed, 4 insertions(+)
>>
>> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
>> index 8daaaad615..fd1738069f 100644
>> --- a/meta/conf/bitbake.conf
>> +++ b/meta/conf/bitbake.conf
>> @@ -450,6 +450,10 @@ DEPLOY_DIR_RPM = "${DEPLOY_DIR}/rpm"
>>  DEPLOY_DIR_DEB = "${DEPLOY_DIR}/deb"
>>  DEPLOY_DIR_IMAGE ?= "${DEPLOY_DIR}/images/${MACHINE}"
>>  DEPLOY_DIR_TOOLS = "${DEPLOY_DIR}/tools"
>> +DEPLOY_DIR_FEED ?= "${DEPLOY_DIR}/feeds/"
>> +DEPLOY_DIR_FEED_IPK = "${DEPLOY_DIR_FEED}/ipk"
>> +DEPLOY_DIR_FEED_RPM = "${DEPLOY_DIR_FEED}/rpm"
>> +DEPLOY_DIR_FEED_DEB = "${DEPLOY_DIR_FEED}/deb"
>>
>>  PKGDATA_DIR = "${TMPDIR}/pkgdata/${MACHINE}"
>>  PKGDATA_DIR_SDK = "${TMPDIR}/pkgdata/${SDK_SYS}"
>> --
>> 2.41.0
>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#185175): https://lists.openembedded.org/g/openembedded-core/message/185175
>> Mute This Topic: https://lists.openembedded.org/mt/100471799/1686489
>> Group Owner: openembedded-core+owner@lists.openembedded.org
>> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>

Re: [oe-core][RFC PATCH v2 1/6] bitbake.conf: Add new DEPLOY_DIR_FEED variables.

[Alexander Kanavin]

I don't know about lockfiles, so your call. Investigate how it's used
elsewhere in poky.

Alex

On Tue, 1 Aug 2023 at 18:18, Charlie Johnston <charlie.johnston@ni.com> wrote:
>
> Ah, good idea. I had the bbclass adding the ${PN} but it definitely
> makes more sense here.
>
> Is it worth it to add [lockfiles] for each package type's directory?
> Or maybe one at the top level "feeds" location?
>
> Thanks!
> Charlie Johnston
>
> On 8/1/23 05:25, Alexander Kanavin wrote:
> > As these directories will be written into by multiple recipes that use
> > the packagefeed class, won't they race and step on each other? Should
> > there be a ${PN} in these definitions?
> >
> > Alex
> >
> > On Mon, 31 Jul 2023 at 23:53, Charlie Johnston <charlie.johnston@ni.com> wrote:
> >>
> >> This change adds a new variable that defines where
> >> feeds should be created when building a packagefeed.
> >> A feed location for each package type is also added
> >> to allow multiple package type feeds to be created
> >> in parallel.
> >>
> >> The location is ${DEPLOY_DIR}/feeds/<pkg_type>
> >>
> >> Signed-off-by: Charlie Johnston <charlie.johnston@ni.com>
> >> ---
> >>  meta/conf/bitbake.conf | 4 ++++
> >>  1 file changed, 4 insertions(+)
> >>
> >> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> >> index 8daaaad615..fd1738069f 100644
> >> --- a/meta/conf/bitbake.conf
> >> +++ b/meta/conf/bitbake.conf
> >> @@ -450,6 +450,10 @@ DEPLOY_DIR_RPM = "${DEPLOY_DIR}/rpm"
> >>  DEPLOY_DIR_DEB = "${DEPLOY_DIR}/deb"
> >>  DEPLOY_DIR_IMAGE ?= "${DEPLOY_DIR}/images/${MACHINE}"
> >>  DEPLOY_DIR_TOOLS = "${DEPLOY_DIR}/tools"
> >> +DEPLOY_DIR_FEED ?= "${DEPLOY_DIR}/feeds/"
> >> +DEPLOY_DIR_FEED_IPK = "${DEPLOY_DIR_FEED}/ipk"
> >> +DEPLOY_DIR_FEED_RPM = "${DEPLOY_DIR_FEED}/rpm"
> >> +DEPLOY_DIR_FEED_DEB = "${DEPLOY_DIR_FEED}/deb"
> >>
> >>  PKGDATA_DIR = "${TMPDIR}/pkgdata/${MACHINE}"
> >>  PKGDATA_DIR_SDK = "${TMPDIR}/pkgdata/${SDK_SYS}"
> >> --
> >> 2.41.0
> >>
> >>
> >> -=-=-=-=-=-=-=-=-=-=-=-
> >> Links: You receive all messages sent to this group.
> >> View/Reply Online (#185175): https://lists.openembedded.org/g/openembedded-core/message/185175
> >> Mute This Topic: https://lists.openembedded.org/mt/100471799/1686489
> >> Group Owner: openembedded-core+owner@lists.openembedded.org
> >> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> >> -=-=-=-=-=-=-=-=-=-=-=-
> >>
>