[PATCH 1/3] rpm: add support for remote package feeds via PACKAGE_FEED_URIS variable

[PATCH 1/3] rpm: add support for remote package feeds via PACKAGE_FEED_URIS variable

[Alexander Kanavin]

I've used a previous patch (which was never merged) by
Humberto Ibarra <humberto.ibarra.lopez@intel.com> as a model
for how to do runtime testing of this feature (e.g. we need to boot
an image, run dnf on it, and check that it is indeed able to
access the remote repo over http). Here's his original commit message:

=====
Testing that feeds specified with PACKAGE_FEED_URIS var are set
correctly has two parts. First a build with this var set is required,
and then smart update needs to be issued in the running taget.

The previous is not a common selftest practice because this is a
simple test, but requires building and running a specific image,
which takes a lot of time. testimage is not a good fit either,
since the images tested there do not have the PACKAGE_FEED_URIS
var set.

For this test, the runtime-test module is being used, which is a
selftest module but runs a testimage command. The var and test
environment were set in runtime-perf.py and the actual test is
done in a new testcase added to meta-selftest layer.
=====

[YOCTO #10872]

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
---
 meta-selftest/lib/oeqa/runtime/cases/selftest.py | 35 ++++++++++++++++++++++++
 meta/lib/oe/package_manager.py                   | 12 +++++++-
 meta/lib/oeqa/selftest/runtime-test.py           |  5 ++++
 3 files changed, 51 insertions(+), 1 deletion(-)

diff --git a/meta-selftest/lib/oeqa/runtime/cases/selftest.py b/meta-selftest/lib/oeqa/runtime/cases/selftest.py
index 19de7406236..e872ed85b12 100644
--- a/meta-selftest/lib/oeqa/runtime/cases/selftest.py
+++ b/meta-selftest/lib/oeqa/runtime/cases/selftest.py
@@ -1,5 +1,7 @@
 from oeqa.runtime.case import OERuntimeTestCase
 from oeqa.core.decorator.depends import OETestDepends
+from oeqa.runtime.cases.dnf import DnfTest
+from oeqa.utils.httpserver import HTTPService
 
 class Selftest(OERuntimeTestCase):
 
@@ -29,3 +31,36 @@ class Selftest(OERuntimeTestCase):
 
         (status, output) = self.target.run("socat -V")
         self.assertNotEqual(status, 0, msg="socat is still installed")
+
+
+class DnfSelftest(DnfTest):
+
+    @classmethod
+    def setUpClass(cls):
+        cls.repo_server = HTTPService(os.path.join(cls.tc.td['WORKDIR'], 'oe-rootfs-repo'),
+                                      cls.tc.target.server_ip)
+        cls.repo_server.port = 8080
+        cls.repo_server.start()
+
+    @classmethod
+    def tearDownClass(cls):
+        cls.repo_server.stop()
+
+    @OETestDepends(['ssh.SSHTest.test_ssh'])
+    def test_verify_package_feeds(self):
+        """
+        Summary: Check correct setting of PACKAGE_FEED_URIS var
+        Expected: 1. Feeds were correctly set for dnf
+                  2. Update recovers packages from host's repo
+        Author: Humberto Ibarra <humberto.ibarra.lopez@intel.com>
+        Author: Alexander Kanavin <alexander.kanavin@intel.com>
+        """
+        import re
+
+        output_makecache = self.dnf('makecache')
+        self.assertTrue(re.match(r".*Metadata cache created", output_makecache, re.DOTALL) is not None, msg = "dnf makecache failed: %s" %(output_makecache))
+
+        output_repoinfo = self.dnf('repoinfo')
+        matchobj = re.match(r".*Repo-pkgs\s*:\s*(?P<n_pkgs>[0-9]+)", output_repoinfo, re.DOTALL)
+        self.assertTrue(matchobj is not None, msg = "Could not find the amount of packages in dnf repoinfo output: %s" %(output_repoinfo))
+        self.assertTrue(int(matchobj.group('n_pkgs')) > 0, msg = "Amount of remote packages is not more than zero: %s\n" %(output_repoinfo))
diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index d51609189d0..b016bc32dc5 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -533,10 +533,20 @@ class RpmPM(PackageManager):
         bb.utils.unlockfile(lf)
 
     def insert_feeds_uris(self, feed_uris, feed_base_paths, feed_archs):
+        from urllib.parse import urlparse
+
         if feed_uris == "":
             return
 
-        raise NotImplementedError("Adding remote dnf feeds not yet supported.")
+        bb.utils.mkdirhier(oe.path.join(self.target_rootfs, "etc", "yum.repos.d"))
+        remote_uris = self.construct_uris(feed_uris.split(), feed_base_paths.split())
+        for uri in remote_uris:
+            repo_name = "oe-remote-repo" + "-".join(urlparse(uri).path.split("/"))
+            if feed_archs is not None:
+                repo_uris = [uri + "/" + arch for arch in feed_archs]
+            else:
+                repo_uris = [uri]
+            open(oe.path.join(self.target_rootfs, "etc", "yum.repos.d", repo_name + ".repo"), 'w').write("[%s]\nbaseurl=%s\n" % (repo_name, " ".join(repo_uris)))
 
     def _prepare_pkg_transaction(self):
         os.environ['D'] = self.target_rootfs
diff --git a/meta/lib/oeqa/selftest/runtime-test.py b/meta/lib/oeqa/selftest/runtime-test.py
index e8b483d7f8b..d23cb7cf46f 100644
--- a/meta/lib/oeqa/selftest/runtime-test.py
+++ b/meta/lib/oeqa/selftest/runtime-test.py
@@ -108,14 +108,19 @@ class TestImage(oeSelfTest):
         Summary: Check install packages functionality for testimage/testexport.
         Expected: 1. Import tests from a directory other than meta.
                   2. Check install/uninstall of socat.
+                  3. Check that remote package feeds can be accessed
         Product: oe-core
         Author: Mariano Lopez <mariano.lopez@intel.com>
+        Author: Alexander Kanavin <alexander.kanavin@intel.com>
         """
         if get_bb_var('DISTRO') == 'poky-tiny':
             self.skipTest('core-image-full-cmdline not buildable for poky-tiny')
 
         features = 'INHERIT += "testimage"\n'
         features += 'TEST_SUITES = "ping ssh selftest"\n'
+        features += 'PACKAGE_FEED_URIS = "http://192.168.7.1:8080"\n'
+        features += 'EXTRA_IMAGE_FEATURES += "package-management"\n'
+        features += 'PACKAGE_CLASSES = "package_rpm"'
         self.write_config(features)
 
         # Build core-image-sato and testimage
-- 
2.11.0

[PATCH 2/3] lib/oe/package_manager: import rpm signing key to rpmdb

[Alexander Kanavin]

From: Markus Lehtonen <markus.lehtonen@linux.intel.com>

Import the gpg key used in rpm signing into rpmdb. This makes it
possible again to create images when rpm signing is enabled.

Also, instruct dnf to enforce signature check if rpm signing is enabled.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
---
 meta/lib/oe/package_manager.py | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index b016bc32dc5..eeb4c76071a 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -520,7 +520,14 @@ class RpmPM(PackageManager):
             open(platformconfdir + "macros", 'a').write("%_prefer_color 7")
 
         if self.d.getVar('RPM_SIGN_PACKAGES') == '1':
-            raise NotImplementedError("Signature verification with rpm not yet supported.")
+            pubkey_path = self.d.getVar('RPM_GPG_PUBKEY')
+            rpm_bin = bb.utils.which(os.getenv('PATH'), "rpmkeys")
+            cmd = [rpm_bin, '--root=%s' % self.target_rootfs, '--import', pubkey_path]
+            try:
+                subprocess.check_output(cmd, stderr=subprocess.STDOUT)
+            except subprocess.CalledProcessError as e:
+                bb.fatal("Importing GPG key failed. Command '%s' "
+                        "returned %d:\n%s" % (' '.join(cmd), e.returncode, e.output.decode("utf-8")))
 
     def create_configs(self):
         self._configure_dnf()
@@ -570,7 +577,8 @@ class RpmPM(PackageManager):
         output = self._invoke_dnf((["--skip-broken"] if attempt_only else []) +
                          (["-x", ",".join(exclude_pkgs)] if len(exclude_pkgs) > 0 else []) +
                          (["--setopt=install_weak_deps=False"] if self.d.getVar('NO_RECOMMENDATIONS') == 1 else []) +
-                         ["--nogpgcheck", "install"] +
+                         (["--nogpgcheck"] if self.d.getVar('RPM_SIGN_PACKAGES') != '1' else ["--setopt=gpgcheck=True"]) +
+                         ["install"] +
                          pkgs)
 
         failed_scriptlets_pkgnames = collections.OrderedDict()
-- 
2.11.0

[PATCH 3/3] selftest/signing.py: verify that images can be created from signed packages

[Alexander Kanavin]

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
---
 meta/lib/oeqa/selftest/signing.py | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/meta/lib/oeqa/selftest/signing.py b/meta/lib/oeqa/selftest/signing.py
index 38b2fca8cf9..8a106aa80f6 100644
--- a/meta/lib/oeqa/selftest/signing.py
+++ b/meta/lib/oeqa/selftest/signing.py
@@ -34,8 +34,10 @@ class Signing(oeSelfTest):
         """
         Summary:     Test that packages can be signed in the package feed
         Expected:    Package should be signed with the correct key
+        Expected:    Images can be created from signed packages
         Product:     oe-core
         Author:      Daniel Istrate <daniel.alexandrux.istrate@intel.com>
+        Author:      Alexander Kanavin <alexander.kanavin@intel.com>
         AutomatedBy: Daniel Istrate <daniel.alexandrux.istrate@intel.com>
         """
         import oe.packagedata
@@ -85,6 +87,12 @@ class Signing(oeSelfTest):
         self.assertIn('rsa sha1 (md5) pgp md5 OK', ret.output, 'Package signed incorrectly.')
         shutil.rmtree(rpmdb)
 
+        #Check that an image can be built from signed packages
+        self.add_command_to_tearDown('bitbake -c clean core-image-minimal')
+        bitbake('-c clean core-image-minimal')
+        bitbake('core-image-minimal')
+
+
     @testcase(1382)
     def test_signing_sstate_archive(self):
         """
-- 
2.11.0

Re: [PATCH 1/3] rpm: add support for remote package feeds via PACKAGE_FEED_URIS variable

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 304 bytes --]

On 14 March 2017 at 17:27, Alexander Kanavin <
alexander.kanavin@linux.intel.com> wrote:

> +        cls.repo_server.port = 8080
>

What if parallel selftests are running at once?  Can this service be
started before the image is created, so the right port can be written into
the image?

Ross

[-- Attachment #2: Type: text/html, Size: 724 bytes --]

Re: [PATCH 1/3] rpm: add support for remote package feeds via PACKAGE_FEED_URIS variable

[Alexander Kanavin]

On 03/15/2017 01:13 PM, Burton, Ross wrote:
>
>     +        cls.repo_server.port = 8080
>
>
> What if parallel selftests are running at once?  Can this service be
> started before the image is created, so the right port can be written
> into the image?

I don't know. I only followed what Humberto wrote in the original patch; 
we need to ask him. I'm also not entirely certain that hardcoding the ip 
address will always work, and did not get an answer to that.

Another, more hackish option is to monkey patch the ephemeral port 
before running dnf into the repo config on the image.

Alex

Re: [PATCH 1/3] rpm: add support for remote package feeds via PACKAGE_FEED_URIS variable

[Patrick Ohly]

On Wed, 2017-03-15 at 13:19 +0200, Alexander Kanavin wrote:
> On 03/15/2017 01:13 PM, Burton, Ross wrote:
> >
> >     +        cls.repo_server.port = 8080
> >
> >
> > What if parallel selftests are running at once?  Can this service be
> > started before the image is created, so the right port can be written
> > into the image?
> 
> I don't know. I only followed what Humberto wrote in the original patch; 
> we need to ask him. I'm also not entirely certain that hardcoding the ip 
> address will always work, and did not get an answer to that.
> 
> Another, more hackish option is to monkey patch the ephemeral port 
> before running dnf into the repo config on the image.

Or perhaps use ssh with port forwarding? Then the host-side port can be
picked randomly, while on the target side it can be fixed.

I was about to suggest qemu slirp networking because that also does port
mapping, but I think ssh port forwarding is better.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCH 1/3] rpm: add support for remote package feeds via PACKAGE_FEED_URIS variable

[Ibarra Lopez, Humberto]

Sorry, missed the previous mail.

I am not sure how to improve the hardcoding there, I don't like it either, but couldn't find an alternative. The thing is that we need the ip of the host as seen from the target to run testimage, but we don’t have the exact value for that one until qemu is up. The issue here is that we need to give this ip to the conf in the PACKAGE_FEED_URIS before building so we can get an image with this var to later use in testimage.

If someone has a better idea to get rid of this hardcoding, it would be great to know.

-----Original Message-----
From: Alexander Kanavin [mailto:alexander.kanavin@linux.intel.com] 
Sent: Wednesday, March 15, 2017 5:19 AM
To: Burton, Ross <ross.burton@intel.com>
Cc: OE-core <openembedded-core@lists.openembedded.org>; Ibarra Lopez, Humberto <humberto.ibarra.lopez@intel.com>
Subject: Re: [OE-core] [PATCH 1/3] rpm: add support for remote package feeds via PACKAGE_FEED_URIS variable

On 03/15/2017 01:13 PM, Burton, Ross wrote:
>
>     +        cls.repo_server.port = 8080
>
>
> What if parallel selftests are running at once?  Can this service be 
> started before the image is created, so the right port can be written 
> into the image?

I don't know. I only followed what Humberto wrote in the original patch; we need to ask him. I'm also not entirely certain that hardcoding the ip address will always work, and did not get an answer to that.

Another, more hackish option is to monkey patch the ephemeral port before running dnf into the repo config on the image.

Alex

Re: [PATCH 1/3] rpm: add support for remote package feeds via PACKAGE_FEED_URIS variable

[Richard Purdie]

On Wed, 2017-03-15 at 15:38 +0000, Ibarra Lopez, Humberto wrote:
> Sorry, missed the previous mail.
> 
> I am not sure how to improve the hardcoding there, I don't like it
> either, but couldn't find an alternative. The thing is that we need
> the ip of the host as seen from the target to run testimage, but we
> don’t have the exact value for that one until qemu is up. The issue
> here is that we need to give this ip to the conf in the
> PACKAGE_FEED_URIS before building so we can get an image with this
> var to later use in testimage.
> 
> If someone has a better idea to get rid of this hardcoding, it would
> be great to know.

We could run a sed command inside the image after it boots but before
triggering the tests to put the correct IP/port number into the file?

Not a perfect test but simple and probably good enough for this?

Cheers,

Richard

Re: [PATCH 1/3] rpm: add support for remote package feeds via PACKAGE_FEED_URIS variable

[Ibarra Lopez, Humberto]

Yeah, it does sound off but we could use it. We could put a mock value in the ip to the PACKAGE_FEED_URIS var to build this image and once it boths update that value with the real one. We would have to check that the mock value was correctly set before updating so we can test the var being correctly set; and then check that the feeds are updated correctly once a valid ip is there. I will look into this.

Humberto

-----Original Message-----
From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org] 
Sent: Wednesday, March 15, 2017 10:08 AM
To: Ibarra Lopez, Humberto <humberto.ibarra.lopez@intel.com>; Alexander Kanavin <alexander.kanavin@linux.intel.com>; Burton, Ross <ross.burton@intel.com>
Cc: OE-core <openembedded-core@lists.openembedded.org>
Subject: Re: [OE-core] [PATCH 1/3] rpm: add support for remote package feeds via PACKAGE_FEED_URIS variable

On Wed, 2017-03-15 at 15:38 +0000, Ibarra Lopez, Humberto wrote:
> Sorry, missed the previous mail.
> 
> I am not sure how to improve the hardcoding there, I don't like it 
> either, but couldn't find an alternative. The thing is that we need 
> the ip of the host as seen from the target to run testimage, but we 
> don’t have the exact value for that one until qemu is up. The issue 
> here is that we need to give this ip to the conf in the 
> PACKAGE_FEED_URIS before building so we can get an image with this var 
> to later use in testimage.
> 
> If someone has a better idea to get rid of this hardcoding, it would 
> be great to know.

We could run a sed command inside the image after it boots but before triggering the tests to put the correct IP/port number into the file?

Not a perfect test but simple and probably good enough for this?

Cheers,

Richard