package_manager: support for signed DEB package feeds

package_manager: support for signed DEB package feeds

[Ferry Toth]

[PATCH v4 1/2] apt: add apt selftest to test signed package feeds
[PATCH v4 2/2] package_manager: fix missing dependency on gnupg when

Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default.
Currently when building images this requirement is worked around by using [allow-insecure=yes] and
equivalently when performing selftest.
    
Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign deb package feeds"
(already in master) enable signed deb package feeds. When called from 
`oe-selftest -r runtime_test.TestImage.test_testimage_apt` this patch adds a runtime test for apt 
derived from the test_testimage_dnf test. It creates a signed deb package feed, runs a qemu 
image to install the key and performs some package management. To be able to install the key
the gnupg package is added to the testimage.

Changes in V4:
 - Add fix to make gnupg-native a dependency else hosttools is used and 
   `oe-selftest -r runtime_test.TestImage.test_testimage_apt` fails on Ubuntu 16.04 used 
   on the autobuilder (Alexandre Belloni)

Changes in V3:
 - When called from `bitbake core-image-sato -c testimage` package feed is unsigned. Auto-detect
   this case and behave as before (Richard Purdie)

Changes in V2:
 - Added runtime test for signed deb package feeds (Richard Purdie)

[PATCH v4 1/2] apt: add apt selftest to test signed package feeds

[Ferry Toth]

From: Ferry Toth <ftoth@exalondelft.nl>

Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default.
Currently when building images this requirement is worked around by using [allow-insecure=yes] and
equivalently when performing selftest.

Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign DEB package feeds"
enable signed DEB package feeds. This patch adds a runtime test for apt derived from the test_testimage_dnf
test. It creates a signed deb package feed, runs a qemu image to install the key and performs some package
management. To be able to install the key the gnupg package is added to the testimage.

Signed-off-by: Ferry Toth <ftoth@exalondelft.nl>
---
 meta/lib/oeqa/runtime/cases/apt.py           | 38 ++++++++++++++++----
 meta/lib/oeqa/selftest/cases/runtime_test.py | 38 ++++++++++++++++++++
 2 files changed, 69 insertions(+), 7 deletions(-)

diff --git a/meta/lib/oeqa/runtime/cases/apt.py b/meta/lib/oeqa/runtime/cases/apt.py
index 53745df93f..574a34f148 100644
--- a/meta/lib/oeqa/runtime/cases/apt.py
+++ b/meta/lib/oeqa/runtime/cases/apt.py
@@ -21,7 +21,7 @@ class AptRepoTest(AptTest):
 
     @classmethod
     def setUpClass(cls):
-        service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], 'all')
+        service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], '')
         cls.repo_server = HTTPService(service_repo,
                                       '0.0.0.0', port=cls.tc.target.server_port,
                                       logger=cls.tc.logger)
@@ -34,20 +34,44 @@ class AptRepoTest(AptTest):
     def setup_source_config_for_package_install(self):
         apt_get_source_server = 'http://%s:%s/' % (self.tc.target.server_ip, self.repo_server.port)
         apt_get_sourceslist_dir = '/etc/apt/'
-        self.target.run('cd %s; echo deb [ allow-insecure=yes ] %s ./ > sources.list' % (apt_get_sourceslist_dir, apt_get_source_server))
+        self.target.run('cd %s; echo deb [ allow-insecure=yes ] %s/all ./ > sources.list' % (apt_get_sourceslist_dir, apt_get_source_server))
+
+    def setup_source_config_for_package_install_signed(self):
+        apt_get_source_server = 'http:\/\/%s:%s' % (self.tc.target.server_ip, self.repo_server.port)
+        apt_get_sourceslist_dir = '/etc/apt/'
+        self.target.run("cd %s; cp sources.list sources.list.bak; sed -i 's/\[trusted=yes\] http:\/\/bogus_ip:bogus_port/%s/g' sources.list" % (apt_get_sourceslist_dir, apt_get_source_server))
 
     def cleanup_source_config_for_package_install(self):
         apt_get_sourceslist_dir = '/etc/apt/'
         self.target.run('cd %s; rm sources.list' % (apt_get_sourceslist_dir))
 
+    def cleanup_source_config_for_package_install_signed(self):
+        apt_get_sourceslist_dir = '/etc/apt/'
+        self.target.run('cd %s; mv sources.list.bak sources.list' % (apt_get_sourceslist_dir))
+
+    def setup_key(self):
+        # the key is found on the target /etc/pki/packagefeed-gpg/
+        # named PACKAGEFEED-GPG-KEY-poky-branch
+        self.target.run('cd %s; apt-key add P*' % ('/etc/pki/packagefeed-gpg'))
+
     @skipIfNotFeature('package-management',
                       'Test requires package-management to be in IMAGE_FEATURES')
     @skipIfNotDataVar('IMAGE_PKGTYPE', 'deb',
                       'DEB is not the primary package manager')
     @OEHasPackage(['apt'])
     def test_apt_install_from_repo(self):
-        self.setup_source_config_for_package_install()
-        self.pkg('update')
-        self.pkg('remove --yes run-postinsts-dev')
-        self.pkg('install --yes --allow-unauthenticated run-postinsts-dev')
-        self.cleanup_source_config_for_package_install()
+        if not self.tc.td.get('PACKAGE_FEED_GPG_NAME'):
+            self.setup_source_config_for_package_install()
+            self.pkg('update')
+            self.pkg('remove --yes run-postinsts-dev')
+            self.pkg('install --yes --allow-unauthenticated run-postinsts-dev')
+            self.cleanup_source_config_for_package_install()
+        else:
+            # when we are here a key has been set to sign the package feed and
+            # public key and gnupg installed on the image by test_testimage_apt
+            self.setup_source_config_for_package_install_signed()
+            self.setup_key()
+            self.pkg('update')
+            self.pkg('install --yes run-postinsts-dev')
+            self.pkg('remove --yes run-postinsts-dev')
+            self.cleanup_source_config_for_package_install_signed()
diff --git a/meta/lib/oeqa/selftest/cases/runtime_test.py b/meta/lib/oeqa/selftest/cases/runtime_test.py
index 2ad89490fc..3ece617cb0 100644
--- a/meta/lib/oeqa/selftest/cases/runtime_test.py
+++ b/meta/lib/oeqa/selftest/cases/runtime_test.py
@@ -162,6 +162,44 @@ class TestImage(OESelftestTestCase):
         bitbake('core-image-full-cmdline socat')
         bitbake('-c testimage core-image-full-cmdline')
 
+    def test_testimage_apt(self):
+        """
+        Summary: Check package feeds functionality for apt
+        Expected: 1. Check that remote package feeds can be accessed
+        Product: oe-core
+        Author: Ferry Toth <fntoth@gmail.com>
+        """
+        if get_bb_var('DISTRO') == 'poky-tiny':
+            self.skipTest('core-image-full-cmdline not buildable for poky-tiny')
+
+        features = 'INHERIT += "testimage"\n'
+        features += 'TEST_SUITES = "ping ssh apt.AptRepoTest.test_apt_install_from_repo"\n'
+        # We don't yet know what the server ip and port will be - they will be patched
+        # in at the start of the on-image test
+        features += 'PACKAGE_FEED_URIS = "http://bogus_ip:bogus_port"\n'
+        features += 'EXTRA_IMAGE_FEATURES += "package-management"\n'
+        features += 'PACKAGE_CLASSES = "package_deb"\n'
+        # We need  gnupg on the target to install keys
+        features += 'IMAGE_INSTALL:append:pn-core-image-full-cmdline = " gnupg"\n'
+
+        bitbake('gnupg-native -c addto_recipe_sysroot')
+
+        # Enable package feed signing
+        self.gpg_home = tempfile.mkdtemp(prefix="oeqa-feed-sign-")
+        self.track_for_cleanup(self.gpg_home)
+        signing_key_dir = os.path.join(self.testlayer_path, 'files', 'signing')
+        runCmd('gpgconf --list-dirs --homedir %s; gpg -v --batch --homedir %s --import %s' % (self.gpg_home, self.gpg_home, os.path.join(signing_key_dir, 'key.secret')), native_sysroot=get_bb_var("RECIPE_SYSROOT_NATIVE", "gnupg-native"), shell=True)
+        features += 'INHERIT += "sign_package_feed"\n'
+        features += 'PACKAGE_FEED_GPG_NAME = "testuser"\n'
+        features += 'PACKAGE_FEED_GPG_PASSPHRASE_FILE = "%s"\n' % os.path.join(signing_key_dir, 'key.passphrase')
+        features += 'GPG_PATH = "%s"\n' % self.gpg_home
+        features += 'PSEUDO_IGNORE_PATHS .= ",%s"\n' % self.gpg_home
+        self.write_config(features)
+
+        # Build core-image-sato and testimage
+        bitbake('core-image-full-cmdline socat')
+        bitbake('-c testimage core-image-full-cmdline')
+
     def test_testimage_virgl_gtk_sdl(self):
         """
         Summary: Check host-assisted accelerate OpenGL functionality in qemu with gtk and SDL frontends
-- 
2.32.0

[PATCH v4 2/2] package_manager: fix missing dependency on gnupg when signing deb package feeds

[Ferry Toth]

From: Ferry Toth <ftoth@exalondelft.nl>

When signing the deb package feed gpg tools are a soft requirement. If gnupg-native
is not declared a dependancy the version from hosttools is used. Unfortunately the
gpg-agent version from Ubuntu 16.04 on the autobuilders is incompatible with the package_index task
and fails during oe-selftest. Fix by making gnupg-native a dependency.

Fixes: 0b4231b5 "package_manager: sign DEB package feeds"
Reported-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Suggested-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Ferry Toth <ftoth@exalondelft.nl>
---
 meta/classes/sign_package_feed.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/sign_package_feed.bbclass b/meta/classes/sign_package_feed.bbclass
index 16bcd147aa..f1504c2225 100644
--- a/meta/classes/sign_package_feed.bbclass
+++ b/meta/classes/sign_package_feed.bbclass
@@ -27,6 +27,7 @@ inherit sanity
 PACKAGE_FEED_SIGN = '1'
 PACKAGE_FEED_GPG_BACKEND ?= 'local'
 PACKAGE_FEED_GPG_SIGNATURE_TYPE ?= 'ASC'
+PACKAGEINDEXDEPS += "gnupg-native:do_populate_sysroot"
 
 # Make feed signing key to be present in rootfs
 FEATURE_PACKAGES_package-management:append = " signing-keys-packagefeed"
-- 
2.32.0

Re: package_manager: support for signed DEB package feeds

[Richard Purdie]

On Wed, 2022-04-13 at 22:37 +0200, Ferry Toth wrote:
> [PATCH v4 1/2] apt: add apt selftest to test signed package feeds
> [PATCH v4 2/2] package_manager: fix missing dependency on gnupg when
> 
> Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default.
> Currently when building images this requirement is worked around by using [allow-insecure=yes] and
> equivalently when performing selftest.
>     
> Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign deb package feeds"
> (already in master) enable signed deb package feeds. When called from 
> `oe-selftest -r runtime_test.TestImage.test_testimage_apt` this patch adds a runtime test for apt 
> derived from the test_testimage_dnf test. It creates a signed deb package feed, runs a qemu 
> image to install the key and performs some package management. To be able to install the key
> the gnupg package is added to the testimage.
> 
> Changes in V4:
>  - Add fix to make gnupg-native a dependency else hosttools is used and 
>    `oe-selftest -r runtime_test.TestImage.test_testimage_apt` fails on Ubuntu 16.04 used 
>    on the autobuilder (Alexandre Belloni)
> 
> Changes in V3:
>  - When called from `bitbake core-image-sato -c testimage` package feed is unsigned. Auto-detect
>    this case and behave as before (Richard Purdie)
> 
> Changes in V2:
>  - Added runtime test for signed deb package feeds (Richard Purdie)

This has now merged, thanks for working through the details with this! The test
should allow the functionality to stay working and is extremely useful/helpful.

Cheers,

Richard

Re: package_manager: support for signed DEB package feeds

[Ferry Toth]

Hi,

Op 19-04-2022 om 15:21 schreef Richard Purdie:
> On Wed, 2022-04-13 at 22:37 +0200, Ferry Toth wrote:
>> [PATCH v4 1/2] apt: add apt selftest to test signed package feeds
>> [PATCH v4 2/2] package_manager: fix missing dependency on gnupg when
>>
>> Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default.
>> Currently when building images this requirement is worked around by using [allow-insecure=yes] and
>> equivalently when performing selftest.
>>      
>> Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign deb package feeds"
>> (already in master) enable signed deb package feeds. When called from
>> `oe-selftest -r runtime_test.TestImage.test_testimage_apt` this patch adds a runtime test for apt
>> derived from the test_testimage_dnf test. It creates a signed deb package feed, runs a qemu
>> image to install the key and performs some package management. To be able to install the key
>> the gnupg package is added to the testimage.
>>
>> Changes in V4:
>>   - Add fix to make gnupg-native a dependency else hosttools is used and
>>     `oe-selftest -r runtime_test.TestImage.test_testimage_apt` fails on Ubuntu 16.04 used
>>     on the autobuilder (Alexandre Belloni)
>>
>> Changes in V3:
>>   - When called from `bitbake core-image-sato -c testimage` package feed is unsigned. Auto-detect
>>     this case and behave as before (Richard Purdie)
>>
>> Changes in V2:
>>   - Added runtime test for signed deb package feeds (Richard Purdie)
> 
> This has now merged, thanks for working through the details with this! The test
> should allow the functionality to stay working and is extremely useful/helpful.
I watched it go through the CI bots and I must say I'm impressed with 
the process. Thanks for guiding me through this and merging.

> Cheers,
> 
> Richard
> 

package_manager: support for signed DEB package feeds

[Ferry Toth]

[PATCH v3 1/1] apt: add apt selftest to test signed package feeds

Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default.
Currently when building images this requirement is worked around by using [allow-insecure=yes] and
equivalently when performing selftest.
    
Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign deb package feeds"
(already in master) enable signed deb package feeds. When called from 
`oe-selftest -r runtime_test.TestImage.test_testimage_apt` this patch adds a runtime test for apt 
derived from the  test_testimage_dnf test. It creates a signed deb package feed, runs a qemu 
image to install the key and performs some package management. To be able to install the key
the gnupg package is added to the testimage.

Changes in V3:
 - When called from `bitbake core-image-sato -c testimage` package feed is unsigned. Auto-detect
   this case and behave as before (Richard Purdie)

Changes in V2:
 - Added runtime test for signed deb package feeds (Richard Purdie)

package_manager: support for signed DEB package feeds

[Ferry Toth]

[PATCH v3 1/1] apt: add apt selftest to test signed package feeds

Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default.
Currently when building images this requirement is worked around by using [allow-insecure=yes] and
equivalently when performing selftest.
    
Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign deb package feeds"
(in master) enabled signed deb package feeds. 

This patch adds a runtime test for apt derived from the test_testimage_dnf test. When called from 
`oe-selftest -r runtime_test.TestImage.test_testimage_apt` it creates a signed deb package feed, 
runs a qemu image to install the key and performs some package management. To be able to install 
the key the gnupg package is added to the testimage.

Changes in V3:
 - Changed to original behavior when called from `bitbake core-image-sato -c testimage` and no 
   signed feed was created (Richard Purdie)

Changes in V2:
 - Added runtime test for signed deb package feeds (Richard Purdie)

package_manager: support for signed DEB package feeds

[Ferry Toth]

[PATCH v2 0/3] package_manager: support for signed DEB package feeds
[PATCH v2 1/3] gpg-sign: Add parameters to gpg signature function
[PATCH v2 2/3] package_manager: sign DEB package feeds
[PATCH v2 3/3] apt: add apt selftest to test signed package feeds

Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default.
Currently when building images this requirement is worked around by using [allow-insecure=yes] and
equivalently when performing selftest.
    
Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign deb package feeds"
enable signed deb package feeds. This patch adds a runtime test for apt derived from the test_testimage_dnf
test. It creates a signed deb package feed, runs a qemu image to install the key and performs some package
management. To be able to install the key the gnupg package is added to the testimage.
    
These patches makes deb a first class citizen as ipk and rpm.

Patches have been in use in meta-intel-edison since Gatesgarth, 
see https://edison-fw.github.io/meta-intel-edison/5.0-Creating-a-deb-repository.html

Changes in V2:
 - Added runtime test for signed deb package feeds (Richard Purdie)