Failure Inheriting rpm_sign

Failure Inheriting rpm_sign

[Chris Trobridge]

I am getting "Exception: OSError: [Errno 7] Argument list too long" for sign_rpm in the do_package_write_rpm tasks for the 
linux-yocto and glibc-locale recipes.

This is building core-image-minimal (and also my own image) with morty (5aa481d) on Fedora 25.

I have enabled the rpm signing with:

INHERIT += " sign_rpm"
RPM_GPG_NAME = "{name}"
RPM_GPG_PASSPHRASE = "{passphrase}"
IMAGE_INSTALL_append = " signing-keys-rpm"

The error message makes some sense in as much as these recipes produce a lot of packages (for example, glibc-locale produces 1791 packages) and the command line in the log is pretty big, although reading around I didn't find a consensus on what the max command line should be.

The code to sign rpms is in meta/lib/oe/gpg_sign.py b/meta/lib/oe/gpg_sign.py and it builds up one command line with all the packages.

I changed the code (patch appended) to sign each rpm in a separate command and the build completed successfully.  The signing operations take a large amount of time so I think this might be a reasonable change but you may have other concerns.

Regards,
Chris

diff --git a/meta/lib/oe/gpg_sign.py b/meta/lib/oe/gpg_sign.py
index 38eb0cb..a386b1f 100644
--- a/meta/lib/oe/gpg_sign.py
+++ b/meta/lib/oe/gpg_sign.py
@@ -29,17 +29,18 @@ class LocalSigner(object):
     def sign_rpms(self, files, keyid, passphrase):
         """Sign RPM files"""
 
-        cmd = self.rpm_bin + " --addsign --define '_gpg_name %s'  " % keyid
-        cmd += "--define '_gpg_passphrase %s' " % passphrase
-        if self.gpg_bin:
-            cmd += "--define '%%__gpg %s' " % self.gpg_bin
-        if self.gpg_path:
-            cmd += "--define '_gpg_path %s' " % self.gpg_path
-        cmd += ' '.join(files)
-
-        status, output = oe.utils.getstatusoutput(cmd)
-        if status:
-            raise bb.build.FuncFailed("Failed to sign RPM packages: %s" % output)
+        for file in files:
+            cmd = self.rpm_bin + " --addsign --define '_gpg_name %s'  " % keyid
+            cmd += "--define '_gpg_passphrase %s' " % passphrase
+            if self.gpg_bin:
+                cmd += "--define '%%__gpg %s' " % self.gpg_bin
+            if self.gpg_path:
+                cmd += "--define '_gpg_path %s' " % self.gpg_path
+            cmd += file
+
+            status, output = oe.utils.getstatusoutput(cmd)
+            if status:
+                raise bb.build.FuncFailed("Failed to sign RPM packages: %s" % output)
 
     def detach_sign(self, input_file, keyid, passphrase_file, passphrase=None, armor=True):
         """Create a detached signature of a file"""

Re: Failure Inheriting rpm_sign

[Khem Raj]

On Fri, Jan 6, 2017 at 3:52 AM, Chris Trobridge
<christrobridge@hotmail.com> wrote:
> I am getting "Exception: OSError: [Errno 7] Argument list too long" for sign_rpm in the do_package_write_rpm tasks for the
> linux-yocto and glibc-locale recipes.
>
> This is building core-image-minimal (and also my own image) with morty (5aa481d) on Fedora 25.
>
> I have enabled the rpm signing with:
>
> INHERIT += " sign_rpm"
> RPM_GPG_NAME = "{name}"
> RPM_GPG_PASSPHRASE = "{passphrase}"
> IMAGE_INSTALL_append = " signing-keys-rpm"
>
> The error message makes some sense in as much as these recipes produce a lot of packages (for example, glibc-locale produces 1791 packages) and the command line in the log is pretty big, although reading around I didn't find a consensus on what the max command line should be.
>
> The code to sign rpms is in meta/lib/oe/gpg_sign.py b/meta/lib/oe/gpg_sign.py and it builds up one command line with all the packages.
>
> I changed the code (patch appended) to sign each rpm in a separate command and the build completed successfully.  The signing operations take a large amount of time so I think this might be a reasonable change but you may have other concerns.

This certainly is useful, perhaps the signing bits can be moved to individual
recipe packaging tasks that way it may be parallelized a bit

>
> Regards,
> Chris
>
> diff --git a/meta/lib/oe/gpg_sign.py b/meta/lib/oe/gpg_sign.py
> index 38eb0cb..a386b1f 100644
> --- a/meta/lib/oe/gpg_sign.py
> +++ b/meta/lib/oe/gpg_sign.py
> @@ -29,17 +29,18 @@ class LocalSigner(object):
>      def sign_rpms(self, files, keyid, passphrase):
>          """Sign RPM files"""
>
> -        cmd = self.rpm_bin + " --addsign --define '_gpg_name %s'  " % keyid
> -        cmd += "--define '_gpg_passphrase %s' " % passphrase
> -        if self.gpg_bin:
> -            cmd += "--define '%%__gpg %s' " % self.gpg_bin
> -        if self.gpg_path:
> -            cmd += "--define '_gpg_path %s' " % self.gpg_path
> -        cmd += ' '.join(files)
> -
> -        status, output = oe.utils.getstatusoutput(cmd)
> -        if status:
> -            raise bb.build.FuncFailed("Failed to sign RPM packages: %s" % output)
> +        for file in files:
> +            cmd = self.rpm_bin + " --addsign --define '_gpg_name %s'  " % keyid
> +            cmd += "--define '_gpg_passphrase %s' " % passphrase
> +            if self.gpg_bin:
> +                cmd += "--define '%%__gpg %s' " % self.gpg_bin
> +            if self.gpg_path:
> +                cmd += "--define '_gpg_path %s' " % self.gpg_path
> +            cmd += file
> +
> +            status, output = oe.utils.getstatusoutput(cmd)
> +            if status:
> +                raise bb.build.FuncFailed("Failed to sign RPM packages: %s" % output)
>
>      def detach_sign(self, input_file, keyid, passphrase_file, passphrase=None, armor=True):
>          """Create a detached signature of a file"""
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto

Re: Failure Inheriting rpm_sign

[Chris Trobridge]

On Mon, 2017-01-09 at 10:47 -0800, Khem Raj wrote:
> On Fri, Jan 6, 2017 at 3:52 AM, Chris Trobridge
> <christrobridge@hotmail.com> wrote:
> > I am getting "Exception: OSError: [Errno 7] Argument list too long"
> > for sign_rpm in the do_package_write_rpm tasks for the
> > linux-yocto and glibc-locale recipes.
> > 
> > This is building core-image-minimal (and also my own image) with
> > morty (5aa481d) on Fedora 25.
> > 
> > I have enabled the rpm signing with:
> > 
> > INHERIT += " sign_rpm"
> > RPM_GPG_NAME = "{name}"
> > RPM_GPG_PASSPHRASE = "{passphrase}"
> > IMAGE_INSTALL_append = " signing-keys-rpm"
> > 
> > The error message makes some sense in as much as these recipes
> > produce a lot of packages (for example, glibc-locale produces 1791
> > packages) and the command line in the log is pretty big, although
> > reading around I didn't find a consensus on what the max command
> > line should be.
> > 
> > The code to sign rpms is in meta/lib/oe/gpg_sign.py
> > b/meta/lib/oe/gpg_sign.py and it builds up one command line with
> > all the packages.
> > 
> > I changed the code (patch appended) to sign each rpm in a separate
> > command and the build completed successfully.  The signing
> > operations take a large amount of time so I think this might be a
> > reasonable change but you may have other concerns.
> 
> This certainly is useful, perhaps the signing bits can be moved to
> individual
> recipe packaging tasks that way it may be parallelized a bit
> 

Thanks Raj,

Something needs to be done as, unless I've messed up somewhere, you
cannot build even core-image-minimal with rpm signing enabled so the
sign_rpm class is effectively broken.

The change I made works, but it's true is less efficient than signing
rpms individually.  The expense of the signature generation meant it
wasn't inefficient to sign each package in a recipe with a separate
command.

However, looking in package_rpm.bbclass, the end of do_package_rpm()
has:

if d.getVar('RPM_SIGN_PACKAGES', True) == '1':
    bb.build.exec_func("sign_rpm", d)

So, to avoid confusion, all the rpms in one recipe are packaged in
task, and then that task calls the function  sign all the packages.  I
don't know if there's a way for do_package_rpm() to spawn tasks to sign
each package individually.

I also found I needed 'IMAGE_INSTALL_append = " signing-keys-rpm"'
local.conf, to deploy the public key but in sign_rpms.bbclass there is:

do_package_index[depends] += "signing-keys:do_deploy"
do_rootfs[depends] += "signing-keys:do_populate_sysroot"

It may be this isn't quite what is required.

Regards,
Chris

Re: Failure Inheriting rpm_sign

[Khem Raj]

On Wed, Jan 11, 2017 at 4:33 AM, Chris Trobridge
<christrobridge@hotmail.com> wrote:
> On Mon, 2017-01-09 at 10:47 -0800, Khem Raj wrote:
>> On Fri, Jan 6, 2017 at 3:52 AM, Chris Trobridge
>> <christrobridge@hotmail.com> wrote:
>> > I am getting "Exception: OSError: [Errno 7] Argument list too long"
>> > for sign_rpm in the do_package_write_rpm tasks for the
>> > linux-yocto and glibc-locale recipes.
>> >
>> > This is building core-image-minimal (and also my own image) with
>> > morty (5aa481d) on Fedora 25.
>> >
>> > I have enabled the rpm signing with:
>> >
>> > INHERIT += " sign_rpm"
>> > RPM_GPG_NAME = "{name}"
>> > RPM_GPG_PASSPHRASE = "{passphrase}"
>> > IMAGE_INSTALL_append = " signing-keys-rpm"
>> >
>> > The error message makes some sense in as much as these recipes
>> > produce a lot of packages (for example, glibc-locale produces 1791
>> > packages) and the command line in the log is pretty big, although
>> > reading around I didn't find a consensus on what the max command
>> > line should be.
>> >
>> > The code to sign rpms is in meta/lib/oe/gpg_sign.py
>> > b/meta/lib/oe/gpg_sign.py and it builds up one command line with
>> > all the packages.
>> >
>> > I changed the code (patch appended) to sign each rpm in a separate
>> > command and the build completed successfully.  The signing
>> > operations take a large amount of time so I think this might be a
>> > reasonable change but you may have other concerns.
>>
>> This certainly is useful, perhaps the signing bits can be moved to
>> individual
>> recipe packaging tasks that way it may be parallelized a bit
>>
>
> Thanks Raj,
>
> Something needs to be done as, unless I've messed up somewhere, you
> cannot build even core-image-minimal with rpm signing enabled so the
> sign_rpm class is effectively broken.

Its possible. I personally dont use rpm package management system, so
dont have first hand usecase here. may be you can open a bug as well


>
> The change I made works, but it's true is less efficient than signing
> rpms individually.  The expense of the signature generation meant it
> wasn't inefficient to sign each package in a recipe with a separate
> command.

existing logic must have worked at some point of time. It just is that
some bug has creeped in over period of time, may be due to it being
a less tested combination

>
> However, looking in package_rpm.bbclass, the end of do_package_rpm()
> has:
>
> if d.getVar('RPM_SIGN_PACKAGES', True) == '1':
>     bb.build.exec_func("sign_rpm", d)
>
> So, to avoid confusion, all the rpms in one recipe are packaged in
> task, and then that task calls the function  sign all the packages.  I
> don't know if there's a way for do_package_rpm() to spawn tasks to sign
> each package individually.

I think it could be made so that all rpms coming out of a given recipe are
signed together

>
> I also found I needed 'IMAGE_INSTALL_append = " signing-keys-rpm"'
> local.conf, to deploy the public key but in sign_rpms.bbclass there is:
>
> do_package_index[depends] += "signing-keys:do_deploy"
> do_rootfs[depends] += "signing-keys:do_populate_sysroot"
>
> It may be this isn't quite what is required.

Perhaps turning this into a distro feature is a better option.

>
> Regards,
> Chris
>

Re: Failure Inheriting rpm_sign

[Markus Lehtonen]

On 11/01/2017, 14.33, "Chris Trobridge" <yocto-bounces@yoctoproject.org on
behalf of christrobridge@hotmail.com> wrote:

>On Mon, 2017-01-09 at 10:47 -0800, Khem Raj wrote:
>> On Fri, Jan 6, 2017 at 3:52 AM, Chris Trobridge
>> <christrobridge@hotmail.com> wrote:
>> > I am getting "Exception: OSError: [Errno 7] Argument list too long"
>> > for sign_rpm in the do_package_write_rpm tasks for the
>> > linux-yocto and glibc-locale recipes.
>> > 
>> > This is building core-image-minimal (and also my own image) with
>> > morty (5aa481d) on Fedora 25.
>> > 
>> > I have enabled the rpm signing with:
>> > 
>> > INHERIT += " sign_rpm"
>> > RPM_GPG_NAME = "{name}"
>> > RPM_GPG_PASSPHRASE = "{passphrase}"
>> > IMAGE_INSTALL_append = " signing-keys-rpm"
>> > 
>> > The error message makes some sense in as much as these recipes
>> > produce a lot of packages (for example, glibc-locale produces 1791
>> > packages) and the command line in the log is pretty big, although
>> > reading around I didn't find a consensus on what the max command
>> > line should be.
>> > 
>> > The code to sign rpms is in meta/lib/oe/gpg_sign.py
>> > b/meta/lib/oe/gpg_sign.py and it builds up one command line with
>> > all the packages.
>> > 
>> > I changed the code (patch appended) to sign each rpm in a separate
>> > command and the build completed successfully.  The signing
>> > operations take a large amount of time so I think this might be a
>> > reasonable change but you may have other concerns.
>> 
>> This certainly is useful, perhaps the signing bits can be moved to
>> individual
>> recipe packaging tasks that way it may be parallelized a bit
>> 
>
>Thanks Raj,
>
>Something needs to be done as, unless I've messed up somewhere, you
>cannot build even core-image-minimal with rpm signing enabled so the
>sign_rpm class is effectively broken.

Signing should not break this way. Could you create a bug about this.



>The change I made works, but it's true is less efficient than signing
>rpms individually.  The expense of the signature generation meant it
>wasn't inefficient to sign each package in a recipe with a separate
>command.
>
>However, looking in package_rpm.bbclass, the end of do_package_rpm()
>has:
>
>if d.getVar('RPM_SIGN_PACKAGES', True) == '1':
>    bb.build.exec_func("sign_rpm", d)
>
>So, to avoid confusion, all the rpms in one recipe are packaged in
>task, and then that task calls the function  sign all the packages.  I
>don't know if there's a way for do_package_rpm() to spawn tasks to sign
>each package individually.

Probably an easy solution would be to sign packages in batches of, say,
100 packages. That way almost all recipes would be signed in one go. Few
packages would require multiple invocations of rpm but that shouldn't be
significant total overhead.



>I also found I needed 'IMAGE_INSTALL_append = " signing-keys-rpm"'
>local.conf, to deploy the public key but in sign_rpms.bbclass there is:

Yes, you need to add that package if you want to get the signing keys
installed in the image.



>
>do_package_index[depends] += "signing-keys:do_deploy"
>do_rootfs[depends] += "signing-keys:do_populate_sysroot"
>
>It may be this isn't quite what is required.

These are needed at build time and they do not install anything in the
final image. The first one is needed for creating rpm package feeds (or
repositories). The second one is needed needed for rpm-native to have the
correct keys when it is installing packages to create the rootfs.



Thanks,
   Markus