[PATCH 0/3] RPM enhancements for kbuild

[PATCH 0/3] RPM enhancements for kbuild

[Mike Marciniszyn]

This series of patches enhances the kernel RPM mechanism
as follows:
- executes /sbin/installkernel in a %post
- installs the firmware into a KERNELRELEASE specific directory
- adds a kernel-devel package

The installkernel allows for booting the target server
after the RPM has been installed.  A "make install" would
have done that.

The unique firmware directory avoids conflicts with distribution
firmware rpm's as recommended by Doug Ledford.

The kernel-devel package allows for external module builds
against the kernel.  A "make modules" install would have
supported such a build on a single server. The capability is lost
through the rpm mechanism because the build server source
directory is no longer appropriate for the target server.

---

Mike Marciniszyn (3):
      kbuild: add %post section to create initramfs and grub hooks
      kbuilld: install firmware files in kernel relative directory
      kbuild: add generation of kernel-devel


 scripts/package/mkspec |   44 ++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 40 insertions(+), 4 deletions(-)

-- 
Mike

[PATCH 1/3] kbuild: add %post section to create initramfs and grub hooks

[Mike Marciniszyn]

/sbin/installkernel is used to insure grub hooks are
inserted and the initramfs is created on the
target system.

The invokation installkernel will work with any
kernel as long as:
- /sbin/installkernel exists
- the kernel and sysem map files are readable

Signed-off-by: Mike Marciniszyn <mike.marciniszyn@intel.com>
---
 scripts/package/mkspec |    9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/scripts/package/mkspec b/scripts/package/mkspec
index fdd3fbf..0aa6a24 100755
--- a/scripts/package/mkspec
+++ b/scripts/package/mkspec
@@ -112,6 +112,15 @@ echo ""
 echo "%clean"
 echo 'rm -rf $RPM_BUILD_ROOT'
 echo ""
+echo "%post"
+echo "if [ -x /sbin/installkernel -a -r /boot/vmlinuz-$KERNELRELEASE -a -r /boot/System.map-$KERNELRELEASE ]; then"
+echo "cp /boot/vmlinuz-$KERNELRELEASE /boot/vmlinuz-$KERNELRELEASE-rpm"
+echo "cp /boot/System.map-$KERNELRELEASE /boot/System.map-$KERNELRELEASE-rpm"
+echo "rm -f /boot/vmlinuz-$KERNELRELEASE /boot/System.map-$KERNELRELEASE"
+echo "/sbin/installkernel $KERNELRELEASE /boot/vmlinuz-$KERNELRELEASE-rpm /boot/System.map-$KERNELRELEASE-rpm"
+echo "rm -f /boot/vmlinuz-$KERNELRELEASE-rpm /boot/System.map-$KERNELRELEASE-rpm"
+echo "fi"
+echo ""
 echo "%files"
 echo '%defattr (-, root, root)'
 echo "%dir /lib/modules"

[PATCH 2/3] kbuilld: install firmware files in kernel relative directory

[Mike Marciniszyn]

Prior to this patch, firmware files were being installed
in /lib/firmware with a potential conflict with already
installed kernels.

firmware files are now installed in /lib/firmware/<release>
and packaged with the rest of the modules.

Signed-off-by: Mike Marciniszyn <mike.marciniszyn@intel.com>
---
 scripts/package/mkspec |    9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/scripts/package/mkspec b/scripts/package/mkspec
index 0aa6a24..514aeb2 100755
--- a/scripts/package/mkspec
+++ b/scripts/package/mkspec
@@ -77,13 +77,14 @@ echo "%install"
 echo 'KBUILD_IMAGE=$(make image_name)'
 echo "%ifarch ia64"
 echo 'mkdir -p $RPM_BUILD_ROOT/boot/efi $RPM_BUILD_ROOT/lib/modules'
-echo 'mkdir -p $RPM_BUILD_ROOT/lib/firmware'
 echo "%else"
 echo 'mkdir -p $RPM_BUILD_ROOT/boot $RPM_BUILD_ROOT/lib/modules'
-echo 'mkdir -p $RPM_BUILD_ROOT/lib/firmware'
 echo "%endif"
+echo 'mkdir -p $RPM_BUILD_ROOT'"/lib/firmware/$KERNELRELEASE"
 
-echo 'INSTALL_MOD_PATH=$RPM_BUILD_ROOT make %{?_smp_mflags} KBUILD_SRC= modules_install'
+echo 'INSTALL_MOD_PATH=$RPM_BUILD_ROOT make %{?_smp_mflags} KBUILD_SRC= mod-fw= modules_install'
+echo 'INSTALL_FW_PATH=$RPM_BUILD_ROOT'"/lib/firmware/$KERNELRELEASE"
+echo 'make INSTALL_FW_PATH=$INSTALL_FW_PATH' firmware_install
 echo "%ifarch ia64"
 echo 'cp $KBUILD_IMAGE $RPM_BUILD_ROOT'"/boot/efi/vmlinuz-$KERNELRELEASE"
 echo 'ln -s '"efi/vmlinuz-$KERNELRELEASE" '$RPM_BUILD_ROOT'"/boot/"
@@ -125,7 +126,7 @@ echo "%files"
 echo '%defattr (-, root, root)'
 echo "%dir /lib/modules"
 echo "/lib/modules/$KERNELRELEASE"
-echo "/lib/firmware"
+echo "/lib/firmware/$KERNELRELEASE"
 echo "/boot/*"
 echo ""
 echo "%files headers"

[PATCH 3/3] kbuild: add generation of kernel-devel

[Mike Marciniszyn]

Change the spec file to generate a kernel-devel module
allowing for compilation of external kernel modules.

Signed-off-by: Mike Marciniszyn <mike.marciniszyn@intel.com>
---
 scripts/package/mkspec |   26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/scripts/package/mkspec b/scripts/package/mkspec
index 514aeb2..65131df 100755
--- a/scripts/package/mkspec
+++ b/scripts/package/mkspec
@@ -59,6 +59,14 @@ echo "header files define structures and constants that are needed for"
 echo "building most standard programs and are also needed for rebuilding the"
 echo "glibc package."
 echo ""
+echo "%package devel"
+echo "Summary: Development package for building kernel modules to match the $__KERNELRELEASE kernel"
+echo "Group: System Environment/Kernel"
+echo "AutoReqProv: no"
+echo "%description -n kernel-devel"
+echo "This package provides kernel headers and makefiles sufficient to build modules"
+echo "against the $__KERNELRELEASE kernel package."
+echo ""
 
 if ! $PREBUILT; then
 echo "%prep"
@@ -109,6 +117,11 @@ echo 'mv vmlinux.bz2 $RPM_BUILD_ROOT'"/boot/vmlinux-$KERNELRELEASE.bz2"
 echo 'mv vmlinux.orig vmlinux'
 echo "%endif"
 
+echo 'rm -rf $RPM_BUILD_ROOT'"/lib/modules/$KERNELRELEASE/{build,source}"
+echo "mkdir -p "'$RPM_BUILD_ROOT'"/usr/src/kernels/$KERNELRELEASE"
+echo "EXCLUDES=\"--exclude-vcs --exclude .tmp_versions --exclude=*vmlinux* --exclude=*.o --exclude=*.ko --exclude=*.cmd --exclude=Documentation --exclude=firmware --exclude .config.old --exclude .missing-syscalls.d\""
+echo "tar "'$EXCLUDES'" -cf- . | (cd "'$RPM_BUILD_ROOT'"/usr/src/kernels/$KERNELRELEASE;tar xvf -)"
+
 echo ""
 echo "%clean"
 echo 'rm -rf $RPM_BUILD_ROOT'
@@ -122,6 +135,15 @@ echo "/sbin/installkernel $KERNELRELEASE /boot/vmlinuz-$KERNELRELEASE-rpm /boot/
 echo "rm -f /boot/vmlinuz-$KERNELRELEASE-rpm /boot/System.map-$KERNELRELEASE-rpm"
 echo "fi"
 echo ""
+echo "%post devel"
+echo "cd /lib/modules/$KERNELRELEASE"
+echo "ln -sf /usr/src/kernels/$KERNELRELEASE build"
+echo "ln -sf /usr/src/kernels/$KERNELRELEASE source"
+echo ""
+echo "%postun devel"
+echo "cd /lib/modules/$KERNELRELEASE"
+echo "rm -f build source"
+echo ""
 echo "%files"
 echo '%defattr (-, root, root)'
 echo "%dir /lib/modules"
@@ -133,3 +155,7 @@ echo "%files headers"
 echo '%defattr (-, root, root)'
 echo "/usr/include"
 echo ""
+echo "%files devel"
+echo '%defattr (-, root, root)'
+echo "/usr/src/kernels/$KERNELRELEASE"
+echo ""

Re: [PATCH 3/3] kbuild: add generation of kernel-devel

[Michal Marek]

On 24.6.2013 16:38, Mike Marciniszyn wrote:
> Change the spec file to generate a kernel-devel module
> allowing for compilation of external kernel modules.
> 
> Signed-off-by: Mike Marciniszyn <mike.marciniszyn@intel.com>
> ---
>  scripts/package/mkspec |   26 ++++++++++++++++++++++++++
>  1 file changed, 26 insertions(+)
> 
> diff --git a/scripts/package/mkspec b/scripts/package/mkspec
> index 514aeb2..65131df 100755
> --- a/scripts/package/mkspec
> +++ b/scripts/package/mkspec
> @@ -59,6 +59,14 @@ echo "header files define structures and constants that are needed for"
>  echo "building most standard programs and are also needed for rebuilding the"
>  echo "glibc package."
>  echo ""
> +echo "%package devel"
> +echo "Summary: Development package for building kernel modules to match the $__KERNELRELEASE kernel"
> +echo "Group: System Environment/Kernel"
> +echo "AutoReqProv: no"
> +echo "%description -n kernel-devel"
> +echo "This package provides kernel headers and makefiles sufficient to build modules"
> +echo "against the $__KERNELRELEASE kernel package."
> +echo ""
>  
>  if ! $PREBUILT; then
>  echo "%prep"
> @@ -109,6 +117,11 @@ echo 'mv vmlinux.bz2 $RPM_BUILD_ROOT'"/boot/vmlinux-$KERNELRELEASE.bz2"
>  echo 'mv vmlinux.orig vmlinux'
>  echo "%endif"
>  
> +echo 'rm -rf $RPM_BUILD_ROOT'"/lib/modules/$KERNELRELEASE/{build,source}"
> +echo "mkdir -p "'$RPM_BUILD_ROOT'"/usr/src/kernels/$KERNELRELEASE"
> +echo "EXCLUDES=\"--exclude-vcs --exclude .tmp_versions --exclude=*vmlinux* --exclude=*.o --exclude=*.ko --exclude=*.cmd --exclude=Documentation --exclude=firmware --exclude .config.old --exclude .missing-syscalls.d\""
> +echo "tar "'$EXCLUDES'" -cf- . | (cd "'$RPM_BUILD_ROOT'"/usr/src/kernels/$KERNELRELEASE;tar xvf -)"

For consistency and compatibility with older tar versions, please use
$RCS_TAR_EXCLUDE instead of --exclude-vcs. Also, you only need .config,
scripts/, include/ and the Makefiles to build external modules. But
that's an optimization that can be done in a later patch.

> +echo "%post devel"
> +echo "cd /lib/modules/$KERNELRELEASE"
> +echo "ln -sf /usr/src/kernels/$KERNELRELEASE build"
> +echo "ln -sf /usr/src/kernels/$KERNELRELEASE source"
> +echo ""
> +echo "%postun devel"
> +echo "cd /lib/modules/$KERNELRELEASE"
> +echo "rm -f build source"

These symlinks should be part of the RPM payload.

Michal

RE: [PATCH 3/3] kbuild: add generation of kernel-devel

[Marciniszyn, Mike]

> For consistency and compatibility with older tar versions, please use
> $RCS_TAR_EXCLUDE instead of --exclude-vcs. Also, you only need .config,
> scripts/, include/ and the Makefiles to build external modules. But that's an
> optimization that can be done in a later patch.
> 

I can't find any reference to the above string in kernel Makefiles or tar info?

Do you have more details?

> > +echo "%post devel"
> > +echo "cd /lib/modules/$KERNELRELEASE"
> > +echo "ln -sf /usr/src/kernels/$KERNELRELEASE build"
> > +echo "ln -sf /usr/src/kernels/$KERNELRELEASE source"
> > +echo ""
> > +echo "%postun devel"
> > +echo "cd /lib/modules/$KERNELRELEASE"
> > +echo "rm -f build source"
> 
> These symlinks should be part of the RPM payload.

I guess the %post* sections would then be not needed...

Mike

RE: [PATCH 3/3] kbuild: add generation of kernel-devel

[Marciniszyn, Mike]

> I can't find any reference to the above string in kernel Makefiles or tar info?
> 
> Do you have more details?
> 

Did you mean RCS_FIND_IGNORE?

Mike

RE: [PATCH 3/3] kbuild: add generation of kernel-devel

[Marciniszyn, Mike]

 
> > I can't find any reference to the above string in kernel Makefiles or tar info?
> >
> > Do you have more details?
> >
> 
> Did you mean RCS_FIND_IGNORE?
> 

Ok.   RCS_TAR_IGNORE seems to be the correct variable.

Mike

Re: [PATCH 3/3] kbuild: add generation of kernel-devel

[Michal Marek]

Dne 5.7.2013 15:04, Marciniszyn, Mike napsal(a):
>> For consistency and compatibility with older tar versions, please use
>> $RCS_TAR_EXCLUDE instead of --exclude-vcs. Also, you only need .config,
>> scripts/, include/ and the Makefiles to build external modules. But that's an
>> optimization that can be done in a later patch.
>>
> 
> I can't find any reference to the above string in kernel Makefiles or tar info?
> 
> Do you have more details?

It's $RCS_TAR_IGNORE, sorry.


>>> +echo "%post devel"
>>> +echo "cd /lib/modules/$KERNELRELEASE"
>>> +echo "ln -sf /usr/src/kernels/$KERNELRELEASE build"
>>> +echo "ln -sf /usr/src/kernels/$KERNELRELEASE source"
>>> +echo ""
>>> +echo "%postun devel"
>>> +echo "cd /lib/modules/$KERNELRELEASE"
>>> +echo "rm -f build source"
>>
>> These symlinks should be part of the RPM payload.
> 
> I guess the %post* sections would then be not needed...

Yes.

Michal

Re: [PATCH 3/3] kbuild: add generation of kernel-devel

[Michal Marek]

Dne 5.7.2013 15:17, Marciniszyn, Mike napsal(a):
>  
>>> I can't find any reference to the above string in kernel Makefiles or tar info?
>>>
>>> Do you have more details?
>>>
>>
>> Did you mean RCS_FIND_IGNORE?
>>
> 
> Ok.   RCS_TAR_IGNORE seems to be the correct variable.

So you found it :). Yes, that's what I meant.

Michal

RE: [PATCH 3/3] kbuild: add generation of kernel-devel

[Marciniszyn, Mike]

> For consistency and compatibility with older tar versions, please use
> $RCS_TAR_EXCLUDE instead of --exclude-vcs. Also, you only need .config,
> scripts/, include/ and the Makefiles to build external modules. But that's an
> optimization that can be done in a later patch.
> 

In v2 of the patch.

> 
> These symlinks should be part of the RPM payload.
> 

This is also in v2 of the same patch.  The %post and %postun scriptlets are no longer there.

These is one minor oddity in that the symlinks are in the main package because of that packages file selection.   Those symlinks will dangle until the devel package is installed.

It might be possible to adjust the file lists of both, but I was wary of creating a more explicit list in the main package because of a potential maintenance issue as "install" payload changes.

Mike

Re: [PATCH 3/3] kbuild: add generation of kernel-devel

[Michal Marek]

On 8.7.2013 13:55, Marciniszyn, Mike wrote:
>> For consistency and compatibility with older tar versions, please
>> use $RCS_TAR_EXCLUDE instead of --exclude-vcs. Also, you only need
>> .config, scripts/, include/ and the Makefiles to build external
>> modules. But that's an optimization that can be done in a later
>> patch.
>> 
> 
> In v2 of the patch.
> 
>> 
>> These symlinks should be part of the RPM payload.
>> 
> 
> This is also in v2 of the same patch.  The %post and %postun
> scriptlets are no longer there.
> 
> These is one minor oddity in that the symlinks are in the main
> package because of that packages file selection.   Those symlinks
> will dangle until the devel package is installed.
> 
> It might be possible to adjust the file lists of both, but I was wary
> of creating a more explicit list in the main package because of a
> potential maintenance issue as "install" payload changes.

You can say '%exclude /some/path' to exclude a path that would otherwise
match an earlier pattern. But I do not know since when RPM supports
this. If there is a risk of breaking make rpm on older systems, then
let's leave it as is.

Michal

RE: [PATCH 3/3] kbuild: add generation of kernel-devel

[Marciniszyn, Mike]

 
> You can say '%exclude /some/path' to exclude a path that would otherwise
> match an earlier pattern. But I do not know since when RPM supports this. If
> there is a risk of breaking make rpm on older systems, then let's leave it as is.

This looks to be an RPM 4.0.3 feature.


I agree.

Mike

Re: [PATCH 3/3] kbuild: add generation of kernel-devel

[Doug Ledford]

On 07/08/13 09:15, Marciniszyn, Mike wrote:
>  
>> You can say '%exclude /some/path' to exclude a path that would otherwise
>> match an earlier pattern. But I do not know since when RPM supports this. If
>> there is a risk of breaking make rpm on older systems, then let's leave it as is.
> 
> This looks to be an RPM 4.0.3 feature.
> 
> 
> I agree.
> 
> Mike
> 

rpm 4.0.3 is *ancient*.  An %exclude is perfectly acceptable and
definitely preferable to a broken payload.

-- 
Doug Ledford <dledford@redhat.com>
              GPG KeyID: 0E572FDD
	      http://people.redhat.com/dledford

RE: [PATCH 3/3] kbuild: add generation of kernel-devel

[Marciniszyn, Mike]

> 
> rpm 4.0.3 is *ancient*.  An %exclude is perfectly acceptable and definitely
> preferable to a broken payload.
> 

The top of the mkspec shell script has this comment:
#       Output a simple RPM spec file that uses no fancy features requiring
#       RPM v4. This is intended to work with any RPM distro.

Are there old distros that still care about this?

Mike

Re: [PATCH 3/3] kbuild: add generation of kernel-devel

[Michal Marek]

On 8.7.2013 15:39, Marciniszyn, Mike wrote:
>>
>> rpm 4.0.3 is *ancient*.  An %exclude is perfectly acceptable and definitely
>> preferable to a broken payload.
>>
> 
> The top of the mkspec shell script has this comment:
> #       Output a simple RPM spec file that uses no fancy features requiring
> #       RPM v4. This is intended to work with any RPM distro.
> 
> Are there old distros that still care about this?

If it's that old, I don't think we need to care. At least not for
building the RPM, because the compiler on such old systems is probably
not going to be able to compile the kernel. I wasn't sure if %exclude
wasn't added in the last five years or so.

Michal

RE: [PATCH 3/3] kbuild: add generation of kernel-devel

[Marciniszyn, Mike]

> > Are there old distros that still care about this?
> 
> If it's that old, I don't think we need to care. At least not for building the RPM,
> because the compiler on such old systems is probably not going to be able to
> compile the kernel. I wasn't sure if %exclude wasn't added in the last five years
> or so.
> 

Ok.   I will takes Doug's advice and try the %exclude.

Look for a v3 in a bit.  I will adjust the mkspec comment indicated a new min RPM level, 4.0.3.

Mike

Re: [PATCH 3/3] kbuild: add generation of kernel-devel

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 943 bytes --]

On 7/8/2013 10:45 AM, Michal Marek wrote:
> On 8.7.2013 15:39, Marciniszyn, Mike wrote:
>>>
>>> rpm 4.0.3 is *ancient*.  An %exclude is perfectly acceptable and definitely
>>> preferable to a broken payload.
>>>
>>
>> The top of the mkspec shell script has this comment:
>> #       Output a simple RPM spec file that uses no fancy features requiring
>> #       RPM v4. This is intended to work with any RPM distro.
>>
>> Are there old distros that still care about this?
> 
> If it's that old, I don't think we need to care. At least not for
> building the RPM, because the compiler on such old systems is probably
> not going to be able to compile the kernel. I wasn't sure if %exclude
> wasn't added in the last five years or so.

Quick google search confirms that rpm 4.0.3 was released in December of
2001.  I think it's safe to say we can use %exclude and not worry about
anything so ancient it can't support it.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 899 bytes --]

RE: [PATCH 3/3] kbuild: add generation of kernel-devel

[Marciniszyn, Mike]

> Quick google search confirms that rpm 4.0.3 was released in December of
> 2001.  I think it's safe to say we can use %exclude and not worry about anything
> so ancient it can't support it.
> 

Version 3 of the fix now uses the exclude.

Mike

Re: [PATCH 1/3] kbuild: add %post section to create initramfs and grub hooks

[Michal Marek]

On 24.6.2013 16:38, Mike Marciniszyn wrote:
> /sbin/installkernel is used to insure grub hooks are
> inserted and the initramfs is created on the
> target system.
> 
> The invokation installkernel will work with any
> kernel as long as:
> - /sbin/installkernel exists
> - the kernel and sysem map files are readable
> 
> Signed-off-by: Mike Marciniszyn <mike.marciniszyn@intel.com>

Applied to kbuild.git#misc.

Michal

Re: [PATCH 2/3] kbuilld: install firmware files in kernel relative directory

[Michal Marek]

On 24.6.2013 16:38, Mike Marciniszyn wrote:
> Prior to this patch, firmware files were being installed
> in /lib/firmware with a potential conflict with already
> installed kernels.
> 
> firmware files are now installed in /lib/firmware/<release>
> and packaged with the rest of the modules.
> 
> Signed-off-by: Mike Marciniszyn <mike.marciniszyn@intel.com>

Applied to kbuild.git#misc.

Michal