[PATCH 0/2] kernel/multlib: fixes and performance tweak

[PATCH 0/2] kernel/multlib: fixes and performance tweak

[Bruce Ashfield]

Richard/Saul,

I bumped these couple of patches to the front of my queue, since one is
for a multilib issue we just ran into, and the other is a performance
tweak that I've been running with for a while (and it hasn't caused
problems).

The first change is a minor change to the packaging of the kernel to save
some time during the install phase. No use cases have been broken here, but
it's worth having more eyes on this to see if we've missed something. Every
little bit can help.

Note: I haven't checked this against your latest series, so I hope this
isn't redundant at this point.

[PATCH 1/2] kernel: avoid copying unnecessary files during do_install

    kernel: avoid copying unnecessary files during do_install
    
    kernel_do_install() populates $kerneldir with files needed to build
    external modules. To accomplish this there are several copy commands
    to get source from the kernel source tree and build trees after which
    a 'clean' is performed. Since we are copying from the build tree we
    get about 1G of .o and .cmd files copied over only to have them
    removed when we clean. This adds additional IO overhead as well as
    overhead caused by pseudo. By avoiding copying these files in the
    first place we get multiple gains:
      * avoid initial copy
      * avoid file deletes during clean
      * reduce pseudo overhead
    
    Additionally we are making use of cpio vs cp which tends to be
    significantly faster at performing copies.
    
    With these changes I observe a 15-30% decrease in the time to complete
    the do_install() operation on the kernel.
    
    [YOCTO #3517]

The second change was done partly by myself and the rest by Mark Hatle. We
ran into linux-dummy showing up as a multlib prefixed dependency when building
in our configurations. The reason was that linux-dummy doesn't inherit 
kernel.bbclass and hence wasn't being skipped. Rather than brining the 
extra overhead and potential side effects from kernel.bbclass, I decided
to check for recipes that provide virtual/kernel and skip based on that
test, versus the include. 

** I don't have enough background to know if there's a potential problem with
this, but it definitely passes our testing after the change.

Mark also found that we needed to skip kernel-module based packages from being
extended with the multib prefix and that makes up the second part of the fix.

[PATCH 2/2] multilib: skip packages that provide virtual/kernel

    multilib: skip packages that provide virtual/kernel
    
    Rather than keying on recipes that inherit kernel.bbclass, we should
    be checking for providers of virtual/kernel when skipping kernel
    recipes in multlib builds.
    
    Not all providers of virtual/kernel inherit kernel.bbclass (notably
    linux-dummy), so checking on the provider is a more complete check.
    
    We need to be sure to check for inheritance of module-base as well, this
    allows for packages that provides modules to avoid the multilib renaming.

cc: Mark Hatle <mark.hatle@windriver.com>
cc: Mark Asselstine <mark.asselstine@windriver.com>

Cheers,

Bruce

The following changes since commit 8d80483e2f0f755a61087be1dffaf595fbc467fd:

  bitbake: bitbake-layers: print the recipe's depends that crosses a layer boundary (2013-01-31 14:00:03 +0000)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib zedd/kernel
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=zedd/kernel

Bruce Ashfield (1):
  multilib: skip packages that provide virtual/kernel

Mark Asselstine (1):
  kernel: avoid copying unnecessary files during do_install

 meta/classes/kernel.bbclass   |    7 +++++--
 meta/classes/multilib.bbclass |    4 +++-
 meta/lib/oe/classextend.py    |    2 +-
 3 files changed, 9 insertions(+), 4 deletions(-)

-- 
1.7.10.4

[PATCH 1/2] kernel: avoid copying unnecessary files during do_install

[Bruce Ashfield]

From: Mark Asselstine <mark.asselstine@windriver.com>

kernel_do_install() populates $kerneldir with files needed to build
external modules. To accomplish this there are several copy commands
to get source from the kernel source tree and build trees after which
a 'clean' is performed. Since we are copying from the build tree we
get about 1G of .o and .cmd files copied over only to have them
removed when we clean. This adds additional IO overhead as well as
overhead caused by pseudo. By avoiding copying these files in the
first place we get multiple gains:
  * avoid initial copy
  * avoid file deletes during clean
  * reduce pseudo overhead

Additionally we are making use of cpio vs cp which tends to be
significantly faster at performing copies.

With these changes I observe a 15-30% decrease in the time to complete
the do_install() operation on the kernel.

[YOCTO #3517]

Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
---
 meta/classes/kernel.bbclass |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index e2a582b..cc61be6 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -143,10 +143,13 @@ kernel_do_install() {
 	# work and sysroots can be on different partitions, so we can't rely on
 	# hardlinking, unfortunately.
 	#
-	cp -fR * $kerneldir
+	find . -depth -not -name "*.cmd" -not -name "*.o" -not -path "./.*" -print0 | cpio --null -pdu $kerneldir
 	cp .config $kerneldir
 	if [ "${S}" != "${B}" ]; then
-		cp -fR ${S}/* $kerneldir
+		pwd="$PWD"
+		cd "${S}"
+		find . -depth -not -path "./.*" -print0 | cpio --null -pdu $kerneldir
+		cd "$pwd"
 	fi
 	install -m 0644 ${KERNEL_OUTPUT} $kerneldir/${KERNEL_IMAGETYPE}
 	install -m 0644 System.map $kerneldir/System.map-${KERNEL_VERSION}
-- 
1.7.10.4

[PATCH 2/2] multilib: skip packages that provide virtual/kernel

[Bruce Ashfield]

Rather than keying on recipes that inherit kernel.bbclass, we should
be checking for providers of virtual/kernel when skipping kernel
recipes in multlib builds.

Not all providers of virtual/kernel inherit kernel.bbclass (notably
linux-dummy), so checking on the provider is a more complete check.

We need to be sure to check for inheritance of module-base as well, this
allows for packages that provides modules to avoid the multilib renaming.

Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
 meta/classes/multilib.bbclass |    4 +++-
 meta/lib/oe/classextend.py    |    2 +-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/meta/classes/multilib.bbclass b/meta/classes/multilib.bbclass
index f5f3c69..f1696b6 100644
--- a/meta/classes/multilib.bbclass
+++ b/meta/classes/multilib.bbclass
@@ -10,7 +10,9 @@ python multilib_virtclass_handler () {
     e.data.setVar('STAGING_KERNEL_DIR', e.data.getVar('STAGING_KERNEL_DIR', True))
 
     # There should only be one kernel in multilib configs
-    if bb.data.inherits_class('kernel', e.data) or bb.data.inherits_class('module-base', e.data):
+    # We also skip multilib setup for module packages.
+    provides = (e.data.getVar("PROVIDES", True) or "").split()
+    if "virtual/kernel" in provides or bb.data.inherits_class('module-base', e.data):
         raise bb.parse.SkipPackage("We shouldn't have multilib variants for the kernel")
 
     if bb.data.inherits_class('image', e.data):
diff --git a/meta/lib/oe/classextend.py b/meta/lib/oe/classextend.py
index 19e1254..61cbb8f 100644
--- a/meta/lib/oe/classextend.py
+++ b/meta/lib/oe/classextend.py
@@ -5,7 +5,7 @@ class ClassExtender(object):
         self.pkgs_mapping = []
 
     def extend_name(self, name):
-        if name.startswith("kernel-module"):
+        if name.startswith("kernel-module") or name == "virtual/kernel":
             return name
         if name.startswith("rtld"):
             return name
-- 
1.7.10.4