[RFC] Rebuild external kernel modules on kernel change

[RFC] Rebuild external kernel modules on kernel change

[Koen Kooi]

Hi,

For beagleboard I have a few things I need to rebuild everytime the 
kernel changes:

* powervr kerneldrivers
* sdma kernel module
* dmai kernel module
* codec-engine

And I have roughly two kinds of kernel changes:

1) version upgrade (e.g. 2.6.29 -> 2.6.29)
2) config changes (e.g. enable ethernet bridging)

The first type of change could be solved by putting KERNEL_VERSION in PV 
or PR, but that needs a non-trivial amount of python since the 
information isn't available at parsing time (exactly like debian.bbclass).
The second kind of change is a lot harder to detect, unless we start 
storing md5sums for kernel defconfigs.

I have a lowtech proposal for this:

-----

conf/bitbake.conf:
# Define a PR for kernels that machines can override so things like
# modules get rebuilt
MACHINE_KERNEL_PR ?= "r0"

conf/machine/beagleboard.conf:
# Increase this everytime the kernel changes
MACHINE_KERNEL_PR = "r39"

classes/kernel.bbclass:
# A machine.conf or local.conf can increase MACHINE_KERNEL_PR to force
# rebuilds for kernel and external modules
PR = "${MACHINE_KERNEL_PR}"

class/module-base.bbclass:
# A machine.conf or local.conf can increase MACHINE_KERNEL_PR to force
# rebuilds for kernel and external modules
PR = "${MACHINE_KERNEL_PR}"

-----

I don't really like this method, but I'm having a hard time coming up 
with a decent solution that:

a) works
b) requires less or equal manual work
c) keeps PR in sync between different buildhosts

regards,

Koen

Re: [RFC] Rebuild external kernel modules on kernel change

[Frans Meulenbroeks]

Hm,

I don't like that one needs to touch conf/machine/beagleboard.conf
each time the kernel is updated.
This is likely to be forgotten at some time.
Can't we use something in the linux-omap-*.bb file ?
Btw, I always bump PR if I change something in the config.

Isn't it possible to add rebuilding those modules to e.g.. virtual/kernel?

Frans

2009/4/4 Koen Kooi <k.kooi@student.utwente.nl>:
> Hi,
>
> For beagleboard I have a few things I need to rebuild everytime the kernel
> changes:
>
> * powervr kerneldrivers
> * sdma kernel module
> * dmai kernel module
> * codec-engine
>
> And I have roughly two kinds of kernel changes:
>
> 1) version upgrade (e.g. 2.6.29 -> 2.6.29)
> 2) config changes (e.g. enable ethernet bridging)
>
> The first type of change could be solved by putting KERNEL_VERSION in PV or
> PR, but that needs a non-trivial amount of python since the information
> isn't available at parsing time (exactly like debian.bbclass).
> The second kind of change is a lot harder to detect, unless we start storing
> md5sums for kernel defconfigs.
>
> I have a lowtech proposal for this:
>
> -----
>
> conf/bitbake.conf:
> # Define a PR for kernels that machines can override so things like
> # modules get rebuilt
> MACHINE_KERNEL_PR ?= "r0"
>
> conf/machine/beagleboard.conf:
> # Increase this everytime the kernel changes
> MACHINE_KERNEL_PR = "r39"
>
> classes/kernel.bbclass:
> # A machine.conf or local.conf can increase MACHINE_KERNEL_PR to force
> # rebuilds for kernel and external modules
> PR = "${MACHINE_KERNEL_PR}"
>
> class/module-base.bbclass:
> # A machine.conf or local.conf can increase MACHINE_KERNEL_PR to force
> # rebuilds for kernel and external modules
> PR = "${MACHINE_KERNEL_PR}"
>
> -----
>
> I don't really like this method, but I'm having a hard time coming up with a
> decent solution that:
>
> a) works
> b) requires less or equal manual work
> c) keeps PR in sync between different buildhosts
>
> regards,
>
> Koen
>
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>

Re: [RFC] Rebuild external kernel modules on kernel change

[Koen Kooi]

On 04-04-09 17:51, Frans Meulenbroeks wrote:
> Hm,
>
> I don't like that one needs to touch conf/machine/beagleboard.conf
> each time the kernel is updated.
> This is likely to be forgotten at some time.
> Can't we use something in the linux-omap-*.bb file ?

You can't set global vars in recipes.

regards,

Koen


> Btw, I always bump PR if I change something in the config.
>
> Isn't it possible to add rebuilding those modules to e.g.. virtual/kernel?
>
> Frans
>
> 2009/4/4 Koen Kooi<k.kooi@student.utwente.nl>:
>> Hi,
>>
>> For beagleboard I have a few things I need to rebuild everytime the kernel
>> changes:
>>
>> * powervr kerneldrivers
>> * sdma kernel module
>> * dmai kernel module
>> * codec-engine
>>
>> And I have roughly two kinds of kernel changes:
>>
>> 1) version upgrade (e.g. 2.6.29 ->  2.6.29)
>> 2) config changes (e.g. enable ethernet bridging)
>>
>> The first type of change could be solved by putting KERNEL_VERSION in PV or
>> PR, but that needs a non-trivial amount of python since the information
>> isn't available at parsing time (exactly like debian.bbclass).
>> The second kind of change is a lot harder to detect, unless we start storing
>> md5sums for kernel defconfigs.
>>
>> I have a lowtech proposal for this:
>>
>> -----
>>
>> conf/bitbake.conf:
>> # Define a PR for kernels that machines can override so things like
>> # modules get rebuilt
>> MACHINE_KERNEL_PR ?= "r0"
>>
>> conf/machine/beagleboard.conf:
>> # Increase this everytime the kernel changes
>> MACHINE_KERNEL_PR = "r39"
>>
>> classes/kernel.bbclass:
>> # A machine.conf or local.conf can increase MACHINE_KERNEL_PR to force
>> # rebuilds for kernel and external modules
>> PR = "${MACHINE_KERNEL_PR}"
>>
>> class/module-base.bbclass:
>> # A machine.conf or local.conf can increase MACHINE_KERNEL_PR to force
>> # rebuilds for kernel and external modules
>> PR = "${MACHINE_KERNEL_PR}"
>>
>> -----
>>
>> I don't really like this method, but I'm having a hard time coming up with a
>> decent solution that:
>>
>> a) works
>> b) requires less or equal manual work
>> c) keeps PR in sync between different buildhosts
>>
>> regards,
>>
>> Koen
>>
>>
>> _______________________________________________
>> Openembedded-devel mailing list
>> Openembedded-devel@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>>

Re: [RFC] Rebuild external kernel modules on kernel change

[Otavio Salvador]

On Sat, Apr 4, 2009 at 10:31 AM, Koen Kooi <k.kooi@student.utwente.nl> wrote:
> Hi,
[...]
> I don't really like this method, but I'm having a hard time coming up with a
> decent solution that:
[...]

I agree that is not beauty but it works.

I also belive it could be used in other recipes too as initramfs-images. This
way forcing the rebuild of it if the kernel is changed and machine revision
is bumped.

Fully support it.

+1

-- 
Otavio Salvador                  O.S. Systems
E-mail: otavio@ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854         http://projetos.ossystems.com.br

Re: [RFC] Rebuild external kernel modules on kernel change

[Frans Meulenbroeks]

2009/4/4 Koen Kooi <k.kooi@student.utwente.nl>:
> On 04-04-09 17:51, Frans Meulenbroeks wrote:
>>
>> Hm,
>>
>> I don't like that one needs to touch conf/machine/beagleboard.conf
>> each time the kernel is updated.
>> This is likely to be forgotten at some time.
>> Can't we use something in the linux-omap-*.bb file ?
>
> You can't set global vars in recipes.

Thinking of it the real issue is that we do have DEPENDS that takes
care that if A needs B, B is build before A (and I guess that after
updating B a bitbake of A will result in a recompile, even if A's PR
is not changed.

Maybe a solution could be to be able to rebuild a package plus all its
dependencies (the information is there in the DEPENDS lines).

Frans.

Re: [RFC] Rebuild external kernel modules on kernel change

[Jeremy Lainé]

H







> conf/machine/beagleboard.conf:
> # Increase this everytime the kernel changes
> MACHINE_KERNEL_PR = "r39"
> 
> classes/kernel.bbclass:
> # A machine.conf or local.conf can increase MACHINE_KERNEL_PR to force
> # rebuilds for kernel and external modules
> PR = "${MACHINE_KERNEL_PR}"
> 
> class/module-base.bbclass:
> # A machine.conf or local.conf can increase MACHINE_KERNEL_PR to force
> # rebuilds for kernel and external modules
> PR = "${MACHINE_KERNEL_PR}"
> 
> -----
> 
> I don't really like this method, but I'm having a hard time coming up
> with a decent solution that:
> 
> a) works
> b) requires less or equal manual work
> c) keeps PR in sync between different buildhosts
> 
> regards,
> 
> Koen
> 
> 
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel


-- 
Jeremy LAINE
Bolloré telecom | 11bis, rue Scribe | F-75009 Paris

Re: [RFC] Rebuild external kernel modules on kernel change

[Koen Kooi]

On 04-04-09 22:18, Frans Meulenbroeks wrote:
> 2009/4/4 Koen Kooi<k.kooi@student.utwente.nl>:
>> On 04-04-09 17:51, Frans Meulenbroeks wrote:
>>>
>>> Hm,
>>>
>>> I don't like that one needs to touch conf/machine/beagleboard.conf
>>> each time the kernel is updated.
>>> This is likely to be forgotten at some time.
>>> Can't we use something in the linux-omap-*.bb file ?
>>
>> You can't set global vars in recipes.
>
> Thinking of it the real issue is that we do have DEPENDS that takes
> care that if A needs B, B is build before A (and I guess that after
> updating B a bitbake of A will result in a recompile, even if A's PR
> is not changed.
>
> Maybe a solution could be to be able to rebuild a package plus all its
> dependencies (the information is there in the DEPENDS lines).

We can already do that with BB_STAMP_POLICY, the problem present is that 
it will generate packages with the same PV and PR, so it won't show up 
as upgrade, so users get a broken installation after the package manager 
upgrades the kernel, but not the modules since PV and PR didn't change 
for those.

regards,

Koen

Re: [RFC] Rebuild external kernel modules on kernel change

[Koen Kooi]

On 04-04-09 19:46, Otavio Salvador wrote:
> On Sat, Apr 4, 2009 at 10:31 AM, Koen Kooi<k.kooi@student.utwente.nl>  wrote:
>> Hi,
> [...]
>> I don't really like this method, but I'm having a hard time coming up with a
>> decent solution that:
> [...]
>
> I agree that is not beauty but it works.
>
> I also belive it could be used in other recipes too as initramfs-images. This
> way forcing the rebuild of it if the kernel is changed and machine revision
> is bumped.
>
> Fully support it.
>
> +1

Thanks, I'll try cooking up a patch for beagleboard tomorrow to show a 
real life example.

regards,

Koen

Re: [RFC] Rebuild external kernel modules on kernel change

[Koen Kooi]

On 05-04-09 18:43, Koen Kooi wrote:
> On 04-04-09 19:46, Otavio Salvador wrote:
>> On Sat, Apr 4, 2009 at 10:31 AM, Koen Kooi<k.kooi@student.utwente.nl>
>> wrote:
>>> Hi,
>> [...]
>>> I don't really like this method, but I'm having a hard time coming up
>>> with a
>>> decent solution that:
>> [...]
>>
>> I agree that is not beauty but it works.
>>
>> I also belive it could be used in other recipes too as
>> initramfs-images. This
>> way forcing the rebuild of it if the kernel is changed and machine
>> revision
>> is bumped.
>>
>> Fully support it.
>>
>> +1
>
> Thanks, I'll try cooking up a patch for beagleboard tomorrow to show a
> real life example.

Here it is, this will be done in 2 commits:

diff --git a/classes/kernel.bbclass b/classes/kernel.bbclass
index 17e8941..954c407 100644
--- a/classes/kernel.bbclass
+++ b/classes/kernel.bbclass
@@ -64,6 +64,10 @@ export CMDLINE_CONSOLE = 
"console=${@bb.data.getVar("KERNEL_CONSOLE",d,1) or "tt
  KERNEL_VERSION = "${@get_kernelversion('${S}')}"
  KERNEL_MAJOR_VERSION = "${@get_kernelmajorversion('${KERNEL_VERSION}')}"

+# A machine.conf or local.conf can increase MACHINE_KERNEL_PR to force
+# rebuilds for kernel and external modules
+PR = "${MACHINE_KERNEL_PR}"
+
  KERNEL_LOCALVERSION ?= ""


diff --git a/classes/module-base.bbclass b/classes/module-base.bbclass
index c98bace..bc53e1b 100644
--- a/classes/module-base.bbclass
+++ b/classes/module-base.bbclass
@@ -5,6 +5,10 @@ inherit kernel-arch
  export OS = "${TARGET_OS}"
  export CROSS_COMPILE = "${TARGET_PREFIX}"

+# A machine.conf or local.conf can increase MACHINE_KERNEL_PR to force
+# rebuilds for kernel and external modules
+PR = "${MACHINE_KERNEL_PR}"
+

diff --git a/conf/bitbake.conf b/conf/bitbake.conf
index a0d652a..136f01c 100644
--- a/conf/bitbake.conf
+++ b/conf/bitbake.conf
@@ -162,6 +162,10 @@ EXTENDPEVER = 
"${@['','${PE\x7d:'][bb.data.getVar('PE',d,1) > 0]}"
  EXTENDPV = "${EXTENDPEVER}${PV}-${PR}${DISTRO_PR}"
  P = "${PN}-${PV}"

+# Define a PR for kernels that machines can override so things like
+# modules get rebuilt
+MACHINE_KERNEL_PR ?= "r0"
+

diff --git a/conf/machine/beagleboard.conf b/conf/machine/beagleboard.conf
index 78c6301..3d55ebc 100644
--- a/conf/machine/beagleboard.conf
+++ b/conf/machine/beagleboard.conf
@@ -25,6 +25,7 @@ EXTRA_IMAGECMD_jffs2 = "-lnp "
  SERIAL_CONSOLE = "115200 ttyS2"

  PREFERRED_PROVIDER_virtual/kernel = "linux-omap"
+MACHINE_KERNEL_PR = "r20"

  KERNEL_IMAGETYPE = "uImage"

diff --git a/recipes/dsplink/gstreamer-ti_svn.bb 
b/recipes/dsplink/gstreamer-ti_svn.bb
index ab705f3..6944ad5 100644
--- a/recipes/dsplink/gstreamer-ti_svn.bb
+++ b/recipes/dsplink/gstreamer-ti_svn.bb
@@ -4,7 +4,7 @@ SRC_URI = 
"svn://gforge.ti.com/svn/gstreamer_ti/trunk;module=gstreamer_ti;proto=
            "
  SRCREV = "160"

-PR = "r7"
+PR = "${MACHINE_KERNEL_PR}"

diff --git a/recipes/dsplink/ti-codec-engine_2.21.bb 
b/recipes/dsplink/ti-codec-engine_2.21.bb
index 5b1d5e0..f0b0d7e 100644
--- a/recipes/dsplink/ti-codec-engine_2.21.bb
+++ b/recipes/dsplink/ti-codec-engine_2.21.bb
@@ -8,7 +8,6 @@ RDEPENDS = "update-modules"
  inherit module

  # tconf from xdctools dislikes '.' in pwd :/
-PR = "r19"
  PV = "221"


diff --git a/recipes/dsplink/ti-codec-engine_2.23.bb 
b/recipes/dsplink/ti-codec-engine_2.23.bb
index a42c33b..bbf54d4 100644
--- a/recipes/dsplink/ti-codec-engine_2.23.bb
+++ b/recipes/dsplink/ti-codec-engine_2.23.bb
@@ -9,7 +9,6 @@ inherit module
  DEFAULT_PREFERENCE = "-1"

  # tconf from xdctools dislikes '.' in pwd :/
-PR = "r0"
  PV = "223"


diff --git a/recipes/dsplink/ti-dmai_svn.bb b/recipes/dsplink/ti-dmai_svn.bb
index bcbaea8..76fa74a 100644
--- a/recipes/dsplink/ti-dmai_svn.bb
+++ b/recipes/dsplink/ti-dmai_svn.bb
@@ -15,7 +15,7 @@ SRCREV = "36"
  S = 
"${WORKDIR}/BRIJESH_GIT_022309/davinci_multimedia_application_interface/dmai"
  # Yes, the xdc stuff still breaks with a '.' in PWD
  PV = "120+svnr${SRCREV}"
-PR = "r16"
+PR = "${MACHINE_KERNEL_PR}"


diff --git a/recipes/linux/linux-omap_2.6.28.bb 
b/recipes/linux/linux-omap_2.6.28.bb
index bc085c7..92de11e 100644
--- a/recipes/linux/linux-omap_2.6.28.bb
+++ b/recipes/linux/linux-omap_2.6.28.bb
@@ -12,7 +12,6 @@ DEFAULT_PREFERENCE_omap5912osk = "1"
  SRCREV = "79d042a081d3e467c735bb0d9569ed6296f85a3c"

  PV = "2.6.28"
-PR = "r19"


diff --git a/recipes/linux/linux-omap_2.6.29.bb 
b/recipes/linux/linux-omap_2.6.29.bb
index a2391ee..43be953 100644
--- a/recipes/linux/linux-omap_2.6.29.bb
+++ b/recipes/linux/linux-omap_2.6.29.bb
@@ -10,7 +10,7 @@ DEFAULT_PREFERENCE_overo = "1"

  SRCREV = "58cf2f1425abfd3a449f9fe985e48be2d2555022"

-PR = "r7+gitr${SRCREV}"
+PR_append = "+gitr${SRCREV}"


diff --git a/recipes/powervr-drivers/omap3-sgx-modules_1.3.13.1397.bb 
b/recipes/powervr-drivers/omap3-sgx-modules_1.3.13.1397.bb
index 985a0c0..fc9f02f 100644
--- a/recipes/powervr-drivers/omap3-sgx-modules_1.3.13.1397.bb
+++ b/recipes/powervr-drivers/omap3-sgx-modules_1.3.13.1397.bb
@@ -1,8 +1,6 @@
  DESCRIPTION = "Kernel drivers for the PowerVR SGX chipset found in the 
omap3 SoCs"
  LICENSE = "GPLv2"

-PR = "r6"
-
  inherit module

Re: [RFC] Rebuild external kernel modules on kernel change

[Richard Purdie]

On Sat, 2009-04-04 at 15:31 +0200, Koen Kooi wrote:
> For beagleboard I have a few things I need to rebuild everytime the 
> kernel changes:
> 
> * powervr kerneldrivers
> * sdma kernel module
> * dmai kernel module
> * codec-engine
> 
> And I have roughly two kinds of kernel changes:
> 
> 1) version upgrade (e.g. 2.6.29 -> 2.6.29)
> 2) config changes (e.g. enable ethernet bridging)
> 
> The first type of change could be solved by putting KERNEL_VERSION in PV 
> or PR, but that needs a non-trivial amount of python since the 
> information isn't available at parsing time (exactly like debian.bbclass).
> The second kind of change is a lot harder to detect, unless we start 
> storing md5sums for kernel defconfigs.
> 
> I have a lowtech proposal for this:

The thing is if we start doing this for kernels, we'll probably end up
having to do it for other things and the end result will be a mess.

A better fix for this is BB_STAMP_POLICY = "whitelist" which will at
least cause the right things to rebuild.

No, this doesn't bump the PR when packages change but I'd actually like
to see some new code to fix this problem and several others once and for
all. The idea would be to "compare" the current build with any previous
ones and error if bad differences are detected. In this case if PR = X
had already built that would be an "error" which the system may work
around by using PR = "X-Z1" in the output packages. You could teach that
code interesting (even distribution specific) logic e.g. to ignore
changes in system libraries like libc.

This would imply this comparison data would have to come from a shared
source for multiple build machines for a distribution like Angstrom.

No, this solution isn't simple but hacking around the problem just
builds a house on top of quicksand ;-).

Cheers,

Richard

Re: [RFC] Rebuild external kernel modules on kernel change

[Koen Kooi]

On 06-04-09 00:43, Richard Purdie wrote:
> On Sat, 2009-04-04 at 15:31 +0200, Koen Kooi wrote:
>> For beagleboard I have a few things I need to rebuild everytime the
>> kernel changes:
>>
>> * powervr kerneldrivers
>> * sdma kernel module
>> * dmai kernel module
>> * codec-engine
>>
>> And I have roughly two kinds of kernel changes:
>>
>> 1) version upgrade (e.g. 2.6.29 ->  2.6.29)
>> 2) config changes (e.g. enable ethernet bridging)
>>
>> The first type of change could be solved by putting KERNEL_VERSION in PV
>> or PR, but that needs a non-trivial amount of python since the
>> information isn't available at parsing time (exactly like debian.bbclass).
>> The second kind of change is a lot harder to detect, unless we start
>> storing md5sums for kernel defconfigs.
>>
>> I have a lowtech proposal for this:
>
> The thing is if we start doing this for kernels, we'll probably end up
> having to do it for other things and the end result will be a mess.
>
> A better fix for this is BB_STAMP_POLICY = "whitelist" which will at
> least cause the right things to rebuild.
>
> No, this doesn't bump the PR when packages change but I'd actually like
> to see some new code to fix this problem and several others once and for
> all. The idea would be to "compare" the current build with any previous
> ones and error if bad differences are detected. In this case if PR = X
> had already built that would be an "error" which the system may work
> around by using PR = "X-Z1" in the output packages. You could teach that
> code interesting (even distribution specific) logic e.g. to ignore
> changes in system libraries like libc.
>
> This would imply this comparison data would have to come from a shared
> source for multiple build machines for a distribution like Angstrom.
>
> No, this solution isn't simple but hacking around the problem just
> builds a house on top of quicksand ;-).

The problems with a default OE/angstrom setup are:

1) No automatic rebuild of dependant packages
  - Already solved with BB_STAMP_POLICY = "whitelist"

2) No automatic way to increase PR of dependant packages

3) No distributed way to record increase of PR of dependant packages

4) No way to inject kernelversion and PR into external modules
  - Can be solved with code similar to debian.bbclass

if someone can tackle 4) we're further along with making things saner.


regards,

Koen

Re: [RFC] Rebuild external kernel modules on kernel change

[Tom Rini]

On Sat, Apr 04, 2009 at 03:31:25PM +0200, Koen Kooi wrote:

> Hi,
>
> For beagleboard I have a few things I need to rebuild everytime the  
> kernel changes:
>
> * powervr kerneldrivers
> * sdma kernel module
> * dmai kernel module
> * codec-engine
>
> And I have roughly two kinds of kernel changes:
>
> 1) version upgrade (e.g. 2.6.29 -> 2.6.29)

Based on the rest of the email, I assume you mean 2.6.28 -> 2.6.29

> 2) config changes (e.g. enable ethernet bridging)

So, by coincidence I had this thread around still.  For (1), you already
have kernel-abiversion, but maybe we need some helpers to make this more
useful?  For (2) that's not true unless you're changing one of the
variables that actually changes the running kernel abi (which isn't
caught in kernel-abiversion file).  This shouldn't be a frequent
operation.  There is, I suppose a 3rd problem which would be within
version API changes, for new and in-progress features.  This I'm not
sure we need a complex mechanism around for.  Use/publish WIP stuff,
make sure users understand it's WIP and bump PRs as needed, manually.

-- 
Tom Rini

Re: [RFC] Rebuild external kernel modules on kernel change

[Koen Kooi]

On 01-06-09 18:58, Tom Rini wrote:
> On Sat, Apr 04, 2009 at 03:31:25PM +0200, Koen Kooi wrote:
>
>> Hi,
>>
>> For beagleboard I have a few things I need to rebuild everytime the
>> kernel changes:
>>
>> * powervr kerneldrivers
>> * sdma kernel module
>> * dmai kernel module
>> * codec-engine
>>
>> And I have roughly two kinds of kernel changes:
>>
>> 1) version upgrade (e.g. 2.6.29 ->  2.6.29)
>
> Based on the rest of the email, I assume you mean 2.6.28 ->  2.6.29

right.

>> 2) config changes (e.g. enable ethernet bridging)
>
> So, by coincidence I had this thread around still.  For (1), you already
> have kernel-abiversion, but maybe we need some helpers to make this more
> useful?  For (2) that's not true unless you're changing one of the
> variables that actually changes the running kernel abi (which isn't
> caught in kernel-abiversion file).  This shouldn't be a frequent
> operation.

2) is *very* hard to track down, e.g. people don't expect that enabling 
netfilter as modules breaks external wifi drivers since struct sk_buff 
will change (ask nokia 770 tablet users).
I bet there are a lot more cases where kernel runtime abi changes, so 
every kernel PR bump will trigger a rebuild *and* (here's comes the most 
important bit) an automatic PR bump for recipes using module*.bbclass.

So far I've only seen suggestions that will trigger a rebuild of recipes 
using module*bbclass, but no solutions for automagic version changes so 
that the package manager will pick it up as well.

regards,

Koen

Re: [RFC] Rebuild external kernel modules on kernel change

[Phil Blundell]

On Mon, 2009-06-01 at 19:25 +0200, Koen Kooi wrote:
> So far I've only seen suggestions that will trigger a rebuild of recipes 
> using module*bbclass, but no solutions for automagic version changes so 
> that the package manager will pick it up as well.

I don't quite understand why this is so complicated; can't you just
append the kernel's ABI version to either ${PR} or ${PV} (depending on
whether or not you want parallel installability) for the modules?

p.

Re: [RFC] Rebuild external kernel modules on kernel change

[Koen Kooi]

On 01-06-09 20:17, Phil Blundell wrote:
> On Mon, 2009-06-01 at 19:25 +0200, Koen Kooi wrote:
>> So far I've only seen suggestions that will trigger a rebuild of recipes
>> using module*bbclass, but no solutions for automagic version changes so
>> that the package manager will pick it up as well.
>
> I don't quite understand why this is so complicated; can't you just
> append the kernel's ABI version to either ${PR} or ${PV} (depending on
> whether or not you want parallel installability) for the modules?

No, since PV and PR need to be known at parsing time when the kernel 
hasn't been built yet.

regards,

Koen

Re: [RFC] Rebuild external kernel modules on kernel change

[Phil Blundell]

On Mon, 2009-06-01 at 20:45 +0200, Koen Kooi wrote:
> On 01-06-09 20:17, Phil Blundell wrote:
> > On Mon, 2009-06-01 at 19:25 +0200, Koen Kooi wrote:
> >> So far I've only seen suggestions that will trigger a rebuild of recipes
> >> using module*bbclass, but no solutions for automagic version changes so
> >> that the package manager will pick it up as well.
> >
> > I don't quite understand why this is so complicated; can't you just
> > append the kernel's ABI version to either ${PR} or ${PV} (depending on
> > whether or not you want parallel installability) for the modules?
> 
> No, since PV and PR need to be known at parsing time when the kernel 
> hasn't been built yet.

I can't think offhand of any reason why it's important to have PR
available at parse time, but if that's the case then you could splice it
into DISTRO_PR (or some equivalent variable) instead.

My reference to PV above was spurious; it's no different to PR in this
context.  If you want parallel installation then it's actually PKG_xx
that you need to modify (or, conceptually, PN, but fiddling with that
would almost certainly be a bad idea).

p.

Re: [RFC] Rebuild external kernel modules on kernel change

[Phil Blundell]

On Mon, 2009-06-01 at 19:25 +0200, Koen Kooi wrote:
> 2) is *very* hard to track down, e.g. people don't expect that enabling 
> netfilter as modules breaks external wifi drivers since struct sk_buff 
> will change (ask nokia 770 tablet users).
> I bet there are a lot more cases where kernel runtime abi changes, so 
> every kernel PR bump will trigger a rebuild *and* (here's comes the most 
> important bit) an automatic PR bump for recipes using module*.bbclass.

The networking code is indeed particularly bad in that respect.  But in
general Tom is right, the vast majority of kernel config changes
(enabling drivers, adjusting command line parameters or default
settings, that kind of thing) don't have ABI impact and there is no need
to rebuild everything as a result.

The easiest way to deal with that issue is probably to have a separate
KERNEL_PACKAGE_ABIVERSION variable which is set locally in the kernel's
own .bb file, which is appended to the regular kernel-abiversion, and
which you can manually increment when making a change that causes, or is
likely to cause, an ABI change.  If you, as a kernel package maintainer,
want to take a particularly conservative line with your own kernels then
you can simply set KERNEL_PACKAGE_ABIVERSION = "${PR}" and it will act
as you describe above; other maintainers who want to avoid unnecessary
rebuilds can use their own judgement about when to increment it.  

For folks who use MODVERSIONS, it'd probably be easy enough to add an
extra check to detect when the symbol hashes have changed since the last
build and emit an error.  In an ideal world, you'd even be able to
generate a single ABI footprint by, say, taking a sha256 hash of the
catenation of all the symbol versions.  Then, your kernel package could
Provide: kernel-abi-123456abc, all your module packages could Depend: on
the same thing, and (modulo a little bit of package manager tweaking
along roughly the same lines as we use for locales) everything would
"just work".

p.

Re: [RFC] Rebuild external kernel modules on kernel change

[Koen Kooi]

On 01-06-09 22:17, Phil Blundell wrote:
  Then, your kernel package could
> Provide: kernel-abi-123456abc, all your module packages could Depend: on
> the same thing, and (modulo a little bit of package manager tweaking
> along roughly the same lines as we use for locales) everything would
> "just work".

only if you use {o,i}pkg, which isn't good enough.

Re: [RFC] Rebuild external kernel modules on kernel change

[Tom Rini]

On Mon, Jun 01, 2009 at 07:25:11PM +0200, Koen Kooi wrote:
> On 01-06-09 18:58, Tom Rini wrote:
>> On Sat, Apr 04, 2009 at 03:31:25PM +0200, Koen Kooi wrote:
>>
>>> Hi,
>>>
>>> For beagleboard I have a few things I need to rebuild everytime the
>>> kernel changes:
>>>
>>> * powervr kerneldrivers
>>> * sdma kernel module
>>> * dmai kernel module
>>> * codec-engine
>>>
>>> And I have roughly two kinds of kernel changes:
>>>
>>> 1) version upgrade (e.g. 2.6.29 ->  2.6.29)
>>
>> Based on the rest of the email, I assume you mean 2.6.28 ->  2.6.29
>
> right.
>
>>> 2) config changes (e.g. enable ethernet bridging)
>>
>> So, by coincidence I had this thread around still.  For (1), you already
>> have kernel-abiversion, but maybe we need some helpers to make this more
>> useful?  For (2) that's not true unless you're changing one of the
>> variables that actually changes the running kernel abi (which isn't
>> caught in kernel-abiversion file).  This shouldn't be a frequent
>> operation.
>
> 2) is *very* hard to track down, e.g. people don't expect that enabling  
> netfilter as modules breaks external wifi drivers since struct sk_buff  
> will change (ask nokia 770 tablet users).
> I bet there are a lot more cases where kernel runtime abi changes, so  
> every kernel PR bump will trigger a rebuild *and* (here's comes the most  
> important bit) an automatic PR bump for recipes using module*.bbclass.

So we need people to have a better understanding of the kernel, true.  I
think we've all been there at some point.  IIRC, netfilter is the
exception, not the rule.

> So far I've only seen suggestions that will trigger a rebuild of recipes  
> using module*bbclass, but no solutions for automagic version changes so  
> that the package manager will pick it up as well.

I think the problem is we're making an automatic hammer for a manual
screwdriver task, in 2/3. (1) does need better handling but we have that
information around already, in kernel-abiversion, we just need to export
that out.

-- 
Tom Rini

Re: [RFC] Rebuild external kernel modules on kernel change

[Tom Rini]

On Mon, Jun 01, 2009 at 10:52:04PM +0200, Koen Kooi wrote:
> On 01-06-09 22:17, Phil Blundell wrote:
>  Then, your kernel package could
>> Provide: kernel-abi-123456abc, all your module packages could Depend: on
>> the same thing, and (modulo a little bit of package manager tweaking
>> along roughly the same lines as we use for locales) everything would
>> "just work".
>
> only if you use {o,i}pkg, which isn't good enough.

Similar stuff should be doable for deb and rpm.

-- 
Tom Rini

Re: [RFC] Rebuild external kernel modules on kernel change

[Phil Blundell]

On Mon, 2009-06-01 at 22:52 +0200, Koen Kooi wrote:
> On 01-06-09 22:17, Phil Blundell wrote:
> > In an ideal world, you'd even be able to generate a single ABI 
> > footprint by, say, taking a sha256 hash of the catenation of all 
> > the symbol versions.  Then, your kernel package could
> > Provide: kernel-abi-123456abc, all your module packages could Depend: on
> > the same thing, and (modulo a little bit of package manager tweaking
> > along roughly the same lines as we use for locales) everything would
> > "just work".
> 
> only if you use {o,i}pkg, which isn't good enough.

Indeed, in that respect this isn't an ideal world.  The other things
that I wrote in that same email are still applicable, though.

p.