Packaging kernel sources

Packaging kernel sources

[Darren Hart]

Hi all,

I'm working on a project which needs to have the full kernel sources
installed on the target. The kernel-dev package as defined by
kernel.bbclass is heavily pruned to minimize packaging time and size and
is intended to enable building of external modules on the target.

Is there an accepted best-practice for how to get the full source packaged
and installed? I can easily write a new recipe,
linux-custom-source_git.bb, to install the sources, for example, without
impacting the packaging time of "virtual/kernel" package.

It would be nice in some respects for it to all come from the same recipe
though, but I suspect the impact to the common-case where this is not need
would be far too great.

Koen, I believe you had a solution for this with Angstrom?

-- 
Darren Hart
Intel Open Source Technology Center

Re: Packaging kernel sources

[Koen Kooi]

Op 9 sep. 2014, om 17:42 heeft Darren Hart <dvhart@linux.intel.com> het volgende geschreven:

> Hi all,
> 
> I'm working on a project which needs to have the full kernel sources
> installed on the target. The kernel-dev package as defined by
> kernel.bbclass is heavily pruned to minimize packaging time and size and
> is intended to enable building of external modules on the target.
> 
> Is there an accepted best-practice for how to get the full source packaged
> and installed? I can easily write a new recipe,
> linux-custom-source_git.bb, to install the sources, for example, without
> impacting the packaging time of "virtual/kernel" package.
> 
> It would be nice in some respects for it to all come from the same recipe
> though, but I suspect the impact to the common-case where this is not need
> would be far too great.
> 
> Koen, I believe you had a solution for this with Angstrom?

I think I put it in linux.inc in meta-beagleboard, but it's been a few years since I looked at it. I'll have a look at it when I get back from vacation next week. Greetings from Pismo Beach :)

Re: Packaging kernel sources

[Robert Yang]

Good idea, how about use the name linux-source, but add "custom"
in the summary or description.

// Robert

On 09/10/2014 08:42 AM, Darren Hart wrote:
> Hi all,
>
> I'm working on a project which needs to have the full kernel sources
> installed on the target. The kernel-dev package as defined by
> kernel.bbclass is heavily pruned to minimize packaging time and size and
> is intended to enable building of external modules on the target.
>
> Is there an accepted best-practice for how to get the full source packaged
> and installed? I can easily write a new recipe,
> linux-custom-source_git.bb, to install the sources, for example, without
> impacting the packaging time of "virtual/kernel" package.
>
> It would be nice in some respects for it to all come from the same recipe
> though, but I suspect the impact to the common-case where this is not need
> would be far too great.
>
> Koen, I believe you had a solution for this with Angstrom?
>

Re: Packaging kernel sources

[Richard Purdie]

On Tue, 2014-09-09 at 17:42 -0700, Darren Hart wrote:
> I'm working on a project which needs to have the full kernel sources
> installed on the target. The kernel-dev package as defined by
> kernel.bbclass is heavily pruned to minimize packaging time and size and
> is intended to enable building of external modules on the target.
> 
> Is there an accepted best-practice for how to get the full source packaged
> and installed? I can easily write a new recipe,
> linux-custom-source_git.bb, to install the sources, for example, without
> impacting the packaging time of "virtual/kernel" package.
> 
> It would be nice in some respects for it to all come from the same recipe
> though, but I suspect the impact to the common-case where this is not need
> would be far too great.

Personally, I'm leaning towards a couple of big changes in this area:

a) "binning" the kernel-dev package and replacing it with some kind of
separate full source recipe like this.

The benefit is a fully functional on target source which is only built
by people who care about it. This means for most users/builds, we no
longer need to generate that huge package. The downside is a little more
complexity for those that needs this but its not much.


b) binning the separate kernel staging dir and making it work more like
the gcc shared work directory. This means external module builds and the
tools like perf and so on would use this shared source directory.

The benefit would be that we no longer have the huge install step in the
main kernel recipe and the populate_sysroot step shinks in size. 

The downside has more impact here, the problem with shared work is that
it cannot be removed once extracted since the system never knows when
something else may need to use it. For gcc the argument was that we have
so many users (gcc-cross-initial, gcc-cross, gcc-runtime,
gcc-cross-canadian, gcc-crosssdk, gcc-crosssdk-initial and so on) that
the multiple copies were far worse. For the kernel, we can argue that we
have a ton of disk usage from it in the sysroot anyway so this change
just makes things more efficient effectively.

The other issue is that for shared work dirs, the stamps need to be kept
in sync, if they step out, odd things happen (i.e. do_fetch, do_unpack,
do_patch task checksums need to match for linux-yocto, perf, kernel
modules and anything else using it). We may need to add some better
error cases to catch problems. Not an insurmountable problem, just one
that will likely need to be addressed.

I do feel the whole situation with the current kernel size is out of
control and badly affecting user experience.

Cheers,

Richard

Re: Packaging kernel sources

[Bruce Ashfield]

On 14-09-10 04:27 AM, Richard Purdie wrote:
> On Tue, 2014-09-09 at 17:42 -0700, Darren Hart wrote:
>> I'm working on a project which needs to have the full kernel sources
>> installed on the target. The kernel-dev package as defined by
>> kernel.bbclass is heavily pruned to minimize packaging time and size and
>> is intended to enable building of external modules on the target.
>>
>> Is there an accepted best-practice for how to get the full source packaged
>> and installed? I can easily write a new recipe,
>> linux-custom-source_git.bb, to install the sources, for example, without
>> impacting the packaging time of "virtual/kernel" package.
>>
>> It would be nice in some respects for it to all come from the same recipe
>> though, but I suspect the impact to the common-case where this is not need
>> would be far too great.
>
> Personally, I'm leaning towards a couple of big changes in this area:

I'd prefer this sort of larger switch as well.

>
> a) "binning" the kernel-dev package and replacing it with some kind of
> separate full source recipe like this.
>
> The benefit is a fully functional on target source which is only built
> by people who care about it. This means for most users/builds, we no
> longer need to generate that huge package. The downside is a little more
> complexity for those that needs this but its not much.

And just so I'm clear, in this case, the current sysroot, grep, sed,
hack-a-thon version of the kernel would stay as is ?

>
>
> b) binning the separate kernel staging dir and making it work more like
> the gcc shared work directory. This means external module builds and the
> tools like perf and so on would use this shared source directory.
>
> The benefit would be that we no longer have the huge install step in the
> main kernel recipe and the populate_sysroot step shinks in size.
>
> The downside has more impact here, the problem with shared work is that
> it cannot be removed once extracted since the system never knows when
> something else may need to use it. For gcc the argument was that we have
> so many users (gcc-cross-initial, gcc-cross, gcc-runtime,
> gcc-cross-canadian, gcc-crosssdk, gcc-crosssdk-initial and so on) that
> the multiple copies were far worse. For the kernel, we can argue that we
> have a ton of disk usage from it in the sysroot anyway so this change
> just makes things more efficient effectively.
>
> The other issue is that for shared work dirs, the stamps need to be kept
> in sync, if they step out, odd things happen (i.e. do_fetch, do_unpack,
> do_patch task checksums need to match for linux-yocto, perf, kernel
> modules and anything else using it). We may need to add some better
> error cases to catch problems. Not an insurmountable problem, just one
> that will likely need to be addressed.
>
> I do feel the whole situation with the current kernel size is out of
> control and badly affecting user experience.

I like (b) myself, since I'm getting tired of copying the source tree,
having the removals fail as we march forward through kernel versions,
and then needing to modify everything from the SDK to the out of tree
module builds to reconstruct scripts, etc.

We'd have the same "don't patch it" restrictions for something that
builds out of this shared directory (like perf), so there's nothing
new and horrible there.

Cheers,

Bruce

>
> Cheers,
>
> Richard
>
>

Re: Packaging kernel sources

[Richard Purdie]

On Wed, 2014-09-10 at 10:11 -0400, Bruce Ashfield wrote:
> On 14-09-10 04:27 AM, Richard Purdie wrote:
> > On Tue, 2014-09-09 at 17:42 -0700, Darren Hart wrote:
> >> I'm working on a project which needs to have the full kernel sources
> >> installed on the target. The kernel-dev package as defined by
> >> kernel.bbclass is heavily pruned to minimize packaging time and size and
> >> is intended to enable building of external modules on the target.
> >>
> >> Is there an accepted best-practice for how to get the full source packaged
> >> and installed? I can easily write a new recipe,
> >> linux-custom-source_git.bb, to install the sources, for example, without
> >> impacting the packaging time of "virtual/kernel" package.
> >>
> >> It would be nice in some respects for it to all come from the same recipe
> >> though, but I suspect the impact to the common-case where this is not need
> >> would be far too great.
> >
> > Personally, I'm leaning towards a couple of big changes in this area:
> 
> I'd prefer this sort of larger switch as well.
> 
> >
> > a) "binning" the kernel-dev package and replacing it with some kind of
> > separate full source recipe like this.
> >
> > The benefit is a fully functional on target source which is only built
> > by people who care about it. This means for most users/builds, we no
> > longer need to generate that huge package. The downside is a little more
> > complexity for those that needs this but its not much.
> 
> And just so I'm clear, in this case, the current sysroot, grep, sed,
> hack-a-thon version of the kernel would stay as is ?

No, these are two things I think we need to do together. There are two
issues, the on target one and the sysroot mess. a) deals with the
former, b) the latter.

So b) takes care of the hack-a-thon but I think we need a) as well.

> > b) binning the separate kernel staging dir and making it work more like
> > the gcc shared work directory. This means external module builds and the
> > tools like perf and so on would use this shared source directory.
> >
> > The benefit would be that we no longer have the huge install step in the
> > main kernel recipe and the populate_sysroot step shinks in size.
> >
> > The downside has more impact here, the problem with shared work is that
> > it cannot be removed once extracted since the system never knows when
> > something else may need to use it. For gcc the argument was that we have
> > so many users (gcc-cross-initial, gcc-cross, gcc-runtime,
> > gcc-cross-canadian, gcc-crosssdk, gcc-crosssdk-initial and so on) that
> > the multiple copies were far worse. For the kernel, we can argue that we
> > have a ton of disk usage from it in the sysroot anyway so this change
> > just makes things more efficient effectively.
> >
> > The other issue is that for shared work dirs, the stamps need to be kept
> > in sync, if they step out, odd things happen (i.e. do_fetch, do_unpack,
> > do_patch task checksums need to match for linux-yocto, perf, kernel
> > modules and anything else using it). We may need to add some better
> > error cases to catch problems. Not an insurmountable problem, just one
> > that will likely need to be addressed.
> >
> > I do feel the whole situation with the current kernel size is out of
> > control and badly affecting user experience.
> 
> I like (b) myself, since I'm getting tired of copying the source tree,
> having the removals fail as we march forward through kernel versions,
> and then needing to modify everything from the SDK to the out of tree
> module builds to reconstruct scripts, etc.
> 
> We'd have the same "don't patch it" restrictions for something that
> builds out of this shared directory (like perf), so there's nothing
> new and horrible there.

Right...

Cheers,

Richard

Re: Packaging kernel sources

[Bruce Ashfield]

On 14-09-10 10:24 AM, Richard Purdie wrote:
> On Wed, 2014-09-10 at 10:11 -0400, Bruce Ashfield wrote:
>> On 14-09-10 04:27 AM, Richard Purdie wrote:
>>> On Tue, 2014-09-09 at 17:42 -0700, Darren Hart wrote:
>>>> I'm working on a project which needs to have the full kernel sources
>>>> installed on the target. The kernel-dev package as defined by
>>>> kernel.bbclass is heavily pruned to minimize packaging time and size and
>>>> is intended to enable building of external modules on the target.
>>>>
>>>> Is there an accepted best-practice for how to get the full source packaged
>>>> and installed? I can easily write a new recipe,
>>>> linux-custom-source_git.bb, to install the sources, for example, without
>>>> impacting the packaging time of "virtual/kernel" package.
>>>>
>>>> It would be nice in some respects for it to all come from the same recipe
>>>> though, but I suspect the impact to the common-case where this is not need
>>>> would be far too great.
>>>
>>> Personally, I'm leaning towards a couple of big changes in this area:
>>
>> I'd prefer this sort of larger switch as well.
>>
>>>
>>> a) "binning" the kernel-dev package and replacing it with some kind of
>>> separate full source recipe like this.
>>>
>>> The benefit is a fully functional on target source which is only built
>>> by people who care about it. This means for most users/builds, we no
>>> longer need to generate that huge package. The downside is a little more
>>> complexity for those that needs this but its not much.
>>
>> And just so I'm clear, in this case, the current sysroot, grep, sed,
>> hack-a-thon version of the kernel would stay as is ?
>
> No, these are two things I think we need to do together. There are two
> issues, the on target one and the sysroot mess. a) deals with the
> former, b) the latter.
>
> So b) takes care of the hack-a-thon but I think we need a) as well.
>

Aha. With your clarification, I can see the difference now .. I hadn't
had coffee yet. Grabbing a full source copy for the on-target needs,
and not just the source that we've mangled or changed for our build
purposes.

Agreed that this also cleans up the target needs for kernel source and
build.

Bruce

>>> b) binning the separate kernel staging dir and making it work more like
>>> the gcc shared work directory. This means external module builds and the
>>> tools like perf and so on would use this shared source directory.
>>>
>>> The benefit would be that we no longer have the huge install step in the
>>> main kernel recipe and the populate_sysroot step shinks in size.
>>>
>>> The downside has more impact here, the problem with shared work is that
>>> it cannot be removed once extracted since the system never knows when
>>> something else may need to use it. For gcc the argument was that we have
>>> so many users (gcc-cross-initial, gcc-cross, gcc-runtime,
>>> gcc-cross-canadian, gcc-crosssdk, gcc-crosssdk-initial and so on) that
>>> the multiple copies were far worse. For the kernel, we can argue that we
>>> have a ton of disk usage from it in the sysroot anyway so this change
>>> just makes things more efficient effectively.
>>>
>>> The other issue is that for shared work dirs, the stamps need to be kept
>>> in sync, if they step out, odd things happen (i.e. do_fetch, do_unpack,
>>> do_patch task checksums need to match for linux-yocto, perf, kernel
>>> modules and anything else using it). We may need to add some better
>>> error cases to catch problems. Not an insurmountable problem, just one
>>> that will likely need to be addressed.
>>>
>>> I do feel the whole situation with the current kernel size is out of
>>> control and badly affecting user experience.
>>
>> I like (b) myself, since I'm getting tired of copying the source tree,
>> having the removals fail as we march forward through kernel versions,
>> and then needing to modify everything from the SDK to the out of tree
>> module builds to reconstruct scripts, etc.
>>
>> We'd have the same "don't patch it" restrictions for something that
>> builds out of this shared directory (like perf), so there's nothing
>> new and horrible there.
>
> Right...
>
> Cheers,
>
> Richard
>

Re: Packaging kernel sources

[Darren Hart]

On 9/10/14, 1:27, "Richard Purdie" <richard.purdie@linuxfoundation.org>
wrote:

>On Tue, 2014-09-09 at 17:42 -0700, Darren Hart wrote:
>> I'm working on a project which needs to have the full kernel sources
>> installed on the target. The kernel-dev package as defined by
>> kernel.bbclass is heavily pruned to minimize packaging time and size and
>> is intended to enable building of external modules on the target.
>> 
>> Is there an accepted best-practice for how to get the full source
>>packaged
>> and installed? I can easily write a new recipe,
>> linux-custom-source_git.bb, to install the sources, for example, without
>> impacting the packaging time of "virtual/kernel" package.
>> 
>> It would be nice in some respects for it to all come from the same
>>recipe
>> though, but I suspect the impact to the common-case where this is not
>>need
>> would be far too great.
>
>Personally, I'm leaning towards a couple of big changes in this area:
>
>a) "binning" the kernel-dev package and replacing it with some kind of
>separate full source recipe like this.
>
>The benefit is a fully functional on target source which is only built
>by people who care about it. This means for most users/builds, we no
>longer need to generate that huge package. The downside is a little more
>complexity for those that needs this but its not much.

The other downside is that the most common use case (building external
modules) would now require a lot more disk space than with just kernel-dev
(something like 150 MB more iirc).

>
>
>b) binning the separate kernel staging dir and making it work more like
>the gcc shared work directory. This means external module builds and the
>tools like perf and so on would use this shared source directory.

I was thinking along the same lines here as well.

>
>The benefit would be that we no longer have the huge install step in the
>main kernel recipe and the populate_sysroot step shinks in size.
>
>The downside has more impact here, the problem with shared work is that
>it cannot be removed once extracted since the system never knows when
>something else may need to use it. For gcc the argument was that we have
>so many users (gcc-cross-initial, gcc-cross, gcc-runtime,
>gcc-cross-canadian, gcc-crosssdk, gcc-crosssdk-initial and so on) that
>the multiple copies were far worse. For the kernel, we can argue that we
>have a ton of disk usage from it in the sysroot anyway so this change
>just makes things more efficient effectively.
>
>The other issue is that for shared work dirs, the stamps need to be kept
>in sync, if they step out, odd things happen (i.e. do_fetch, do_unpack,
>do_patch task checksums need to match for linux-yocto, perf, kernel
>modules and anything else using it). We may need to add some better
>error cases to catch problems. Not an insurmountable problem, just one
>that will likely need to be addressed.

Good points.

>
>I do feel the whole situation with the current kernel size is out of
>control and badly affecting user experience.


Yup.

-- 
Darren Hart					Open Source Technology Center
darren.hart@intel.com				            Intel Corporation

Re: Packaging kernel sources

[Bruce Ashfield]

On 14-09-10 11:27 AM, Brandt, Todd E wrote:
> I think David brings up a good point about only needing the kernel
> source when something goes wrong. How about a compromise. What if we
> provided a simply utility which pulls in the kernel source and
> recreates the existing kernel image by using git (with the proper
> commit). It could be installed from within the kernel package and be
> generated by the linux-kernel recipe so that it has the proper commit
> hashes (like a simple bash script). That way there's no wasted space.
> I think I might just do that for the heck of it anyway.

We have to respect however the kernel was built, patched, etc. So it
just needs to be whatever was in the ${S} of what was built. Much of
anything else would be recreating the patch process of the kernel
build .. or maybe I'm misunderstanding what you are suggesting.

Bruce




> ________________________________________
> From: Darren Hart [dvhart@linux.intel.com]
> Sent: Wednesday, September 10, 2014 8:13 AM
> To: Richard Purdie; Ashfield, Bruce (Wind River)
> Cc: Openembedded-core@lists.openembedded.org; Brandt, Todd E; Koen Kooi; Tom Zanussi
> Subject: Re: [OE-core] Packaging kernel sources
>
> On 9/10/14, 1:27, "Richard Purdie" <richard.purdie@linuxfoundation.org>
> wrote:
>
>> On Tue, 2014-09-09 at 17:42 -0700, Darren Hart wrote:
>>> I'm working on a project which needs to have the full kernel sources
>>> installed on the target. The kernel-dev package as defined by
>>> kernel.bbclass is heavily pruned to minimize packaging time and size and
>>> is intended to enable building of external modules on the target.
>>>
>>> Is there an accepted best-practice for how to get the full source
>>> packaged
>>> and installed? I can easily write a new recipe,
>>> linux-custom-source_git.bb, to install the sources, for example, without
>>> impacting the packaging time of "virtual/kernel" package.
>>>
>>> It would be nice in some respects for it to all come from the same
>>> recipe
>>> though, but I suspect the impact to the common-case where this is not
>>> need
>>> would be far too great.
>>
>> Personally, I'm leaning towards a couple of big changes in this area:
>>
>> a) "binning" the kernel-dev package and replacing it with some kind of
>> separate full source recipe like this.
>>
>> The benefit is a fully functional on target source which is only built
>> by people who care about it. This means for most users/builds, we no
>> longer need to generate that huge package. The downside is a little more
>> complexity for those that needs this but its not much.
>
> The other downside is that the most common use case (building external
> modules) would now require a lot more disk space than with just kernel-dev
> (something like 150 MB more iirc).
>
>>
>>
>> b) binning the separate kernel staging dir and making it work more like
>> the gcc shared work directory. This means external module builds and the
>> tools like perf and so on would use this shared source directory.
>
> I was thinking along the same lines here as well.
>
>>
>> The benefit would be that we no longer have the huge install step in the
>> main kernel recipe and the populate_sysroot step shinks in size.
>>
>> The downside has more impact here, the problem with shared work is that
>> it cannot be removed once extracted since the system never knows when
>> something else may need to use it. For gcc the argument was that we have
>> so many users (gcc-cross-initial, gcc-cross, gcc-runtime,
>> gcc-cross-canadian, gcc-crosssdk, gcc-crosssdk-initial and so on) that
>> the multiple copies were far worse. For the kernel, we can argue that we
>> have a ton of disk usage from it in the sysroot anyway so this change
>> just makes things more efficient effectively.
>>
>> The other issue is that for shared work dirs, the stamps need to be kept
>> in sync, if they step out, odd things happen (i.e. do_fetch, do_unpack,
>> do_patch task checksums need to match for linux-yocto, perf, kernel
>> modules and anything else using it). We may need to add some better
>> error cases to catch problems. Not an insurmountable problem, just one
>> that will likely need to be addressed.
>
> Good points.
>
>>
>> I do feel the whole situation with the current kernel size is out of
>> control and badly affecting user experience.
>
>
> Yup.
>
> --
> Darren Hart                                     Open Source Technology Center
> darren.hart@intel.com                                       Intel Corporation
>
>
>

Re: Packaging kernel sources

[Peter A. Bigot]

On 09/10/2014 10:13 AM, Darren Hart wrote:
> On 9/10/14, 1:27, "Richard Purdie" <richard.purdie@linuxfoundation.org>
> wrote:
>
>> On Tue, 2014-09-09 at 17:42 -0700, Darren Hart wrote:
>>> I'm working on a project which needs to have the full kernel sources
>>> installed on the target. The kernel-dev package as defined by
>>> kernel.bbclass is heavily pruned to minimize packaging time and size and
>>> is intended to enable building of external modules on the target.
>>>
>>> Is there an accepted best-practice for how to get the full source
>>> packaged
>>> and installed? I can easily write a new recipe,
>>> linux-custom-source_git.bb, to install the sources, for example, without
>>> impacting the packaging time of "virtual/kernel" package.
>>>
>>> It would be nice in some respects for it to all come from the same
>>> recipe
>>> though, but I suspect the impact to the common-case where this is not
>>> need
>>> would be far too great.
>> Personally, I'm leaning towards a couple of big changes in this area:
>>
>> a) "binning" the kernel-dev package and replacing it with some kind of
>> separate full source recipe like this.
>>
>> The benefit is a fully functional on target source which is only built
>> by people who care about it. This means for most users/builds, we no
>> longer need to generate that huge package. The downside is a little more
>> complexity for those that needs this but its not much.
> The other downside is that the most common use case (building external
> modules) would now require a lot more disk space than with just kernel-dev
> (something like 150 MB more iirc).

There are three use cases where I've wanted kernel source on target:

1) Building the whole kernel.  Pretty rare, but could happen. Debian 
appears to provide a linux-source package that provides the whole source 
in /usr/src/linux-source-$(uname -r).  I'd want it to come with the 
contents of ${S} at the point where the compile task was ready to run.   
I'd love it to also come with a shallow git repository already present: 
I can't see anybody wanting to do on-target kernel modifications without 
having SCM underneath, and it takes a long time to "git add --all" a 
kernel tree on a class 4 uSD card.  It doesn't need the entire history, 
just maybe three commits: the upstream stable base release, a record of 
the changes/patches done by kern-tools or other kernel-building recipes, 
and a final commit that archives the as-built .config as a defconfig 
somewhere.

2) Building external modules.  Very common, and AFAICT normally packaged 
as "linux-headers" on debian systems, living in 
/usr/src/linux-headers-$(uname -r).  I think simply renaming kernel-dev 
to this and changing where it unpacks would do.  No need for git here 
since its unlikely people would be modifying the headers.

3) Building device trees.  I haven't figured out how to do this other 
than rebuilding the whole kernel which is a major pain, but inspection 
suggests the current kernel-dev (proposed linux-headers) might have that 
functionality.

Peter

>
>>
>> b) binning the separate kernel staging dir and making it work more like
>> the gcc shared work directory. This means external module builds and the
>> tools like perf and so on would use this shared source directory.
> I was thinking along the same lines here as well.
>
>> The benefit would be that we no longer have the huge install step in the
>> main kernel recipe and the populate_sysroot step shinks in size.
>>
>> The downside has more impact here, the problem with shared work is that
>> it cannot be removed once extracted since the system never knows when
>> something else may need to use it. For gcc the argument was that we have
>> so many users (gcc-cross-initial, gcc-cross, gcc-runtime,
>> gcc-cross-canadian, gcc-crosssdk, gcc-crosssdk-initial and so on) that
>> the multiple copies were far worse. For the kernel, we can argue that we
>> have a ton of disk usage from it in the sysroot anyway so this change
>> just makes things more efficient effectively.
>>
>> The other issue is that for shared work dirs, the stamps need to be kept
>> in sync, if they step out, odd things happen (i.e. do_fetch, do_unpack,
>> do_patch task checksums need to match for linux-yocto, perf, kernel
>> modules and anything else using it). We may need to add some better
>> error cases to catch problems. Not an insurmountable problem, just one
>> that will likely need to be addressed.
> Good points.
>
>> I do feel the whole situation with the current kernel size is out of
>> control and badly affecting user experience.
> Yup.

I don't know that there's much that can be done: I wouldn't want 
anything removing parts of the source tree from a kernel on a 
presumption that the user wouldn't want them installed.  But yeah, the 
thing's a pig.

Re: Packaging kernel sources

[Robert Yang]

Hello,

How's this going on, please ?

// Robert

On 09/10/2014 11:13 PM, Darren Hart wrote:
> On 9/10/14, 1:27, "Richard Purdie" <richard.purdie@linuxfoundation.org>
> wrote:
>
>> On Tue, 2014-09-09 at 17:42 -0700, Darren Hart wrote:
>>> I'm working on a project which needs to have the full kernel sources
>>> installed on the target. The kernel-dev package as defined by
>>> kernel.bbclass is heavily pruned to minimize packaging time and size and
>>> is intended to enable building of external modules on the target.
>>>
>>> Is there an accepted best-practice for how to get the full source
>>> packaged
>>> and installed? I can easily write a new recipe,
>>> linux-custom-source_git.bb, to install the sources, for example, without
>>> impacting the packaging time of "virtual/kernel" package.
>>>
>>> It would be nice in some respects for it to all come from the same
>>> recipe
>>> though, but I suspect the impact to the common-case where this is not
>>> need
>>> would be far too great.
>>
>> Personally, I'm leaning towards a couple of big changes in this area:
>>
>> a) "binning" the kernel-dev package and replacing it with some kind of
>> separate full source recipe like this.
>>
>> The benefit is a fully functional on target source which is only built
>> by people who care about it. This means for most users/builds, we no
>> longer need to generate that huge package. The downside is a little more
>> complexity for those that needs this but its not much.
>
> The other downside is that the most common use case (building external
> modules) would now require a lot more disk space than with just kernel-dev
> (something like 150 MB more iirc).
>
>>
>>
>> b) binning the separate kernel staging dir and making it work more like
>> the gcc shared work directory. This means external module builds and the
>> tools like perf and so on would use this shared source directory.
>
> I was thinking along the same lines here as well.
>
>>
>> The benefit would be that we no longer have the huge install step in the
>> main kernel recipe and the populate_sysroot step shinks in size.
>>
>> The downside has more impact here, the problem with shared work is that
>> it cannot be removed once extracted since the system never knows when
>> something else may need to use it. For gcc the argument was that we have
>> so many users (gcc-cross-initial, gcc-cross, gcc-runtime,
>> gcc-cross-canadian, gcc-crosssdk, gcc-crosssdk-initial and so on) that
>> the multiple copies were far worse. For the kernel, we can argue that we
>> have a ton of disk usage from it in the sysroot anyway so this change
>> just makes things more efficient effectively.
>>
>> The other issue is that for shared work dirs, the stamps need to be kept
>> in sync, if they step out, odd things happen (i.e. do_fetch, do_unpack,
>> do_patch task checksums need to match for linux-yocto, perf, kernel
>> modules and anything else using it). We may need to add some better
>> error cases to catch problems. Not an insurmountable problem, just one
>> that will likely need to be addressed.
>
> Good points.
>
>>
>> I do feel the whole situation with the current kernel size is out of
>> control and badly affecting user experience.
>
>
> Yup.
>