* Packaging kernel sources @ 2014-09-10 0:42 Darren Hart 2014-09-10 2:18 ` Koen Kooi ` (2 more replies) 0 siblings, 3 replies; 11+ messages in thread From: Darren Hart @ 2014-09-10 0:42 UTC (permalink / raw) To: Openembedded-core; +Cc: Brandt, Todd E, Koen Kooi, Tom Zanussi Hi all, I'm working on a project which needs to have the full kernel sources installed on the target. The kernel-dev package as defined by kernel.bbclass is heavily pruned to minimize packaging time and size and is intended to enable building of external modules on the target. Is there an accepted best-practice for how to get the full source packaged and installed? I can easily write a new recipe, linux-custom-source_git.bb, to install the sources, for example, without impacting the packaging time of "virtual/kernel" package. It would be nice in some respects for it to all come from the same recipe though, but I suspect the impact to the common-case where this is not need would be far too great. Koen, I believe you had a solution for this with Angstrom? -- Darren Hart Intel Open Source Technology Center ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Packaging kernel sources 2014-09-10 0:42 Packaging kernel sources Darren Hart @ 2014-09-10 2:18 ` Koen Kooi 2014-09-10 6:28 ` Robert Yang 2014-09-10 8:27 ` Richard Purdie 2 siblings, 0 replies; 11+ messages in thread From: Koen Kooi @ 2014-09-10 2:18 UTC (permalink / raw) To: Darren Hart; +Cc: Brandt, Todd E, Tom Zanussi, Openembedded-core Op 9 sep. 2014, om 17:42 heeft Darren Hart <dvhart@linux.intel.com> het volgende geschreven: > Hi all, > > I'm working on a project which needs to have the full kernel sources > installed on the target. The kernel-dev package as defined by > kernel.bbclass is heavily pruned to minimize packaging time and size and > is intended to enable building of external modules on the target. > > Is there an accepted best-practice for how to get the full source packaged > and installed? I can easily write a new recipe, > linux-custom-source_git.bb, to install the sources, for example, without > impacting the packaging time of "virtual/kernel" package. > > It would be nice in some respects for it to all come from the same recipe > though, but I suspect the impact to the common-case where this is not need > would be far too great. > > Koen, I believe you had a solution for this with Angstrom? I think I put it in linux.inc in meta-beagleboard, but it's been a few years since I looked at it. I'll have a look at it when I get back from vacation next week. Greetings from Pismo Beach :) ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Packaging kernel sources 2014-09-10 0:42 Packaging kernel sources Darren Hart 2014-09-10 2:18 ` Koen Kooi @ 2014-09-10 6:28 ` Robert Yang 2014-09-10 8:27 ` Richard Purdie 2 siblings, 0 replies; 11+ messages in thread From: Robert Yang @ 2014-09-10 6:28 UTC (permalink / raw) To: Darren Hart, Openembedded-core; +Cc: Brandt, Todd E, Koen Kooi, Tom Zanussi Good idea, how about use the name linux-source, but add "custom" in the summary or description. // Robert On 09/10/2014 08:42 AM, Darren Hart wrote: > Hi all, > > I'm working on a project which needs to have the full kernel sources > installed on the target. The kernel-dev package as defined by > kernel.bbclass is heavily pruned to minimize packaging time and size and > is intended to enable building of external modules on the target. > > Is there an accepted best-practice for how to get the full source packaged > and installed? I can easily write a new recipe, > linux-custom-source_git.bb, to install the sources, for example, without > impacting the packaging time of "virtual/kernel" package. > > It would be nice in some respects for it to all come from the same recipe > though, but I suspect the impact to the common-case where this is not need > would be far too great. > > Koen, I believe you had a solution for this with Angstrom? > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Packaging kernel sources 2014-09-10 0:42 Packaging kernel sources Darren Hart 2014-09-10 2:18 ` Koen Kooi 2014-09-10 6:28 ` Robert Yang @ 2014-09-10 8:27 ` Richard Purdie 2014-09-10 14:11 ` Bruce Ashfield 2014-09-10 15:13 ` Darren Hart 2 siblings, 2 replies; 11+ messages in thread From: Richard Purdie @ 2014-09-10 8:27 UTC (permalink / raw) To: Darren Hart, Ashfield, Bruce Cc: Brandt, Todd E, Koen Kooi, Tom Zanussi, Openembedded-core On Tue, 2014-09-09 at 17:42 -0700, Darren Hart wrote: > I'm working on a project which needs to have the full kernel sources > installed on the target. The kernel-dev package as defined by > kernel.bbclass is heavily pruned to minimize packaging time and size and > is intended to enable building of external modules on the target. > > Is there an accepted best-practice for how to get the full source packaged > and installed? I can easily write a new recipe, > linux-custom-source_git.bb, to install the sources, for example, without > impacting the packaging time of "virtual/kernel" package. > > It would be nice in some respects for it to all come from the same recipe > though, but I suspect the impact to the common-case where this is not need > would be far too great. Personally, I'm leaning towards a couple of big changes in this area: a) "binning" the kernel-dev package and replacing it with some kind of separate full source recipe like this. The benefit is a fully functional on target source which is only built by people who care about it. This means for most users/builds, we no longer need to generate that huge package. The downside is a little more complexity for those that needs this but its not much. b) binning the separate kernel staging dir and making it work more like the gcc shared work directory. This means external module builds and the tools like perf and so on would use this shared source directory. The benefit would be that we no longer have the huge install step in the main kernel recipe and the populate_sysroot step shinks in size. The downside has more impact here, the problem with shared work is that it cannot be removed once extracted since the system never knows when something else may need to use it. For gcc the argument was that we have so many users (gcc-cross-initial, gcc-cross, gcc-runtime, gcc-cross-canadian, gcc-crosssdk, gcc-crosssdk-initial and so on) that the multiple copies were far worse. For the kernel, we can argue that we have a ton of disk usage from it in the sysroot anyway so this change just makes things more efficient effectively. The other issue is that for shared work dirs, the stamps need to be kept in sync, if they step out, odd things happen (i.e. do_fetch, do_unpack, do_patch task checksums need to match for linux-yocto, perf, kernel modules and anything else using it). We may need to add some better error cases to catch problems. Not an insurmountable problem, just one that will likely need to be addressed. I do feel the whole situation with the current kernel size is out of control and badly affecting user experience. Cheers, Richard ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Packaging kernel sources 2014-09-10 8:27 ` Richard Purdie @ 2014-09-10 14:11 ` Bruce Ashfield 2014-09-10 14:24 ` Richard Purdie 2014-09-10 15:13 ` Darren Hart 1 sibling, 1 reply; 11+ messages in thread From: Bruce Ashfield @ 2014-09-10 14:11 UTC (permalink / raw) To: Richard Purdie, Darren Hart Cc: Brandt, Todd E, Koen Kooi, Tom Zanussi, Openembedded-core On 14-09-10 04:27 AM, Richard Purdie wrote: > On Tue, 2014-09-09 at 17:42 -0700, Darren Hart wrote: >> I'm working on a project which needs to have the full kernel sources >> installed on the target. The kernel-dev package as defined by >> kernel.bbclass is heavily pruned to minimize packaging time and size and >> is intended to enable building of external modules on the target. >> >> Is there an accepted best-practice for how to get the full source packaged >> and installed? I can easily write a new recipe, >> linux-custom-source_git.bb, to install the sources, for example, without >> impacting the packaging time of "virtual/kernel" package. >> >> It would be nice in some respects for it to all come from the same recipe >> though, but I suspect the impact to the common-case where this is not need >> would be far too great. > > Personally, I'm leaning towards a couple of big changes in this area: I'd prefer this sort of larger switch as well. > > a) "binning" the kernel-dev package and replacing it with some kind of > separate full source recipe like this. > > The benefit is a fully functional on target source which is only built > by people who care about it. This means for most users/builds, we no > longer need to generate that huge package. The downside is a little more > complexity for those that needs this but its not much. And just so I'm clear, in this case, the current sysroot, grep, sed, hack-a-thon version of the kernel would stay as is ? > > > b) binning the separate kernel staging dir and making it work more like > the gcc shared work directory. This means external module builds and the > tools like perf and so on would use this shared source directory. > > The benefit would be that we no longer have the huge install step in the > main kernel recipe and the populate_sysroot step shinks in size. > > The downside has more impact here, the problem with shared work is that > it cannot be removed once extracted since the system never knows when > something else may need to use it. For gcc the argument was that we have > so many users (gcc-cross-initial, gcc-cross, gcc-runtime, > gcc-cross-canadian, gcc-crosssdk, gcc-crosssdk-initial and so on) that > the multiple copies were far worse. For the kernel, we can argue that we > have a ton of disk usage from it in the sysroot anyway so this change > just makes things more efficient effectively. > > The other issue is that for shared work dirs, the stamps need to be kept > in sync, if they step out, odd things happen (i.e. do_fetch, do_unpack, > do_patch task checksums need to match for linux-yocto, perf, kernel > modules and anything else using it). We may need to add some better > error cases to catch problems. Not an insurmountable problem, just one > that will likely need to be addressed. > > I do feel the whole situation with the current kernel size is out of > control and badly affecting user experience. I like (b) myself, since I'm getting tired of copying the source tree, having the removals fail as we march forward through kernel versions, and then needing to modify everything from the SDK to the out of tree module builds to reconstruct scripts, etc. We'd have the same "don't patch it" restrictions for something that builds out of this shared directory (like perf), so there's nothing new and horrible there. Cheers, Bruce > > Cheers, > > Richard > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Packaging kernel sources 2014-09-10 14:11 ` Bruce Ashfield @ 2014-09-10 14:24 ` Richard Purdie 2014-09-10 14:42 ` Bruce Ashfield 0 siblings, 1 reply; 11+ messages in thread From: Richard Purdie @ 2014-09-10 14:24 UTC (permalink / raw) To: Bruce Ashfield Cc: Brandt, Todd E, Darren Hart, Tom Zanussi, Koen Kooi, Openembedded-core On Wed, 2014-09-10 at 10:11 -0400, Bruce Ashfield wrote: > On 14-09-10 04:27 AM, Richard Purdie wrote: > > On Tue, 2014-09-09 at 17:42 -0700, Darren Hart wrote: > >> I'm working on a project which needs to have the full kernel sources > >> installed on the target. The kernel-dev package as defined by > >> kernel.bbclass is heavily pruned to minimize packaging time and size and > >> is intended to enable building of external modules on the target. > >> > >> Is there an accepted best-practice for how to get the full source packaged > >> and installed? I can easily write a new recipe, > >> linux-custom-source_git.bb, to install the sources, for example, without > >> impacting the packaging time of "virtual/kernel" package. > >> > >> It would be nice in some respects for it to all come from the same recipe > >> though, but I suspect the impact to the common-case where this is not need > >> would be far too great. > > > > Personally, I'm leaning towards a couple of big changes in this area: > > I'd prefer this sort of larger switch as well. > > > > > a) "binning" the kernel-dev package and replacing it with some kind of > > separate full source recipe like this. > > > > The benefit is a fully functional on target source which is only built > > by people who care about it. This means for most users/builds, we no > > longer need to generate that huge package. The downside is a little more > > complexity for those that needs this but its not much. > > And just so I'm clear, in this case, the current sysroot, grep, sed, > hack-a-thon version of the kernel would stay as is ? No, these are two things I think we need to do together. There are two issues, the on target one and the sysroot mess. a) deals with the former, b) the latter. So b) takes care of the hack-a-thon but I think we need a) as well. > > b) binning the separate kernel staging dir and making it work more like > > the gcc shared work directory. This means external module builds and the > > tools like perf and so on would use this shared source directory. > > > > The benefit would be that we no longer have the huge install step in the > > main kernel recipe and the populate_sysroot step shinks in size. > > > > The downside has more impact here, the problem with shared work is that > > it cannot be removed once extracted since the system never knows when > > something else may need to use it. For gcc the argument was that we have > > so many users (gcc-cross-initial, gcc-cross, gcc-runtime, > > gcc-cross-canadian, gcc-crosssdk, gcc-crosssdk-initial and so on) that > > the multiple copies were far worse. For the kernel, we can argue that we > > have a ton of disk usage from it in the sysroot anyway so this change > > just makes things more efficient effectively. > > > > The other issue is that for shared work dirs, the stamps need to be kept > > in sync, if they step out, odd things happen (i.e. do_fetch, do_unpack, > > do_patch task checksums need to match for linux-yocto, perf, kernel > > modules and anything else using it). We may need to add some better > > error cases to catch problems. Not an insurmountable problem, just one > > that will likely need to be addressed. > > > > I do feel the whole situation with the current kernel size is out of > > control and badly affecting user experience. > > I like (b) myself, since I'm getting tired of copying the source tree, > having the removals fail as we march forward through kernel versions, > and then needing to modify everything from the SDK to the out of tree > module builds to reconstruct scripts, etc. > > We'd have the same "don't patch it" restrictions for something that > builds out of this shared directory (like perf), so there's nothing > new and horrible there. Right... Cheers, Richard ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Packaging kernel sources 2014-09-10 14:24 ` Richard Purdie @ 2014-09-10 14:42 ` Bruce Ashfield 0 siblings, 0 replies; 11+ messages in thread From: Bruce Ashfield @ 2014-09-10 14:42 UTC (permalink / raw) To: Richard Purdie Cc: Brandt, Todd E, Darren Hart, Tom Zanussi, Koen Kooi, Openembedded-core On 14-09-10 10:24 AM, Richard Purdie wrote: > On Wed, 2014-09-10 at 10:11 -0400, Bruce Ashfield wrote: >> On 14-09-10 04:27 AM, Richard Purdie wrote: >>> On Tue, 2014-09-09 at 17:42 -0700, Darren Hart wrote: >>>> I'm working on a project which needs to have the full kernel sources >>>> installed on the target. The kernel-dev package as defined by >>>> kernel.bbclass is heavily pruned to minimize packaging time and size and >>>> is intended to enable building of external modules on the target. >>>> >>>> Is there an accepted best-practice for how to get the full source packaged >>>> and installed? I can easily write a new recipe, >>>> linux-custom-source_git.bb, to install the sources, for example, without >>>> impacting the packaging time of "virtual/kernel" package. >>>> >>>> It would be nice in some respects for it to all come from the same recipe >>>> though, but I suspect the impact to the common-case where this is not need >>>> would be far too great. >>> >>> Personally, I'm leaning towards a couple of big changes in this area: >> >> I'd prefer this sort of larger switch as well. >> >>> >>> a) "binning" the kernel-dev package and replacing it with some kind of >>> separate full source recipe like this. >>> >>> The benefit is a fully functional on target source which is only built >>> by people who care about it. This means for most users/builds, we no >>> longer need to generate that huge package. The downside is a little more >>> complexity for those that needs this but its not much. >> >> And just so I'm clear, in this case, the current sysroot, grep, sed, >> hack-a-thon version of the kernel would stay as is ? > > No, these are two things I think we need to do together. There are two > issues, the on target one and the sysroot mess. a) deals with the > former, b) the latter. > > So b) takes care of the hack-a-thon but I think we need a) as well. > Aha. With your clarification, I can see the difference now .. I hadn't had coffee yet. Grabbing a full source copy for the on-target needs, and not just the source that we've mangled or changed for our build purposes. Agreed that this also cleans up the target needs for kernel source and build. Bruce >>> b) binning the separate kernel staging dir and making it work more like >>> the gcc shared work directory. This means external module builds and the >>> tools like perf and so on would use this shared source directory. >>> >>> The benefit would be that we no longer have the huge install step in the >>> main kernel recipe and the populate_sysroot step shinks in size. >>> >>> The downside has more impact here, the problem with shared work is that >>> it cannot be removed once extracted since the system never knows when >>> something else may need to use it. For gcc the argument was that we have >>> so many users (gcc-cross-initial, gcc-cross, gcc-runtime, >>> gcc-cross-canadian, gcc-crosssdk, gcc-crosssdk-initial and so on) that >>> the multiple copies were far worse. For the kernel, we can argue that we >>> have a ton of disk usage from it in the sysroot anyway so this change >>> just makes things more efficient effectively. >>> >>> The other issue is that for shared work dirs, the stamps need to be kept >>> in sync, if they step out, odd things happen (i.e. do_fetch, do_unpack, >>> do_patch task checksums need to match for linux-yocto, perf, kernel >>> modules and anything else using it). We may need to add some better >>> error cases to catch problems. Not an insurmountable problem, just one >>> that will likely need to be addressed. >>> >>> I do feel the whole situation with the current kernel size is out of >>> control and badly affecting user experience. >> >> I like (b) myself, since I'm getting tired of copying the source tree, >> having the removals fail as we march forward through kernel versions, >> and then needing to modify everything from the SDK to the out of tree >> module builds to reconstruct scripts, etc. >> >> We'd have the same "don't patch it" restrictions for something that >> builds out of this shared directory (like perf), so there's nothing >> new and horrible there. > > Right... > > Cheers, > > Richard > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Packaging kernel sources 2014-09-10 8:27 ` Richard Purdie 2014-09-10 14:11 ` Bruce Ashfield @ 2014-09-10 15:13 ` Darren Hart [not found] ` <11E08D716F0541429B7042699DD5C1A170875BAD@FMSMSX103.amr.corp.intel.com> ` (2 more replies) 1 sibling, 3 replies; 11+ messages in thread From: Darren Hart @ 2014-09-10 15:13 UTC (permalink / raw) To: Richard Purdie, Ashfield, Bruce Cc: Brandt, Todd E, Koen Kooi, Tom Zanussi, Openembedded-core On 9/10/14, 1:27, "Richard Purdie" <richard.purdie@linuxfoundation.org> wrote: >On Tue, 2014-09-09 at 17:42 -0700, Darren Hart wrote: >> I'm working on a project which needs to have the full kernel sources >> installed on the target. The kernel-dev package as defined by >> kernel.bbclass is heavily pruned to minimize packaging time and size and >> is intended to enable building of external modules on the target. >> >> Is there an accepted best-practice for how to get the full source >>packaged >> and installed? I can easily write a new recipe, >> linux-custom-source_git.bb, to install the sources, for example, without >> impacting the packaging time of "virtual/kernel" package. >> >> It would be nice in some respects for it to all come from the same >>recipe >> though, but I suspect the impact to the common-case where this is not >>need >> would be far too great. > >Personally, I'm leaning towards a couple of big changes in this area: > >a) "binning" the kernel-dev package and replacing it with some kind of >separate full source recipe like this. > >The benefit is a fully functional on target source which is only built >by people who care about it. This means for most users/builds, we no >longer need to generate that huge package. The downside is a little more >complexity for those that needs this but its not much. The other downside is that the most common use case (building external modules) would now require a lot more disk space than with just kernel-dev (something like 150 MB more iirc). > > >b) binning the separate kernel staging dir and making it work more like >the gcc shared work directory. This means external module builds and the >tools like perf and so on would use this shared source directory. I was thinking along the same lines here as well. > >The benefit would be that we no longer have the huge install step in the >main kernel recipe and the populate_sysroot step shinks in size. > >The downside has more impact here, the problem with shared work is that >it cannot be removed once extracted since the system never knows when >something else may need to use it. For gcc the argument was that we have >so many users (gcc-cross-initial, gcc-cross, gcc-runtime, >gcc-cross-canadian, gcc-crosssdk, gcc-crosssdk-initial and so on) that >the multiple copies were far worse. For the kernel, we can argue that we >have a ton of disk usage from it in the sysroot anyway so this change >just makes things more efficient effectively. > >The other issue is that for shared work dirs, the stamps need to be kept >in sync, if they step out, odd things happen (i.e. do_fetch, do_unpack, >do_patch task checksums need to match for linux-yocto, perf, kernel >modules and anything else using it). We may need to add some better >error cases to catch problems. Not an insurmountable problem, just one >that will likely need to be addressed. Good points. > >I do feel the whole situation with the current kernel size is out of >control and badly affecting user experience. Yup. -- Darren Hart Open Source Technology Center darren.hart@intel.com Intel Corporation ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <11E08D716F0541429B7042699DD5C1A170875BAD@FMSMSX103.amr.corp.intel.com>]
* Re: Packaging kernel sources [not found] ` <11E08D716F0541429B7042699DD5C1A170875BAD@FMSMSX103.amr.corp.intel.com> @ 2014-09-10 16:13 ` Bruce Ashfield 0 siblings, 0 replies; 11+ messages in thread From: Bruce Ashfield @ 2014-09-10 16:13 UTC (permalink / raw) To: Brandt, Todd E, Darren Hart, Richard Purdie Cc: Koen Kooi, Tom Zanussi, Openembedded-core On 14-09-10 11:27 AM, Brandt, Todd E wrote: > I think David brings up a good point about only needing the kernel > source when something goes wrong. How about a compromise. What if we > provided a simply utility which pulls in the kernel source and > recreates the existing kernel image by using git (with the proper > commit). It could be installed from within the kernel package and be > generated by the linux-kernel recipe so that it has the proper commit > hashes (like a simple bash script). That way there's no wasted space. > I think I might just do that for the heck of it anyway. We have to respect however the kernel was built, patched, etc. So it just needs to be whatever was in the ${S} of what was built. Much of anything else would be recreating the patch process of the kernel build .. or maybe I'm misunderstanding what you are suggesting. Bruce > ________________________________________ > From: Darren Hart [dvhart@linux.intel.com] > Sent: Wednesday, September 10, 2014 8:13 AM > To: Richard Purdie; Ashfield, Bruce (Wind River) > Cc: Openembedded-core@lists.openembedded.org; Brandt, Todd E; Koen Kooi; Tom Zanussi > Subject: Re: [OE-core] Packaging kernel sources > > On 9/10/14, 1:27, "Richard Purdie" <richard.purdie@linuxfoundation.org> > wrote: > >> On Tue, 2014-09-09 at 17:42 -0700, Darren Hart wrote: >>> I'm working on a project which needs to have the full kernel sources >>> installed on the target. The kernel-dev package as defined by >>> kernel.bbclass is heavily pruned to minimize packaging time and size and >>> is intended to enable building of external modules on the target. >>> >>> Is there an accepted best-practice for how to get the full source >>> packaged >>> and installed? I can easily write a new recipe, >>> linux-custom-source_git.bb, to install the sources, for example, without >>> impacting the packaging time of "virtual/kernel" package. >>> >>> It would be nice in some respects for it to all come from the same >>> recipe >>> though, but I suspect the impact to the common-case where this is not >>> need >>> would be far too great. >> >> Personally, I'm leaning towards a couple of big changes in this area: >> >> a) "binning" the kernel-dev package and replacing it with some kind of >> separate full source recipe like this. >> >> The benefit is a fully functional on target source which is only built >> by people who care about it. This means for most users/builds, we no >> longer need to generate that huge package. The downside is a little more >> complexity for those that needs this but its not much. > > The other downside is that the most common use case (building external > modules) would now require a lot more disk space than with just kernel-dev > (something like 150 MB more iirc). > >> >> >> b) binning the separate kernel staging dir and making it work more like >> the gcc shared work directory. This means external module builds and the >> tools like perf and so on would use this shared source directory. > > I was thinking along the same lines here as well. > >> >> The benefit would be that we no longer have the huge install step in the >> main kernel recipe and the populate_sysroot step shinks in size. >> >> The downside has more impact here, the problem with shared work is that >> it cannot be removed once extracted since the system never knows when >> something else may need to use it. For gcc the argument was that we have >> so many users (gcc-cross-initial, gcc-cross, gcc-runtime, >> gcc-cross-canadian, gcc-crosssdk, gcc-crosssdk-initial and so on) that >> the multiple copies were far worse. For the kernel, we can argue that we >> have a ton of disk usage from it in the sysroot anyway so this change >> just makes things more efficient effectively. >> >> The other issue is that for shared work dirs, the stamps need to be kept >> in sync, if they step out, odd things happen (i.e. do_fetch, do_unpack, >> do_patch task checksums need to match for linux-yocto, perf, kernel >> modules and anything else using it). We may need to add some better >> error cases to catch problems. Not an insurmountable problem, just one >> that will likely need to be addressed. > > Good points. > >> >> I do feel the whole situation with the current kernel size is out of >> control and badly affecting user experience. > > > Yup. > > -- > Darren Hart Open Source Technology Center > darren.hart@intel.com Intel Corporation > > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Packaging kernel sources 2014-09-10 15:13 ` Darren Hart [not found] ` <11E08D716F0541429B7042699DD5C1A170875BAD@FMSMSX103.amr.corp.intel.com> @ 2014-09-10 16:26 ` Peter A. Bigot 2014-10-02 7:25 ` Robert Yang 2 siblings, 0 replies; 11+ messages in thread From: Peter A. Bigot @ 2014-09-10 16:26 UTC (permalink / raw) To: Darren Hart, Richard Purdie, Ashfield, Bruce Cc: Brandt, Todd E, Koen Kooi, Tom Zanussi, Openembedded-core On 09/10/2014 10:13 AM, Darren Hart wrote: > On 9/10/14, 1:27, "Richard Purdie" <richard.purdie@linuxfoundation.org> > wrote: > >> On Tue, 2014-09-09 at 17:42 -0700, Darren Hart wrote: >>> I'm working on a project which needs to have the full kernel sources >>> installed on the target. The kernel-dev package as defined by >>> kernel.bbclass is heavily pruned to minimize packaging time and size and >>> is intended to enable building of external modules on the target. >>> >>> Is there an accepted best-practice for how to get the full source >>> packaged >>> and installed? I can easily write a new recipe, >>> linux-custom-source_git.bb, to install the sources, for example, without >>> impacting the packaging time of "virtual/kernel" package. >>> >>> It would be nice in some respects for it to all come from the same >>> recipe >>> though, but I suspect the impact to the common-case where this is not >>> need >>> would be far too great. >> Personally, I'm leaning towards a couple of big changes in this area: >> >> a) "binning" the kernel-dev package and replacing it with some kind of >> separate full source recipe like this. >> >> The benefit is a fully functional on target source which is only built >> by people who care about it. This means for most users/builds, we no >> longer need to generate that huge package. The downside is a little more >> complexity for those that needs this but its not much. > The other downside is that the most common use case (building external > modules) would now require a lot more disk space than with just kernel-dev > (something like 150 MB more iirc). There are three use cases where I've wanted kernel source on target: 1) Building the whole kernel. Pretty rare, but could happen. Debian appears to provide a linux-source package that provides the whole source in /usr/src/linux-source-$(uname -r). I'd want it to come with the contents of ${S} at the point where the compile task was ready to run. I'd love it to also come with a shallow git repository already present: I can't see anybody wanting to do on-target kernel modifications without having SCM underneath, and it takes a long time to "git add --all" a kernel tree on a class 4 uSD card. It doesn't need the entire history, just maybe three commits: the upstream stable base release, a record of the changes/patches done by kern-tools or other kernel-building recipes, and a final commit that archives the as-built .config as a defconfig somewhere. 2) Building external modules. Very common, and AFAICT normally packaged as "linux-headers" on debian systems, living in /usr/src/linux-headers-$(uname -r). I think simply renaming kernel-dev to this and changing where it unpacks would do. No need for git here since its unlikely people would be modifying the headers. 3) Building device trees. I haven't figured out how to do this other than rebuilding the whole kernel which is a major pain, but inspection suggests the current kernel-dev (proposed linux-headers) might have that functionality. Peter > >> >> b) binning the separate kernel staging dir and making it work more like >> the gcc shared work directory. This means external module builds and the >> tools like perf and so on would use this shared source directory. > I was thinking along the same lines here as well. > >> The benefit would be that we no longer have the huge install step in the >> main kernel recipe and the populate_sysroot step shinks in size. >> >> The downside has more impact here, the problem with shared work is that >> it cannot be removed once extracted since the system never knows when >> something else may need to use it. For gcc the argument was that we have >> so many users (gcc-cross-initial, gcc-cross, gcc-runtime, >> gcc-cross-canadian, gcc-crosssdk, gcc-crosssdk-initial and so on) that >> the multiple copies were far worse. For the kernel, we can argue that we >> have a ton of disk usage from it in the sysroot anyway so this change >> just makes things more efficient effectively. >> >> The other issue is that for shared work dirs, the stamps need to be kept >> in sync, if they step out, odd things happen (i.e. do_fetch, do_unpack, >> do_patch task checksums need to match for linux-yocto, perf, kernel >> modules and anything else using it). We may need to add some better >> error cases to catch problems. Not an insurmountable problem, just one >> that will likely need to be addressed. > Good points. > >> I do feel the whole situation with the current kernel size is out of >> control and badly affecting user experience. > Yup. I don't know that there's much that can be done: I wouldn't want anything removing parts of the source tree from a kernel on a presumption that the user wouldn't want them installed. But yeah, the thing's a pig. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Packaging kernel sources 2014-09-10 15:13 ` Darren Hart [not found] ` <11E08D716F0541429B7042699DD5C1A170875BAD@FMSMSX103.amr.corp.intel.com> 2014-09-10 16:26 ` Peter A. Bigot @ 2014-10-02 7:25 ` Robert Yang 2 siblings, 0 replies; 11+ messages in thread From: Robert Yang @ 2014-10-02 7:25 UTC (permalink / raw) To: Darren Hart, Richard Purdie, Ashfield, Bruce Cc: Brandt, Todd E, Koen Kooi, Tom Zanussi, Openembedded-core Hello, How's this going on, please ? // Robert On 09/10/2014 11:13 PM, Darren Hart wrote: > On 9/10/14, 1:27, "Richard Purdie" <richard.purdie@linuxfoundation.org> > wrote: > >> On Tue, 2014-09-09 at 17:42 -0700, Darren Hart wrote: >>> I'm working on a project which needs to have the full kernel sources >>> installed on the target. The kernel-dev package as defined by >>> kernel.bbclass is heavily pruned to minimize packaging time and size and >>> is intended to enable building of external modules on the target. >>> >>> Is there an accepted best-practice for how to get the full source >>> packaged >>> and installed? I can easily write a new recipe, >>> linux-custom-source_git.bb, to install the sources, for example, without >>> impacting the packaging time of "virtual/kernel" package. >>> >>> It would be nice in some respects for it to all come from the same >>> recipe >>> though, but I suspect the impact to the common-case where this is not >>> need >>> would be far too great. >> >> Personally, I'm leaning towards a couple of big changes in this area: >> >> a) "binning" the kernel-dev package and replacing it with some kind of >> separate full source recipe like this. >> >> The benefit is a fully functional on target source which is only built >> by people who care about it. This means for most users/builds, we no >> longer need to generate that huge package. The downside is a little more >> complexity for those that needs this but its not much. > > The other downside is that the most common use case (building external > modules) would now require a lot more disk space than with just kernel-dev > (something like 150 MB more iirc). > >> >> >> b) binning the separate kernel staging dir and making it work more like >> the gcc shared work directory. This means external module builds and the >> tools like perf and so on would use this shared source directory. > > I was thinking along the same lines here as well. > >> >> The benefit would be that we no longer have the huge install step in the >> main kernel recipe and the populate_sysroot step shinks in size. >> >> The downside has more impact here, the problem with shared work is that >> it cannot be removed once extracted since the system never knows when >> something else may need to use it. For gcc the argument was that we have >> so many users (gcc-cross-initial, gcc-cross, gcc-runtime, >> gcc-cross-canadian, gcc-crosssdk, gcc-crosssdk-initial and so on) that >> the multiple copies were far worse. For the kernel, we can argue that we >> have a ton of disk usage from it in the sysroot anyway so this change >> just makes things more efficient effectively. >> >> The other issue is that for shared work dirs, the stamps need to be kept >> in sync, if they step out, odd things happen (i.e. do_fetch, do_unpack, >> do_patch task checksums need to match for linux-yocto, perf, kernel >> modules and anything else using it). We may need to add some better >> error cases to catch problems. Not an insurmountable problem, just one >> that will likely need to be addressed. > > Good points. > >> >> I do feel the whole situation with the current kernel size is out of >> control and badly affecting user experience. > > > Yup. > ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2014-10-02 7:25 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2014-09-10 0:42 Packaging kernel sources Darren Hart 2014-09-10 2:18 ` Koen Kooi 2014-09-10 6:28 ` Robert Yang 2014-09-10 8:27 ` Richard Purdie 2014-09-10 14:11 ` Bruce Ashfield 2014-09-10 14:24 ` Richard Purdie 2014-09-10 14:42 ` Bruce Ashfield 2014-09-10 15:13 ` Darren Hart [not found] ` <11E08D716F0541429B7042699DD5C1A170875BAD@FMSMSX103.amr.corp.intel.com> 2014-09-10 16:13 ` Bruce Ashfield 2014-09-10 16:26 ` Peter A. Bigot 2014-10-02 7:25 ` Robert Yang
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.