* kernel-devsrc used to be full source - not the case since thud
@ 2019-05-23 13:56 Andrei Gherzan
2019-05-23 14:32 ` Bruce Ashfield
0 siblings, 1 reply; 11+ messages in thread
From: Andrei Gherzan @ 2019-05-23 13:56 UTC (permalink / raw)
To: Yocto discussion list; +Cc: zubair, bruce.ashfield
Hello,
This might have been discussed before. I couldn't find a relevant
thread, but if it is so, just link me to it.
Since thud, more specific since
commit 9af0f1a46bbb6ad9ee8b35957251f4aa826b023f
Author: Bruce Ashfield <bruce.ashfield@windriver.com>
Date: Sat Aug 18 22:50:44 2018 -0400
kernel-devsrc: restructure for out of tree (and on target) module builds
... we switched from a recipe that was deploying the entire source code
to one that provides mainly the kernel headers (but not only). This
change broke people expectations of this recipe while the description is
also confusing: "Development source linux kernel. When built, this
recipe packages the \source of the preferred virtual/kernel provider and
makes it available for full kernel \development or external module builds".
If size is not a problem (which can be the case when you compile on a
builder for example and deploy only a OOT kernel module through other
means), the full kernel source was a painless experience where things like
commit a9471601fedd1f5087304eaa5fd39b98ae220313
Author: Bruce Ashfield <bruce.ashfield@windriver.com>
Date: Thu Aug 30 09:45:41 2018 -0400
kernel-devsrc: fix arm/arm64 target module build
... would not appear. I understand the size impact on target and for
those cases, continuously maintaining this recipe with new
files/resources needed from the kernel, makes sense. So my proposal is
to have two recipes, for example kernel-devsrc and kernel-fullsrc
(kernel-src etc.) so people can choose what they need/want
deploying/using. Or even have another devsrc provider. I'm open to any
implementation detail. I'd just want to have an option for a full kernel
source recipe.
Regards,
--
Andrei Gherzan
gpg: rsa4096/D4D94F67AD0E9640 | t: @agherzan
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel-devsrc used to be full source - not the case since thud
2019-05-23 13:56 kernel-devsrc used to be full source - not the case since thud Andrei Gherzan
@ 2019-05-23 14:32 ` Bruce Ashfield
2019-05-23 14:39 ` Bruce Ashfield
2019-05-23 14:39 ` Andrei Gherzan
0 siblings, 2 replies; 11+ messages in thread
From: Bruce Ashfield @ 2019-05-23 14:32 UTC (permalink / raw)
To: Andrei Gherzan; +Cc: zubair, Yocto discussion list
[-- Attachment #1: Type: text/plain, Size: 2431 bytes --]
On Thu, May 23, 2019 at 9:56 AM Andrei Gherzan <andrei@gherzan.ro> wrote:
> Hello,
>
> This might have been discussed before. I couldn't find a relevant
> thread, but if it is so, just link me to it.
>
> Since thud, more specific since
>
> commit 9af0f1a46bbb6ad9ee8b35957251f4aa826b023f
> Author: Bruce Ashfield <bruce.ashfield@windriver.com>
> Date: Sat Aug 18 22:50:44 2018 -0400
> kernel-devsrc: restructure for out of tree (and on target) module
> builds
>
> ... we switched from a recipe that was deploying the entire source code
> to one that provides mainly the kernel headers (but not only). This
> change broke people expectations of this recipe while the description is
> also confusing: "Development source linux kernel. When built, this
> recipe packages the \source of the preferred virtual/kernel provider and
> makes it available for full kernel \development or external module builds".
>
> If size is not a problem (which can be the case when you compile on a
> builder for example and deploy only a OOT kernel module through other
> means), the full kernel source was a painless experience where things like
>
> commit a9471601fedd1f5087304eaa5fd39b98ae220313
> Author: Bruce Ashfield <bruce.ashfield@windriver.com>
> Date: Thu Aug 30 09:45:41 2018 -0400
> kernel-devsrc: fix arm/arm64 target module build
>
> ... would not appear. I understand the size impact on target and for
> those cases, continuously maintaining this recipe with new
> files/resources needed from the kernel, makes sense. So my proposal is
> to have two recipes, for example kernel-devsrc and kernel-fullsrc
> (kernel-src etc.) so people can choose what they need/want
> deploying/using. Or even have another devsrc provider. I'm open to any
> implementation detail. I'd just want to have an option for a full kernel
> source recipe.
>
This is already planned, and hidden in bugzilla somewhere. I'll have some
new kernel packaging
options available for the fall release.
Bruce
>
> Regards,
>
> --
> Andrei Gherzan
> gpg: rsa4096/D4D94F67AD0E9640 | t: @agherzan
>
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end
- "Use the force Harry" - Gandalf, Star Trek II
[-- Attachment #2: Type: text/html, Size: 3705 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel-devsrc used to be full source - not the case since thud
2019-05-23 14:32 ` Bruce Ashfield
@ 2019-05-23 14:39 ` Bruce Ashfield
2019-05-23 15:00 ` Andrei Gherzan
2019-05-23 14:39 ` Andrei Gherzan
1 sibling, 1 reply; 11+ messages in thread
From: Bruce Ashfield @ 2019-05-23 14:39 UTC (permalink / raw)
To: Andrei Gherzan; +Cc: zubair, Yocto discussion list
[-- Attachment #1: Type: text/plain, Size: 3731 bytes --]
On Thu, May 23, 2019 at 10:32 AM Bruce Ashfield <bruce.ashfield@gmail.com>
wrote:
>
>
> On Thu, May 23, 2019 at 9:56 AM Andrei Gherzan <andrei@gherzan.ro> wrote:
>
>> Hello,
>>
>> This might have been discussed before. I couldn't find a relevant
>> thread, but if it is so, just link me to it.
>>
>> Since thud, more specific since
>>
>> commit 9af0f1a46bbb6ad9ee8b35957251f4aa826b023f
>> Author: Bruce Ashfield <bruce.ashfield@windriver.com>
>> Date: Sat Aug 18 22:50:44 2018 -0400
>> kernel-devsrc: restructure for out of tree (and on target) module
>> builds
>>
>> ... we switched from a recipe that was deploying the entire source code
>> to one that provides mainly the kernel headers (but not only). This
>> change broke people expectations of this recipe while the description is
>> also confusing: "Development source linux kernel. When built, this
>> recipe packages the \source of the preferred virtual/kernel provider and
>> makes it available for full kernel \development or external module
>> builds".
>>
>> If size is not a problem (which can be the case when you compile on a
>> builder for example and deploy only a OOT kernel module through other
>> means), the full kernel source was a painless experience where things like
>>
>> commit a9471601fedd1f5087304eaa5fd39b98ae220313
>> Author: Bruce Ashfield <bruce.ashfield@windriver.com>
>> Date: Thu Aug 30 09:45:41 2018 -0400
>> kernel-devsrc: fix arm/arm64 target module build
>>
>> ... would not appear. I understand the size impact on target and for
>> those cases, continuously maintaining this recipe with new
>> files/resources needed from the kernel, makes sense. So my proposal is
>> to have two recipes, for example kernel-devsrc and kernel-fullsrc
>> (kernel-src etc.) so people can choose what they need/want
>> deploying/using. Or even have another devsrc provider. I'm open to any
>> implementation detail. I'd just want to have an option for a full kernel
>> source recipe.
>>
>
> This is already planned, and hidden in bugzilla somewhere. I'll have some
> new kernel packaging
> options available for the fall release.
>
It looks like the bugs that I was using for development were finally moved
to resolved (they were a bit old, and contained collected information on
various kernel packaging options .. my searching of bugzilla isn't turning
it up at the moment). So I just created a new bug to track the development
for 2.8.
The issue with the multiple kernel source providers is really about test
cycles. The smaller devsrc is for on-target module development and builds
against the exported uapi headers, and that is what the nightly / automated
tests will use. We had issues with both the amount of time it took to
package the entire source, and the amount of space that it took up on the
images. Hence the creation of devsrc.
With new kernel-source and kernel-headers packages (the working names),
they are really provided as references to the running kernel, and will
largely be an exercise left up to the developer to use them as they want.
Cheers,
Bruce
>
> Bruce
>
>
>
>>
>> Regards,
>>
>> --
>> Andrei Gherzan
>> gpg: rsa4096/D4D94F67AD0E9640 | t: @agherzan
>>
>> --
>> _______________________________________________
>> yocto mailing list
>> yocto@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/yocto
>>
>
>
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await thee
> at its end
> - "Use the force Harry" - Gandalf, Star Trek II
>
>
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end
- "Use the force Harry" - Gandalf, Star Trek II
[-- Attachment #2: Type: text/html, Size: 6038 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel-devsrc used to be full source - not the case since thud
2019-05-23 14:32 ` Bruce Ashfield
2019-05-23 14:39 ` Bruce Ashfield
@ 2019-05-23 14:39 ` Andrei Gherzan
2019-05-23 14:48 ` Bruce Ashfield
1 sibling, 1 reply; 11+ messages in thread
From: Andrei Gherzan @ 2019-05-23 14:39 UTC (permalink / raw)
To: Bruce Ashfield; +Cc: zubair, Yocto discussion list
[-- Attachment #1: Type: text/plain, Size: 2645 bytes --]
On 23/05/2019 15.32, Bruce Ashfield wrote:
>
>
> On Thu, May 23, 2019 at 9:56 AM Andrei Gherzan <andrei@gherzan.ro
> <mailto:andrei@gherzan.ro>> wrote:
>
> Hello,
>
> This might have been discussed before. I couldn't find a relevant
> thread, but if it is so, just link me to it.
>
> Since thud, more specific since
>
> commit 9af0f1a46bbb6ad9ee8b35957251f4aa826b023f
> Author: Bruce Ashfield <bruce.ashfield@windriver.com
> <mailto:bruce.ashfield@windriver.com>>
> Date: Sat Aug 18 22:50:44 2018 -0400
> kernel-devsrc: restructure for out of tree (and on target)
> module builds
>
> ... we switched from a recipe that was deploying the entire source
> code
> to one that provides mainly the kernel headers (but not only). This
> change broke people expectations of this recipe while the
> description is
> also confusing: "Development source linux kernel. When built, this
> recipe packages the \source of the preferred virtual/kernel
> provider and
> makes it available for full kernel \development or external module
> builds".
>
> If size is not a problem (which can be the case when you compile on a
> builder for example and deploy only a OOT kernel module through other
> means), the full kernel source was a painless experience where
> things like
>
> commit a9471601fedd1f5087304eaa5fd39b98ae220313
> Author: Bruce Ashfield <bruce.ashfield@windriver.com
> <mailto:bruce.ashfield@windriver.com>>
> Date: Thu Aug 30 09:45:41 2018 -0400
> kernel-devsrc: fix arm/arm64 target module build
>
> ... would not appear. I understand the size impact on target and for
> those cases, continuously maintaining this recipe with new
> files/resources needed from the kernel, makes sense. So my proposal is
> to have two recipes, for example kernel-devsrc and kernel-fullsrc
> (kernel-src etc.) so people can choose what they need/want
> deploying/using. Or even have another devsrc provider. I'm open to any
> implementation detail. I'd just want to have an option for a full
> kernel
> source recipe.
>
>
> This is already planned, and hidden in bugzilla somewhere. I'll have
> some new kernel packaging
> options available for the fall release.
Nice to hear that. Can you share more details on that because we plan to
implement it for BalenaOS and would be a waste to redo what you already
have or do it in different ways and reset afterwards.
--
Andrei Gherzan
gpg: rsa4096/D4D94F67AD0E9640 | t: @agherzan
[-- Attachment #2: Type: text/html, Size: 4606 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel-devsrc used to be full source - not the case since thud
2019-05-23 14:39 ` Andrei Gherzan
@ 2019-05-23 14:48 ` Bruce Ashfield
0 siblings, 0 replies; 11+ messages in thread
From: Bruce Ashfield @ 2019-05-23 14:48 UTC (permalink / raw)
To: Andrei Gherzan; +Cc: zubair, Yocto discussion list
[-- Attachment #1: Type: text/plain, Size: 3048 bytes --]
On Thu, May 23, 2019 at 10:39 AM Andrei Gherzan <andrei@gherzan.ro> wrote:
> On 23/05/2019 15.32, Bruce Ashfield wrote:
>
>
>
> On Thu, May 23, 2019 at 9:56 AM Andrei Gherzan <andrei@gherzan.ro> wrote:
>
>> Hello,
>>
>> This might have been discussed before. I couldn't find a relevant
>> thread, but if it is so, just link me to it.
>>
>> Since thud, more specific since
>>
>> commit 9af0f1a46bbb6ad9ee8b35957251f4aa826b023f
>> Author: Bruce Ashfield <bruce.ashfield@windriver.com>
>> Date: Sat Aug 18 22:50:44 2018 -0400
>> kernel-devsrc: restructure for out of tree (and on target) module
>> builds
>>
>> ... we switched from a recipe that was deploying the entire source code
>> to one that provides mainly the kernel headers (but not only). This
>> change broke people expectations of this recipe while the description is
>> also confusing: "Development source linux kernel. When built, this
>> recipe packages the \source of the preferred virtual/kernel provider and
>> makes it available for full kernel \development or external module
>> builds".
>>
>> If size is not a problem (which can be the case when you compile on a
>> builder for example and deploy only a OOT kernel module through other
>> means), the full kernel source was a painless experience where things like
>>
>> commit a9471601fedd1f5087304eaa5fd39b98ae220313
>> Author: Bruce Ashfield <bruce.ashfield@windriver.com>
>> Date: Thu Aug 30 09:45:41 2018 -0400
>> kernel-devsrc: fix arm/arm64 target module build
>>
>> ... would not appear. I understand the size impact on target and for
>> those cases, continuously maintaining this recipe with new
>> files/resources needed from the kernel, makes sense. So my proposal is
>> to have two recipes, for example kernel-devsrc and kernel-fullsrc
>> (kernel-src etc.) so people can choose what they need/want
>> deploying/using. Or even have another devsrc provider. I'm open to any
>> implementation detail. I'd just want to have an option for a full kernel
>> source recipe.
>>
>
> This is already planned, and hidden in bugzilla somewhere. I'll have some
> new kernel packaging
> options available for the fall release.
>
> Nice to hear that. Can you share more details on that because we plan to
> implement it for BalenaOS and would be a waste to redo what you already
> have or do it in different ways and reset afterwards.
>
The best place for that is in the bugzilla, I (re)created bug 13360 and
will start moving the details into that (they are scattered around).
Can you elaborate on the use case that is broken with devsrc ? I'd also
like to make changes to that package to support more common workflows if
we've missed them. Feel free to add them to the bug, that way I won't lose
track of them again.
Bruce
> --
> Andrei Gherzan
> gpg: rsa4096/D4D94F67AD0E9640 | t: @agherzan
>
>
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end
- "Use the force Harry" - Gandalf, Star Trek II
[-- Attachment #2: Type: text/html, Size: 5839 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel-devsrc used to be full source - not the case since thud
2019-05-23 14:39 ` Bruce Ashfield
@ 2019-05-23 15:00 ` Andrei Gherzan
2019-05-23 15:10 ` Bruce Ashfield
0 siblings, 1 reply; 11+ messages in thread
From: Andrei Gherzan @ 2019-05-23 15:00 UTC (permalink / raw)
To: Bruce Ashfield; +Cc: zubair, Yocto discussion list
[-- Attachment #1: Type: text/plain, Size: 5219 bytes --]
On 23/05/2019 15.39, Bruce Ashfield wrote:
>
>
> On Thu, May 23, 2019 at 10:32 AM Bruce Ashfield
> <bruce.ashfield@gmail.com <mailto:bruce.ashfield@gmail.com>> wrote:
>
>
>
> On Thu, May 23, 2019 at 9:56 AM Andrei Gherzan <andrei@gherzan.ro
> <mailto:andrei@gherzan.ro>> wrote:
>
> Hello,
>
> This might have been discussed before. I couldn't find a relevant
> thread, but if it is so, just link me to it.
>
> Since thud, more specific since
>
> commit 9af0f1a46bbb6ad9ee8b35957251f4aa826b023f
> Author: Bruce Ashfield <bruce.ashfield@windriver.com
> <mailto:bruce.ashfield@windriver.com>>
> Date: Sat Aug 18 22:50:44 2018 -0400
> kernel-devsrc: restructure for out of tree (and on target)
> module builds
>
> ... we switched from a recipe that was deploying the entire
> source code
> to one that provides mainly the kernel headers (but not only).
> This
> change broke people expectations of this recipe while the
> description is
> also confusing: "Development source linux kernel. When built, this
> recipe packages the \source of the preferred virtual/kernel
> provider and
> makes it available for full kernel \development or external
> module builds".
>
> If size is not a problem (which can be the case when you
> compile on a
> builder for example and deploy only a OOT kernel module
> through other
> means), the full kernel source was a painless experience where
> things like
>
> commit a9471601fedd1f5087304eaa5fd39b98ae220313
> Author: Bruce Ashfield <bruce.ashfield@windriver.com
> <mailto:bruce.ashfield@windriver.com>>
> Date: Thu Aug 30 09:45:41 2018 -0400
> kernel-devsrc: fix arm/arm64 target module build
>
> ... would not appear. I understand the size impact on target
> and for
> those cases, continuously maintaining this recipe with new
> files/resources needed from the kernel, makes sense. So my
> proposal is
> to have two recipes, for example kernel-devsrc and kernel-fullsrc
> (kernel-src etc.) so people can choose what they need/want
> deploying/using. Or even have another devsrc provider. I'm
> open to any
> implementation detail. I'd just want to have an option for a
> full kernel
> source recipe.
>
>
> This is already planned, and hidden in bugzilla somewhere. I'll
> have some new kernel packaging
> options available for the fall release.
>
>
> It looks like the bugs that I was using for development were finally
> moved to resolved (they were a bit old, and contained collected
> information on various kernel packaging options .. my searching of
> bugzilla isn't turning it up at the moment). So I just created a new
> bug to track the development for 2.8.
>
> The issue with the multiple kernel source providers is really about
> test cycles. The smaller devsrc is for on-target module development
> and builds against the exported uapi headers, and that is what the
> nightly / automated tests will use. We had issues with both the amount
> of time it took to package the entire source, and the amount of space
> that it took up on the images. Hence the creation of devsrc.
>
> With new kernel-source and kernel-headers packages (the working
> names), they are really provided as references to the running kernel,
> and will largely be an exercise left up to the developer to use them
> as they want.
Right. So is there anything that holds us from creating two recipes -
one for what we currently have and one for what we used to have pre-thud
- find some pretty names and start from that? I'm asking just in case
I'm missing any internal architecture discussions or so.
And about test cycles, I'm not sure I understand why having a provider
vs having two separate recipes impact automated tests. In both cases you
will select whatever you want to test - by using a specific recipe or by
setting a preferred provider (I don't have a strong opinion here but I
was just curious about the time impact on automated tests).
Replying to the other email (this thread was forked). We have maintained
a similar kernel headers approach for a couple of years now. And many
times we found ourselves in the situation where something had to be
added/removed/checked - that was either kernel fork specific or
something that was added due to kernel updates. And this loop of fixes
here and there was obviously completely resolved by just using the "old"
kernel-devsrc which always had "everything" in. A good example is commit
a9471601fedd1f5087304eaa5fd39b98ae220313 I'm mentioning above. This kind
of stuff will always be needed - or at least in our experience was a
pretty often loop (when you try to support one kernel headers recipe
against tens of BSPs, you'll need cases, quirks and so on).
Regards,
--
Andrei Gherzan
gpg: rsa4096/D4D94F67AD0E9640 | t: @agherzan
[-- Attachment #2: Type: text/html, Size: 8858 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel-devsrc used to be full source - not the case since thud
2019-05-23 15:00 ` Andrei Gherzan
@ 2019-05-23 15:10 ` Bruce Ashfield
2019-05-23 15:24 ` Andrei Gherzan
0 siblings, 1 reply; 11+ messages in thread
From: Bruce Ashfield @ 2019-05-23 15:10 UTC (permalink / raw)
To: Andrei Gherzan; +Cc: zubair, Yocto discussion list
[-- Attachment #1: Type: text/plain, Size: 6622 bytes --]
On Thu, May 23, 2019 at 11:00 AM Andrei Gherzan <andrei@gherzan.ro> wrote:
>
> On 23/05/2019 15.39, Bruce Ashfield wrote:
>
>
>
> On Thu, May 23, 2019 at 10:32 AM Bruce Ashfield <bruce.ashfield@gmail.com>
> wrote:
>
>>
>>
>> On Thu, May 23, 2019 at 9:56 AM Andrei Gherzan <andrei@gherzan.ro> wrote:
>>
>>> Hello,
>>>
>>> This might have been discussed before. I couldn't find a relevant
>>> thread, but if it is so, just link me to it.
>>>
>>> Since thud, more specific since
>>>
>>> commit 9af0f1a46bbb6ad9ee8b35957251f4aa826b023f
>>> Author: Bruce Ashfield <bruce.ashfield@windriver.com>
>>> Date: Sat Aug 18 22:50:44 2018 -0400
>>> kernel-devsrc: restructure for out of tree (and on target) module
>>> builds
>>>
>>> ... we switched from a recipe that was deploying the entire source code
>>> to one that provides mainly the kernel headers (but not only). This
>>> change broke people expectations of this recipe while the description is
>>> also confusing: "Development source linux kernel. When built, this
>>> recipe packages the \source of the preferred virtual/kernel provider and
>>> makes it available for full kernel \development or external module
>>> builds".
>>>
>>> If size is not a problem (which can be the case when you compile on a
>>> builder for example and deploy only a OOT kernel module through other
>>> means), the full kernel source was a painless experience where things
>>> like
>>>
>>> commit a9471601fedd1f5087304eaa5fd39b98ae220313
>>> Author: Bruce Ashfield <bruce.ashfield@windriver.com>
>>> Date: Thu Aug 30 09:45:41 2018 -0400
>>> kernel-devsrc: fix arm/arm64 target module build
>>>
>>> ... would not appear. I understand the size impact on target and for
>>> those cases, continuously maintaining this recipe with new
>>> files/resources needed from the kernel, makes sense. So my proposal is
>>> to have two recipes, for example kernel-devsrc and kernel-fullsrc
>>> (kernel-src etc.) so people can choose what they need/want
>>> deploying/using. Or even have another devsrc provider. I'm open to any
>>> implementation detail. I'd just want to have an option for a full kernel
>>> source recipe.
>>>
>>
>> This is already planned, and hidden in bugzilla somewhere. I'll have some
>> new kernel packaging
>> options available for the fall release.
>>
>
> It looks like the bugs that I was using for development were finally moved
> to resolved (they were a bit old, and contained collected information on
> various kernel packaging options .. my searching of bugzilla isn't turning
> it up at the moment). So I just created a new bug to track the development
> for 2.8.
>
> The issue with the multiple kernel source providers is really about test
> cycles. The smaller devsrc is for on-target module development and builds
> against the exported uapi headers, and that is what the nightly / automated
> tests will use. We had issues with both the amount of time it took to
> package the entire source, and the amount of space that it took up on the
> images. Hence the creation of devsrc.
>
> With new kernel-source and kernel-headers packages (the working names),
> they are really provided as references to the running kernel, and will
> largely be an exercise left up to the developer to use them as they want.
>
> Right. So is there anything that holds us from creating two recipes - one
> for what we currently have and one for what we used to have pre-thud - find
> some pretty names and start from that? I'm asking just in case I'm missing
> any internal architecture discussions or so.
>
In some other non-core layer ? Sure :D The pre-thud copying of the kernel
source was a disaster and needed to be killed with fire, as it was. It was
all special cases and I commonly needed to fight with it. What you see in
the current devsrc is not an invention from Yocto, it is based on the
redhat spec files, and also (a bit) on some of the debian packaging of
minimal source.
So no, I'd rather not restore the mess that was the pre-thud devsrc in any
form.
> And about test cycles, I'm not sure I understand why having a provider vs
> having two separate recipes impact automated tests. In both cases you will
> select whatever you want to test - by using a specific recipe or by setting
> a preferred provider (I don't have a strong opinion here but I was just
> curious about the time impact on automated tests).
>
If it goes into core, we would need to run a full set of tests against it
to make sure that it supported development tasks. That would mean on target
kernel module builds, as well as potential some new cases given that the
full source is available (i.e. kernel rebuild). We already do that for
kernel-devrsc, adding multiple providers of something similar means that
we'd then have to test the multiple providers against all the
architectures, across the actively supported reference kernels in any given
release .. something always breaks, so then I'm on the hook for sorting
through all the issues (devsrc was not bug free in its previous form).
Given the amount of test cycles available, that's not necessarily a trivial
thing to add. In the end, it would be up to Richard if thinks something
like multiple provides are a good thing, and to commit the test
resources/cycles to them.
> Replying to the other email (this thread was forked). We have maintained a
> similar kernel headers approach for a couple of years now. And many times
> we found ourselves in the situation where something had to be
> added/removed/checked - that was either kernel fork specific or something
> that was added due to kernel updates. And this loop of fixes here and there
> was obviously completely resolved by just using the "old" kernel-devsrc
> which always had "everything" in. A good example is commit
> a9471601fedd1f5087304eaa5fd39b98ae220313 I'm mentioning above. This kind of
> stuff will always be needed - or at least in our experience was a pretty
> often loop (when you try to support one kernel headers recipe against tens
> of BSPs, you'll need cases, quirks and so on).
>
Those small tweaks really are trivial, I don't consider them to be a big
load. The advantages of the smaller footprint package have outweighed any
tuning we've done since things switched over.
Bruce
> Regards,
>
> --
> Andrei Gherzan
> gpg: rsa4096/D4D94F67AD0E9640 | t: @agherzan
>
>
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end
- "Use the force Harry" - Gandalf, Star Trek II
[-- Attachment #2: Type: text/html, Size: 11687 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel-devsrc used to be full source - not the case since thud
2019-05-23 15:10 ` Bruce Ashfield
@ 2019-05-23 15:24 ` Andrei Gherzan
2019-05-23 16:11 ` Bruce Ashfield
0 siblings, 1 reply; 11+ messages in thread
From: Andrei Gherzan @ 2019-05-23 15:24 UTC (permalink / raw)
To: Bruce Ashfield; +Cc: zubair, Yocto discussion list
[-- Attachment #1: Type: text/plain, Size: 8085 bytes --]
On 23/05/2019 16.10, Bruce Ashfield wrote:
>
>
> On Thu, May 23, 2019 at 11:00 AM Andrei Gherzan <andrei@gherzan.ro
> <mailto:andrei@gherzan.ro>> wrote:
>
>
> On 23/05/2019 15.39, Bruce Ashfield wrote:
>>
>>
>> On Thu, May 23, 2019 at 10:32 AM Bruce Ashfield
>> <bruce.ashfield@gmail.com <mailto:bruce.ashfield@gmail.com>> wrote:
>>
>>
>>
>> On Thu, May 23, 2019 at 9:56 AM Andrei Gherzan
>> <andrei@gherzan.ro <mailto:andrei@gherzan.ro>> wrote:
>>
>> Hello,
>>
>> This might have been discussed before. I couldn't find a
>> relevant
>> thread, but if it is so, just link me to it.
>>
>> Since thud, more specific since
>>
>> commit 9af0f1a46bbb6ad9ee8b35957251f4aa826b023f
>> Author: Bruce Ashfield <bruce.ashfield@windriver.com
>> <mailto:bruce.ashfield@windriver.com>>
>> Date: Sat Aug 18 22:50:44 2018 -0400
>> kernel-devsrc: restructure for out of tree (and on
>> target) module builds
>>
>> ... we switched from a recipe that was deploying the
>> entire source code
>> to one that provides mainly the kernel headers (but not
>> only). This
>> change broke people expectations of this recipe while the
>> description is
>> also confusing: "Development source linux kernel. When
>> built, this
>> recipe packages the \source of the preferred
>> virtual/kernel provider and
>> makes it available for full kernel \development or
>> external module builds".
>>
>> If size is not a problem (which can be the case when you
>> compile on a
>> builder for example and deploy only a OOT kernel module
>> through other
>> means), the full kernel source was a painless experience
>> where things like
>>
>> commit a9471601fedd1f5087304eaa5fd39b98ae220313
>> Author: Bruce Ashfield <bruce.ashfield@windriver.com
>> <mailto:bruce.ashfield@windriver.com>>
>> Date: Thu Aug 30 09:45:41 2018 -0400
>> kernel-devsrc: fix arm/arm64 target module build
>>
>> ... would not appear. I understand the size impact on
>> target and for
>> those cases, continuously maintaining this recipe with new
>> files/resources needed from the kernel, makes sense. So
>> my proposal is
>> to have two recipes, for example kernel-devsrc and
>> kernel-fullsrc
>> (kernel-src etc.) so people can choose what they need/want
>> deploying/using. Or even have another devsrc provider.
>> I'm open to any
>> implementation detail. I'd just want to have an option
>> for a full kernel
>> source recipe.
>>
>>
>> This is already planned, and hidden in bugzilla somewhere.
>> I'll have some new kernel packaging
>> options available for the fall release.
>>
>>
>> It looks like the bugs that I was using for development were
>> finally moved to resolved (they were a bit old, and contained
>> collected information on various kernel packaging options .. my
>> searching of bugzilla isn't turning it up at the moment). So I
>> just created a new bug to track the development for 2.8.
>>
>> The issue with the multiple kernel source providers is really
>> about test cycles. The smaller devsrc is for on-target module
>> development and builds against the exported uapi headers, and
>> that is what the nightly / automated tests will use. We had
>> issues with both the amount of time it took to package the entire
>> source, and the amount of space that it took up on the images.
>> Hence the creation of devsrc.
>>
>> With new kernel-source and kernel-headers packages (the working
>> names), they are really provided as references to the running
>> kernel, and will largely be an exercise left up to the developer
>> to use them as they want.
>
> Right. So is there anything that holds us from creating two
> recipes - one for what we currently have and one for what we used
> to have pre-thud - find some pretty names and start from that? I'm
> asking just in case I'm missing any internal architecture
> discussions or so.
>
>
> In some other non-core layer ? Sure :D The pre-thud copying of the
> kernel source was a disaster and needed to be killed with fire, as it
> was. It was all special cases and I commonly needed to fight with it.
> What you see in the current devsrc is not an invention from Yocto, it
> is based on the redhat spec files, and also (a bit) on some of the
> debian packaging of minimal source.
I am aware of all the other distros having the similar approach (and
they do this dance for disk space optimization) but I don't understand
what were all the special cases you had to fight against? The recipe was
barely touched in years. For example there is only one commit on it in
2017.
>
> So no, I'd rather not restore the mess that was the pre-thud devsrc in
> any form.
>
>
> And about test cycles, I'm not sure I understand why having a
> provider vs having two separate recipes impact automated tests. In
> both cases you will select whatever you want to test - by using a
> specific recipe or by setting a preferred provider (I don't have a
> strong opinion here but I was just curious about the time impact
> on automated tests).
>
>
> If it goes into core, we would need to run a full set of tests against
> it to make sure that it supported development tasks. That would mean
> on target kernel module builds, as well as potential some new cases
> given that the full source is available (i.e. kernel rebuild). We
> already do that for kernel-devrsc, adding multiple providers of
> something similar means that we'd then have to test the multiple
> providers against all the architectures, across the actively supported
> reference kernels in any given release .. something always breaks, so
> then I'm on the hook for sorting through all the issues (devsrc was
> not bug free in its previous form). Given the amount of test cycles
> available, that's not necessarily a trivial thing to add. In the end,
> it would be up to Richard if thinks something like multiple provides
> are a good thing, and to commit the test resources/cycles to them.
I see. You were comparing something else. That makes sense.
>
>
>
> Replying to the other email (this thread was forked). We have
> maintained a similar kernel headers approach for a couple of years
> now. And many times we found ourselves in the situation where
> something had to be added/removed/checked - that was either kernel
> fork specific or something that was added due to kernel updates.
> And this loop of fixes here and there was obviously completely
> resolved by just using the "old" kernel-devsrc which always had
> "everything" in. A good example is commit
> a9471601fedd1f5087304eaa5fd39b98ae220313 I'm mentioning above.
> This kind of stuff will always be needed - or at least in our
> experience was a pretty often loop (when you try to support one
> kernel headers recipe against tens of BSPs, you'll need cases,
> quirks and so on).
>
> Those small tweaks really are trivial, I don't consider them to be a
> big load. The advantages of the smaller footprint package have
> outweighed any tuning we've done since things switched over.
Again, I 100% understand the space advantage. I'm trying to see if there
are any other advantages at all. In a world with infinite disk space :D
--
Andrei Gherzan
gpg: rsa4096/D4D94F67AD0E9640 | t: @agherzan
[-- Attachment #2: Type: text/html, Size: 15752 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel-devsrc used to be full source - not the case since thud
2019-05-23 15:24 ` Andrei Gherzan
@ 2019-05-23 16:11 ` Bruce Ashfield
2019-05-23 16:20 ` Andrei Gherzan
0 siblings, 1 reply; 11+ messages in thread
From: Bruce Ashfield @ 2019-05-23 16:11 UTC (permalink / raw)
To: Andrei Gherzan; +Cc: zubair, Yocto discussion list
[-- Attachment #1: Type: text/plain, Size: 8572 bytes --]
On Thu, May 23, 2019 at 11:24 AM Andrei Gherzan <andrei@gherzan.ro> wrote:
> On 23/05/2019 16.10, Bruce Ashfield wrote:
>
>
>
> On Thu, May 23, 2019 at 11:00 AM Andrei Gherzan <andrei@gherzan.ro> wrote:
>
>>
>> On 23/05/2019 15.39, Bruce Ashfield wrote:
>>
>>
>>
>> On Thu, May 23, 2019 at 10:32 AM Bruce Ashfield <bruce.ashfield@gmail.com>
>> wrote:
>>
>>>
>>>
>>> On Thu, May 23, 2019 at 9:56 AM Andrei Gherzan <andrei@gherzan.ro>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> This might have been discussed before. I couldn't find a relevant
>>>> thread, but if it is so, just link me to it.
>>>>
>>>> Since thud, more specific since
>>>>
>>>> commit 9af0f1a46bbb6ad9ee8b35957251f4aa826b023f
>>>> Author: Bruce Ashfield <bruce.ashfield@windriver.com>
>>>> Date: Sat Aug 18 22:50:44 2018 -0400
>>>> kernel-devsrc: restructure for out of tree (and on target) module
>>>> builds
>>>>
>>>> ... we switched from a recipe that was deploying the entire source code
>>>> to one that provides mainly the kernel headers (but not only). This
>>>> change broke people expectations of this recipe while the description is
>>>> also confusing: "Development source linux kernel. When built, this
>>>> recipe packages the \source of the preferred virtual/kernel provider and
>>>> makes it available for full kernel \development or external module
>>>> builds".
>>>>
>>>> If size is not a problem (which can be the case when you compile on a
>>>> builder for example and deploy only a OOT kernel module through other
>>>> means), the full kernel source was a painless experience where things
>>>> like
>>>>
>>>> commit a9471601fedd1f5087304eaa5fd39b98ae220313
>>>> Author: Bruce Ashfield <bruce.ashfield@windriver.com>
>>>> Date: Thu Aug 30 09:45:41 2018 -0400
>>>> kernel-devsrc: fix arm/arm64 target module build
>>>>
>>>> ... would not appear. I understand the size impact on target and for
>>>> those cases, continuously maintaining this recipe with new
>>>> files/resources needed from the kernel, makes sense. So my proposal is
>>>> to have two recipes, for example kernel-devsrc and kernel-fullsrc
>>>> (kernel-src etc.) so people can choose what they need/want
>>>> deploying/using. Or even have another devsrc provider. I'm open to any
>>>> implementation detail. I'd just want to have an option for a full kernel
>>>> source recipe.
>>>>
>>>
>>> This is already planned, and hidden in bugzilla somewhere. I'll have
>>> some new kernel packaging
>>> options available for the fall release.
>>>
>>
>> It looks like the bugs that I was using for development were finally
>> moved to resolved (they were a bit old, and contained collected information
>> on various kernel packaging options .. my searching of bugzilla isn't
>> turning it up at the moment). So I just created a new bug to track the
>> development for 2.8.
>>
>> The issue with the multiple kernel source providers is really about test
>> cycles. The smaller devsrc is for on-target module development and builds
>> against the exported uapi headers, and that is what the nightly / automated
>> tests will use. We had issues with both the amount of time it took to
>> package the entire source, and the amount of space that it took up on the
>> images. Hence the creation of devsrc.
>>
>> With new kernel-source and kernel-headers packages (the working names),
>> they are really provided as references to the running kernel, and will
>> largely be an exercise left up to the developer to use them as they want.
>>
>> Right. So is there anything that holds us from creating two recipes - one
>> for what we currently have and one for what we used to have pre-thud - find
>> some pretty names and start from that? I'm asking just in case I'm missing
>> any internal architecture discussions or so.
>>
>
> In some other non-core layer ? Sure :D The pre-thud copying of the kernel
> source was a disaster and needed to be killed with fire, as it was. It was
> all special cases and I commonly needed to fight with it. What you see in
> the current devsrc is not an invention from Yocto, it is based on the
> redhat spec files, and also (a bit) on some of the debian packaging of
> minimal source.
>
> I am aware of all the other distros having the similar approach (and they
> do this dance for disk space optimization) but I don't understand what were
> all the special cases you had to fight against? The recipe was barely
> touched in years. For example there is only one commit on it in 2017.
>
There were corner cases popping up with architectures and the newer
kernels. I had inflight changes that were tweaking the nasty find commands,
etc, trying to get everything we needed, minimize the time spent copying
and avoiding QA errors due to cross arch artifacts popping in. Let's just
say, I'm not copying and pruning the same way in a plainer "kernel-source"
package. Since it isn't installed by default, and is opt-in, it doesn't
have to be quite so careful on those fronts.
>
> So no, I'd rather not restore the mess that was the pre-thud devsrc in any
> form.
>
>
>> And about test cycles, I'm not sure I understand why having a provider vs
>> having two separate recipes impact automated tests. In both cases you will
>> select whatever you want to test - by using a specific recipe or by setting
>> a preferred provider (I don't have a strong opinion here but I was just
>> curious about the time impact on automated tests).
>>
>
> If it goes into core, we would need to run a full set of tests against it
> to make sure that it supported development tasks. That would mean on target
> kernel module builds, as well as potential some new cases given that the
> full source is available (i.e. kernel rebuild). We already do that for
> kernel-devrsc, adding multiple providers of something similar means that
> we'd then have to test the multiple providers against all the
> architectures, across the actively supported reference kernels in any given
> release .. something always breaks, so then I'm on the hook for sorting
> through all the issues (devsrc was not bug free in its previous form).
> Given the amount of test cycles available, that's not necessarily a trivial
> thing to add. In the end, it would be up to Richard if thinks something
> like multiple provides are a good thing, and to commit the test
> resources/cycles to them.
>
> I see. You were comparing something else. That makes sense.
>
>
>
>
>
>> Replying to the other email (this thread was forked). We have maintained
>> a similar kernel headers approach for a couple of years now. And many times
>> we found ourselves in the situation where something had to be
>> added/removed/checked - that was either kernel fork specific or something
>> that was added due to kernel updates. And this loop of fixes here and there
>> was obviously completely resolved by just using the "old" kernel-devsrc
>> which always had "everything" in. A good example is commit
>> a9471601fedd1f5087304eaa5fd39b98ae220313 I'm mentioning above. This kind of
>> stuff will always be needed - or at least in our experience was a pretty
>> often loop (when you try to support one kernel headers recipe against tens
>> of BSPs, you'll need cases, quirks and so on).
>>
> Those small tweaks really are trivial, I don't consider them to be a big
> load. The advantages of the smaller footprint package have outweighed any
> tuning we've done since things switched over.
>
> Again, I 100% understand the space advantage. I'm trying to see if there
> are any other advantages at all. In a world with infinite disk space :D
>
Nothing super dramatic :D The curated / whitelist approach does make sure
that nothing new slips into the package until we've tested and ok'd it
(with the downsides you are pointing out) .. so we have an easier route to
patch / workaround issues as they pop up (I remember before uapi and hand
sanitizing the kernel headers .. and wouldn't want to be that extreme ever
again).
But I completely agree, having a simple, full copy of the source for
situations where build/packaging time or diskspace are not a requirement is
a good thing. I'll sort through the last symlink / provider issues and get
something out ASAP.
Bruce
> --
> Andrei Gherzan
> gpg: rsa4096/D4D94F67AD0E9640 | t: @agherzan
>
>
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end
- "Use the force Harry" - Gandalf, Star Trek II
[-- Attachment #2: Type: text/html, Size: 17800 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel-devsrc used to be full source - not the case since thud
2019-05-23 16:11 ` Bruce Ashfield
@ 2019-05-23 16:20 ` Andrei Gherzan
2019-05-23 16:23 ` Bruce Ashfield
0 siblings, 1 reply; 11+ messages in thread
From: Andrei Gherzan @ 2019-05-23 16:20 UTC (permalink / raw)
To: Bruce Ashfield; +Cc: zubair, Yocto discussion list
[-- Attachment #1: Type: text/plain, Size: 10317 bytes --]
On 23/05/2019 17.11, Bruce Ashfield wrote:
>
>
> On Thu, May 23, 2019 at 11:24 AM Andrei Gherzan <andrei@gherzan.ro
> <mailto:andrei@gherzan.ro>> wrote:
>
> On 23/05/2019 16.10, Bruce Ashfield wrote:
>>
>>
>> On Thu, May 23, 2019 at 11:00 AM Andrei Gherzan
>> <andrei@gherzan.ro <mailto:andrei@gherzan.ro>> wrote:
>>
>>
>> On 23/05/2019 15.39, Bruce Ashfield wrote:
>>>
>>>
>>> On Thu, May 23, 2019 at 10:32 AM Bruce Ashfield
>>> <bruce.ashfield@gmail.com <mailto:bruce.ashfield@gmail.com>>
>>> wrote:
>>>
>>>
>>>
>>> On Thu, May 23, 2019 at 9:56 AM Andrei Gherzan
>>> <andrei@gherzan.ro <mailto:andrei@gherzan.ro>> wrote:
>>>
>>> Hello,
>>>
>>> This might have been discussed before. I couldn't
>>> find a relevant
>>> thread, but if it is so, just link me to it.
>>>
>>> Since thud, more specific since
>>>
>>> commit 9af0f1a46bbb6ad9ee8b35957251f4aa826b023f
>>> Author: Bruce Ashfield <bruce.ashfield@windriver.com
>>> <mailto:bruce.ashfield@windriver.com>>
>>> Date: Sat Aug 18 22:50:44 2018 -0400
>>> kernel-devsrc: restructure for out of tree (and
>>> on target) module builds
>>>
>>> ... we switched from a recipe that was deploying the
>>> entire source code
>>> to one that provides mainly the kernel headers (but
>>> not only). This
>>> change broke people expectations of this recipe
>>> while the description is
>>> also confusing: "Development source linux kernel.
>>> When built, this
>>> recipe packages the \source of the preferred
>>> virtual/kernel provider and
>>> makes it available for full kernel \development or
>>> external module builds".
>>>
>>> If size is not a problem (which can be the case when
>>> you compile on a
>>> builder for example and deploy only a OOT kernel
>>> module through other
>>> means), the full kernel source was a painless
>>> experience where things like
>>>
>>> commit a9471601fedd1f5087304eaa5fd39b98ae220313
>>> Author: Bruce Ashfield <bruce.ashfield@windriver.com
>>> <mailto:bruce.ashfield@windriver.com>>
>>> Date: Thu Aug 30 09:45:41 2018 -0400
>>> kernel-devsrc: fix arm/arm64 target module build
>>>
>>> ... would not appear. I understand the size impact
>>> on target and for
>>> those cases, continuously maintaining this recipe
>>> with new
>>> files/resources needed from the kernel, makes sense.
>>> So my proposal is
>>> to have two recipes, for example kernel-devsrc and
>>> kernel-fullsrc
>>> (kernel-src etc.) so people can choose what they
>>> need/want
>>> deploying/using. Or even have another devsrc
>>> provider. I'm open to any
>>> implementation detail. I'd just want to have an
>>> option for a full kernel
>>> source recipe.
>>>
>>>
>>> This is already planned, and hidden in bugzilla
>>> somewhere. I'll have some new kernel packaging
>>> options available for the fall release.
>>>
>>>
>>> It looks like the bugs that I was using for development were
>>> finally moved to resolved (they were a bit old, and
>>> contained collected information on various kernel packaging
>>> options .. my searching of bugzilla isn't turning it up at
>>> the moment). So I just created a new bug to track the
>>> development for 2.8.
>>>
>>> The issue with the multiple kernel source providers is
>>> really about test cycles. The smaller devsrc is for
>>> on-target module development and builds against the exported
>>> uapi headers, and that is what the nightly / automated tests
>>> will use. We had issues with both the amount of time it took
>>> to package the entire source, and the amount of space that
>>> it took up on the images. Hence the creation of devsrc.
>>>
>>> With new kernel-source and kernel-headers packages (the
>>> working names), they are really provided as references to
>>> the running kernel, and will largely be an exercise left up
>>> to the developer to use them as they want.
>>
>> Right. So is there anything that holds us from creating two
>> recipes - one for what we currently have and one for what we
>> used to have pre-thud - find some pretty names and start from
>> that? I'm asking just in case I'm missing any internal
>> architecture discussions or so.
>>
>>
>> In some other non-core layer ? Sure :D The pre-thud copying of
>> the kernel source was a disaster and needed to be killed with
>> fire, as it was. It was all special cases and I commonly needed
>> to fight with it. What you see in the current devsrc is not an
>> invention from Yocto, it is based on the redhat spec files, and
>> also (a bit) on some of the debian packaging of minimal source.
>
> I am aware of all the other distros having the similar approach
> (and they do this dance for disk space optimization) but I don't
> understand what were all the special cases you had to fight
> against? The recipe was barely touched in years. For example there
> is only one commit on it in 2017.
>
>
> There were corner cases popping up with architectures and the newer
> kernels. I had inflight changes that were tweaking the nasty find
> commands, etc, trying to get everything we needed, minimize the time
> spent copying and avoiding QA errors due to cross arch artifacts
> popping in. Let's just say, I'm not copying and pruning the same way
> in a plainer "kernel-source" package. Since it isn't installed by
> default, and is opt-in, it doesn't have to be quite so careful on
> those fronts.
>
>
>
>>
>> So no, I'd rather not restore the mess that was the pre-thud
>> devsrc in any form.
>>
>>
>> And about test cycles, I'm not sure I understand why having a
>> provider vs having two separate recipes impact automated
>> tests. In both cases you will select whatever you want to
>> test - by using a specific recipe or by setting a preferred
>> provider (I don't have a strong opinion here but I was just
>> curious about the time impact on automated tests).
>>
>>
>> If it goes into core, we would need to run a full set of tests
>> against it to make sure that it supported development tasks. That
>> would mean on target kernel module builds, as well as potential
>> some new cases given that the full source is available (i.e.
>> kernel rebuild). We already do that for kernel-devrsc, adding
>> multiple providers of something similar means that we'd then have
>> to test the multiple providers against all the architectures,
>> across the actively supported reference kernels in any given
>> release .. something always breaks, so then I'm on the hook for
>> sorting through all the issues (devsrc was not bug free in its
>> previous form). Given the amount of test cycles available,
>> that's not necessarily a trivial thing to add. In the end, it
>> would be up to Richard if thinks something like multiple provides
>> are a good thing, and to commit the test resources/cycles to them.
> I see. You were comparing something else. That makes sense.
>>
>
>>
>>
>> Replying to the other email (this thread was forked). We have
>> maintained a similar kernel headers approach for a couple of
>> years now. And many times we found ourselves in the situation
>> where something had to be added/removed/checked - that was
>> either kernel fork specific or something that was added due
>> to kernel updates. And this loop of fixes here and there was
>> obviously completely resolved by just using the "old"
>> kernel-devsrc which always had "everything" in. A good
>> example is commit a9471601fedd1f5087304eaa5fd39b98ae220313
>> I'm mentioning above. This kind of stuff will always be
>> needed - or at least in our experience was a pretty often
>> loop (when you try to support one kernel headers recipe
>> against tens of BSPs, you'll need cases, quirks and so on).
>>
>> Those small tweaks really are trivial, I don't consider them to
>> be a big load. The advantages of the smaller footprint package
>> have outweighed any tuning we've done since things switched over.
> Again, I 100% understand the space advantage. I'm trying to see if
> there are any other advantages at all. In a world with infinite
> disk space :D
>
>
> Nothing super dramatic :D The curated / whitelist approach does make
> sure that nothing new slips into the package until we've tested and
> ok'd it (with the downsides you are pointing out) .. so we have an
> easier route to patch / workaround issues as they pop up (I remember
> before uapi and hand sanitizing the kernel headers .. and wouldn't
> want to be that extreme ever again).
>
> But I completely agree, having a simple, full copy of the source for
> situations where build/packaging time or diskspace are not a
> requirement is a good thing. I'll sort through the last symlink /
> provider issues and get something out ASAP.
Thanks Bruce. Would you please CC me when you push that? Just to make
sure I don't miss it.
--
Andrei Gherzan
gpg: rsa4096/D4D94F67AD0E9640 | t: @agherzan
[-- Attachment #2: Type: text/html, Size: 23614 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel-devsrc used to be full source - not the case since thud
2019-05-23 16:20 ` Andrei Gherzan
@ 2019-05-23 16:23 ` Bruce Ashfield
0 siblings, 0 replies; 11+ messages in thread
From: Bruce Ashfield @ 2019-05-23 16:23 UTC (permalink / raw)
To: Andrei Gherzan; +Cc: zubair, Yocto discussion list
[-- Attachment #1: Type: text/plain, Size: 9078 bytes --]
On Thu, May 23, 2019 at 12:20 PM Andrei Gherzan <andrei@gherzan.ro> wrote:
>
> On 23/05/2019 17.11, Bruce Ashfield wrote:
>
>
>
> On Thu, May 23, 2019 at 11:24 AM Andrei Gherzan <andrei@gherzan.ro> wrote:
>
>> On 23/05/2019 16.10, Bruce Ashfield wrote:
>>
>>
>>
>> On Thu, May 23, 2019 at 11:00 AM Andrei Gherzan <andrei@gherzan.ro>
>> wrote:
>>
>>>
>>> On 23/05/2019 15.39, Bruce Ashfield wrote:
>>>
>>>
>>>
>>> On Thu, May 23, 2019 at 10:32 AM Bruce Ashfield <
>>> bruce.ashfield@gmail.com> wrote:
>>>
>>>>
>>>>
>>>> On Thu, May 23, 2019 at 9:56 AM Andrei Gherzan <andrei@gherzan.ro>
>>>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> This might have been discussed before. I couldn't find a relevant
>>>>> thread, but if it is so, just link me to it.
>>>>>
>>>>> Since thud, more specific since
>>>>>
>>>>> commit 9af0f1a46bbb6ad9ee8b35957251f4aa826b023f
>>>>> Author: Bruce Ashfield <bruce.ashfield@windriver.com>
>>>>> Date: Sat Aug 18 22:50:44 2018 -0400
>>>>> kernel-devsrc: restructure for out of tree (and on target) module
>>>>> builds
>>>>>
>>>>> ... we switched from a recipe that was deploying the entire source code
>>>>> to one that provides mainly the kernel headers (but not only). This
>>>>> change broke people expectations of this recipe while the description
>>>>> is
>>>>> also confusing: "Development source linux kernel. When built, this
>>>>> recipe packages the \source of the preferred virtual/kernel provider
>>>>> and
>>>>> makes it available for full kernel \development or external module
>>>>> builds".
>>>>>
>>>>> If size is not a problem (which can be the case when you compile on a
>>>>> builder for example and deploy only a OOT kernel module through other
>>>>> means), the full kernel source was a painless experience where things
>>>>> like
>>>>>
>>>>> commit a9471601fedd1f5087304eaa5fd39b98ae220313
>>>>> Author: Bruce Ashfield <bruce.ashfield@windriver.com>
>>>>> Date: Thu Aug 30 09:45:41 2018 -0400
>>>>> kernel-devsrc: fix arm/arm64 target module build
>>>>>
>>>>> ... would not appear. I understand the size impact on target and for
>>>>> those cases, continuously maintaining this recipe with new
>>>>> files/resources needed from the kernel, makes sense. So my proposal is
>>>>> to have two recipes, for example kernel-devsrc and kernel-fullsrc
>>>>> (kernel-src etc.) so people can choose what they need/want
>>>>> deploying/using. Or even have another devsrc provider. I'm open to any
>>>>> implementation detail. I'd just want to have an option for a full
>>>>> kernel
>>>>> source recipe.
>>>>>
>>>>
>>>> This is already planned, and hidden in bugzilla somewhere. I'll have
>>>> some new kernel packaging
>>>> options available for the fall release.
>>>>
>>>
>>> It looks like the bugs that I was using for development were finally
>>> moved to resolved (they were a bit old, and contained collected information
>>> on various kernel packaging options .. my searching of bugzilla isn't
>>> turning it up at the moment). So I just created a new bug to track the
>>> development for 2.8.
>>>
>>> The issue with the multiple kernel source providers is really about test
>>> cycles. The smaller devsrc is for on-target module development and builds
>>> against the exported uapi headers, and that is what the nightly / automated
>>> tests will use. We had issues with both the amount of time it took to
>>> package the entire source, and the amount of space that it took up on the
>>> images. Hence the creation of devsrc.
>>>
>>> With new kernel-source and kernel-headers packages (the working names),
>>> they are really provided as references to the running kernel, and will
>>> largely be an exercise left up to the developer to use them as they want.
>>>
>>> Right. So is there anything that holds us from creating two recipes -
>>> one for what we currently have and one for what we used to have pre-thud -
>>> find some pretty names and start from that? I'm asking just in case I'm
>>> missing any internal architecture discussions or so.
>>>
>>
>> In some other non-core layer ? Sure :D The pre-thud copying of the kernel
>> source was a disaster and needed to be killed with fire, as it was. It was
>> all special cases and I commonly needed to fight with it. What you see in
>> the current devsrc is not an invention from Yocto, it is based on the
>> redhat spec files, and also (a bit) on some of the debian packaging of
>> minimal source.
>>
>> I am aware of all the other distros having the similar approach (and they
>> do this dance for disk space optimization) but I don't understand what were
>> all the special cases you had to fight against? The recipe was barely
>> touched in years. For example there is only one commit on it in 2017.
>>
>
> There were corner cases popping up with architectures and the newer
> kernels. I had inflight changes that were tweaking the nasty find commands,
> etc, trying to get everything we needed, minimize the time spent copying
> and avoiding QA errors due to cross arch artifacts popping in. Let's just
> say, I'm not copying and pruning the same way in a plainer "kernel-source"
> package. Since it isn't installed by default, and is opt-in, it doesn't
> have to be quite so careful on those fronts.
>
>
>
>>
>> So no, I'd rather not restore the mess that was the pre-thud devsrc in
>> any form.
>>
>>
>>> And about test cycles, I'm not sure I understand why having a provider
>>> vs having two separate recipes impact automated tests. In both cases you
>>> will select whatever you want to test - by using a specific recipe or by
>>> setting a preferred provider (I don't have a strong opinion here but I was
>>> just curious about the time impact on automated tests).
>>>
>>
>> If it goes into core, we would need to run a full set of tests against it
>> to make sure that it supported development tasks. That would mean on target
>> kernel module builds, as well as potential some new cases given that the
>> full source is available (i.e. kernel rebuild). We already do that for
>> kernel-devrsc, adding multiple providers of something similar means that
>> we'd then have to test the multiple providers against all the
>> architectures, across the actively supported reference kernels in any given
>> release .. something always breaks, so then I'm on the hook for sorting
>> through all the issues (devsrc was not bug free in its previous form).
>> Given the amount of test cycles available, that's not necessarily a trivial
>> thing to add. In the end, it would be up to Richard if thinks something
>> like multiple provides are a good thing, and to commit the test
>> resources/cycles to them.
>>
>> I see. You were comparing something else. That makes sense.
>>
>>
>>
>>
>>
>>> Replying to the other email (this thread was forked). We have maintained
>>> a similar kernel headers approach for a couple of years now. And many times
>>> we found ourselves in the situation where something had to be
>>> added/removed/checked - that was either kernel fork specific or something
>>> that was added due to kernel updates. And this loop of fixes here and there
>>> was obviously completely resolved by just using the "old" kernel-devsrc
>>> which always had "everything" in. A good example is commit
>>> a9471601fedd1f5087304eaa5fd39b98ae220313 I'm mentioning above. This kind of
>>> stuff will always be needed - or at least in our experience was a pretty
>>> often loop (when you try to support one kernel headers recipe against tens
>>> of BSPs, you'll need cases, quirks and so on).
>>>
>> Those small tweaks really are trivial, I don't consider them to be a big
>> load. The advantages of the smaller footprint package have outweighed any
>> tuning we've done since things switched over.
>>
>> Again, I 100% understand the space advantage. I'm trying to see if there
>> are any other advantages at all. In a world with infinite disk space :D
>>
>
> Nothing super dramatic :D The curated / whitelist approach does make sure
> that nothing new slips into the package until we've tested and ok'd it
> (with the downsides you are pointing out) .. so we have an easier route to
> patch / workaround issues as they pop up (I remember before uapi and hand
> sanitizing the kernel headers .. and wouldn't want to be that extreme ever
> again).
>
> But I completely agree, having a simple, full copy of the source for
> situations where build/packaging time or diskspace are not a requirement is
> a good thing. I'll sort through the last symlink / provider issues and get
> something out ASAP.
>
> Thanks Bruce. Would you please CC me when you push that? Just to make sure
> I don't miss it.
>
Absolutely. I'll make sure you are copied.
Bruce
> --
> Andrei Gherzan
> gpg: rsa4096/D4D94F67AD0E9640 | t: @agherzan
>
>
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end
- "Use the force Harry" - Gandalf, Star Trek II
[-- Attachment #2: Type: text/html, Size: 23408 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2019-05-23 16:23 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-05-23 13:56 kernel-devsrc used to be full source - not the case since thud Andrei Gherzan
2019-05-23 14:32 ` Bruce Ashfield
2019-05-23 14:39 ` Bruce Ashfield
2019-05-23 15:00 ` Andrei Gherzan
2019-05-23 15:10 ` Bruce Ashfield
2019-05-23 15:24 ` Andrei Gherzan
2019-05-23 16:11 ` Bruce Ashfield
2019-05-23 16:20 ` Andrei Gherzan
2019-05-23 16:23 ` Bruce Ashfield
2019-05-23 14:39 ` Andrei Gherzan
2019-05-23 14:48 ` Bruce Ashfield
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.