use of yocto-cfg-fragments

use of yocto-cfg-fragments

[Sverdlin, Alexander]

Hello Bruce!

Trying to understand the usage of SRC_URI type=kmeta/kernel-cache in
commits
5c2129118797 ("virtualization/config: allow conditional use of yocto-cfg-fragments")
and 5be8686e659c ("kernel: fix conditional application of fragments").

I have an issue implementing some locally-maintained fragments, I'm adding
"file://features/debug;type=kmeta;destsuffix=features/debug" to SRC_URI in my
kernel recipe derived from kernel-yocto class and this actually works
fine for me until I try to combine this with meta-virtualization.

My current understanding is that
meta-virtualization/recipes-kernel/linux/linux-yocto_virtualization.inc
makes an assumption that any type=kmeta will point to upstream
git://git.yoctoproject.org/yocto-kernel-cache but this doesn't hold
for my use case.

The only type=kmeta is my local file:// storage and the following
linux-yocto_virtualization.inc part leads to fetch failure (fetch
from Internet not expected/forbidden):

# if kernel-yocto has been inherited (how we can check for configuration
# fragment merging suport at the moment, then add a dependency on the
# configuration fragment repository. This allows us to be sure that our
# features can be enabled via the fragments
do_kernel_metadata[depends] += "${@['', 'yocto-cfg-fragments-native:do_populate_sysroot'][(bb.data.inherits_class('kernel-yocto', d))]}"

Summary: 1 task failed:
virtual:native:.../projects/ccp-sd-virt/../../meta-virtualization/recipes-kernel/linux/yocto-cfg-fragments.bb:do_fetch

When I read the comment "if the kernel-yocto meta-data routine automatically
starts to add the recipe-sysroot-native..." I get a feeling that
this feature is still "work in progress". Are there any further developments
I could test in this direction?

What would be a proper resolution of the above type=kmeta usage conflict from
your PoV?

-- 
Alexander Sverdlin
Siemens AG
www.siemens.com

Re: use of yocto-cfg-fragments

[Bruce Ashfield]

Hi Alexander!

On Thu, Nov 2, 2023 at 6:24 AM Sverdlin, Alexander
<alexander.sverdlin@siemens.com> wrote:
>
> Hello Bruce!
>
> Trying to understand the usage of SRC_URI type=kmeta/kernel-cache in
> commits
> 5c2129118797 ("virtualization/config: allow conditional use of yocto-cfg-fragments")
> and 5be8686e659c ("kernel: fix conditional application of fragments").
>
> I have an issue implementing some locally-maintained fragments, I'm adding
> "file://features/debug;type=kmeta;destsuffix=features/debug" to SRC_URI in my
> kernel recipe derived from kernel-yocto class and this actually works
> fine for me until I try to combine this with meta-virtualization.
>
> My current understanding is that
> meta-virtualization/recipes-kernel/linux/linux-yocto_virtualization.inc
> makes an assumption that any type=kmeta will point to upstream
> git://git.yoctoproject.org/yocto-kernel-cache but this doesn't hold
> for my use case.

close.

The support assumes that if something is on the src_uri named
"kernel-cache" that it is a complete, similarly formatted repository
to the upstream / centralized one that I maintain

If that is found, it is preferred over the yocto-kernel-cache provided
by the yocto-cfg-fragments recipe.

That's why I have the similarly formatted stipulation above, since
it is locating a fragment based on a name and relative path. I could
easily drop that stipulation by doing more advanced searching,
but that hasn't been required so far.

All the above assumes that I don't have bugs in the implementation :)

side note: I just updated the yocto-cfg-fragments recipe to the latest
6.5 SRCREVs.

>
> The only type=kmeta is my local file:// storage and the following
> linux-yocto_virtualization.inc part leads to fetch failure (fetch
> from Internet not expected/forbidden):
>
> # if kernel-yocto has been inherited (how we can check for configuration
> # fragment merging suport at the moment, then add a dependency on the
> # configuration fragment repository. This allows us to be sure that our
> # features can be enabled via the fragments
> do_kernel_metadata[depends] += "${@['', 'yocto-cfg-fragments-native:do_populate_sysroot'][(bb.data.inherits_class('kernel-yocto', d))]}"
>
> Summary: 1 task failed:
> virtual:native:.../projects/ccp-sd-virt/../../meta-virtualization/recipes-kernel/linux/yocto-cfg-fragments.bb:do_fetch

At a glance, I'm not seeing how your local kmeta is causing
the full fragments recipe to fail to fetch.

>
> When I read the comment "if the kernel-yocto meta-data routine automatically
> starts to add the recipe-sysroot-native..." I get a feeling that
> this feature is still "work in progress". Are there any further developments
> I could test in this direction?

There's nothing that would fix that issue. The in-progress part of that
is that I plan to remove all of the local fragments in the virtualization
layer and rely on the cloned kernel-cache from that recipe.

I don't suppose there's a layer I can look at which has your fragments
and the related kernel bbappend ? I'd like to reproduce the issue
locally so I can debug it in more detail.

Bruce

>
> What would be a proper resolution of the above type=kmeta usage conflict from
> your PoV?
>
> --
> Alexander Sverdlin
> Siemens AG
> www.siemens.com



-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: use of yocto-cfg-fragments

[Sverdlin, Alexander]

Hello Bruce,

thanks for the quick reply!

On Thu, 2023-11-02 at 15:13 -0400, Bruce Ashfield wrote:
> > My current understanding is that
> > meta-virtualization/recipes-kernel/linux/linux-yocto_virtualization.inc
> > makes an assumption that any type=kmeta will point to upstream
> > git://git.yoctoproject.org/yocto-kernel-cache but this doesn't hold
> > for my use case.
> 
> close.
> 
> The support assumes that if something is on the src_uri named
> "kernel-cache" that it is a complete, similarly formatted repository
> to the upstream / centralized one that I maintain
> 
> If that is found, it is preferred over the yocto-kernel-cache provided
> by the yocto-cfg-fragments recipe.

[...]

> > The only type=kmeta is my local file:// storage and the following
> > linux-yocto_virtualization.inc part leads to fetch failure (fetch
> > from Internet not expected/forbidden):
> > 
> > # if kernel-yocto has been inherited (how we can check for configuration
> > # fragment merging suport at the moment, then add a dependency on the
> > # configuration fragment repository. This allows us to be sure that our
> > # features can be enabled via the fragments
> > do_kernel_metadata[depends] += "${@['', 'yocto-cfg-fragments-native:do_populate_sysroot'][(bb.data.inherits_class('kernel-yocto', d))]}"
> > 
> > Summary: 1 task failed:
> > virtual:native:.../projects/ccp-sd-virt/../../meta-virtualization/recipes-kernel/linux/yocto-cfg-fragments.bb:do_fetch
> 
> At a glance, I'm not seeing how your local kmeta is causing
> the full fragments recipe to fail to fetch.
> 

Fetch failure is expected, as there is simply no access to outside world and
all expected packages are mirrored.
Now the real issue is that providing local type=kmeta pulls
git://git.yoctoproject.org/yocto-kernel-cache and currently we require no
of the upstream fragments, so there is no point in mirroring the yocto-kernel-cache
repo just to satisfy fetch as we don't require these features.

Currently I see no way to control this behaviour...

> > When I read the comment "if the kernel-yocto meta-data routine automatically
> > starts to add the recipe-sysroot-native..." I get a feeling that
> > this feature is still "work in progress". Are there any further developments
> > I could test in this direction?
> 
> There's nothing that would fix that issue. The in-progress part of that
> is that I plan to remove all of the local fragments in the virtualization
> layer and rely on the cloned kernel-cache from that recipe.
> 
> I don't suppose there's a layer I can look at which has your fragments
> and the related kernel bbappend ? I'd like to reproduce the issue
> locally so I can debug it in more detail.

I cannot easily point you to something publicly available... But roughly it's a
combination of meta-virtualization and a kernel recipe derived from kernel-yocto.bbclass.

Now I long as one doesn't have any type=kmeta, meta-virtualization doesn't pull
git://git.yoctoproject.org/yocto-kernel-cache and adding
"file://features/debug;type=kmeta;destsuffix=features/debug" to SRC_URI causes
yocto-kernel-cache to be pulled even though there is no logical reason for this.

-- 
Alexander Sverdlin
Siemens AG
www.siemens.com

Re: use of yocto-cfg-fragments

[Bruce Ashfield]

On Fri, Nov 3, 2023 at 5:15 AM Sverdlin, Alexander
<alexander.sverdlin@siemens.com> wrote:
>
> Hello Bruce,
>
> thanks for the quick reply!
>
> On Thu, 2023-11-02 at 15:13 -0400, Bruce Ashfield wrote:
> > > My current understanding is that
> > > meta-virtualization/recipes-kernel/linux/linux-yocto_virtualization.inc
> > > makes an assumption that any type=kmeta will point to upstream
> > > git://git.yoctoproject.org/yocto-kernel-cache but this doesn't hold
> > > for my use case.
> >
> > close.
> >
> > The support assumes that if something is on the src_uri named
> > "kernel-cache" that it is a complete, similarly formatted repository
> > to the upstream / centralized one that I maintain
> >
> > If that is found, it is preferred over the yocto-kernel-cache provided
> > by the yocto-cfg-fragments recipe.
>
> [...]
>
> > > The only type=kmeta is my local file:// storage and the following
> > > linux-yocto_virtualization.inc part leads to fetch failure (fetch
> > > from Internet not expected/forbidden):
> > >
> > > # if kernel-yocto has been inherited (how we can check for configuration
> > > # fragment merging suport at the moment, then add a dependency on the
> > > # configuration fragment repository. This allows us to be sure that our
> > > # features can be enabled via the fragments
> > > do_kernel_metadata[depends] += "${@['', 'yocto-cfg-fragments-native:do_populate_sysroot'][(bb.data.inherits_class('kernel-yocto', d))]}"
> > >
> > > Summary: 1 task failed:
> > > virtual:native:.../projects/ccp-sd-virt/../../meta-virtualization/recipes-kernel/linux/yocto-cfg-fragments.bb:do_fetch
> >
> > At a glance, I'm not seeing how your local kmeta is causing
> > the full fragments recipe to fail to fetch.
> >
>
> Fetch failure is expected, as there is simply no access to outside world and
> all expected packages are mirrored.
> Now the real issue is that providing local type=kmeta pulls
> git://git.yoctoproject.org/yocto-kernel-cache and currently we require no
> of the upstream fragments, so there is no point in mirroring the yocto-kernel-cache
> repo just to satisfy fetch as we don't require these features.

if you are enabling the virtualization distro feature, and are including
meta-virtualization, that repository is a requirement. That's how it
ensures that subtle runtime errors don't creep in with incorrect kernel
configurations.

So why not just mirror it ?

>
> Currently I see no way to control this behaviour...
>
> > > When I read the comment "if the kernel-yocto meta-data routine automatically
> > > starts to add the recipe-sysroot-native..." I get a feeling that
> > > this feature is still "work in progress". Are there any further developments
> > > I could test in this direction?
> >
> > There's nothing that would fix that issue. The in-progress part of that
> > is that I plan to remove all of the local fragments in the virtualization
> > layer and rely on the cloned kernel-cache from that recipe.
> >
> > I don't suppose there's a layer I can look at which has your fragments
> > and the related kernel bbappend ? I'd like to reproduce the issue
> > locally so I can debug it in more detail.
>
> I cannot easily point you to something publicly available... But roughly it's a
> combination of meta-virtualization and a kernel recipe derived from kernel-yocto.bbclass.
>
> Now I long as one doesn't have any type=kmeta, meta-virtualization doesn't pull
> git://git.yoctoproject.org/yocto-kernel-cache and adding
> "file://features/debug;type=kmeta;destsuffix=features/debug" to SRC_URI causes
> yocto-kernel-cache to be pulled even though there is no logical reason for this.

That's not what my local tests show. It is keying off an inherit of
kernel-yocto,
not kmeta, but even by extension, seeing that fragment merging is available
(whether keyed of kmeta or the inherit of kernel-yocto), that is the logic as to
why it pulls it in (and the use of it is what I explained above).

Unless I have a way to see the issue locally, I can't easily address it (if
required).

Bruce

>
> --
> Alexander Sverdlin
> Siemens AG
> www.siemens.com



-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [meta-virtualization] use of yocto-cfg-fragments

[Bruce Ashfield]

On Fri, Nov 3, 2023 at 8:52 AM Bruce Ashfield via
lists.yoctoproject.org
<bruce.ashfield=gmail.com@lists.yoctoproject.org> wrote:
>
> On Fri, Nov 3, 2023 at 5:15 AM Sverdlin, Alexander
> <alexander.sverdlin@siemens.com> wrote:
> >
> > Hello Bruce,
> >
> > thanks for the quick reply!
> >
> > On Thu, 2023-11-02 at 15:13 -0400, Bruce Ashfield wrote:
> > > > My current understanding is that
> > > > meta-virtualization/recipes-kernel/linux/linux-yocto_virtualization.inc
> > > > makes an assumption that any type=kmeta will point to upstream
> > > > git://git.yoctoproject.org/yocto-kernel-cache but this doesn't hold
> > > > for my use case.
> > >
> > > close.
> > >
> > > The support assumes that if something is on the src_uri named
> > > "kernel-cache" that it is a complete, similarly formatted repository
> > > to the upstream / centralized one that I maintain
> > >
> > > If that is found, it is preferred over the yocto-kernel-cache provided
> > > by the yocto-cfg-fragments recipe.
> >
> > [...]
> >
> > > > The only type=kmeta is my local file:// storage and the following
> > > > linux-yocto_virtualization.inc part leads to fetch failure (fetch
> > > > from Internet not expected/forbidden):
> > > >
> > > > # if kernel-yocto has been inherited (how we can check for configuration
> > > > # fragment merging suport at the moment, then add a dependency on the
> > > > # configuration fragment repository. This allows us to be sure that our
> > > > # features can be enabled via the fragments
> > > > do_kernel_metadata[depends] += "${@['', 'yocto-cfg-fragments-native:do_populate_sysroot'][(bb.data.inherits_class('kernel-yocto', d))]}"
> > > >
> > > > Summary: 1 task failed:
> > > > virtual:native:.../projects/ccp-sd-virt/../../meta-virtualization/recipes-kernel/linux/yocto-cfg-fragments.bb:do_fetch
> > >
> > > At a glance, I'm not seeing how your local kmeta is causing
> > > the full fragments recipe to fail to fetch.
> > >
> >
> > Fetch failure is expected, as there is simply no access to outside world and
> > all expected packages are mirrored.
> > Now the real issue is that providing local type=kmeta pulls
> > git://git.yoctoproject.org/yocto-kernel-cache and currently we require no
> > of the upstream fragments, so there is no point in mirroring the yocto-kernel-cache
> > repo just to satisfy fetch as we don't require these features.
>
> if you are enabling the virtualization distro feature, and are including
> meta-virtualization, that repository is a requirement. That's how it
> ensures that subtle runtime errors don't creep in with incorrect kernel
> configurations.
>
> So why not just mirror it ?
>
> >
> > Currently I see no way to control this behaviour...
> >
> > > > When I read the comment "if the kernel-yocto meta-data routine automatically
> > > > starts to add the recipe-sysroot-native..." I get a feeling that
> > > > this feature is still "work in progress". Are there any further developments
> > > > I could test in this direction?
> > >
> > > There's nothing that would fix that issue. The in-progress part of that
> > > is that I plan to remove all of the local fragments in the virtualization
> > > layer and rely on the cloned kernel-cache from that recipe.
> > >
> > > I don't suppose there's a layer I can look at which has your fragments
> > > and the related kernel bbappend ? I'd like to reproduce the issue
> > > locally so I can debug it in more detail.
> >
> > I cannot easily point you to something publicly available... But roughly it's a
> > combination of meta-virtualization and a kernel recipe derived from kernel-yocto.bbclass.
> >
> > Now I long as one doesn't have any type=kmeta, meta-virtualization doesn't pull
> > git://git.yoctoproject.org/yocto-kernel-cache and adding
> > "file://features/debug;type=kmeta;destsuffix=features/debug" to SRC_URI causes
> > yocto-kernel-cache to be pulled even though there is no logical reason for this.
>
> That's not what my local tests show. It is keying off an inherit of
> kernel-yocto,
> not kmeta, but even by extension, seeing that fragment merging is available
> (whether keyed of kmeta or the inherit of kernel-yocto), that is the logic as to
> why it pulls it in (and the use of it is what I explained above).

Cancel that comment, I do see it now. And yes, if there's any kmeta fragments
in the SRC_URI that is used to construct the kernel include file name, which
then double checks if kernel-yocto is inherited, and the combination of the two
indicates that the fragments should be made available for the packages in the
layer to be properly supported at runtime.

There should be a way for you to manipulate the do_kernel_metadata[depends]
value in your own bbappend to remove the dependency on the repo.

I can also try and think up another variable or a way to exclude that
dependency
.. but again, it is an intended dependency, and I was trying for a lighter touch
with the existing checks, and I don't want to make them too complex or introduce
a bug where the fragments stop being pulled in where they have currently been
pulled as a dependency.

Bruce

>
> Unless I have a way to see the issue locally, I can't easily address it (if
> required).
>
> Bruce
>
> >
> > --
> > Alexander Sverdlin
> > Siemens AG
> > www.siemens.com
>
>
>
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#8416): https://lists.yoctoproject.org/g/meta-virtualization/message/8416
> Mute This Topic: https://lists.yoctoproject.org/mt/102338788/1050810
> Group Owner: meta-virtualization+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/meta-virtualization/leave/6693017/1050810/978058475/xyzzy [bruce.ashfield@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [meta-virtualization] use of yocto-cfg-fragments

[Sverdlin, Alexander]

Hello Bruce,

thanks for looking into this!

On Fri, 2023-11-03 at 09:00 -0400, Bruce Ashfield wrote:
> > > I cannot easily point you to something publicly available... But roughly it's a
> > > combination of meta-virtualization and a kernel recipe derived from kernel-yocto.bbclass.
> > > 
> > > Now I long as one doesn't have any type=kmeta, meta-virtualization doesn't pull
> > > git://git.yoctoproject.org/yocto-kernel-cache and adding
> > > "file://features/debug;type=kmeta;destsuffix=features/debug" to SRC_URI causes
> > > yocto-kernel-cache to be pulled even though there is no logical reason for this.
> > 
> > That's not what my local tests show. It is keying off an inherit of
> > kernel-yocto,
> > not kmeta, but even by extension, seeing that fragment merging is available
> > (whether keyed of kmeta or the inherit of kernel-yocto), that is the logic as to
> > why it pulls it in (and the use of it is what I explained above).
> 
> Cancel that comment, I do see it now. And yes, if there's any kmeta fragments
> in the SRC_URI that is used to construct the kernel include file name, which
> then double checks if kernel-yocto is inherited, and the combination of the two
> indicates that the fragments should be made available for the packages in the
> layer to be properly supported at runtime.
> 
> There should be a way for you to manipulate the do_kernel_metadata[depends]
> value in your own bbappend to remove the dependency on the repo.
> 
> I can also try and think up another variable or a way to exclude that
> dependency
> .. but again, it is an intended dependency, and I was trying for a lighter touch
> with the existing checks, and I don't want to make them too complex or introduce
> a bug where the fragments stop being pulled in where they have currently been
> pulled as a dependency.

This unfortunately means undoing the whole linux-yocto_virtualization.inc
with all appended KERNEL_MODULE_AUTOLOAD and KERNEL_FEATURES and turns to be
quite messy at the end.

So the logic of pulling the whole linux-yocto_virtualization.inc
if d.getVar('SRC_URI').find('type=kmeta')
is not so obvious... For those re-using type=kmeta for something else than
yocto-kernel-cache.

We tend to trust our kernel configs and want to know which modules we load...
For instance if we only do containers, we still might want meta-virtualization,
but not necessary KVM, right?

-- 
Alexander Sverdlin
Siemens AG
www.siemens.com

Re: [meta-virtualization] use of yocto-cfg-fragments

[Bruce Ashfield]

On Fri, Nov 3, 2023 at 9:14 AM Sverdlin, Alexander
<alexander.sverdlin@siemens.com> wrote:
>
> Hello Bruce,
>
> thanks for looking into this!
>
> On Fri, 2023-11-03 at 09:00 -0400, Bruce Ashfield wrote:
> > > > I cannot easily point you to something publicly available... But roughly it's a
> > > > combination of meta-virtualization and a kernel recipe derived from kernel-yocto.bbclass.
> > > >
> > > > Now I long as one doesn't have any type=kmeta, meta-virtualization doesn't pull
> > > > git://git.yoctoproject.org/yocto-kernel-cache and adding
> > > > "file://features/debug;type=kmeta;destsuffix=features/debug" to SRC_URI causes
> > > > yocto-kernel-cache to be pulled even though there is no logical reason for this.
> > >
> > > That's not what my local tests show. It is keying off an inherit of
> > > kernel-yocto,
> > > not kmeta, but even by extension, seeing that fragment merging is available
> > > (whether keyed of kmeta or the inherit of kernel-yocto), that is the logic as to
> > > why it pulls it in (and the use of it is what I explained above).
> >
> > Cancel that comment, I do see it now. And yes, if there's any kmeta fragments
> > in the SRC_URI that is used to construct the kernel include file name, which
> > then double checks if kernel-yocto is inherited, and the combination of the two
> > indicates that the fragments should be made available for the packages in the
> > layer to be properly supported at runtime.
> >
> > There should be a way for you to manipulate the do_kernel_metadata[depends]
> > value in your own bbappend to remove the dependency on the repo.
> >
> > I can also try and think up another variable or a way to exclude that
> > dependency
> > .. but again, it is an intended dependency, and I was trying for a lighter touch
> > with the existing checks, and I don't want to make them too complex or introduce
> > a bug where the fragments stop being pulled in where they have currently been
> > pulled as a dependency.
>
> This unfortunately means undoing the whole linux-yocto_virtualization.inc
> with all appended KERNEL_MODULE_AUTOLOAD and KERNEL_FEATURES and turns to be
> quite messy at the end.
>
> So the logic of pulling the whole linux-yocto_virtualization.inc
> if d.getVar('SRC_URI').find('type=kmeta')
> is not so obvious... For those re-using type=kmeta for something else than
> yocto-kernel-cache.

In theory, I created the type=kmeta for just that purpose, kernel-cache and
fragment extensions, so keying on it is at least consistent :)

But I do agree that it is pulling in (potential) support for ALL of the features
in meta-virt. That's a balance between having too many fine grained
distro features, versus the one big switch "virtualization".

It could be finer grained, let me see what I can come up with to slim things
down a bit .. but as it relates to the kernel-cache, even a more fine grained
trigger will want the kernel-cache for either virtualization,
containers, xen, etc.

I'll follow up later today, once I've thought about this a bit more.

Bruce

>
> We tend to trust our kernel configs and want to know which modules we load...
> For instance if we only do containers, we still might want meta-virtualization,
> but not necessary KVM, right?
>
> --
> Alexander Sverdlin
> Siemens AG
> www.siemens.com



-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [meta-virtualization] use of yocto-cfg-fragments

[Sverdlin, Alexander]

Hello Bruce,

On Fri, 2023-11-03 at 09:59 -0400, Bruce Ashfield wrote:
> In theory, I created the type=kmeta for just that purpose, kernel-cache and
> fragment extensions, so keying on it is at least consistent :)

I had this feeling regarding type=kmeta, trying to understand the limited
documentation and how to use it with file:// and nested directories.

However, I find it extremely useful to have custom .scc files with
the whole verification machinery in kernel-yocto.bbclass.

I doubt that each and every use case must be upstreamed and go
via yocto-kernel-cache, I believe users may have valid but very
specific .scc tailored for their use cases.

> > I can also try and think up another variable or a way to exclude that
> > dependency

So maybe another variable could be an answer.
Because it's hard to distinguish real type-kmeta yocto-kernel-cache
(especially if people would have their local, maybe partial mirrors)
from just couple of .scc files.

> But I do agree that it is pulling in (potential) support for ALL of the features
> in meta-virt. That's a balance between having too many fine grained
> distro features, versus the one big switch "virtualization".
> 
> It could be finer grained, let me see what I can come up with to slim things
> down a bit .. but as it relates to the kernel-cache, even a more fine grained
> trigger will want the kernel-cache for either virtualization,
> containers, xen, etc.
> 
> I'll follow up later today, once I've thought about this a bit more.

-- 
Alexander Sverdlin
Siemens AG
www.siemens.com

Re: [meta-virtualization] use of yocto-cfg-fragments

[Bruce Ashfield]

On Fri, Nov 3, 2023 at 10:21 AM Sverdlin, Alexander
<alexander.sverdlin@siemens.com> wrote:
>
> Hello Bruce,
>
> On Fri, 2023-11-03 at 09:59 -0400, Bruce Ashfield wrote:
> > In theory, I created the type=kmeta for just that purpose, kernel-cache and
> > fragment extensions, so keying on it is at least consistent :)
>
> I had this feeling regarding type=kmeta, trying to understand the limited
> documentation and how to use it with file:// and nested directories.
>
> However, I find it extremely useful to have custom .scc files with
> the whole verification machinery in kernel-yocto.bbclass.
>
> I doubt that each and every use case must be upstreamed and go
> via yocto-kernel-cache, I believe users may have valid but very
> specific .scc tailored for their use cases.

Right. I want to support your use case, and encourage the sharing of
useful fragments.  Definitely not a requirement to upstream everything.

But I did make the shared fragments a requirement for meta-virt with
the intention of deleting the local ones. That's why you see that
dependency kick in.

Having the concepts and tools used and being useful is a good first step
and I'm good with that, and like I said, I do want to support it.

>
> > > I can also try and think up another variable or a way to exclude that
> > > dependency
>
> So maybe another variable could be an answer.
> Because it's hard to distinguish real type-kmeta yocto-kernel-cache
> (especially if people would have their local, maybe partial mirrors)
> from just couple of .scc files.

Very likely this is the solution, have a way to opt-out. Knowing that
you can possibly run into issues with KERNEL_FEATURE validation
or with runtime if you do opt out. I'll probably emit a similar warning
to the one you get if you include meta-virt but don't enable the
virtualization distro feature.

Bruce

>
> > But I do agree that it is pulling in (potential) support for ALL of the features
> > in meta-virt. That's a balance between having too many fine grained
> > distro features, versus the one big switch "virtualization".
> >
> > It could be finer grained, let me see what I can come up with to slim things
> > down a bit .. but as it relates to the kernel-cache, even a more fine grained
> > trigger will want the kernel-cache for either virtualization,
> > containers, xen, etc.
> >
> > I'll follow up later today, once I've thought about this a bit more.
>
> --
> Alexander Sverdlin
> Siemens AG
> www.siemens.com



-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

[PATCH RFC] kernel: allow KERNEL_META_TYPE redefinition

[A. Sverdlin]

From: Alexander Sverdlin <alexander.sverdlin@siemens.com>

There are use cases for type=kmeta in kernel-yocto.bbclass-derived SRC_URI
which do not point to git://git.yoctoproject.org/yocto-kernel-cache but to
file:// local storage containing just some .scc files. It's not obvious
that they shall pull in the whole automatic kernel configuration of
meta-virtualization. Make this behaviour configurable by allowing
KERNEL_META_TYPE to be defined in local.conf or distro config.
---
 recipes-kernel/linux/linux-%.bbappend | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/recipes-kernel/linux/linux-%.bbappend b/recipes-kernel/linux/linux-%.bbappend
index 5f31578..c2fc91c 100644
--- a/recipes-kernel/linux/linux-%.bbappend
+++ b/recipes-kernel/linux/linux-%.bbappend
@@ -5,6 +5,6 @@ LINUX_MAJOR = "${@(d.getVar('LINUX_VERSION') or "x.y").split('.')[0]}"
 LINUX_MINOR = "${@(d.getVar('LINUX_VERSION') or "x.y").split('.')[1]}"
 
 
-KERNEL_META_TYPE = "${@'yocto' if d.getVar('SRC_URI').find('type=kmeta') > 0 else 'none'}"
+KERNEL_META_TYPE ?= "${@'yocto' if d.getVar('SRC_URI').find('type=kmeta') > 0 else 'none'}"
 
 include ${@bb.utils.contains('DISTRO_FEATURES', 'virtualization', 'linux-${KERNEL_META_TYPE}_${LINUX_MAJOR}.${LINUX_MINOR}_virtualization.inc', '', d)}
-- 
2.41.0

Re: [PATCH RFC] kernel: allow KERNEL_META_TYPE redefinition

[Bruce Ashfield]

On Mon, Nov 6, 2023 at 4:21 AM A. Sverdlin
<alexander.sverdlin@siemens.com> wrote:
>
> From: Alexander Sverdlin <alexander.sverdlin@siemens.com>
>
> There are use cases for type=kmeta in kernel-yocto.bbclass-derived SRC_URI
> which do not point to git://git.yoctoproject.org/yocto-kernel-cache but to
> file:// local storage containing just some .scc files. It's not obvious
> that they shall pull in the whole automatic kernel configuration of
> meta-virtualization. Make this behaviour configurable by allowing
> KERNEL_META_TYPE to be defined in local.conf or distro config.

This is an ok temporary solution, but I'd only merge something like
this with a bbwarning also being emitted at the same time.

As I keep looking at it, there's very little in the .inc file that isn't
controlled by a distro feature, so there's only small tweaks needed
to make sure that only fragments and kernel features that are
quite well targeted are enabled.

Having the kernel-cache as a dependency is not much to ask,
combined with the ability to use them to configure kernels
properly and audit the results.

The hours (more like weeks and months) spent debugging runtime
issues due to incorrectly configured kernels has been huge over
the lifetime of meta-virtualization. Asking for a few fragments to
be downloaded (which can be mirrored) is minor in the scope of
things.

Which is why if a kernel type that can merge fragments, but yet
is disabling the feature specification, the layer should emit a
warning. I'd then have another variable which inhibits the warning
message. Since that is something I can ask about in bug reports
and until the warning message is re-enabled, the reporter is on
their own.

Cheers,

Bruce

> ---
>  recipes-kernel/linux/linux-%.bbappend | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/recipes-kernel/linux/linux-%.bbappend b/recipes-kernel/linux/linux-%.bbappend
> index 5f31578..c2fc91c 100644
> --- a/recipes-kernel/linux/linux-%.bbappend
> +++ b/recipes-kernel/linux/linux-%.bbappend
> @@ -5,6 +5,6 @@ LINUX_MAJOR = "${@(d.getVar('LINUX_VERSION') or "x.y").split('.')[0]}"
>  LINUX_MINOR = "${@(d.getVar('LINUX_VERSION') or "x.y").split('.')[1]}"
>
>
> -KERNEL_META_TYPE = "${@'yocto' if d.getVar('SRC_URI').find('type=kmeta') > 0 else 'none'}"
> +KERNEL_META_TYPE ?= "${@'yocto' if d.getVar('SRC_URI').find('type=kmeta') > 0 else 'none'}"
>
>  include ${@bb.utils.contains('DISTRO_FEATURES', 'virtualization', 'linux-${KERNEL_META_TYPE}_${LINUX_MAJOR}.${LINUX_MINOR}_virtualization.inc', '', d)}
> --
> 2.41.0
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [meta-virtualization] [PATCH RFC] kernel: allow KERNEL_META_TYPE redefinition

[Bruce Ashfield]

On Mon, Nov 6, 2023 at 8:41 AM Bruce Ashfield via
lists.yoctoproject.org
<bruce.ashfield=gmail.com@lists.yoctoproject.org> wrote:
>
> On Mon, Nov 6, 2023 at 4:21 AM A. Sverdlin
> <alexander.sverdlin@siemens.com> wrote:
> >
> > From: Alexander Sverdlin <alexander.sverdlin@siemens.com>
> >
> > There are use cases for type=kmeta in kernel-yocto.bbclass-derived SRC_URI
> > which do not point to git://git.yoctoproject.org/yocto-kernel-cache but to
> > file:// local storage containing just some .scc files. It's not obvious
> > that they shall pull in the whole automatic kernel configuration of
> > meta-virtualization. Make this behaviour configurable by allowing
> > KERNEL_META_TYPE to be defined in local.conf or distro config.
>
> This is an ok temporary solution, but I'd only merge something like
> this with a bbwarning also being emitted at the same time.
>
> As I keep looking at it, there's very little in the .inc file that isn't
> controlled by a distro feature, so there's only small tweaks needed
> to make sure that only fragments and kernel features that are
> quite well targeted are enabled.
>
> Having the kernel-cache as a dependency is not much to ask,
> combined with the ability to use them to configure kernels
> properly and audit the results.
>
> The hours (more like weeks and months) spent debugging runtime
> issues due to incorrectly configured kernels has been huge over
> the lifetime of meta-virtualization. Asking for a few fragments to
> be downloaded (which can be mirrored) is minor in the scope of
> things.
>
> Which is why if a kernel type that can merge fragments, but yet
> is disabling the feature specification, the layer should emit a
> warning. I'd then have another variable which inhibits the warning
> message. Since that is something I can ask about in bug reports
> and until the warning message is re-enabled, the reporter is on
> their own.

My solution is now in master-next:

https://git.yoctoproject.org/meta-virtualization/commit/?h=master-next&id=45c2297a916516e5fc7b82dab46c4719590813b7

Bruce

>
> Cheers,
>
> Bruce
>
> > ---
> >  recipes-kernel/linux/linux-%.bbappend | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/recipes-kernel/linux/linux-%.bbappend b/recipes-kernel/linux/linux-%.bbappend
> > index 5f31578..c2fc91c 100644
> > --- a/recipes-kernel/linux/linux-%.bbappend
> > +++ b/recipes-kernel/linux/linux-%.bbappend
> > @@ -5,6 +5,6 @@ LINUX_MAJOR = "${@(d.getVar('LINUX_VERSION') or "x.y").split('.')[0]}"
> >  LINUX_MINOR = "${@(d.getVar('LINUX_VERSION') or "x.y").split('.')[1]}"
> >
> >
> > -KERNEL_META_TYPE = "${@'yocto' if d.getVar('SRC_URI').find('type=kmeta') > 0 else 'none'}"
> > +KERNEL_META_TYPE ?= "${@'yocto' if d.getVar('SRC_URI').find('type=kmeta') > 0 else 'none'}"
> >
> >  include ${@bb.utils.contains('DISTRO_FEATURES', 'virtualization', 'linux-${KERNEL_META_TYPE}_${LINUX_MAJOR}.${LINUX_MINOR}_virtualization.inc', '', d)}
> > --
> > 2.41.0
> >
>
>
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#8425): https://lists.yoctoproject.org/g/meta-virtualization/message/8425
> Mute This Topic: https://lists.yoctoproject.org/mt/102416724/1050810
> Group Owner: meta-virtualization+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/meta-virtualization/leave/6693017/1050810/978058475/xyzzy [bruce.ashfield@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [meta-virtualization] [PATCH RFC] kernel: allow KERNEL_META_TYPE redefinition

[Sverdlin, Alexander]

Hi Bruce!

On Mon, 2023-11-06 at 19:57 -0500, Bruce Ashfield wrote:
> > As I keep looking at it, there's very little in the .inc file that isn't
> > controlled by a distro feature, so there's only small tweaks needed
> > to make sure that only fragments and kernel features that are
> > quite well targeted are enabled.
> > 
> > Having the kernel-cache as a dependency is not much to ask,
> > combined with the ability to use them to configure kernels
> > properly and audit the results.
> > 
> > The hours (more like weeks and months) spent debugging runtime
> > issues due to incorrectly configured kernels has been huge over
> > the lifetime of meta-virtualization. Asking for a few fragments to
> > be downloaded (which can be mirrored) is minor in the scope of
> > things.
> > 
> > Which is why if a kernel type that can merge fragments, but yet
> > is disabling the feature specification, the layer should emit a
> > warning. I'd then have another variable which inhibits the warning
> > message. Since that is something I can ask about in bug reports
> > and until the warning message is re-enabled, the reporter is on
> > their own.
> 
> My solution is now in master-next:
> 
> https://git.yoctoproject.org/meta-virtualization/commit/?h=master-next&id=45c2297a916516e5fc7b82dab46c4719590813b7

I've tested your solution and it solves the issue on my side,
thank you for addressing it so quickly!

There is a problem in the warning code though, namely a typo here:

skip_kernel_include_enabled = d.getVar('SKIP_META_VIRT_KERNEL_INLUDE')

if it would be

skip_kernel_include_enabled = d.getVar('SKIP_META_VIRT_KERNEL_INCLUDE')

the warning would be emitted as expected. After the change both emitting
and silencing the warning works fine.

-- 
Alexander Sverdlin
Siemens AG
www.siemens.com

Re: [meta-virtualization] [PATCH RFC] kernel: allow KERNEL_META_TYPE redefinition

[Bruce Ashfield]

On Tue, Nov 7, 2023 at 5:10 AM Sverdlin, Alexander
<alexander.sverdlin@siemens.com> wrote:
>
> Hi Bruce!
>
> On Mon, 2023-11-06 at 19:57 -0500, Bruce Ashfield wrote:
> > > As I keep looking at it, there's very little in the .inc file that isn't
> > > controlled by a distro feature, so there's only small tweaks needed
> > > to make sure that only fragments and kernel features that are
> > > quite well targeted are enabled.
> > >
> > > Having the kernel-cache as a dependency is not much to ask,
> > > combined with the ability to use them to configure kernels
> > > properly and audit the results.
> > >
> > > The hours (more like weeks and months) spent debugging runtime
> > > issues due to incorrectly configured kernels has been huge over
> > > the lifetime of meta-virtualization. Asking for a few fragments to
> > > be downloaded (which can be mirrored) is minor in the scope of
> > > things.
> > >
> > > Which is why if a kernel type that can merge fragments, but yet
> > > is disabling the feature specification, the layer should emit a
> > > warning. I'd then have another variable which inhibits the warning
> > > message. Since that is something I can ask about in bug reports
> > > and until the warning message is re-enabled, the reporter is on
> > > their own.
> >
> > My solution is now in master-next:
> >
> > https://git.yoctoproject.org/meta-virtualization/commit/?h=master-next&id=45c2297a916516e5fc7b82dab46c4719590813b7
>
> I've tested your solution and it solves the issue on my side,
> thank you for addressing it so quickly!
>
> There is a problem in the warning code though, namely a typo here:
>
> skip_kernel_include_enabled = d.getVar('SKIP_META_VIRT_KERNEL_INLUDE')
>
> if it would be
>
> skip_kernel_include_enabled = d.getVar('SKIP_META_VIRT_KERNEL_INCLUDE')
>
> the warning would be emitted as expected. After the change both emitting
> and silencing the warning works fine.

oops!

I just force pushed the fix to master-next.

Bruce

>
> --
> Alexander Sverdlin
> Siemens AG
> www.siemens.com



-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [meta-virtualization] [PATCH RFC] kernel: allow KERNEL_META_TYPE redefinition

[Sverdlin, Alexander]

Thanks Bruce,

On Tue, 2023-11-07 at 08:23 -0500, Bruce Ashfield wrote:
> > > > Which is why if a kernel type that can merge fragments, but yet
> > > > is disabling the feature specification, the layer should emit a
> > > > warning. I'd then have another variable which inhibits the warning
> > > > message. Since that is something I can ask about in bug reports
> > > > and until the warning message is re-enabled, the reporter is on
> > > > their own.
> > > 
> > > My solution is now in master-next:
> > > 
> > > https://git.yoctoproject.org/meta-virtualization/commit/?h=master-next&id=45c2297a916516e5fc7b82dab46c4719590813b7
> > 
> > I've tested your solution and it solves the issue on my side,
> > thank you for addressing it so quickly!
> > 
> > There is a problem in the warning code though, namely a typo here:
> > 
> > skip_kernel_include_enabled = d.getVar('SKIP_META_VIRT_KERNEL_INLUDE')
> > 
> > if it would be
> > 
> > skip_kernel_include_enabled = d.getVar('SKIP_META_VIRT_KERNEL_INCLUDE')
> > 
> > the warning would be emitted as expected. After the change both emitting
> > and silencing the warning works fine.
> 
> oops!
> 
> I just force pushed the fix to master-next.

Tested-by: Alexander Sverdlin <alexander.sverdlin@siemens.com>

if it matters

-- 
Alexander Sverdlin
Siemens AG
www.siemens.com

Re: [meta-virtualization] [PATCH RFC] kernel: allow KERNEL_META_TYPE redefinition

[Bruce Ashfield]

On Tue, Nov 7, 2023 at 8:57 AM Sverdlin, Alexander
<alexander.sverdlin@siemens.com> wrote:
>
> Thanks Bruce,
>
> On Tue, 2023-11-07 at 08:23 -0500, Bruce Ashfield wrote:
> > > > > Which is why if a kernel type that can merge fragments, but yet
> > > > > is disabling the feature specification, the layer should emit a
> > > > > warning. I'd then have another variable which inhibits the warning
> > > > > message. Since that is something I can ask about in bug reports
> > > > > and until the warning message is re-enabled, the reporter is on
> > > > > their own.
> > > >
> > > > My solution is now in master-next:
> > > >
> > > > https://git.yoctoproject.org/meta-virtualization/commit/?h=master-next&id=45c2297a916516e5fc7b82dab46c4719590813b7
> > >
> > > I've tested your solution and it solves the issue on my side,
> > > thank you for addressing it so quickly!
> > >
> > > There is a problem in the warning code though, namely a typo here:
> > >
> > > skip_kernel_include_enabled = d.getVar('SKIP_META_VIRT_KERNEL_INLUDE')
> > >
> > > if it would be
> > >
> > > skip_kernel_include_enabled = d.getVar('SKIP_META_VIRT_KERNEL_INCLUDE')
> > >
> > > the warning would be emitted as expected. After the change both emitting
> > > and silencing the warning works fine.
> >
> > oops!
> >
> > I just force pushed the fix to master-next.
>
> Tested-by: Alexander Sverdlin <alexander.sverdlin@siemens.com>
>
> if it matters

It does! Thanks for the confirmation.

Bruce

>
> --
> Alexander Sverdlin
> Siemens AG
> www.siemens.com



-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II