meta-openembedded CVEs, patch stats and AUH

meta-openembedded CVEs, patch stats and AUH

[Richard Purdie]

Patch Metrics and CVEs
----------------------

I'm now pleased to say we have CVE and patch metrics being generated
(Thanks Yoann and Ross):

https://autobuilder.yocto.io/pub/non-release/patchmetrics-meta-oe/

which for example generates this:

https://autobuilder.yocto.io/pub/non-release/patchmetrics-meta-oe/cve-status-master.txt

(268 CVEs)

This doesn't work for the LTS branches since the layer override being
used to generate it on master won't work for kirkstone/dunfell.

Those reports are generated live onto the site daily but aren't emailed
anywhere at this time.


Mirroring
---------

I covered mirroring for meta-oe layers here:

https://lists.openembedded.org/g/openembedded-devel/message/105754

Basically it works but we need to decide whether we can fix the
warnings to keep this enabled for dunfell/kirkstone.


Reproducibility
---------------

We can now test reproducibility for meta-oe on the autobuilder:
https://autobuilder.yoctoproject.org/typhoon/#/builders/155

In order to make this useful we really need a list of known issues so
that we can then exclude them and start showing regressions.


AUH
---

This is now working and generating updates. It takes a while to run and
generates a lot of emails.

https://autobuilder.yoctoproject.org/typhoon/#/builders/159/builds/8
(18 hours so far)

meta-oe summary:
https://lists.yoctoproject.org/g/test-list/message/425
TOTAL: attempted=206 succeeded=42(20.39%) failed=164(79.61%)

meta-python summary:
https://lists.yoctoproject.org/g/test-list/message/426
TOTAL: attempted=92 succeeded=50(54.35%) failed=42(45.65%)

meta-perl summary:
https://lists.yoctoproject.org/g/test-list/message/434
TOTAL: attempted=7 succeeded=5(71.43%) failed=2(28.57%)

meta-networking summary:
https://lists.yoctoproject.org/g/test-list/message/767
TOTAL: attempted=43 succeeded=8(18.60%) failed=35(81.40%)

so around ~350 emails of which ~105 were successful upgrades. I'm not
sure whether we want to enable that to send to openembedded-devel?
There are lot of failures and there are a lot of recipes where the
upstream status couldn't be checked. Is anyone interested in working on
improving things in either of those areas?

There are also various backend improvements we'd love to make to the
AUH code now we've seen how it works in practise in this role.


Summary
-------

All things considered, this shows what our tooling is capable of and
we've a good baseline. Whether/how we enable/use this going forward
depends a lot on whether people are willing to step up and help
maintain/process the results. It can't/won't be Khem/I trying to make
and keep it all working so this is where we need help from the wider
community.

Re: meta-openembedded CVEs, patch stats and AUH

[Khem Raj]

Awesome work !

On Thu, Nov 2, 2023 at 6:49 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> Patch Metrics and CVEs
> ----------------------
>
> I'm now pleased to say we have CVE and patch metrics being generated
> (Thanks Yoann and Ross):
>
> https://autobuilder.yocto.io/pub/non-release/patchmetrics-meta-oe/
>
> which for example generates this:
>
> https://autobuilder.yocto.io/pub/non-release/patchmetrics-meta-oe/cve-status-master.txt
>
> (268 CVEs)
>
> This doesn't work for the LTS branches since the layer override being
> used to generate it on master won't work for kirkstone/dunfell.
>
> Those reports are generated live onto the site daily but aren't emailed
> anywhere at this time.
>
>
> Mirroring
> ---------
>
> I covered mirroring for meta-oe layers here:
>
> https://lists.openembedded.org/g/openembedded-devel/message/105754
>
> Basically it works but we need to decide whether we can fix the
> warnings to keep this enabled for dunfell/kirkstone.
>
>
> Reproducibility
> ---------------
>
> We can now test reproducibility for meta-oe on the autobuilder:
> https://autobuilder.yoctoproject.org/typhoon/#/builders/155
>
> In order to make this useful we really need a list of known issues so
> that we can then exclude them and start showing regressions.
>
>
> AUH
> ---
>
> This is now working and generating updates. It takes a while to run and
> generates a lot of emails.
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/159/builds/8
> (18 hours so far)
>
> meta-oe summary:
> https://lists.yoctoproject.org/g/test-list/message/425
> TOTAL: attempted=206 succeeded=42(20.39%) failed=164(79.61%)
>
> meta-python summary:
> https://lists.yoctoproject.org/g/test-list/message/426
> TOTAL: attempted=92 succeeded=50(54.35%) failed=42(45.65%)
>
> meta-perl summary:
> https://lists.yoctoproject.org/g/test-list/message/434
> TOTAL: attempted=7 succeeded=5(71.43%) failed=2(28.57%)
>
> meta-networking summary:
> https://lists.yoctoproject.org/g/test-list/message/767
> TOTAL: attempted=43 succeeded=8(18.60%) failed=35(81.40%)
>
> so around ~350 emails of which ~105 were successful upgrades. I'm not
> sure whether we want to enable that to send to openembedded-devel?
> There are lot of failures and there are a lot of recipes where the
> upstream status couldn't be checked. Is anyone interested in working on
> improving things in either of those areas?
>
> There are also various backend improvements we'd love to make to the
> AUH code now we've seen how it works in practise in this role.
>
>
> Summary
> -------
>
> All things considered, this shows what our tooling is capable of and
> we've a good baseline. Whether/how we enable/use this going forward
> depends a lot on whether people are willing to step up and help
> maintain/process the results. It can't/won't be Khem/I trying to make
> and keep it all working so this is where we need help from the wider
> community.
>

Re: [oe] meta-openembedded CVEs, patch stats and AUH

[Tim Orling]

[-- Attachment #1: Type: text/plain, Size: 5337 bytes --]

On Thu, Nov 2, 2023 at 10:51 AM Khem Raj <raj.khem@gmail.com> wrote:

> Awesome work !


I agree. This is very exciting.


>
> On Thu, Nov 2, 2023 at 6:49 AM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> >
> > Patch Metrics and CVEs
> > ----------------------
> >
> > I'm now pleased to say we have CVE and patch metrics being generated
> > (Thanks Yoann and Ross):
> >
> > https://autobuilder.yocto.io/pub/non-release/patchmetrics-meta-oe/
> >
> > which for example generates this:
> >
> >
> https://autobuilder.yocto.io/pub/non-release/patchmetrics-meta-oe/cve-status-master.txt
> >
> > (268 CVEs)
> >
> > This doesn't work for the LTS branches since the layer override being
> > used to generate it on master won't work for kirkstone/dunfell.
> >
> > Those reports are generated live onto the site daily but aren't emailed
> > anywhere at this time.
> >
> >
> > Mirroring
> > ---------
> >
> > I covered mirroring for meta-oe layers here:
> >
> > https://lists.openembedded.org/g/openembedded-devel/message/105754
> >
> > Basically it works but we need to decide whether we can fix the
> > warnings to keep this enabled for dunfell/kirkstone.
> >
> >
> > Reproducibility
> > ---------------
> >
> > We can now test reproducibility for meta-oe on the autobuilder:
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/155
> >
> > In order to make this useful we really need a list of known issues so
> > that we can then exclude them and start showing regressions.
> >
> >
> > AUH
> > ---
> >
> > This is now working and generating updates. It takes a while to run and
> > generates a lot of emails.
> >
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/159/builds/8
> > (18 hours so far)
> >
> > meta-oe summary:
> > https://lists.yoctoproject.org/g/test-list/message/425
> > TOTAL: attempted=206 succeeded=42(20.39%) failed=164(79.61%)
> >
> > meta-python summary:
> > https://lists.yoctoproject.org/g/test-list/message/426
> > TOTAL: attempted=92 succeeded=50(54.35%) failed=42(45.65%)
> >
> > meta-perl summary:
> > https://lists.yoctoproject.org/g/test-list/message/434
> > TOTAL: attempted=7 succeeded=5(71.43%) failed=2(28.57%)
> >



> It is good to see this capability on the AB. I previously ran similar AUH
jobs for Perl and Python recipes (including meta-oe via inherit filters) on
some hardware I had access to (thank you Mark Hatley).

We saw some uptick in contributions, mostly to meta-python, as a result of
my prior auh@moto-timo.dev emails. We do not see many folks addressing the
AUH devtool/etc failures and those that do not even succeed to check
upstream status are somewhat masked because no email output is sent (it
only goes to stdout).


> > meta-networking summary:
> > https://lists.yoctoproject.org/g/test-list/message/767
> > TOTAL: attempted=43 succeeded=8(18.60%) failed=35(81.40%)
> >
> > so around ~350 emails of which ~105 were successful upgrades. I'm not
> > sure whether we want to enable that to send to openembedded-devel?
> > There are lot of failures and there are a lot of recipes where the
> > upstream status couldn't be checked. Is anyone interested in working on
> > improving things in either of those areas?
> >
> > There are also various backend improvements we'd love to make to the
> > AUH code now we've seen how it works in practise in this role.
> >
> >
> > Summary
> > -------
> >
> > All things considered, this shows what our tooling is capable of and
> > we've a good baseline. Whether/how we enable/use this going forward
> > depends a lot on whether people are willing to step up and help
> > maintain/process the results. It can't/won't be Khem/I trying to make
> > and keep it all working so this is where we need help from the wider
> > community.
> >


I don’t think anyone (including myself) that are “layer maintainers” can
handle this volume of updates either. I used to try to keep up, but the
flood of even successful updates is just too great. On average, each recipe
takes 10-60 minutes to upgrade, verify changes, add some helpful git commit
information, build and run time test. Some contributors send upgrades that
build, but runtime testing is lacking and e.g. dependency or license
changes are missed. Ptest could help with catching runtime failures, but we
still need people willing to review and fix the failures. Ptests can also
add a lot of additional dependencies (for which we do not have recipes) and
someone still needs to be responding to any failures. The devtool and other
AUH failures, such as po4a for several years, tell us perhaps some pruning
of recipes is timely.

The quality of these layers is a community effort. There is no magic team
waiting to do the work. It is all of you. It is all of us.

—Tim


>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#105844):
> https://lists.openembedded.org/g/openembedded-devel/message/105844
> Mute This Topic: https://lists.openembedded.org/mt/102342283/924729
> Group Owner: openembedded-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-devel/unsub [
> ticotimo@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 9348 bytes --]

Re: [oe] meta-openembedded CVEs, patch stats and AUH

[Marta Rybczynska]

On Thu, Nov 2, 2023 at 2:49 PM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> Patch Metrics and CVEs
> ----------------------
>
> I'm now pleased to say we have CVE and patch metrics being generated
> (Thanks Yoann and Ross):
>
> https://autobuilder.yocto.io/pub/non-release/patchmetrics-meta-oe/
>
> which for example generates this:
>
> https://autobuilder.yocto.io/pub/non-release/patchmetrics-meta-oe/cve-status-master.txt
>
> (268 CVEs)
>
> This doesn't work for the LTS branches since the layer override being
> used to generate it on master won't work for kirkstone/dunfell.
>
> Those reports are generated live onto the site daily but aren't emailed
> anywhere at this time.
>
>
> Mirroring
> ---------
>
> I covered mirroring for meta-oe layers here:
>
> https://lists.openembedded.org/g/openembedded-devel/message/105754
>
> Basically it works but we need to decide whether we can fix the
> warnings to keep this enabled for dunfell/kirkstone.
>
>
> Reproducibility
> ---------------
>
> We can now test reproducibility for meta-oe on the autobuilder:
> https://autobuilder.yoctoproject.org/typhoon/#/builders/155
>
> In order to make this useful we really need a list of known issues so
> that we can then exclude them and start showing regressions.
>
>
> AUH
> ---
>
> This is now working and generating updates. It takes a while to run and
> generates a lot of emails.
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/159/builds/8
> (18 hours so far)
>
> meta-oe summary:
> https://lists.yoctoproject.org/g/test-list/message/425
> TOTAL: attempted=206 succeeded=42(20.39%) failed=164(79.61%)
>
> meta-python summary:
> https://lists.yoctoproject.org/g/test-list/message/426
> TOTAL: attempted=92 succeeded=50(54.35%) failed=42(45.65%)
>
> meta-perl summary:
> https://lists.yoctoproject.org/g/test-list/message/434
> TOTAL: attempted=7 succeeded=5(71.43%) failed=2(28.57%)
>
> meta-networking summary:
> https://lists.yoctoproject.org/g/test-list/message/767
> TOTAL: attempted=43 succeeded=8(18.60%) failed=35(81.40%)
>
> so around ~350 emails of which ~105 were successful upgrades. I'm not
> sure whether we want to enable that to send to openembedded-devel?
> There are lot of failures and there are a lot of recipes where the
> upstream status couldn't be checked. Is anyone interested in working on
> improving things in either of those areas?
>
> There are also various backend improvements we'd love to make to the
> AUH code now we've seen how it works in practise in this role.
>
>
> Summary
> -------
>
> All things considered, this shows what our tooling is capable of and
> we've a good baseline. Whether/how we enable/use this going forward
> depends a lot on whether people are willing to step up and help
> maintain/process the results. It can't/won't be Khem/I trying to make
> and keep it all working so this is where we need help from the wider
> community.
>

This is a really nice work! For CVEs I was mostly aware of the situation, but
not for all layers. In short: from my first analysis quite many of the reported
issues are false postives related to NVD mismatches. Need curation and a
nice chat with NVD :)

I'm happy also to see the first AUH results.

To sum up, a nice advancement!

Kind regards,
Marta