Outreachy internship project - license tracing enhancement

Outreachy internship project - license tracing enhancement

[Paul Eggleton]

Hi folks

I'd like to propose we put forward the following project proposal for an Outreachy internship (https://www.outreachy.org/communities/cfp/). I'm prepared to be the mentor for the project and Microsoft will provide funding. (Note that we haven't got our community re-registered with Outreachy or set this up as an intern project proposal yet - deadline for YP community registration is September 17th and for project submissions is September 24th). Here's the brief:

-------------
Yocto Project License tracing enhancement
 
The Yocto Project build system is typically used to build customised Linux images from source for embedded applications. Along with the image, a manifest of packages and their corresponding licenses is prepared, however the accuracy of the license information is dependent on the accuracy of the metadata we have for each package (i.e. what is in the recipe file). As part of the build, we have an internal mapping from output files to source files which is currently used to prepare source packages to aid in debugging, however with the presence of SPDX headers in source files it could also be used to allow tracing the license of sources used in building a package/image to help improve our metadata and future license manifests. A proof-of concept implementation of this has been put together [1] - during this internship a successful intern will:
1) take the proof-of-concept implementation and get it to a state where it can be merged into the poky repository
2) use the functionality to examine the accuracy of our license tagging (LICENSE fields in recipes); look for errors / noise in the comparison, and produce a simple report with the results
3) run a check over sources in a world build looking for percentage coverage of SPDX headers, and run it for several past releases to see the change over time
 
Bonus: assess the current state of meta-spdx-scanner; investigate what it would take to produce SPDX documents from build output (would likely require integration with Fossology).

[1] http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=rpurdie/license-experiments-osls
-------------

I'm making the assumption that we're OK with merging the PoC functionality in rather than just keeping it separate and using it for analysis - let me know if otherwise. What I'd really like to know is do people think that this is sufficient for a 3-month internship, assuming that the intern has limited to moderate familiarity with our codebase? Do we need to flesh it out further? Any modifications that you'd suggest to the work?

Thanks
Paul

Re: [yocto] Outreachy internship project - license tracing enhancement

[akuster]

On 9/9/20 3:51 AM, Paul Eggleton via lists.yoctoproject.org wrote:
> Hi folks
>
> I'd like to propose we put forward the following project proposal for an Outreachy internship (https://www.outreachy.org/communities/cfp/). I'm prepared to be the mentor for the project and Microsoft will provide funding. (Note that we haven't got our community re-registered with Outreachy or set this up as an intern project proposal yet - deadline for YP community registration is September 17th and for project submissions is September 24th). Here's the brief:

This sounds great.

Are you looking for YP to take action on  registration or are you
handling this?

-armin
>
> -------------
> Yocto Project License tracing enhancement
>  
> The Yocto Project build system is typically used to build customised Linux images from source for embedded applications. Along with the image, a manifest of packages and their corresponding licenses is prepared, however the accuracy of the license information is dependent on the accuracy of the metadata we have for each package (i.e. what is in the recipe file). As part of the build, we have an internal mapping from output files to source files which is currently used to prepare source packages to aid in debugging, however with the presence of SPDX headers in source files it could also be used to allow tracing the license of sources used in building a package/image to help improve our metadata and future license manifests. A proof-of concept implementation of this has been put together [1] - during this internship a successful intern will:
> 1) take the proof-of-concept implementation and get it to a state where it can be merged into the poky repository
> 2) use the functionality to examine the accuracy of our license tagging (LICENSE fields in recipes); look for errors / noise in the comparison, and produce a simple report with the results
> 3) run a check over sources in a world build looking for percentage coverage of SPDX headers, and run it for several past releases to see the change over time
>  
> Bonus: assess the current state of meta-spdx-scanner; investigate what it would take to produce SPDX documents from build output (would likely require integration with Fossology).
>
> [1] http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=rpurdie/license-experiments-osls
> -------------
>
> I'm making the assumption that we're OK with merging the PoC functionality in rather than just keeping it separate and using it for analysis - let me know if otherwise. What I'd really like to know is do people think that this is sufficient for a 3-month internship, assuming that the intern has limited to moderate familiarity with our codebase? Do we need to flesh it out further? Any modifications that you'd suggest to the work?
>
> Thanks
> Paul
>
> 

Re: [yocto] Outreachy internship project - license tracing enhancement

[Nicolas Dechesne]

[-- Attachment #1: Type: text/plain, Size: 5009 bytes --]

Hey Paul,

First of all, thank you so much for doing this. It's very much appreciated,
I am very happy to see our community members willing to contribute to such
a great internship program. As I mentioned to you separately, I will do my
best to support and help you!

On Wed, Sep 9, 2020 at 6:55 PM Paul Eggleton <Paul.Eggleton@microsoft.com>
wrote:

> Hi Armin
>
> I think Nicolas will need to do the re-registration as he's the named
> Outreachy coordinator for YP.
>

I don't need to re-register, I need to confirm that the YP community (which
is already registered) will participate in this round, and I need to
indicate how the funding is done. I will reach out privately to discuss the
funding side of things (there are a couple of questions/information to
provide on their form).

That said, if anyone else is willing to sponsor another internship,
please let me know here or privately!


> Thanks
> Paul
>
> -----Original Message-----
> From: akuster808 <akuster808@gmail.com>
> Sent: Thursday, 10 September 2020 2:29 am
> To: Paul Eggleton <Paul.Eggleton@microsoft.com>;
> yocto@lists.yoctoproject.org
> Cc: Nicolas Dechesne <nicolas.dechesne@linaro.org>; Richard Purdie (
> richard.purdie@linuxfoundation.org) <richard.purdie@linuxfoundation.org>
> Subject: Re: [yocto] Outreachy internship project - license tracing
> enhancement
>
>
>
> On 9/9/20 3:51 AM, Paul Eggleton via lists.yoctoproject.org wrote:
> > Hi folks
> >
> > I'd like to propose we put forward the following project proposal for an
> Outreachy internship (
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.outreachy.org%2Fcommunities%2Fcfp%2F&amp;data=02%7C01%7Cpaul.eggleton%40microsoft.com%7C3a6b0143434b454bbab708d854cca58f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637352585960663229&amp;sdata=2L0gT8udq5P5bxEcPIsrjVxHXADmt9kEDMZxwk1ul5o%3D&amp;reserved=0).
> I'm prepared to be the mentor for the project and Microsoft will provide
> funding. (Note that we haven't got our community re-registered with
> Outreachy or set this up as an intern project proposal yet - deadline for
> YP community registration is September 17th and for project submissions is
> September 24th). Here's the brief:
>
> This sounds great.
>
> Are you looking for YP to take action on  registration or are you handling
> this?
>
> -armin
> >
> > -------------
> > Yocto Project License tracing enhancement
> >
> > The Yocto Project build system is typically used to build customised
> Linux images from source for embedded applications. Along with the image, a
> manifest of packages and their corresponding licenses is prepared, however
> the accuracy of the license information is dependent on the accuracy of the
> metadata we have for each package (i.e. what is in the recipe file). As
> part of the build, we have an internal mapping from output files to source
> files which is currently used to prepare source packages to aid in
> debugging, however with the presence of SPDX headers in source files it
> could also be used to allow tracing the license of sources used in building
> a package/image to help improve our metadata and future license manifests.
> A proof-of concept implementation of this has been put together [1] -
> during this internship a successful intern will:
> > 1) take the proof-of-concept implementation and get it to a state
> > where it can be merged into the poky repository
> > 2) use the functionality to examine the accuracy of our license
> > tagging (LICENSE fields in recipes); look for errors / noise in the
> > comparison, and produce a simple report with the results
> > 3) run a check over sources in a world build looking for percentage
> > coverage of SPDX headers, and run it for several past releases to see
> > the change over time
> >
> > Bonus: assess the current state of meta-spdx-scanner; investigate what
> it would take to produce SPDX documents from build output (would likely
> require integration with Fossology).
> >
> > [1]
> > https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgit.y
> > octoproject.org%2Fcgit%2Fcgit.cgi%2Fpoky-contrib%2Flog%2F%3Fh%3Drpurdi
> > e%2Flicense-experiments-osls&amp;data=02%7C01%7Cpaul.eggleton%40micros
> > oft.com%7C3a6b0143434b454bbab708d854cca58f%7C72f988bf86f141af91ab2d7cd
> > 011db47%7C1%7C0%7C637352585960673189&amp;sdata=ZoCsEf0BYvCs5GalpYUgbv%
> > 2Ff5qt1Lagho3NBML5qgUg%3D&amp;reserved=0
> > -------------
> >
> > I'm making the assumption that we're OK with merging the PoC
> functionality in rather than just keeping it separate and using it for
> analysis - let me know if otherwise. What I'd really like to know is do
> people think that this is sufficient for a 3-month internship, assuming
> that the intern has limited to moderate familiarity with our codebase? Do
> we need to flesh it out further? Any modifications that you'd suggest to
> the work?
> >
> > Thanks
> > Paul
> >
> > 
>
>

[-- Attachment #2: Type: text/html, Size: 6853 bytes --]