[RFC PATCH 0/2] buildhistory: record SRC_URI

[RFC PATCH 0/2] buildhistory: record SRC_URI

[Paul Eggleton]

Record SRC_URI for all built recipes in buildhistory. Since we also
want this recorded for native recipes, enable writing out the "latest"
file under packages/ for native recipes as well.


Please review the following changes for suitability for inclusion. If you have
any objections or suggestions for improvement, please respond to the patches. If
you agree with the changes, please provide your Acked-by.

The following changes since commit 1f26495884b8f567aecadc5936651846dfeed3f5:

  siteinfo: Recognize bigendian sh3be and sh4be (2020-10-17 23:22:15 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/buildhistory-src-uri
  http://cgit.openembedded.org/openembedded-core-contrib/log/?h=paule/buildhistory-src-uri

Paul Eggleton (2):
  classes/buildhistory: record SRC_URI
  classes/buildhistory: also save recipe info for native recipes

 meta/classes/buildhistory.bbclass | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

-- 
1.8.3.1

[RFC PATCH 1/2] classes/buildhistory: record SRC_URI

[Paul Eggleton]

From: Paul Eggleton <paul.eggleton@microsoft.com>

It can be useful to record SRC_URI into buildhistory for the purposes of
tracking exactly which sources got built (we already have SRCREV) as
well as getting an indication when changes to the SRC_URI relate to
changes in the output.

Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com>
---
 meta/classes/buildhistory.bbclass | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 0f26c3c..7d5e3eb 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -116,6 +116,7 @@ python buildhistory_emit_pkghistory() {
             self.srcrev = ""
             self.layer = ""
             self.config = ""
+            self.src_uri = ""
 
 
     class PackageInfo:
@@ -258,6 +259,7 @@ python buildhistory_emit_pkghistory() {
     rcpinfo.packages = packages
     rcpinfo.layer = layer
     rcpinfo.config = sortlist(oe.utils.squashspaces(d.getVar('PACKAGECONFIG') or ""))
+    rcpinfo.src_uri = oe.utils.squashspaces(d.getVar('SRC_URI') or "")
     write_recipehistory(rcpinfo, d)
 
     bb.build.exec_func("read_subpackage_metadata", d)
@@ -368,6 +370,7 @@ def write_recipehistory(rcpinfo, d):
         f.write(u"PACKAGES = %s\n" %  rcpinfo.packages)
         f.write(u"LAYER = %s\n" %  rcpinfo.layer)
         f.write(u"CONFIG = %s\n" %  rcpinfo.config)
+        f.write(u"SRC_URI = %s\n" %  rcpinfo.src_uri)
 
     write_latest_srcrev(d, pkghistdir)
 
-- 
1.8.3.1

[RFC PATCH 2/2] classes/buildhistory: also save recipe info for native recipes

[Paul Eggleton]

From: Paul Eggleton <paul.eggleton@microsoft.com>

If we want to also collect SRC_URI for native recipes we need to ensure
that the code that writes out all of the recipe info is called - there
isn't a do_packagedata for native recipes so we need to piggyback on
do_populate_sysroot instead.

Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com>
---
 meta/classes/buildhistory.bbclass | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 7d5e3eb..6d04d8c 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -90,8 +90,7 @@ buildhistory_emit_sysroot() {
 python buildhistory_emit_pkghistory() {
     if d.getVar('BB_CURRENTTASK') in ['populate_sysroot', 'populate_sysroot_setscene']:
         bb.build.exec_func("buildhistory_emit_sysroot", d)
-
-    if not d.getVar('BB_CURRENTTASK') in ['packagedata', 'packagedata_setscene']:
+    elif not d.getVar('BB_CURRENTTASK') in ['packagedata', 'packagedata_setscene']:
         return 0
 
     if not "package" in (d.getVar('BUILDHISTORY_FEATURES') or "").split():
@@ -229,8 +228,9 @@ python buildhistory_emit_pkghistory() {
                     break
     except IOError as e:
         if e.errno == errno.ENOENT:
-            # Probably a -cross recipe, just ignore
-            return 0
+            if not bb.data.inherits_class('native', d):
+                # Probably a -cross recipe, just ignore
+                return 0
         else:
             raise
 
-- 
1.8.3.1

Re: [OE-core] [RFC PATCH 0/2] buildhistory: record SRC_URI

[Paul Eggleton]

On Monday, 19 October 2020 17:03:55 NZDT Paul Eggleton wrote:
> Please review the following changes for suitability for inclusion. If you
> have any objections or suggestions for improvement, please respond to the
> patches. If you agree with the changes, please provide your Acked-by.

Hmm, this should not have been RFC. Consider this a regular patch series.

Thanks
Paul

Re: [OE-core] [RFC PATCH 1/2] classes/buildhistory: record SRC_URI

[Mikko Rapeli]

On Sun, Oct 18, 2020 at 09:03:56PM -0700, Paul Eggleton wrote:
> From: Paul Eggleton <paul.eggleton@microsoft.com>
> 
> It can be useful to record SRC_URI into buildhistory for the purposes of
> tracking exactly which sources got built (we already have SRCREV) as
> well as getting an indication when changes to the SRC_URI relate to
> changes in the output.
> 
> Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com>

I have similar patch in our poky trees. Also have patches
to export LICENSE and CVE_PRODUCT to buildhistory. These are used
by some post-build QA check scripts.

Acked-by: Mikko Rapeli <mikko.rapeli@bmw.de>

Cheers,

-Mikko

> ---
>  meta/classes/buildhistory.bbclass | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
> index 0f26c3c..7d5e3eb 100644
> --- a/meta/classes/buildhistory.bbclass
> +++ b/meta/classes/buildhistory.bbclass
> @@ -116,6 +116,7 @@ python buildhistory_emit_pkghistory() {
>              self.srcrev = ""
>              self.layer = ""
>              self.config = ""
> +            self.src_uri = ""
>
>
>      class PackageInfo:
> @@ -258,6 +259,7 @@ python buildhistory_emit_pkghistory() {
>      rcpinfo.packages = packages
>      rcpinfo.layer = layer
>      rcpinfo.config = sortlist(oe.utils.squashspaces(d.getVar('PACKAGECONFIG') or ""))
> +    rcpinfo.src_uri = oe.utils.squashspaces(d.getVar('SRC_URI') or "")
>      write_recipehistory(rcpinfo, d)
>
>      bb.build.exec_func("read_subpackage_metadata", d)
> @@ -368,6 +370,7 @@ def write_recipehistory(rcpinfo, d):
>          f.write(u"PACKAGES = %s\n" %  rcpinfo.packages)
>          f.write(u"LAYER = %s\n" %  rcpinfo.layer)
>          f.write(u"CONFIG = %s\n" %  rcpinfo.config)
> +        f.write(u"SRC_URI = %s\n" %  rcpinfo.src_uri)
>
>      write_latest_srcrev(d, pkghistdir)
>
> -- 
> 1.8.3.1
> 

> 
> 
> 

Re: [OE-core] [RFC PATCH 1/2] classes/buildhistory: record SRC_URI

[Richard Purdie]

On Mon, 2020-10-26 at 06:43 +0000, Mikko Rapeli wrote:
> On Sun, Oct 18, 2020 at 09:03:56PM -0700, Paul Eggleton wrote:
> > From: Paul Eggleton <paul.eggleton@microsoft.com>
> > 
> > It can be useful to record SRC_URI into buildhistory for the
> > purposes of
> > tracking exactly which sources got built (we already have SRCREV)
> > as
> > well as getting an indication when changes to the SRC_URI relate to
> > changes in the output.
> > 
> > Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com>
> 
> I have similar patch in our poky trees. Also have patches
> to export LICENSE and CVE_PRODUCT to buildhistory. These are used
> by some post-build QA check scripts.
> 
> Acked-by: Mikko Rapeli <mikko.rapeli@bmw.de>

I'm actually fairly against some of these kinds of changes.
Buildhistory is meant to be there to highlight changes in the output
over time. Using it to create manifests and for license checking
purposes is not what it was designed for.

I just feel if the original author of the class thinks its a good idea
I need to give up :/.

Cheers,

Richard

Re: [OE-core] [RFC PATCH 1/2] classes/buildhistory: record SRC_URI

[Mikko Rapeli]

On Wed, Oct 28, 2020 at 02:21:22PM +0000, Richard Purdie wrote:
> On Mon, 2020-10-26 at 06:43 +0000, Mikko Rapeli wrote:
> > On Sun, Oct 18, 2020 at 09:03:56PM -0700, Paul Eggleton wrote:
> > > From: Paul Eggleton <paul.eggleton@microsoft.com>
> > > 
> > > It can be useful to record SRC_URI into buildhistory for the
> > > purposes of
> > > tracking exactly which sources got built (we already have SRCREV)
> > > as
> > > well as getting an indication when changes to the SRC_URI relate to
> > > changes in the output.
> > > 
> > > Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com>
> > 
> > I have similar patch in our poky trees. Also have patches
> > to export LICENSE and CVE_PRODUCT to buildhistory. These are used
> > by some post-build QA check scripts.
> > 
> > Acked-by: Mikko Rapeli <mikko.rapeli@bmw.de>
> 
> I'm actually fairly against some of these kinds of changes.
> Buildhistory is meant to be there to highlight changes in the output
> over time. Using it to create manifests and for license checking
> purposes is not what it was designed for.
> 
> I just feel if the original author of the class thinks its a good idea
> I need to give up :/.

Surely changes in LICENSE and SRC_URI over time are important for an overview
of changes?

At least to me they are pretty much essential. I'm using buildhistory diffs to
see what changes in major yocto updates for example once builds are passing.

Cheers,

-Mikko

> 
> Cheers,
> 
> Richard

Re: [OE-core] [RFC PATCH 1/2] classes/buildhistory: record SRC_URI

[Richard Purdie]

On Thu, 2020-10-29 at 09:18 +0000, Mikko.Rapeli@bmw.de wrote:
> On Wed, Oct 28, 2020 at 02:21:22PM +0000, Richard Purdie wrote:
> > On Mon, 2020-10-26 at 06:43 +0000, Mikko Rapeli wrote:
> > > On Sun, Oct 18, 2020 at 09:03:56PM -0700, Paul Eggleton wrote:
> > > > From: Paul Eggleton <paul.eggleton@microsoft.com>
> > > > 
> > > > It can be useful to record SRC_URI into buildhistory for the
> > > > purposes of
> > > > tracking exactly which sources got built (we already have
> > > > SRCREV)
> > > > as
> > > > well as getting an indication when changes to the SRC_URI
> > > > relate to
> > > > changes in the output.
> > > > 
> > > > Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com>
> > > 
> > > I have similar patch in our poky trees. Also have patches
> > > to export LICENSE and CVE_PRODUCT to buildhistory. These are used
> > > by some post-build QA check scripts.
> > > 
> > > Acked-by: Mikko Rapeli <mikko.rapeli@bmw.de>
> > 
> > I'm actually fairly against some of these kinds of changes.
> > Buildhistory is meant to be there to highlight changes in the
> > output
> > over time. Using it to create manifests and for license checking
> > purposes is not what it was designed for.
> > 
> > I just feel if the original author of the class thinks its a good
> > idea
> > I need to give up :/.
> 
> Surely changes in LICENSE and SRC_URI over time are important for an
> overview of changes?

LICENSE is important and I understand that one. SRC_URI is trickier.
The recipe version is in some ways more relevant that SRC_URI. I guess
with the latter, you can look at the change in the set of patches or
perhaps source of the code but I'd suspect the version is the more
useful thing people are looking for.

There is then the temptation for people to use the buildhistory data
for license auditing which worries me as its not what its intended for.

> At least to me they are pretty much essential. I'm using buildhistory
> diffs to see what changes in major yocto updates for example once
> builds are passing.

The question is whether the history is of the inputs, the outputs or
tangential information like the license information. Traditionally it
was focused very specifically on the output, we've then added more and
more input data and people are using it for licensing (which there is
supposed to be separate code for). It makes it hard to maintain and
develop since the usage becomes so varied. I appreciate few people care
about that though as it's someone else's problem though :(. I try and
keep things true to their design and people end up unhappy. Where we
deviate, I try and ensure its at least a conscious decision, I'm still
not sure it is in this case.

Cheers,

Richard