repost: how to create a SPDX "notice file" from a build?

repost: how to create a SPDX "notice file" from a build?

[rpjday]

  i asked about this a couple months ago but didn't see any replies,
so i'll ask again with a little more detail.

  colleague wants to, from YP (actually petalinux but should be
irrelevant), some sort of SPDX "notice file", along the lines of what
can be generated by black duck. it doesn't need to be identical, but
it would be useful to at least have a first pass that people can look
at and say what they want tweaked.

  is there an example of how to (using the spdx.bbclass class file, i
assume) do something like this? thanks.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                         http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: repost: how to create a SPDX "notice file" from a build?

[rpjday]

On Fri, 22 Nov 2019, Robert P. J. Day wrote:

>   i asked about this a couple months ago but didn't see any replies,
> so i'll ask again with a little more detail.
>
>   colleague wants to, from YP (actually petalinux but should be
> irrelevant), some sort of SPDX "notice file", along the lines of what
> can be generated by black duck. it doesn't need to be identical, but
> it would be useful to at least have a first pass that people can look
> at and say what they want tweaked.
>
>   is there an example of how to (using the spdx.bbclass class file, i
> assume) do something like this? thanks.

  a lilttle more progress, and then an error -- i installed fossology
on my ubuntu 18.04 dev box, then added

  INHERIT += "spdx"

to local.conf, and tried to build a simple qemuarmx86-64, and very
quickly had a number of native packages fail with:

//////////// start //////////

DEBUG: Executing python function do_spdx
ERROR: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:do_spdx(d)
     0003:
File: '/home/rpjday/oe/dist/layers/poky/meta/classes/spdx.bbclass', lineno: 83, function: do_spdx
     0079:        foss_full_spdx = d.getVar('FOSS_FULL_SPDX') == "true" or False
     0080:        foss_command = "wget %s --post-file=%s %s"\
     0081:            % (foss_flags, info['tar_file'], foss_server)
     0082:
 *** 0083:        foss_result = run_fossology(foss_command, foss_full_spdx)
     0084:        if foss_result is not None:
     0085:            (foss_package_info, foss_file_info, foss_license_info) = foss_result
     0086:            spdx_file_info = create_spdx_doc(local_file_info, foss_file_info)
     0087:            ## write to cache
File: '/home/rpjday/oe/dist/layers/poky/meta/classes/spdx.bbclass', lineno: 230, function: run_fossology
     0226:    # Package info
     0227:    package_info = {}
     0228:    if full_spdx:
     0229:        # All mandatory, only one occurrence
 *** 0230:        package_info['PackageCopyrightText'] = re.findall('PackageCopyrightText: (.*?</text>)', foss_output, re.S)[0]
     0231:        package_info['PackageLicenseDeclared'] = re.findall('PackageLicenseDeclared: (.*)', foss_output)[0]
     0232:        package_info['PackageLicenseConcluded'] = re.findall('PackageLicenseConcluded: (.*)', foss_output)[0]
     0233:        # These may be more than one
     0234:        package_info['PackageLicenseInfoFromFiles'] = re.findall('PackageLicenseInfoFromFiles: (.*)', foss_output)
Exception: IndexError: list index out of range

ERROR: list index out of range
DEBUG: Python function do_spdx finished
ERROR: Function failed: do_spdx

/////////// end /////////

  i have absolutely no idea what to think of this, and am open to
suggestions. does anyone have a working scenario to simply demonstrate
the usage of spdx.bbclass?

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                         http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: [yocto] repost: how to create a SPDX "notice file" from a build?

[Khem Raj]

On Fri, Nov 22, 2019 at 9:03 AM rpjday@crashcourse.ca
<rpjday@crashcourse.ca> wrote:
>
> On Fri, 22 Nov 2019, Robert P. J. Day wrote:
>
> >   i asked about this a couple months ago but didn't see any replies,
> > so i'll ask again with a little more detail.
> >
> >   colleague wants to, from YP (actually petalinux but should be
> > irrelevant), some sort of SPDX "notice file", along the lines of what
> > can be generated by black duck. it doesn't need to be identical, but
> > it would be useful to at least have a first pass that people can look
> > at and say what they want tweaked.
> >
> >   is there an example of how to (using the spdx.bbclass class file, i
> > assume) do something like this? thanks.
>
>   a lilttle more progress, and then an error -- i installed fossology
> on my ubuntu 18.04 dev box, then added
>
>   INHERIT += "spdx"
>
> to local.conf, and tried to build a simple qemuarmx86-64, and very
> quickly had a number of native packages fail with:
>
> //////////// start //////////
>
> DEBUG: Executing python function do_spdx
> ERROR: Error executing a python function in exec_python_func() autogenerated:
>
> The stack trace of python calls that resulted in this exception/failure was:
> File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
>      0001:
>  *** 0002:do_spdx(d)
>      0003:
> File: '/home/rpjday/oe/dist/layers/poky/meta/classes/spdx.bbclass', lineno: 83, function: do_spdx
>      0079:        foss_full_spdx = d.getVar('FOSS_FULL_SPDX') == "true" or False
>      0080:        foss_command = "wget %s --post-file=%s %s"\
>      0081:            % (foss_flags, info['tar_file'], foss_server)
>      0082:
>  *** 0083:        foss_result = run_fossology(foss_command, foss_full_spdx)
>      0084:        if foss_result is not None:
>      0085:            (foss_package_info, foss_file_info, foss_license_info) = foss_result
>      0086:            spdx_file_info = create_spdx_doc(local_file_info, foss_file_info)
>      0087:            ## write to cache
> File: '/home/rpjday/oe/dist/layers/poky/meta/classes/spdx.bbclass', lineno: 230, function: run_fossology
>      0226:    # Package info
>      0227:    package_info = {}
>      0228:    if full_spdx:
>      0229:        # All mandatory, only one occurrence
>  *** 0230:        package_info['PackageCopyrightText'] = re.findall('PackageCopyrightText: (.*?</text>)', foss_output, re.S)[0]
>      0231:        package_info['PackageLicenseDeclared'] = re.findall('PackageLicenseDeclared: (.*)', foss_output)[0]
>      0232:        package_info['PackageLicenseConcluded'] = re.findall('PackageLicenseConcluded: (.*)', foss_output)[0]
>      0233:        # These may be more than one
>      0234:        package_info['PackageLicenseInfoFromFiles'] = re.findall('PackageLicenseInfoFromFiles: (.*)', foss_output)
> Exception: IndexError: list index out of range
>
> ERROR: list index out of range
> DEBUG: Python function do_spdx finished
> ERROR: Function failed: do_spdx
>
> /////////// end /////////
>
>   i have absolutely no idea what to think of this, and am open to
> suggestions. does anyone have a working scenario to simply demonstrate
> the usage of spdx.bbclass?

it once generated SPDX manifest from patched sources for package and
file level it has sadly regressed, so would need some love as you see
above.

>
> rday
>
> --
>
> ========================================================================
> Robert P. J. Day                                 Ottawa, Ontario, CANADA
>                          http://crashcourse.ca
>
> Twitter:                                       http://twitter.com/rpjday
> LinkedIn:                               http://ca.linkedin.com/in/rpjday
> ========================================================================
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
>
> View/Reply Online (#47387): https://lists.yoctoproject.org/g/yocto/message/47387
> Mute This Topic: https://lists.yoctoproject.org/mt/61664060/1997914
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub  [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-

Re: [yocto] repost: how to create a SPDX "notice file" from a build?

[akuster808]

[-- Attachment #1: Type: text/plain, Size: 3675 bytes --]



On 11/22/19 9:03 AM, rpjday@crashcourse.ca wrote:
> On Fri, 22 Nov 2019, Robert P. J. Day wrote:
>
>>   i asked about this a couple months ago but didn't see any replies,
>> so i'll ask again with a little more detail.
>>
>>   colleague wants to, from YP (actually petalinux but should be
>> irrelevant), some sort of SPDX "notice file", along the lines of what
>> can be generated by black duck. it doesn't need to be identical, but
>> it would be useful to at least have a first pass that people can look
>> at and say what they want tweaked.
>>
>>   is there an example of how to (using the spdx.bbclass class file, i
>> assume) do something like this? thanks.
>   a lilttle more progress, and then an error -- i installed fossology
> on my ubuntu 18.04 dev box, then added
>
>   INHERIT += "spdx"
>
> to local.conf, and tried to build a simple qemuarmx86-64, and very
> quickly had a number of native packages fail with:
>
> //////////// start //////////
>
> DEBUG: Executing python function do_spdx
> ERROR: Error executing a python function in exec_python_func() autogenerated:
>
> The stack trace of python calls that resulted in this exception/failure was:
> File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
>      0001:
>  *** 0002:do_spdx(d)
>      0003:
> File: '/home/rpjday/oe/dist/layers/poky/meta/classes/spdx.bbclass', lineno: 83, function: do_spdx
>      0079:        foss_full_spdx = d.getVar('FOSS_FULL_SPDX') == "true" or False
>      0080:        foss_command = "wget %s --post-file=%s %s"\
>      0081:            % (foss_flags, info['tar_file'], foss_server)
>      0082:
>  *** 0083:        foss_result = run_fossology(foss_command, foss_full_spdx)
>      0084:        if foss_result is not None:
>      0085:            (foss_package_info, foss_file_info, foss_license_info) = foss_result
>      0086:            spdx_file_info = create_spdx_doc(local_file_info, foss_file_info)
>      0087:            ## write to cache
> File: '/home/rpjday/oe/dist/layers/poky/meta/classes/spdx.bbclass', lineno: 230, function: run_fossology
>      0226:    # Package info
>      0227:    package_info = {}
>      0228:    if full_spdx:
>      0229:        # All mandatory, only one occurrence
>  *** 0230:        package_info['PackageCopyrightText'] = re.findall('PackageCopyrightText: (.*?</text>)', foss_output, re.S)[0]
>      0231:        package_info['PackageLicenseDeclared'] = re.findall('PackageLicenseDeclared: (.*)', foss_output)[0]
>      0232:        package_info['PackageLicenseConcluded'] = re.findall('PackageLicenseConcluded: (.*)', foss_output)[0]
>      0233:        # These may be more than one
>      0234:        package_info['PackageLicenseInfoFromFiles'] = re.findall('PackageLicenseInfoFromFiles: (.*)', foss_output)
> Exception: IndexError: list index out of range
>
> ERROR: list index out of range
> DEBUG: Python function do_spdx finished
> ERROR: Function failed: do_spdx
>
> /////////// end /////////
>
>   i have absolutely no idea what to think of this, and am open to
> suggestions. does anyone have a working scenario to simply demonstrate
> the usage of spdx.bbclass?

Would you mind opening a Yocto defect.

- armin
> rday
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
>
> View/Reply Online (#47387): https://lists.yoctoproject.org/g/yocto/message/47387
> Mute This Topic: https://lists.yoctoproject.org/mt/61664060/1024635
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub  [akuster@mvista.com]
> -=-=-=-=-=-=-=-=-=-=-=-


[-- Attachment #2: Type: text/html, Size: 4976 bytes --]

Re: [yocto] repost: how to create a SPDX "notice file" from a build?

[Richard Purdie]

On Fri, 2019-11-22 at 09:59 -0800, akuster808 wrote:
> 
> 
> On 11/22/19 9:03 AM, rpjday@crashcourse.ca wrote:
> > On Fri, 22 Nov 2019, Robert P. J. Day wrote:
> > 
> > 
> > /////////// end /////////
> > 
> >   i have absolutely no idea what to think of this, and am open to
> > suggestions. does anyone have a working scenario to simply
> > demonstrate
> > the usage of spdx.bbclass?
>  
> Would you mind opening a Yocto defect.

That code hasn't been touched in a while and needs some serious
attention. The underlying tools and processes have changed so much it
may be a case of starting again and we should perhaps consider removing
that class...

Cheers,

Richard

Re: [yocto] repost: how to create a SPDX "notice file" from a build?

[rpjday]

On Sat, 23 Nov 2019, Richard Purdie wrote:

> On Fri, 2019-11-22 at 09:59 -0800, akuster808 wrote:
> >
> >
> > On 11/22/19 9:03 AM, rpjday@crashcourse.ca wrote:
> > > On Fri, 22 Nov 2019, Robert P. J. Day wrote:
> > >
> > >
> > > /////////// end /////////
> > >
> > >   i have absolutely no idea what to think of this, and am open to
> > > suggestions. does anyone have a working scenario to simply
> > > demonstrate
> > > the usage of spdx.bbclass?
> >
> > Would you mind opening a Yocto defect.
>
> That code hasn't been touched in a while and needs some serious
> attention. The underlying tools and processes have changed so much it
> may be a case of starting again and we should perhaps consider removing
> that class...

  yeah, that the impression i got ... is there a newer alternative to
that, or just the standard license info that's been generated all this
time?

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                         http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: [yocto] repost: how to create a SPDX "notice file" from a build?

[Mark Hatle]

On 11/23/19 6:01 AM, Richard Purdie wrote:
> On Fri, 2019-11-22 at 09:59 -0800, akuster808 wrote:
>>
>>
>> On 11/22/19 9:03 AM, rpjday@crashcourse.ca wrote:
>>> On Fri, 22 Nov 2019, Robert P. J. Day wrote:
>>>
>>>
>>> /////////// end /////////
>>>
>>>   i have absolutely no idea what to think of this, and am open to
>>> suggestions. does anyone have a working scenario to simply
>>> demonstrate
>>> the usage of spdx.bbclass?
>>  
>> Would you mind opening a Yocto defect.
> 
> That code hasn't been touched in a while and needs some serious
> attention. The underlying tools and processes have changed so much it
> may be a case of starting again and we should perhaps consider removing
> that class...

I think the use-cases have changed over time, even though parts and pieces are
still valid.  There are really a few groups to consider.

1) (old case) someone is building a system and wants to construct SPDX files for
the things they are building.  Contacting, uploading, getting a report from
fossology may still be the best way of doing this.

2) (new case) things could be shipped with prebuilt SPDX files (based on
fossology run by the system, maintainer, an addon layer, OSV, etc..)
In this case we would want to simply tie a recipe to an SPDX and be able to
correlate them.

3) In either case, we have a list of SPDX files, but that doesn't meet Robert's
question.  Something needs to process these SPDX files and generate notice files
and similar.  To me this is an external tool, that could optionally be invoked
at image creation time (or by the user directly.)

Further, a 4th case.. what is the license of the components I've actually
deployed.  I've wanted to do this for a long time, but using the dwarf debug
information you can determine what files were actually used to construct the
binaries in your images.  From that you can go back to the SPDX files and
correlated to exactly what was deployed including file level copyright, notice,
and license requirements (not just recipe) and produce an incredibly accurate
report.  Add to this that SPDX has the ability for custom fields that can be
used to track other IP issues like patents, legal concerns, etc.  And you could
construct a report in a form for the legal organization of a company to review
prior to product shipment.

Right now, we have an old way to do 1, but it doesn't solve Robert's issue --
even if it DID work.  and no way to do the rest (that I am aware of).

--Mark

> Cheers,
> 
> Richard
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> 
> View/Reply Online (#47394): https://lists.yoctoproject.org/g/yocto/message/47394
> Mute This Topic: https://lists.yoctoproject.org/mt/61664060/3616948
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub  [mark.hatle@kernel.crashing.org]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: [yocto] repost: how to create a SPDX "notice file" from a build?

[rpjday]

On Sat, 23 Nov 2019, Richard Purdie wrote:

> On Fri, 2019-11-22 at 09:59 -0800, akuster808 wrote:
> >
> >
> > On 11/22/19 9:03 AM, rpjday@crashcourse.ca wrote:
> > > On Fri, 22 Nov 2019, Robert P. J. Day wrote:
> > >
> > >
> > > /////////// end /////////
> > >
> > >   i have absolutely no idea what to think of this, and am open to
> > > suggestions. does anyone have a working scenario to simply
> > > demonstrate
> > > the usage of spdx.bbclass?
> >
> > Would you mind opening a Yocto defect.
>
> That code hasn't been touched in a while and needs some serious
> attention. The underlying tools and processes have changed so much
> it may be a case of starting again and we should perhaps consider
> removing that class...

  as one more followup to this, first, i was asked by a colleague
about the possibility of generating an "SPDX notice file," and as i
don't know what the format of one of those things is, it's hard for me
to know how to answer. it seems clear at this point that spdx.bbclass
is not going to cut it, as richard points out.

  more to the point, has anyone had to do something that seems to
resemble what i was asked about? i'm open to suggestions as to what
*is* supported so i can try it out and hand it over and at least say,
"is this close to what you're looking for?"

  thanks for any pointers.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                         http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: [yocto] repost: how to create a SPDX "notice file" from a build?

[Mark Hatle]

(resending, the list rejected the first reply)

On 11/24/19 4:11 AM, rpjday@crashcourse.ca wrote:
> On Sat, 23 Nov 2019, Richard Purdie wrote:
> 
>> On Fri, 2019-11-22 at 09:59 -0800, akuster808 wrote:
>>>
>>>
>>> On 11/22/19 9:03 AM, rpjday@crashcourse.ca wrote:
>>>> On Fri, 22 Nov 2019, Robert P. J. Day wrote:
>>>>
>>>>
>>>> /////////// end /////////
>>>>
>>>>   i have absolutely no idea what to think of this, and am open to
>>>> suggestions. does anyone have a working scenario to simply
>>>> demonstrate
>>>> the usage of spdx.bbclass?
>>>
>>> Would you mind opening a Yocto defect.
>>
>> That code hasn't been touched in a while and needs some serious
>> attention. The underlying tools and processes have changed so much
>> it may be a case of starting again and we should perhaps consider
>> removing that class...
> 
>   as one more followup to this, first, i was asked by a colleague
> about the possibility of generating an "SPDX notice file," and as i
> don't know what the format of one of those things is, it's hard for me
> to know how to answer. it seems clear at this point that spdx.bbclass
> is not going to cut it, as richard points out.
> 
>   more to the point, has anyone had to do something that seems to
> resemble what i was asked about? i'm open to suggestions as to what
> *is* supported so i can try it out and hand it over and at least say,
> "is this close to what you're looking for?"

I know individual companies have created their own tools to do this, but since
it's both required and could include liability (if you get it wrong), I'm not
aware of any open source way of doing this.

If there is one, it would likely be published as part of the SPDX
project/mailing lists.

--Mark

>   thanks for any pointers.
> 
> rday
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> 
> View/Reply Online (#47399): https://lists.yoctoproject.org/g/yocto/message/47399
> Mute This Topic: https://lists.yoctoproject.org/mt/61664060/3616948
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub  [mark.hatle@kernel.crashing.org]
> -=-=-=-=-=-=-=-=-=-=-=-
>