* repost: how to create a SPDX "notice file" from a build? @ 2019-11-22 14:54 rpjday 2019-11-22 17:03 ` rpjday 0 siblings, 1 reply; 9+ messages in thread From: rpjday @ 2019-11-22 14:54 UTC (permalink / raw) To: Yocto discussion list i asked about this a couple months ago but didn't see any replies, so i'll ask again with a little more detail. colleague wants to, from YP (actually petalinux but should be irrelevant), some sort of SPDX "notice file", along the lines of what can be generated by black duck. it doesn't need to be identical, but it would be useful to at least have a first pass that people can look at and say what they want tweaked. is there an example of how to (using the spdx.bbclass class file, i assume) do something like this? thanks. rday -- ======================================================================== Robert P. J. Day Ottawa, Ontario, CANADA http://crashcourse.ca Twitter: http://twitter.com/rpjday LinkedIn: http://ca.linkedin.com/in/rpjday ======================================================================== ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: repost: how to create a SPDX "notice file" from a build? 2019-11-22 14:54 repost: how to create a SPDX "notice file" from a build? rpjday @ 2019-11-22 17:03 ` rpjday 2019-11-22 17:57 ` [yocto] " Khem Raj 2019-11-22 17:59 ` akuster808 0 siblings, 2 replies; 9+ messages in thread From: rpjday @ 2019-11-22 17:03 UTC (permalink / raw) To: Yocto discussion list On Fri, 22 Nov 2019, Robert P. J. Day wrote: > i asked about this a couple months ago but didn't see any replies, > so i'll ask again with a little more detail. > > colleague wants to, from YP (actually petalinux but should be > irrelevant), some sort of SPDX "notice file", along the lines of what > can be generated by black duck. it doesn't need to be identical, but > it would be useful to at least have a first pass that people can look > at and say what they want tweaked. > > is there an example of how to (using the spdx.bbclass class file, i > assume) do something like this? thanks. a lilttle more progress, and then an error -- i installed fossology on my ubuntu 18.04 dev box, then added INHERIT += "spdx" to local.conf, and tried to build a simple qemuarmx86-64, and very quickly had a number of native packages fail with: //////////// start ////////// DEBUG: Executing python function do_spdx ERROR: Error executing a python function in exec_python_func() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 0001: *** 0002:do_spdx(d) 0003: File: '/home/rpjday/oe/dist/layers/poky/meta/classes/spdx.bbclass', lineno: 83, function: do_spdx 0079: foss_full_spdx = d.getVar('FOSS_FULL_SPDX') == "true" or False 0080: foss_command = "wget %s --post-file=%s %s"\ 0081: % (foss_flags, info['tar_file'], foss_server) 0082: *** 0083: foss_result = run_fossology(foss_command, foss_full_spdx) 0084: if foss_result is not None: 0085: (foss_package_info, foss_file_info, foss_license_info) = foss_result 0086: spdx_file_info = create_spdx_doc(local_file_info, foss_file_info) 0087: ## write to cache File: '/home/rpjday/oe/dist/layers/poky/meta/classes/spdx.bbclass', lineno: 230, function: run_fossology 0226: # Package info 0227: package_info = {} 0228: if full_spdx: 0229: # All mandatory, only one occurrence *** 0230: package_info['PackageCopyrightText'] = re.findall('PackageCopyrightText: (.*?</text>)', foss_output, re.S)[0] 0231: package_info['PackageLicenseDeclared'] = re.findall('PackageLicenseDeclared: (.*)', foss_output)[0] 0232: package_info['PackageLicenseConcluded'] = re.findall('PackageLicenseConcluded: (.*)', foss_output)[0] 0233: # These may be more than one 0234: package_info['PackageLicenseInfoFromFiles'] = re.findall('PackageLicenseInfoFromFiles: (.*)', foss_output) Exception: IndexError: list index out of range ERROR: list index out of range DEBUG: Python function do_spdx finished ERROR: Function failed: do_spdx /////////// end ///////// i have absolutely no idea what to think of this, and am open to suggestions. does anyone have a working scenario to simply demonstrate the usage of spdx.bbclass? rday -- ======================================================================== Robert P. J. Day Ottawa, Ontario, CANADA http://crashcourse.ca Twitter: http://twitter.com/rpjday LinkedIn: http://ca.linkedin.com/in/rpjday ======================================================================== ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [yocto] repost: how to create a SPDX "notice file" from a build? 2019-11-22 17:03 ` rpjday @ 2019-11-22 17:57 ` Khem Raj 2019-11-22 17:59 ` akuster808 1 sibling, 0 replies; 9+ messages in thread From: Khem Raj @ 2019-11-22 17:57 UTC (permalink / raw) To: rpjday; +Cc: Yocto discussion list On Fri, Nov 22, 2019 at 9:03 AM rpjday@crashcourse.ca <rpjday@crashcourse.ca> wrote: > > On Fri, 22 Nov 2019, Robert P. J. Day wrote: > > > i asked about this a couple months ago but didn't see any replies, > > so i'll ask again with a little more detail. > > > > colleague wants to, from YP (actually petalinux but should be > > irrelevant), some sort of SPDX "notice file", along the lines of what > > can be generated by black duck. it doesn't need to be identical, but > > it would be useful to at least have a first pass that people can look > > at and say what they want tweaked. > > > > is there an example of how to (using the spdx.bbclass class file, i > > assume) do something like this? thanks. > > a lilttle more progress, and then an error -- i installed fossology > on my ubuntu 18.04 dev box, then added > > INHERIT += "spdx" > > to local.conf, and tried to build a simple qemuarmx86-64, and very > quickly had a number of native packages fail with: > > //////////// start ////////// > > DEBUG: Executing python function do_spdx > ERROR: Error executing a python function in exec_python_func() autogenerated: > > The stack trace of python calls that resulted in this exception/failure was: > File: 'exec_python_func() autogenerated', lineno: 2, function: <module> > 0001: > *** 0002:do_spdx(d) > 0003: > File: '/home/rpjday/oe/dist/layers/poky/meta/classes/spdx.bbclass', lineno: 83, function: do_spdx > 0079: foss_full_spdx = d.getVar('FOSS_FULL_SPDX') == "true" or False > 0080: foss_command = "wget %s --post-file=%s %s"\ > 0081: % (foss_flags, info['tar_file'], foss_server) > 0082: > *** 0083: foss_result = run_fossology(foss_command, foss_full_spdx) > 0084: if foss_result is not None: > 0085: (foss_package_info, foss_file_info, foss_license_info) = foss_result > 0086: spdx_file_info = create_spdx_doc(local_file_info, foss_file_info) > 0087: ## write to cache > File: '/home/rpjday/oe/dist/layers/poky/meta/classes/spdx.bbclass', lineno: 230, function: run_fossology > 0226: # Package info > 0227: package_info = {} > 0228: if full_spdx: > 0229: # All mandatory, only one occurrence > *** 0230: package_info['PackageCopyrightText'] = re.findall('PackageCopyrightText: (.*?</text>)', foss_output, re.S)[0] > 0231: package_info['PackageLicenseDeclared'] = re.findall('PackageLicenseDeclared: (.*)', foss_output)[0] > 0232: package_info['PackageLicenseConcluded'] = re.findall('PackageLicenseConcluded: (.*)', foss_output)[0] > 0233: # These may be more than one > 0234: package_info['PackageLicenseInfoFromFiles'] = re.findall('PackageLicenseInfoFromFiles: (.*)', foss_output) > Exception: IndexError: list index out of range > > ERROR: list index out of range > DEBUG: Python function do_spdx finished > ERROR: Function failed: do_spdx > > /////////// end ///////// > > i have absolutely no idea what to think of this, and am open to > suggestions. does anyone have a working scenario to simply demonstrate > the usage of spdx.bbclass? it once generated SPDX manifest from patched sources for package and file level it has sadly regressed, so would need some love as you see above. > > rday > > -- > > ======================================================================== > Robert P. J. Day Ottawa, Ontario, CANADA > http://crashcourse.ca > > Twitter: http://twitter.com/rpjday > LinkedIn: http://ca.linkedin.com/in/rpjday > ======================================================================== > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#47387): https://lists.yoctoproject.org/g/yocto/message/47387 > Mute This Topic: https://lists.yoctoproject.org/mt/61664060/1997914 > Group Owner: yocto+owner@lists.yoctoproject.org > Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [raj.khem@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [yocto] repost: how to create a SPDX "notice file" from a build? 2019-11-22 17:03 ` rpjday 2019-11-22 17:57 ` [yocto] " Khem Raj @ 2019-11-22 17:59 ` akuster808 2019-11-23 12:01 ` Richard Purdie 1 sibling, 1 reply; 9+ messages in thread From: akuster808 @ 2019-11-22 17:59 UTC (permalink / raw) To: rpjday, Yocto discussion list [-- Attachment #1: Type: text/plain, Size: 3675 bytes --] On 11/22/19 9:03 AM, rpjday@crashcourse.ca wrote: > On Fri, 22 Nov 2019, Robert P. J. Day wrote: > >> i asked about this a couple months ago but didn't see any replies, >> so i'll ask again with a little more detail. >> >> colleague wants to, from YP (actually petalinux but should be >> irrelevant), some sort of SPDX "notice file", along the lines of what >> can be generated by black duck. it doesn't need to be identical, but >> it would be useful to at least have a first pass that people can look >> at and say what they want tweaked. >> >> is there an example of how to (using the spdx.bbclass class file, i >> assume) do something like this? thanks. > a lilttle more progress, and then an error -- i installed fossology > on my ubuntu 18.04 dev box, then added > > INHERIT += "spdx" > > to local.conf, and tried to build a simple qemuarmx86-64, and very > quickly had a number of native packages fail with: > > //////////// start ////////// > > DEBUG: Executing python function do_spdx > ERROR: Error executing a python function in exec_python_func() autogenerated: > > The stack trace of python calls that resulted in this exception/failure was: > File: 'exec_python_func() autogenerated', lineno: 2, function: <module> > 0001: > *** 0002:do_spdx(d) > 0003: > File: '/home/rpjday/oe/dist/layers/poky/meta/classes/spdx.bbclass', lineno: 83, function: do_spdx > 0079: foss_full_spdx = d.getVar('FOSS_FULL_SPDX') == "true" or False > 0080: foss_command = "wget %s --post-file=%s %s"\ > 0081: % (foss_flags, info['tar_file'], foss_server) > 0082: > *** 0083: foss_result = run_fossology(foss_command, foss_full_spdx) > 0084: if foss_result is not None: > 0085: (foss_package_info, foss_file_info, foss_license_info) = foss_result > 0086: spdx_file_info = create_spdx_doc(local_file_info, foss_file_info) > 0087: ## write to cache > File: '/home/rpjday/oe/dist/layers/poky/meta/classes/spdx.bbclass', lineno: 230, function: run_fossology > 0226: # Package info > 0227: package_info = {} > 0228: if full_spdx: > 0229: # All mandatory, only one occurrence > *** 0230: package_info['PackageCopyrightText'] = re.findall('PackageCopyrightText: (.*?</text>)', foss_output, re.S)[0] > 0231: package_info['PackageLicenseDeclared'] = re.findall('PackageLicenseDeclared: (.*)', foss_output)[0] > 0232: package_info['PackageLicenseConcluded'] = re.findall('PackageLicenseConcluded: (.*)', foss_output)[0] > 0233: # These may be more than one > 0234: package_info['PackageLicenseInfoFromFiles'] = re.findall('PackageLicenseInfoFromFiles: (.*)', foss_output) > Exception: IndexError: list index out of range > > ERROR: list index out of range > DEBUG: Python function do_spdx finished > ERROR: Function failed: do_spdx > > /////////// end ///////// > > i have absolutely no idea what to think of this, and am open to > suggestions. does anyone have a working scenario to simply demonstrate > the usage of spdx.bbclass? Would you mind opening a Yocto defect. - armin > rday > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#47387): https://lists.yoctoproject.org/g/yocto/message/47387 > Mute This Topic: https://lists.yoctoproject.org/mt/61664060/1024635 > Group Owner: yocto+owner@lists.yoctoproject.org > Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [akuster@mvista.com] > -=-=-=-=-=-=-=-=-=-=-=- [-- Attachment #2: Type: text/html, Size: 4976 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [yocto] repost: how to create a SPDX "notice file" from a build? 2019-11-22 17:59 ` akuster808 @ 2019-11-23 12:01 ` Richard Purdie 2019-11-23 12:02 ` rpjday ` (2 more replies) 0 siblings, 3 replies; 9+ messages in thread From: Richard Purdie @ 2019-11-23 12:01 UTC (permalink / raw) To: akuster808, rpjday, Yocto discussion list On Fri, 2019-11-22 at 09:59 -0800, akuster808 wrote: > > > On 11/22/19 9:03 AM, rpjday@crashcourse.ca wrote: > > On Fri, 22 Nov 2019, Robert P. J. Day wrote: > > > > > > /////////// end ///////// > > > > i have absolutely no idea what to think of this, and am open to > > suggestions. does anyone have a working scenario to simply > > demonstrate > > the usage of spdx.bbclass? > > Would you mind opening a Yocto defect. That code hasn't been touched in a while and needs some serious attention. The underlying tools and processes have changed so much it may be a case of starting again and we should perhaps consider removing that class... Cheers, Richard ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [yocto] repost: how to create a SPDX "notice file" from a build? 2019-11-23 12:01 ` Richard Purdie @ 2019-11-23 12:02 ` rpjday 2019-11-23 15:53 ` Mark Hatle 2019-11-24 10:11 ` rpjday 2 siblings, 0 replies; 9+ messages in thread From: rpjday @ 2019-11-23 12:02 UTC (permalink / raw) To: Richard Purdie; +Cc: akuster808, Yocto discussion list On Sat, 23 Nov 2019, Richard Purdie wrote: > On Fri, 2019-11-22 at 09:59 -0800, akuster808 wrote: > > > > > > On 11/22/19 9:03 AM, rpjday@crashcourse.ca wrote: > > > On Fri, 22 Nov 2019, Robert P. J. Day wrote: > > > > > > > > > /////////// end ///////// > > > > > > i have absolutely no idea what to think of this, and am open to > > > suggestions. does anyone have a working scenario to simply > > > demonstrate > > > the usage of spdx.bbclass? > > > > Would you mind opening a Yocto defect. > > That code hasn't been touched in a while and needs some serious > attention. The underlying tools and processes have changed so much it > may be a case of starting again and we should perhaps consider removing > that class... yeah, that the impression i got ... is there a newer alternative to that, or just the standard license info that's been generated all this time? rday -- ======================================================================== Robert P. J. Day Ottawa, Ontario, CANADA http://crashcourse.ca Twitter: http://twitter.com/rpjday LinkedIn: http://ca.linkedin.com/in/rpjday ======================================================================== ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [yocto] repost: how to create a SPDX "notice file" from a build? 2019-11-23 12:01 ` Richard Purdie 2019-11-23 12:02 ` rpjday @ 2019-11-23 15:53 ` Mark Hatle 2019-11-24 10:11 ` rpjday 2 siblings, 0 replies; 9+ messages in thread From: Mark Hatle @ 2019-11-23 15:53 UTC (permalink / raw) To: yocto On 11/23/19 6:01 AM, Richard Purdie wrote: > On Fri, 2019-11-22 at 09:59 -0800, akuster808 wrote: >> >> >> On 11/22/19 9:03 AM, rpjday@crashcourse.ca wrote: >>> On Fri, 22 Nov 2019, Robert P. J. Day wrote: >>> >>> >>> /////////// end ///////// >>> >>> i have absolutely no idea what to think of this, and am open to >>> suggestions. does anyone have a working scenario to simply >>> demonstrate >>> the usage of spdx.bbclass? >> >> Would you mind opening a Yocto defect. > > That code hasn't been touched in a while and needs some serious > attention. The underlying tools and processes have changed so much it > may be a case of starting again and we should perhaps consider removing > that class... I think the use-cases have changed over time, even though parts and pieces are still valid. There are really a few groups to consider. 1) (old case) someone is building a system and wants to construct SPDX files for the things they are building. Contacting, uploading, getting a report from fossology may still be the best way of doing this. 2) (new case) things could be shipped with prebuilt SPDX files (based on fossology run by the system, maintainer, an addon layer, OSV, etc..) In this case we would want to simply tie a recipe to an SPDX and be able to correlate them. 3) In either case, we have a list of SPDX files, but that doesn't meet Robert's question. Something needs to process these SPDX files and generate notice files and similar. To me this is an external tool, that could optionally be invoked at image creation time (or by the user directly.) Further, a 4th case.. what is the license of the components I've actually deployed. I've wanted to do this for a long time, but using the dwarf debug information you can determine what files were actually used to construct the binaries in your images. From that you can go back to the SPDX files and correlated to exactly what was deployed including file level copyright, notice, and license requirements (not just recipe) and produce an incredibly accurate report. Add to this that SPDX has the ability for custom fields that can be used to track other IP issues like patents, legal concerns, etc. And you could construct a report in a form for the legal organization of a company to review prior to product shipment. Right now, we have an old way to do 1, but it doesn't solve Robert's issue -- even if it DID work. and no way to do the rest (that I am aware of). --Mark > Cheers, > > Richard > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#47394): https://lists.yoctoproject.org/g/yocto/message/47394 > Mute This Topic: https://lists.yoctoproject.org/mt/61664060/3616948 > Group Owner: yocto+owner@lists.yoctoproject.org > Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [mark.hatle@kernel.crashing.org] > -=-=-=-=-=-=-=-=-=-=-=- > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [yocto] repost: how to create a SPDX "notice file" from a build? 2019-11-23 12:01 ` Richard Purdie 2019-11-23 12:02 ` rpjday 2019-11-23 15:53 ` Mark Hatle @ 2019-11-24 10:11 ` rpjday 2019-11-24 16:37 ` Mark Hatle 2 siblings, 1 reply; 9+ messages in thread From: rpjday @ 2019-11-24 10:11 UTC (permalink / raw) To: Richard Purdie; +Cc: akuster808, Yocto discussion list On Sat, 23 Nov 2019, Richard Purdie wrote: > On Fri, 2019-11-22 at 09:59 -0800, akuster808 wrote: > > > > > > On 11/22/19 9:03 AM, rpjday@crashcourse.ca wrote: > > > On Fri, 22 Nov 2019, Robert P. J. Day wrote: > > > > > > > > > /////////// end ///////// > > > > > > i have absolutely no idea what to think of this, and am open to > > > suggestions. does anyone have a working scenario to simply > > > demonstrate > > > the usage of spdx.bbclass? > > > > Would you mind opening a Yocto defect. > > That code hasn't been touched in a while and needs some serious > attention. The underlying tools and processes have changed so much > it may be a case of starting again and we should perhaps consider > removing that class... as one more followup to this, first, i was asked by a colleague about the possibility of generating an "SPDX notice file," and as i don't know what the format of one of those things is, it's hard for me to know how to answer. it seems clear at this point that spdx.bbclass is not going to cut it, as richard points out. more to the point, has anyone had to do something that seems to resemble what i was asked about? i'm open to suggestions as to what *is* supported so i can try it out and hand it over and at least say, "is this close to what you're looking for?" thanks for any pointers. rday -- ======================================================================== Robert P. J. Day Ottawa, Ontario, CANADA http://crashcourse.ca Twitter: http://twitter.com/rpjday LinkedIn: http://ca.linkedin.com/in/rpjday ======================================================================== ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [yocto] repost: how to create a SPDX "notice file" from a build? 2019-11-24 10:11 ` rpjday @ 2019-11-24 16:37 ` Mark Hatle 0 siblings, 0 replies; 9+ messages in thread From: Mark Hatle @ 2019-11-24 16:37 UTC (permalink / raw) To: yocto (resending, the list rejected the first reply) On 11/24/19 4:11 AM, rpjday@crashcourse.ca wrote: > On Sat, 23 Nov 2019, Richard Purdie wrote: > >> On Fri, 2019-11-22 at 09:59 -0800, akuster808 wrote: >>> >>> >>> On 11/22/19 9:03 AM, rpjday@crashcourse.ca wrote: >>>> On Fri, 22 Nov 2019, Robert P. J. Day wrote: >>>> >>>> >>>> /////////// end ///////// >>>> >>>> i have absolutely no idea what to think of this, and am open to >>>> suggestions. does anyone have a working scenario to simply >>>> demonstrate >>>> the usage of spdx.bbclass? >>> >>> Would you mind opening a Yocto defect. >> >> That code hasn't been touched in a while and needs some serious >> attention. The underlying tools and processes have changed so much >> it may be a case of starting again and we should perhaps consider >> removing that class... > > as one more followup to this, first, i was asked by a colleague > about the possibility of generating an "SPDX notice file," and as i > don't know what the format of one of those things is, it's hard for me > to know how to answer. it seems clear at this point that spdx.bbclass > is not going to cut it, as richard points out. > > more to the point, has anyone had to do something that seems to > resemble what i was asked about? i'm open to suggestions as to what > *is* supported so i can try it out and hand it over and at least say, > "is this close to what you're looking for?" I know individual companies have created their own tools to do this, but since it's both required and could include liability (if you get it wrong), I'm not aware of any open source way of doing this. If there is one, it would likely be published as part of the SPDX project/mailing lists. --Mark > thanks for any pointers. > > rday > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#47399): https://lists.yoctoproject.org/g/yocto/message/47399 > Mute This Topic: https://lists.yoctoproject.org/mt/61664060/3616948 > Group Owner: yocto+owner@lists.yoctoproject.org > Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [mark.hatle@kernel.crashing.org] > -=-=-=-=-=-=-=-=-=-=-=- > ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2019-11-24 16:37 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-11-22 14:54 repost: how to create a SPDX "notice file" from a build? rpjday 2019-11-22 17:03 ` rpjday 2019-11-22 17:57 ` [yocto] " Khem Raj 2019-11-22 17:59 ` akuster808 2019-11-23 12:01 ` Richard Purdie 2019-11-23 12:02 ` rpjday 2019-11-23 15:53 ` Mark Hatle 2019-11-24 10:11 ` rpjday 2019-11-24 16:37 ` Mark Hatle
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.