[Buildroot] Saving

[Buildroot] Saving

[Nicolas Carrier]

Hello!
I'm having a look at legal-info generation, the source saving step saves neither local nor override
packages.

This is a problem, because it's those packages, when forked from OSS, which would require publishing
the modified sources.
Is there a technical / philosophical / whatever reason for no supporting them?


If none, I'd be willing to develop support for that, if anyone has pointers as how to proceed /
where to patch / mistakes to avoid, don't hesitate to share :)

FTR, I'm using a (lightweight) fork of 2022.02.

Thank you by advance!
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Saving

[Arnout Vandecappelle]

On 12/01/2023 16:46, Nicolas Carrier wrote:
> Hello!
> I'm having a look at legal-info generation, the source saving step saves neither local nor override
> packages.
> 
> This is a problem, because it's those packages, when forked from OSS, which would require publishing
> the modified sources.
> Is there a technical / philosophical / whatever reason for no supporting them?

  I think the reason is purely technical: for these packages, we don't have a 
tarball available that can be readily used. Sine we now have post_process_repack 
in support/download/helpers, it's actually fairly easy to add this feature.

  A secondary reason is that the idea for "local" packages was that they're 
probably proprietary. However, in reality there isn't too much correlation 
between download method and proprietary. And we anyway have FOO_REDISTRIBUTE to 
handle this.

  Regards,
  Arnout


> If none, I'd be willing to develop support for that, if anyone has pointers as how to proceed /
> where to patch / mistakes to avoid, don't hesitate to share :)
> 
> FTR, I'm using a (lightweight) fork of 2022.02.
> 
> Thank you by advance!
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Saving

[Nicolas Carrier]

First of all, sorry for the poor title, I proof read my email 4 times but forgot the title :)

> On 12/01/2023 16:46, Nicolas Carrier wrote:
> > Hello!
> > I'm having a look at legal-info generation, the source saving step saves neither local nor
> > override
> > packages.
> > 
> > This is a problem, because it's those packages, when forked from OSS, which would require
> > publishing
> > the modified sources.
> > Is there a technical / philosophical / whatever reason for no supporting them?
> 
>   I think the reason is purely technical: for these packages, we don't have a
> tarball available that can be readily used. Sine we now have post_process_repack
> in support/download/helpers, it's actually fairly easy to add this feature.
> 

It is not clear to me yet how I can use post_process_repack. I'll try to grok the code tomorrow and
see if I can understand that.

>   A secondary reason is that the idea for "local" packages was that they're
> probably proprietary. However, in reality there isn't too much correlation
> between download method and proprietary. And we anyway have FOO_REDISTRIBUTE to
> handle this.

Yes, we have a mix of proprietary and open source packages, for example, linuxptp or the linux
kernel, for which we have a couple of patches we'd like to redistribute properly.

But even for proprietary ones, if they could be handled "properly" that is, with _REDISTRIBUTE = NO
being honored and avoiding the generation of a warning, that would be helpful when wanting to list
the real legal-info warnings in order to fix them.

> 
>   Regards,
>   Arnout
> 

Thank you very much for your answer!

> 
> > If none, I'd be willing to develop support for that, if anyone has pointers as how to proceed /
> > where to patch / mistakes to avoid, don't hesitate to share :)
> > 
> > FTR, I'm using a (lightweight) fork of 2022.02.
> > 
> > Thank you by advance!
> > _______________________________________________
> > buildroot mailing list
> > buildroot@buildroot.org
> > https://lists.buildroot.org/mailman/listinfo/buildroot

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Saving

[Yann E. MORIN]

Nicolas, All,

On 2023-01-12 16:55 +0000, Nicolas Carrier spake thusly:
> > On 12/01/2023 16:46, Nicolas Carrier wrote:
> > > I'm having a look at legal-info generation, the source saving step saves neither local nor
> > > override
> > > packages.
> > > 
> > > This is a problem, because it's those packages, when forked from OSS, which would require
> > > publishing
> > > the modified sources.
> > > Is there a technical / philosophical / whatever reason for no supporting them?
> >   I think the reason is purely technical: for these packages, we don't have a
> > tarball available that can be readily used. Sine we now have post_process_repack
> > in support/download/helpers, it's actually fairly easy to add this feature.
> It is not clear to me yet how I can use post_process_repack. I'll try to grok the code tomorrow and
> see if I can understand that.

So, first, I'd say that override packages really should not be accoutned
for: override is really to be used during development. If one want to
have local packages, there is the 'local' _SITE_METHOD for that. Which
brings up the second topic...

Now: IANAL, and all disclaimers that may apply, the folowing are only my
uninformed opinion, do not base legal decisions solely on those, consult
an actual lawyer, etc...

Then, for "local" packages, I would think we could assume that they are
part of the Buildroot tree (or of a br2-external tree). [0]

One could argue that, to comply with licenses like the GPL (v2 or v3,
lesser or not) of said open-source pacakages, the "scripts used to
control compilation and installation of the executable" are part of the
compliance delivery, and in that case, one may argue that Buildroot (and
the associated br2-external tree if any) would fall into that category,
and thus would have to be provided for to-the-letter and to-the-spirit
compliance eith the license of said packages.

So, the sources for those components are thus available as part of the
delivery of the Buildroot (and associated br2-external if any)
archive(s).

Ergo, there would be no reason to actually handle those local packages
in a specific way because they are already handled.

[0] if they are not in the same repository as Buildroot (or a
br2-external tree), then there is a mechanism to bring them in-tree,
like git submodules, and thus that means there is a mechanism to update
the version pointed to for those packages, which is exactly akin to
updating the _VERSION in the .mk, so there is no benefit to that (yeah,
I know "repo", but that's a whole other discussion...) So, assuming
"local" packages are in-tree is just good sense.

> >   A secondary reason is that the idea for "local" packages was that they're
> > probably proprietary. However, in reality there isn't too much correlation
> > between download method and proprietary. And we anyway have FOO_REDISTRIBUTE to
> > handle this.
> 
> Yes, we have a mix of proprietary and open source packages, for example, linuxptp or the linux
> kernel, for which we have a couple of patches we'd like to redistribute properly.

But then you have patches. Just redistribute the unpatched (original)
linuxptp and linux archives, and your Buildroot tree and br2-external if
any, and you should be all set to comply with the requirement of the
licenses of those packages...

> But even for proprietary ones, if they could be handled "properly" that is, with _REDISTRIBUTE = NO
> being honored and avoiding the generation of a warning, that would be helpful when wanting to list
> the real legal-info warnings in order to fix them.

What warning are you referring to?

I once in the past tried to push changes that to the legal-info infra,
that would also allow for a package to declare that it should simply be
ignored completely from the list of packages stored in legal-info.
However, we discussed that back in the day, and I also now totally
adhere to that conclusion: filtering things should be done with local
scripts that are run on the output of legal-info.

In any case, the output of legal-info is not guaranteed to be either
correct or exhaustive (not that we should not pursue that goal, but we
can't guarantee it), and will always require that a human look at it for
completness and correctness before it can be used as a means of
compliance.

Regards,
Yann E. MORIN.

> >   Regards,
> >   Arnout
> > 
> 
> Thank you very much for your answer!
> 
> > 
> > > If none, I'd be willing to develop support for that, if anyone has pointers as how to proceed /
> > > where to patch / mistakes to avoid, don't hesitate to share :)
> > > 
> > > FTR, I'm using a (lightweight) fork of 2022.02.
> > > 
> > > Thank you by advance!
> > > _______________________________________________
> > > buildroot mailing list
> > > buildroot@buildroot.org
> > > https://lists.buildroot.org/mailman/listinfo/buildroot
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Saving

[Nicolas Carrier]

Yann, All,

Thank you for your answer.

On Fri, 2023-01-13 at 08:42 +0100, Yann E. MORIN wrote:
> CAUTION: This email originated from outside of the organization.
> Do not click links or open attachments unless you recognize the sender and know the content is
> safe.
> 
> Nicolas, All,
> 
> On 2023-01-12 16:55 +0000, Nicolas Carrier spake thusly:
> > > On 12/01/2023 16:46, Nicolas Carrier wrote:
> > > > I'm having a look at legal-info generation, the source saving step saves neither local nor
> > > > override
> > > > packages.
> > > > 
> > > > This is a problem, because it's those packages, when forked from OSS, which would require
> > > > publishing
> > > > the modified sources.
> > > > Is there a technical / philosophical / whatever reason for no supporting them?
> > >   I think the reason is purely technical: for these packages, we don't have a
> > > tarball available that can be readily used. Sine we now have post_process_repack
> > > in support/download/helpers, it's actually fairly easy to add this feature.
> > It is not clear to me yet how I can use post_process_repack. I'll try to grok the code tomorrow
> > and
> > see if I can understand that.
> 
> So, first, I'd say that override packages really should not be accoutned
> for: override is really to be used during development. If one want to
> have local packages, there is the 'local' _SITE_METHOD for that. Which
> brings up the second topic...

We have 3 "override" packages, linux, uboot and linuxptp.
Since they are already provided by buildroot and because we want to have the sources in the
workspace (for ease of development), we have no other choice but to use OVERRIDE.
I don't know if you remember, but we already had a discussion at the buildroot dev con in Lyon some
time ago, about the possibility to redefine a package already provided by buildroot and I was
advised to use override instead, in this kind of situation.
I had also pushed for having a config for uboot and the kernel, to allow to use a local tree without
needing OVERRIDE, but that was rejected.

Anyways, 'local' can't be a solution here, since we don't control the _SITE_METHOD for overridden
packages. Or maybe I missed something :)

> 
> Now: IANAL, and all disclaimers that may apply, the folowing are only my
> uninformed opinion, do not base legal decisions solely on those, consult
> an actual lawyer, etc...
> 
> Then, for "local" packages, I would think we could assume that they are
> part of the Buildroot tree (or of a br2-external tree). [0]
> 
> One could argue that, to comply with licenses like the GPL (v2 or v3,
> lesser or not) of said open-source pacakages, the "scripts used to
> control compilation and installation of the executable" are part of the
> compliance delivery, and in that case, one may argue that Buildroot (and
> the associated br2-external tree if any) would fall into that category,
> and thus would have to be provided for to-the-letter and to-the-spirit
> compliance eith the license of said packages.

I agree, we already "wrap" the legal-info target to add extra components such as buildroot or the
toolchain. But even that, it would be great if buildroot could take care of it...

> 
> So, the sources for those components are thus available as part of the
> delivery of the Buildroot (and associated br2-external if any)
> archive(s).

Not in our case, because:
* we won't provide the full external tree anyway, only what's necessary for compliance
* the sources aren't inside the br2-external, or buildroot
* even if it was the case, we have a mix of open source / proprietary, so we'd need a filtering
method, and REDISTRIBUTE = no seems like the most logical candidate.

> 
> Ergo, there would be no reason to actually handle those local packages
> in a specific way because they are already handled.
> 
> [0] if they are not in the same repository as Buildroot (or a
> br2-external tree), then there is a mechanism to bring them in-tree,
> like git submodules, and thus that means there is a mechanism to update
> the version pointed to for those packages, which is exactly akin to
> updating the _VERSION in the .mk, so there is no benefit to that (yeah,
> I know "repo", but that's a whole other discussion...) So, assuming
> "local" packages are in-tree is just good sense.
> 

I'm not sure I completely understand what you're referring to, but here's what I can say:
We're exactly in the "repo" situation: buildroot, our br-external and all our source code are "at
the same level", there is no git (or other) repository nesting.
For all our local (proprietary or not) + override packages, we have no VERSION, as the integration
part is delegated to repo.

> > >   A secondary reason is that the idea for "local" packages was that they're
> > > probably proprietary. However, in reality there isn't too much correlation
> > > between download method and proprietary. And we anyway have FOO_REDISTRIBUTE to
> > > handle this.
> > 
> > Yes, we have a mix of proprietary and open source packages, for example, linuxptp or the linux
> > kernel, for which we have a couple of patches we'd like to redistribute properly.
> 
> But then you have patches. Just redistribute the unpatched (original)
> linuxptp and linux archives, and your Buildroot tree and br2-external if
> any, and you should be all set to comply with the requirement of the
> licenses of those packages...

Sorry, I wasn't clear enough, we don't have patches per se, we have git repositories cloned in our
workspace and containing our modifications in our branches.
And the whole point is to let buildroot handle the archiving since it already has all the
infrastructure.

> 
> > But even for proprietary ones, if they could be handled "properly" that is, with _REDISTRIBUTE =
> > NO
> > being honored and avoiding the generation of a warning, that would be helpful when wanting to
> > list
> > the real legal-info warnings in order to fix them.
> 
> What warning are you referring to?

This kind of warning:
WARNING: liblogreport: sources not saved (local packages not handled)
which, I think, shouldn't be printed since our liblogreport (proprietary) package has:
LIBLOGREPORT_REDISTRIBUTE = NO

> 
> I once in the past tried to push changes that to the legal-info infra,
> that would also allow for a package to declare that it should simply be
> ignored completely from the list of packages stored in legal-info.
> However, we discussed that back in the day, and I also now totally
> adhere to that conclusion: filtering things should be done with local
> scripts that are run on the output of legal-info.

For proprietary local packages, I don't really want to add a new filtering feature, but more to
enforce the semantics of the already existing _REDISTRIBUTE feature, to local packages too.

> 
> In any case, the output of legal-info is not guaranteed to be either
> correct or exhaustive (not that we should not pursue that goal, but we
> can't guarantee it), and will always require that a human look at it for
> completness and correctness before it can be used as a means of
> compliance.

Agreed, yet I'd to reduce as much as possible the amount of manual intervention.


Here I have 2 aims:
 * reduce the amount of work needed to manually inspect the legal-info output, when I'm trying to
fix real legal info issues (missing LICENCES_FILES...)
 * comply the best I can, to the floss license terms leveraging Buildroot mechanisms whenever
possible

I know I'll always be able to add mechanisms in wrapper around Buildroot, but I'd prefer to help
trying to improve Buildroot instead :)


And here I see 3 "problems" I'd like to solve, by decreasing priority:
 * be able to redistribute source of "override" packages
 * be able to redistribute source of "local" open source packages
 * be able to ignore proprietary packages in legal-info

If I was able to do that, I'd be left with (nearly) only the warnings which really require a proper
fix to submit.



Anyway, I'll try to submit a couple of patches so that we can start discussions on the actual
technical solutions.

> 
> Regards,
> Yann E. MORIN.
> 
> > >   Regards,
> > >   Arnout
> > > 
> > 
> > Thank you very much for your answer!
> > 
> > > 
> > > > If none, I'd be willing to develop support for that, if anyone has pointers as how to
> > > > proceed /
> > > > where to patch / mistakes to avoid, don't hesitate to share :)
> > > > 
> > > > FTR, I'm using a (lightweight) fork of 2022.02.
> > > > 
> > > > Thank you by advance!
> > > > _______________________________________________
> > > > buildroot mailing list
> > > > buildroot@buildroot.org
> > > > https://lists.buildroot.org/mailman/listinfo/buildroot
> > 
> > _______________________________________________
> > buildroot mailing list
> > buildroot@buildroot.org
> > https://lists.buildroot.org/mailman/listinfo/buildroot
> 
> --
> .-----------------.--------------------.------------------.--------------------.
> >  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> > +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
> > +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
> > http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> '------------------------------^-------^------------------^--------------------'

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot