[Buildroot] [PATCH v5 0/3] Add tainting support to buildroot

[Angelo Compagnucci <angelo@amarulasolutions.com>]

Yann, All,
On Mon, Sep 10, 2018 at 7:07 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
>
> Angelo, All,
>
> On 2018-09-10 19:10 +0200, Angelo Compagnucci spake thusly:
> > Il giorno lun 10 set 2018 alle ore 17:00 Yann E. MORIN
> > <yann.morin.1998@free.fr> ha scritto:
> > I can agree with you, but what do you propose for handling package
> > managers: Simply put a !BR2_REPRODUCIBLE for a package that needs
> > dependencies resolved and cannot handle dependencies cleanly?
>
> Yes, see below...
>
> > > [0] For example, consider an explicit and complete list such as (spitted
> > >     for readability):
> > >         BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL="
> > >             http://internal-server/node/mod-1
> > >             http://internal-server/node/mod-2
> > >             http://internal-server/node/mod-3
> > >             http://internal-server/node/mod-4
> > >         "
> > >     and that all the dependencies of those modules *are* present in that
> > >     list, leaving npm no chance to download anything.
> >
> > As I said previously, hiding npm inside nodejs is wrong. What if you
> > have a couple nodejs based software and for one you have a clear
> > picture of dependencies but not for the other?
> > Npm should decoupled and used each time a nodejs software needs it.
>
> npm (the utility) comes from nodejs (the source tree). So, the npm we
> use in Buildroot comes from host-nodejs (the Buildroot package). Thus,
> the package that calls npm (i.e. nodejs today) is already decorelated
> from the package that installs npm.
>
> And you can already use npm from a generic-package in Buildroot as it is
> today...
>
> There are two ways to install stuff with npm in Buildroot:
>
>   - add a pacakge name (URL, name and version, or just name) in the
>     existing BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL config option;
>
>   - add a new generic-package that calls npm as part of its
>     _INSTALL_CMDS.
>
> In either case you may or may not have reproducibility. In the former,
> Buildroot can not decide either way, because only a human can tell, so
> your patch 3 can't be applied, because it is wring.
>
> As for the latter case, the human was in charge of writing the .mk and
> Config.in. Whether they write TAINT=YES in the .mk, or depends on
> !BR2_REPRODUCIBLE in the Config.in has the same effect. As we already
> have BR2_REPRODUCIBLE, we don't need TAINT.
>
> Or I don't understand the problem...
>
> > This is what I did with watchtower:
> [--SNIP--]
> > +WATCHTOWER_TAINTS = YES
>
> If you are going so far as to explicitly write this line, just hide the
> package behing BR2_REPRODUCIBLE instead, yes.

Ok, I'm sold. I'm not interested on how we do it, it's way more
important that we define a way to add packages based on package
managers.

> And even better yet, for packages we would include in Buildroot, provide
> a clean and reproducible list of packages for the dependencies for those
> packages. Although this would be the ideal situation, if that is not
> possible, then hide behind !BR2_REPRODUCIBLE.

Yes it could, only if the dependency chain is not nested and it easy
to walk. That's not always true.

> For packages that stay in a br2-external or a random fork elsewhere, I
> don't think we should care, as I explained previously.
>
> For a parallel: if I do a build without threads, then packages than need
> threads are hidden (with a comment). Similarly, if I want a reproducible
> build, the packages that can't be reproducible are hidden (with a
> comment).
>
> By the way, this has nothing to do with whether they are packages coming
> from a "package manager". Even "traditional" packages may want to do
> unclean downloads at build time; for example, some care had to be taken
> for asterisk, or it would download some sound stuff during the install
> step (although that was reproducible).
>
> > +cd $(WATCHTOWER_SRC_PATH) && GOPATH="$(@D)/$(WATCHTOWER_WORKSPACE)" $(HOST_DIR)/usr/bin/glide install
>
> What would prevent having a script like we have in utils/scancpan or
> utils/scanpypi to do the same processing for Go packages? Or npm
> modules?

This is not possible imho, at least fo golang packages. Golang
packages is basically git repositories, we could end up packaging a
good part of github!

> Yes, some people are afraid we get thousands of them. But we don't know
> how far we'll go. Today, we're pretty happy adding random python modules.
> Why would it be different for the other "package managers"?

Well, the main difference is that python packages is self contained,
npm packages for example not, they can depend on other modules in a
very intricate way.
You should write something that recursively walks all the dependency
tree and outputs a package for everything. Probably we should only
embrace the shift paradigm of having packages that downloads their own
dependencies.

There is also another reason why python packages are added frequently:
we have no way to call a host-pip to resolve dependencies. I'm sure
that there would be way less python packages if we had a host-pip
package.

Sincerely, Angelo

> If we find that this is going "way too far", we can revisit that later.
>
> Regards,
> Yann E. MORIN.
>
> --
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
> | +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> '------------------------------^-------^------------------^--------------------'