* [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot @ 2018-09-05 22:22 Angelo Compagnucci 2018-09-05 22:22 ` [Buildroot] [PATCH v5 1/3] Makefile: add tainting support Angelo Compagnucci ` (5 more replies) 0 siblings, 6 replies; 37+ messages in thread From: Angelo Compagnucci @ 2018-09-05 22:22 UTC (permalink / raw) To: buildroot Packages that need to resolve dependencies internally and use a package manager would harm the reproducibility of a build, moreover they escape the legal infrastructure not giving enough informations on licensing. This patch adds a tainting mechanism in the form of a variable FOO_TAINTS that can be used to signal that a package harms the reproducibility or licensing under certain conditions. This opens the door to include per language dependency managers in buildroot. Angelo Compagnucci (3): Makefile: add tainting support docs/manual: adding infos about tainting package/nodejs: taint the build on external modules Makefile | 10 ++++++++++ docs/manual/adding-packages-generic.txt | 6 ++++++ docs/manual/legal-notice.txt | 12 ++++++++++++ package/nodejs/nodejs.mk | 1 + package/pkg-generic.mk | 15 +++++++++++++++ 5 files changed, 44 insertions(+) -- 2.7.4 ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 1/3] Makefile: add tainting support 2018-09-05 22:22 [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot Angelo Compagnucci @ 2018-09-05 22:22 ` Angelo Compagnucci 2018-09-06 7:44 ` Thomas Petazzoni 2018-09-05 22:22 ` [Buildroot] [PATCH v5 2/3] docs/manual: adding infos about tainting Angelo Compagnucci ` (4 subsequent siblings) 5 siblings, 1 reply; 37+ messages in thread From: Angelo Compagnucci @ 2018-09-05 22:22 UTC (permalink / raw) To: buildroot Packages who harms the build reproducibility or licensing can declare FOO_TAINTS variable. If a package taints the build it will be added to a list of tainting packages. The build ends with a warning if the tainting packages list is not empty. Moreover, legal info will show a warning in presence of a tainting package. Tainting of host packages is not supported right now. Signed-off-by: Angelo Compagnucci <angelo@amarulasolutions.com> Signed-off-by: Angelo Compagnucci <angelo.compagnucci@gmail.com> --- Makefile | 10 ++++++++++ package/pkg-generic.mk | 15 +++++++++++++++ 2 files changed, 25 insertions(+) diff --git a/Makefile b/Makefile index 9d66bba..ad61130 100644 --- a/Makefile +++ b/Makefile @@ -758,12 +758,21 @@ endif touch $(TARGET_DIR)/usr +.PHONY: check-tainted +check-tainted: +ifneq ($(BR2_TAINTED_BY),) + $(error Your buildroot configuration is tainted by: $(BR2_TAINTED_BY)) +else + @echo "Your buildroot configuration is not tainted" +endif + .PHONY: target-post-image target-post-image: $(TARGETS_ROOTFS) target-finalize @rm -f $(ROOTFS_COMMON_TAR) @$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \ $(call MESSAGE,"Executing post-image script $(s)"); \ $(EXTRA_ENV) $(s) $(BINARIES_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep)) + $(if $(BR2_TAINTED_BY), at echo "WARNING: Your buildroot configuration is tainted by: $(BR2_TAINTED_BY).") .PHONY: source source: $(foreach p,$(PACKAGES),$(p)-all-source) @@ -1070,6 +1079,7 @@ help: @echo ' source - download all sources needed for offline-build' @echo ' external-deps - list external packages used' @echo ' legal-info - generate info about license compliance' + @echo ' check-tainted - check if any selected package harms build reproducibility or licensing' @echo ' printvars - dump all the internal variables' @echo @echo ' make V=0|1 - 0 => quiet build (default), 1 => verbose build' diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 91b61c6..465916c 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -870,6 +870,15 @@ else $(2)_KCONFIG_VAR = BR2_PACKAGE_$(2) endif +ifdef $(2)_TAINTS +ifeq ($$($$($(2)_KCONFIG_VAR)),y) +BR2_TAINTED_BY+=$$($(2)_RAWNAME) +endif +ifeq ($$($(2)_TYPE),host) +$$(error "Host package $(1) has $(2)_TAINTS set: not supported) +endif +endif + # legal-info: declare dependencies and set values used later for the manifest ifneq ($$($(2)_LICENSE_FILES),) $(2)_MANIFEST_LICENSE_FILES = $$($(2)_LICENSE_FILES) @@ -909,6 +918,12 @@ else $(Q)$$(foreach F,$$($(2)_LICENSE_FILES),$$(call legal-license-file,$$($(2)_RAWNAME),$$($(2)_BASENAME_RAW),$$($(2)_PKGDIR),$$(F),$$($(2)_DIR)/$$(F),$$(call UPPERCASE,$(4)))$$(sep)) endif # license files +ifeq ($$(call qstrip,$$($(2)_TAINTS)),YES) +ifeq ($$($$($(2)_KCONFIG_VAR)),y) + $(Q)$$(call legal-warning-pkg,$$($(2)_RAWNAME),unknown license for additional modules or dependencies) +endif +endif + ifeq ($$($(2)_SITE_METHOD),local) # Packages without a tarball: don't save and warn @$$(call legal-warning-nosource,$$($(2)_RAWNAME),local) -- 2.7.4 ^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 1/3] Makefile: add tainting support 2018-09-05 22:22 ` [Buildroot] [PATCH v5 1/3] Makefile: add tainting support Angelo Compagnucci @ 2018-09-06 7:44 ` Thomas Petazzoni 2018-09-06 7:46 ` Angelo Compagnucci 0 siblings, 1 reply; 37+ messages in thread From: Thomas Petazzoni @ 2018-09-06 7:44 UTC (permalink / raw) To: buildroot Hello, On Thu, 6 Sep 2018 00:22:11 +0200, Angelo Compagnucci wrote: > Packages who harms the build reproducibility or licensing can declare > FOO_TAINTS variable. If a package taints the build it will be added > to a list of tainting packages. The build ends with a warning if the > tainting packages list is not empty. Moreover, legal info will show > a warning in presence of a tainting package. > > Tainting of host packages is not supported right now. > > Signed-off-by: Angelo Compagnucci <angelo@amarulasolutions.com> > Signed-off-by: Angelo Compagnucci <angelo.compagnucci@gmail.com> Why two SoBs ? Just one matching the From: is good enough. > +ifdef $(2)_TAINTS > +ifeq ($$($$($(2)_KCONFIG_VAR)),y) > +BR2_TAINTED_BY+=$$($(2)_RAWNAME) Spaces around += No need to resend just for that. Just explain why you have two SoB, if I can keep just one, I can fix that up when applying. Thanks, Thomas -- Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 1/3] Makefile: add tainting support 2018-09-06 7:44 ` Thomas Petazzoni @ 2018-09-06 7:46 ` Angelo Compagnucci 0 siblings, 0 replies; 37+ messages in thread From: Angelo Compagnucci @ 2018-09-06 7:46 UTC (permalink / raw) To: buildroot On Thu, Sep 6, 2018 at 9:44 AM, Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote: > Hello, > > On Thu, 6 Sep 2018 00:22:11 +0200, Angelo Compagnucci wrote: >> Packages who harms the build reproducibility or licensing can declare >> FOO_TAINTS variable. If a package taints the build it will be added >> to a list of tainting packages. The build ends with a warning if the >> tainting packages list is not empty. Moreover, legal info will show >> a warning in presence of a tainting package. >> >> Tainting of host packages is not supported right now. >> >> Signed-off-by: Angelo Compagnucci <angelo@amarulasolutions.com> >> Signed-off-by: Angelo Compagnucci <angelo.compagnucci@gmail.com> > > Why two SoBs ? Just one matching the From: is good enough. > >> +ifdef $(2)_TAINTS >> +ifeq ($$($$($(2)_KCONFIG_VAR)),y) >> +BR2_TAINTED_BY+=$$($(2)_RAWNAME) > > Spaces around += > > No need to resend just for that. Just explain why you have two SoB, if > I can keep just one, I can fix that up when applying. Sorry Thomas, I was on another pc and forget about different git configuration. You can keep angelo at amarulasolutions.com one please. Thanks! > > Thanks, > > Thomas > -- > Thomas Petazzoni, CTO, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 2/3] docs/manual: adding infos about tainting 2018-09-05 22:22 [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot Angelo Compagnucci 2018-09-05 22:22 ` [Buildroot] [PATCH v5 1/3] Makefile: add tainting support Angelo Compagnucci @ 2018-09-05 22:22 ` Angelo Compagnucci 2018-09-09 8:00 ` Yann E. MORIN 2018-09-05 22:22 ` [Buildroot] [PATCH v5 3/3] package/nodejs: taint the build on external modules Angelo Compagnucci ` (3 subsequent siblings) 5 siblings, 1 reply; 37+ messages in thread From: Angelo Compagnucci @ 2018-09-05 22:22 UTC (permalink / raw) To: buildroot From: Angelo Compagnucci <angelo@amarulasolutions.com> Adding documentation about the usage of LIBFOO_TAINTS and "make check-tainted". Signed-off-by: Angelo Compagnucci <angelo@amarulasolutions.com> Signed-off-by: Angelo Compagnucci <angelo.compagnucci@gmail.com> --- docs/manual/adding-packages-generic.txt | 6 ++++++ docs/manual/legal-notice.txt | 12 ++++++++++++ 2 files changed, 18 insertions(+) diff --git a/docs/manual/adding-packages-generic.txt b/docs/manual/adding-packages-generic.txt index 7be1754..6495157 100644 --- a/docs/manual/adding-packages-generic.txt +++ b/docs/manual/adding-packages-generic.txt @@ -445,6 +445,12 @@ not and can not work as people would expect it should: to let you know, and +not saved+ will appear in the +license files+ field of the manifest file for this package. +* +LIBFOO_TAINTS+ shoud be set to YES if a package taints a Buildroot + configuration. A Buildroot configuration is tainted when a packages uses + external dependencies for which Buildroot cannot clearly recover licensing + informations. If a configuration is tainted, it means that the licensing + information produced by +make legal-info+ could not be accurate. + * +LIBFOO_ACTUAL_SOURCE_TARBALL+ only applies to packages whose +LIBFOO_SITE+ / +LIBTOO_SOURCE+ pair points to an archive that does not actually contain source code, but binary code. This a very diff --git a/docs/manual/legal-notice.txt b/docs/manual/legal-notice.txt index 6975328..7fde09a 100644 --- a/docs/manual/legal-notice.txt +++ b/docs/manual/legal-notice.txt @@ -73,6 +73,18 @@ distribution is required). When you run +make legal-info+, Buildroot produces warnings in the +README+ file to inform you of relevant material that could not be saved. +Furthermore, a Buildroot configuration could be tainted from a package that uses +some custom external dependencies from the Buildroot tree. An example could be +a package manager for a software stack that downloads the required dependencies +during the building of a package. In such cases, Buildroot cannot check the +licensing of the downloaded software and thus giving accurate licensing +informations. +To check if your configuration is tainted, run: + +-------------------- +make check-tainted +-------------------- + Finally, keep in mind that the output of +make legal-info+ is based on declarative statements in each of the packages recipes. The Buildroot developers try to do their best to keep those declarative statements as -- 2.7.4 ^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 2/3] docs/manual: adding infos about tainting 2018-09-05 22:22 ` [Buildroot] [PATCH v5 2/3] docs/manual: adding infos about tainting Angelo Compagnucci @ 2018-09-09 8:00 ` Yann E. MORIN 0 siblings, 0 replies; 37+ messages in thread From: Yann E. MORIN @ 2018-09-09 8:00 UTC (permalink / raw) To: buildroot Angelo, All, On 2018-09-06 00:22 +0200, Angelo Compagnucci spake thusly: > From: Angelo Compagnucci <angelo@amarulasolutions.com> > > Adding documentation about the usage of LIBFOO_TAINTS and > "make check-tainted". > > Signed-off-by: Angelo Compagnucci <angelo@amarulasolutions.com> > Signed-off-by: Angelo Compagnucci <angelo.compagnucci@gmail.com> > --- > docs/manual/adding-packages-generic.txt | 6 ++++++ > docs/manual/legal-notice.txt | 12 ++++++++++++ > 2 files changed, 18 insertions(+) > > diff --git a/docs/manual/adding-packages-generic.txt b/docs/manual/adding-packages-generic.txt > index 7be1754..6495157 100644 > --- a/docs/manual/adding-packages-generic.txt > +++ b/docs/manual/adding-packages-generic.txt > @@ -445,6 +445,12 @@ not and can not work as people would expect it should: > to let you know, and +not saved+ will appear in the +license files+ field > of the manifest file for this package. > > +* +LIBFOO_TAINTS+ shoud be set to YES if a package taints a Buildroot > + configuration. A Buildroot configuration is tainted when a packages uses > + external dependencies for which Buildroot cannot clearly recover licensing > + informations. If a configuration is tainted, it means that the licensing > + information produced by +make legal-info+ could not be accurate. In your cover-letter, you said: FOO_TAINTS [...] can be used to signal that a package harms the reproducibility or licensing under certain conditions. But here, you only consider the licensing problem. As I already explained in my reply to the cover letter, I believe the licensing problem is already covered by the existing licensing infrastructure: FOO_LICENSE := $(FOO_LICENSE), Unknown (unreproducible external data) (which is a bit different but better than what I suggested in the cover letter.) Regards, Yann E. MORIN. > * +LIBFOO_ACTUAL_SOURCE_TARBALL+ only applies to packages whose > +LIBFOO_SITE+ / +LIBTOO_SOURCE+ pair points to an archive that does > not actually contain source code, but binary code. This a very > diff --git a/docs/manual/legal-notice.txt b/docs/manual/legal-notice.txt > index 6975328..7fde09a 100644 > --- a/docs/manual/legal-notice.txt > +++ b/docs/manual/legal-notice.txt > @@ -73,6 +73,18 @@ distribution is required). > When you run +make legal-info+, Buildroot produces warnings in the +README+ > file to inform you of relevant material that could not be saved. > > +Furthermore, a Buildroot configuration could be tainted from a package that uses > +some custom external dependencies from the Buildroot tree. An example could be > +a package manager for a software stack that downloads the required dependencies > +during the building of a package. In such cases, Buildroot cannot check the > +licensing of the downloaded software and thus giving accurate licensing > +informations. > +To check if your configuration is tainted, run: > + > +-------------------- > +make check-tainted > +-------------------- > + > Finally, keep in mind that the output of +make legal-info+ is based on > declarative statements in each of the packages recipes. The Buildroot > developers try to do their best to keep those declarative statements as > -- > 2.7.4 > > _______________________________________________ > buildroot mailing list > buildroot at busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 3/3] package/nodejs: taint the build on external modules 2018-09-05 22:22 [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot Angelo Compagnucci 2018-09-05 22:22 ` [Buildroot] [PATCH v5 1/3] Makefile: add tainting support Angelo Compagnucci 2018-09-05 22:22 ` [Buildroot] [PATCH v5 2/3] docs/manual: adding infos about tainting Angelo Compagnucci @ 2018-09-05 22:22 ` Angelo Compagnucci 2018-09-09 7:49 ` Yann E. MORIN 2018-09-06 7:42 ` [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot Thomas Petazzoni ` (2 subsequent siblings) 5 siblings, 1 reply; 37+ messages in thread From: Angelo Compagnucci @ 2018-09-05 22:22 UTC (permalink / raw) To: buildroot This patch enables the tainting of the build when an external module is added. Signed-off-by: Angelo Compagnucci <angelo@amarulasolutions.com> Signed-off-by: Angelo Compagnucci <angelo.compagnucci@gmail.com> --- package/nodejs/nodejs.mk | 1 + 1 file changed, 1 insertion(+) diff --git a/package/nodejs/nodejs.mk b/package/nodejs/nodejs.mk index e2c94ba..322a1ec 100644 --- a/package/nodejs/nodejs.mk +++ b/package/nodejs/nodejs.mk @@ -160,6 +160,7 @@ NPM = $(TARGET_CONFIGURE_OPTS) \ # We can only call NPM if there's something to install. # ifneq ($(NODEJS_MODULES_LIST),) +NODEJS_TAINTS = YES define NODEJS_INSTALL_MODULES # If you're having trouble with module installation, adding -d to the # npm install call below and setting npm_config_rollback=false can both -- 2.7.4 ^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 3/3] package/nodejs: taint the build on external modules 2018-09-05 22:22 ` [Buildroot] [PATCH v5 3/3] package/nodejs: taint the build on external modules Angelo Compagnucci @ 2018-09-09 7:49 ` Yann E. MORIN 2018-09-09 12:17 ` Angelo Compagnucci 0 siblings, 1 reply; 37+ messages in thread From: Yann E. MORIN @ 2018-09-09 7:49 UTC (permalink / raw) To: buildroot Angelo, All, On 2018-09-06 00:22 +0200, Angelo Compagnucci spake thusly: > This patch enables the tainting of the build when an > external module is added. > > Signed-off-by: Angelo Compagnucci <angelo@amarulasolutions.com> > Signed-off-by: Angelo Compagnucci <angelo.compagnucci@gmail.com> > --- > package/nodejs/nodejs.mk | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/package/nodejs/nodejs.mk b/package/nodejs/nodejs.mk > index e2c94ba..322a1ec 100644 > --- a/package/nodejs/nodejs.mk > +++ b/package/nodejs/nodejs.mk > @@ -160,6 +160,7 @@ NPM = $(TARGET_CONFIGURE_OPTS) \ > # We can only call NPM if there's something to install. > # > ifneq ($(NODEJS_MODULES_LIST),) > +NODEJS_TAINTS = YES That is not even true. If I have a Buildroot .config that contains: BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL="http://myserver/node-mods/VERSION/foo" Then this is 100% reproducible, because *I* manage and guarantee that it is, as this is *my* repository. I could even have: BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL="$(BR2_EXTERANL_MY_TREE_PATH)/mods/foo" which is also reproducible, by way of being in my git-versioned br2-external tree. And if you were to have: BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL="foo at 1.2.3" then it would also be preproducible, because the version is specified. So, no, we can't set the tainted flag as soon as an external set of npm modules are used; this would be plain wrong for people that already do the "right thing". Regards, Yann E. MORIN. > define NODEJS_INSTALL_MODULES > # If you're having trouble with module installation, adding -d to the > # npm install call below and setting npm_config_rollback=false can both > -- > 2.7.4 > > _______________________________________________ > buildroot mailing list > buildroot at busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 3/3] package/nodejs: taint the build on external modules 2018-09-09 7:49 ` Yann E. MORIN @ 2018-09-09 12:17 ` Angelo Compagnucci 2018-09-09 13:01 ` Yann E. MORIN 0 siblings, 1 reply; 37+ messages in thread From: Angelo Compagnucci @ 2018-09-09 12:17 UTC (permalink / raw) To: buildroot Yann, All, On Sun, Sep 9, 2018 at 8:49 AM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > Angelo, All, > > On 2018-09-06 00:22 +0200, Angelo Compagnucci spake thusly: > > This patch enables the tainting of the build when an > > external module is added. > > > > Signed-off-by: Angelo Compagnucci <angelo@amarulasolutions.com> > > Signed-off-by: Angelo Compagnucci <angelo.compagnucci@gmail.com> > > --- > > package/nodejs/nodejs.mk | 1 + > > 1 file changed, 1 insertion(+) > > > > diff --git a/package/nodejs/nodejs.mk b/package/nodejs/nodejs.mk > > index e2c94ba..322a1ec 100644 > > --- a/package/nodejs/nodejs.mk > > +++ b/package/nodejs/nodejs.mk > > @@ -160,6 +160,7 @@ NPM = $(TARGET_CONFIGURE_OPTS) \ > > # We can only call NPM if there's something to install. > > # > > ifneq ($(NODEJS_MODULES_LIST),) > > +NODEJS_TAINTS = YES > > That is not even true. > > If I have a Buildroot .config that contains: > > BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL="http://myserver/node-mods/VERSION/foo" > > Then this is 100% reproducible, because *I* manage and guarantee that it > is, as this is *my* repository. > > I could even have: > > BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL="$(BR2_EXTERANL_MY_TREE_PATH)/mods/foo" > > which is also reproducible, by way of being in my git-versioned br2-external > tree. > > And if you were to have: > > BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL="foo at 1.2.3" > > then it would also be preproducible, because the version is specified. > > So, no, we can't set the tainted flag as soon as an external set of npm > modules are used; this would be plain wrong for people that already do > the "right thing". Unfortunately, that's not true. Yo have to trust the entire dependency chain to make an assumptin like that, but in my experience there is always some dependency in the chain that points to master. A couple of years ago a simple npm package almost broke the internet, I rember my company had tought time tring to find a workaround for that single package[1]. So I honestly think we cannot trust a chain of tens or hundreds of dependencies that points to random git repositories around the internet. [1] http://mentalfloss.com/article/77951/how-11-lost-lines-code-almost-broke-internet > > Regards, > Yann E. MORIN. > > > define NODEJS_INSTALL_MODULES > > # If you're having trouble with module installation, adding -d to the > > # npm install call below and setting npm_config_rollback=false can both > > -- > > 2.7.4 > > > > _______________________________________________ > > buildroot mailing list > > buildroot at busybox.net > > http://lists.busybox.net/mailman/listinfo/buildroot > > -- > .-----------------.--------------------.------------------.--------------------. > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 3/3] package/nodejs: taint the build on external modules 2018-09-09 12:17 ` Angelo Compagnucci @ 2018-09-09 13:01 ` Yann E. MORIN 2018-09-09 13:29 ` Angelo Compagnucci 0 siblings, 1 reply; 37+ messages in thread From: Yann E. MORIN @ 2018-09-09 13:01 UTC (permalink / raw) To: buildroot Angelo, All, On 2018-09-09 13:17 +0100, Angelo Compagnucci spake thusly: > On Sun, Sep 9, 2018 at 8:49 AM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > On 2018-09-06 00:22 +0200, Angelo Compagnucci spake thusly: > > > This patch enables the tainting of the build when an > > > external module is added. > > > > > > Signed-off-by: Angelo Compagnucci <angelo@amarulasolutions.com> > > > Signed-off-by: Angelo Compagnucci <angelo.compagnucci@gmail.com> > > > --- > > > package/nodejs/nodejs.mk | 1 + > > > 1 file changed, 1 insertion(+) > > > > > > diff --git a/package/nodejs/nodejs.mk b/package/nodejs/nodejs.mk > > > index e2c94ba..322a1ec 100644 > > > --- a/package/nodejs/nodejs.mk > > > +++ b/package/nodejs/nodejs.mk > > > @@ -160,6 +160,7 @@ NPM = $(TARGET_CONFIGURE_OPTS) \ > > > # We can only call NPM if there's something to install. > > > # > > > ifneq ($(NODEJS_MODULES_LIST),) > > > +NODEJS_TAINTS = YES > > > > That is not even true. > > > > If I have a Buildroot .config that contains: > > > > BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL="http://myserver/node-mods/VERSION/foo" > > > > Then this is 100% reproducible, because *I* manage and guarantee that it > > is, as this is *my* repository. > > > > I could even have: > > > > BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL="$(BR2_EXTERANL_MY_TREE_PATH)/mods/foo" > > > > which is also reproducible, by way of being in my git-versioned br2-external > > tree. > > > > And if you were to have: > > > > BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL="foo at 1.2.3" > > > > then it would also be preproducible, because the version is specified. > > > > So, no, we can't set the tainted flag as soon as an external set of npm > > modules are used; this would be plain wrong for people that already do > > the "right thing". > > Unfortunately, that's not true. Yo have to trust the entire dependency > chain to make an assumptin like that, True, but there *are* cases where this is trusted, e.g. when said chain is listed in the modules list, and that each dependency is specified by version, and/or that the modules are identified by an exact URL pointing to an internal, stable, reproducible repository. I only provided a few trivial examples to explain where the assumption you made about taint is wrong. What I mean is that there cases where using additional nodejs modules will taint the build (as per your tainting definition), but there are also cases where said modules will not taint the build (again, by your definition of taint). To get to the bottom of it: we have no way to know whether a set of additional nodejs modules will taint or not the build. As such, marking the build as tainted as soon as such modules are used is not correct. So, if I were to use a list of modules for which I know that they are reproducible, why would you want to mark my build as tainted when it is not? Your patch would always mark my builds as tainted, and an actual taint would be undetectable. > but in my experience there is > always some dependency in the chain that points to master. > A couple of years ago a simple npm package almost broke the internet, > I rember my company had tought time tring to find a workaround for > that single package[1]. Yeah, I know about the leftpad fiasco. It really made me laugh out loud for a good while! And I'm still amused that people are stil keen on using this, well... how could I put it and stay politically correct? There is only a 4-letter word that comes to mind... ;-] > So I honestly think we cannot trust a chain of tens or hundreds of > dependencies that points to random git repositories around the > internet. No we can't. But what your patch does is also wrong for people who actually *know* what they are doing, see above. Regards, Yann E. MORIN. > [1] http://mentalfloss.com/article/77951/how-11-lost-lines-code-almost-broke-internet > > > > > Regards, > > Yann E. MORIN. > > > > > define NODEJS_INSTALL_MODULES > > > # If you're having trouble with module installation, adding -d to the > > > # npm install call below and setting npm_config_rollback=false can both > > > -- > > > 2.7.4 > > > > > > _______________________________________________ > > > buildroot mailing list > > > buildroot at busybox.net > > > http://lists.busybox.net/mailman/listinfo/buildroot > > > > -- > > .-----------------.--------------------.------------------.--------------------. > > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > > '------------------------------^-------^------------------^--------------------' -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 3/3] package/nodejs: taint the build on external modules 2018-09-09 13:01 ` Yann E. MORIN @ 2018-09-09 13:29 ` Angelo Compagnucci 0 siblings, 0 replies; 37+ messages in thread From: Angelo Compagnucci @ 2018-09-09 13:29 UTC (permalink / raw) To: buildroot Yann, All, On Sun, Sep 9, 2018 at 2:01 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > Angelo, All, > > On 2018-09-09 13:17 +0100, Angelo Compagnucci spake thusly: > > On Sun, Sep 9, 2018 at 8:49 AM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > > On 2018-09-06 00:22 +0200, Angelo Compagnucci spake thusly: > > > > This patch enables the tainting of the build when an > > > > external module is added. > > > > > > > > Signed-off-by: Angelo Compagnucci <angelo@amarulasolutions.com> > > > > Signed-off-by: Angelo Compagnucci <angelo.compagnucci@gmail.com> > > > > --- > > > > package/nodejs/nodejs.mk | 1 + > > > > 1 file changed, 1 insertion(+) > > > > > > > > diff --git a/package/nodejs/nodejs.mk b/package/nodejs/nodejs.mk > > > > index e2c94ba..322a1ec 100644 > > > > --- a/package/nodejs/nodejs.mk > > > > +++ b/package/nodejs/nodejs.mk > > > > @@ -160,6 +160,7 @@ NPM = $(TARGET_CONFIGURE_OPTS) \ > > > > # We can only call NPM if there's something to install. > > > > # > > > > ifneq ($(NODEJS_MODULES_LIST),) > > > > +NODEJS_TAINTS = YES > > > > > > That is not even true. > > > > > > If I have a Buildroot .config that contains: > > > > > > BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL="http://myserver/node-mods/VERSION/foo" > > > > > > Then this is 100% reproducible, because *I* manage and guarantee that it > > > is, as this is *my* repository. > > > > > > I could even have: > > > > > > BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL="$(BR2_EXTERANL_MY_TREE_PATH)/mods/foo" > > > > > > which is also reproducible, by way of being in my git-versioned br2-external > > > tree. > > > > > > And if you were to have: > > > > > > BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL="foo at 1.2.3" > > > > > > then it would also be preproducible, because the version is specified. > > > > > > So, no, we can't set the tainted flag as soon as an external set of npm > > > modules are used; this would be plain wrong for people that already do > > > the "right thing". > > > > Unfortunately, that's not true. Yo have to trust the entire dependency > > chain to make an assumptin like that, > > True, but there *are* cases where this is trusted, e.g. when said chain > is listed in the modules list, and that each dependency is specified by > version, and/or that the modules are identified by an exact URL pointing > to an internal, stable, reproducible repository. > > I only provided a few trivial examples to explain where the assumption > you made about taint is wrong. > > What I mean is that there cases where using additional nodejs modules > will taint the build (as per your tainting definition), but there are > also cases where said modules will not taint the build (again, by your > definition of taint). > > To get to the bottom of it: we have no way to know whether a set of > additional nodejs modules will taint or not the build. As such, marking > the build as tainted as soon as such modules are used is not correct. > > So, if I were to use a list of modules for which I know that they are > reproducible, why would you want to mark my build as tainted when it is > not? Your patch would always mark my builds as tainted, and an actual > taint would be undetectable. If you are an experienced developer and are adding a package X to buildroot maintaining the reproducibility, you can package your set of manageable dependencies and get rid of the problem, you can even call the package manager from the inside of your .mk and do the job. The use case here is for all the other people that wants to use something big and cannot trust all of their dependencies. Try to have a look at any modern php applications based on composer, it's really not easy to find our rosebud. So, in the end, I think that we cannot trust a package manager in any way. If we absolutely regret adding package managers we could end the discussion here. Else, if we provide package managers, we cannot trust em when used. > > but in my experience there is > > always some dependency in the chain that points to master. > > A couple of years ago a simple npm package almost broke the internet, > > I rember my company had tought time tring to find a workaround for > > that single package[1]. > > Yeah, I know about the leftpad fiasco. It really made me laugh out loud > for a good while! And I'm still amused that people are stil keen on > using this, well... how could I put it and stay politically correct? > There is only a 4-letter word that comes to mind... ;-] Mee too and luckly I was not a software developer there :) . > > So I honestly think we cannot trust a chain of tens or hundreds of > > dependencies that points to random git repositories around the > > internet. > > No we can't. But what your patch does is also wrong for people who > actually *know* what they are doing, see above. > > Regards, > Yann E. MORIN. > > > [1] http://mentalfloss.com/article/77951/how-11-lost-lines-code-almost-broke-internet > > > > > > > > Regards, > > > Yann E. MORIN. > > > > > > > define NODEJS_INSTALL_MODULES > > > > # If you're having trouble with module installation, adding -d to the > > > > # npm install call below and setting npm_config_rollback=false can both > > > > -- > > > > 2.7.4 > > > > > > > > _______________________________________________ > > > > buildroot mailing list > > > > buildroot at busybox.net > > > > http://lists.busybox.net/mailman/listinfo/buildroot > > > > > > -- > > > .-----------------.--------------------.------------------.--------------------. > > > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > > > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > > > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > > > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > > > '------------------------------^-------^------------------^--------------------' > > -- > .-----------------.--------------------.------------------.--------------------. > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot 2018-09-05 22:22 [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot Angelo Compagnucci ` (2 preceding siblings ...) 2018-09-05 22:22 ` [Buildroot] [PATCH v5 3/3] package/nodejs: taint the build on external modules Angelo Compagnucci @ 2018-09-06 7:42 ` Thomas Petazzoni 2018-09-09 7:36 ` Yann E. MORIN 2018-11-01 12:14 ` Arnout Vandecappelle 5 siblings, 0 replies; 37+ messages in thread From: Thomas Petazzoni @ 2018-09-06 7:42 UTC (permalink / raw) To: buildroot Hello, On Thu, 6 Sep 2018 00:22:10 +0200, Angelo Compagnucci wrote: > Packages that need to resolve dependencies internally > and use a package manager would harm the reproducibility > of a build, moreover they escape the legal infrastructure > not giving enough informations on licensing. > > This patch adds a tainting mechanism in the form of a > variable FOO_TAINTS that can be used to signal that > a package harms the reproducibility or licensing under > certain conditions. > > This opens the door to include per language dependency > managers in buildroot. > > Angelo Compagnucci (3): > Makefile: add tainting support > docs/manual: adding infos about tainting > package/nodejs: taint the build on external modules It would be good to include a changelog of what has been changed between the different iterations of your series. Thomas -- Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot 2018-09-05 22:22 [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot Angelo Compagnucci ` (3 preceding siblings ...) 2018-09-06 7:42 ` [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot Thomas Petazzoni @ 2018-09-09 7:36 ` Yann E. MORIN 2018-09-09 12:10 ` Thomas Petazzoni 2018-11-01 12:14 ` Arnout Vandecappelle 5 siblings, 1 reply; 37+ messages in thread From: Yann E. MORIN @ 2018-09-09 7:36 UTC (permalink / raw) To: buildroot Angelo, Thomas, All, On 2018-09-06 00:22 +0200, Angelo Compagnucci spake thusly: > Packages that need to resolve dependencies internally > and use a package manager would harm the reproducibility > of a build, moreover they escape the legal infrastructure > not giving enough informations on licensing. So, I am not in favour of having support for such package managers in Buildroot, to begin with. As you say, the reproducibility of a build, as offered by Buildroot, is really a very important point. In my opinion, reproducibility is even paramount. Whatever gets in the way should be banned. > This patch adds a tainting mechanism in the form of a > variable FOO_TAINTS that can be used to signal that > a package harms the reproducibility or licensing under > certain conditions. When I read that, it comes that the tainting mechanism serves two purposes: first, mark (non-)reproducibility, and second, mark incorrect and/or incomplete licensing information. This does not sound nice to me. For the licensing information, I would just rely on the existing licensing infra, and be done with it, i.e. add : ifneq ($(FOO_NON_REPRODUCIBLE_EXTERNAL_DATA),) FOO_LICENSE += Unknown (non reproducible external data) endif I.e., no need for the tainting mechanism to represent the licensing problem, as we already have what is needed to cover it. Now, what would be the purpose for this "tainted" flag? I understand clearly what it provides, indeed, technically, but what would it serve to the user? If I were to use a non-reproduible set of data ffor;, say, nodejs, then I know that this is not handled by Buildroot, and thus it is not reproducible. I don't need to be told it, escept maybe as a note in the manual: "If you use external data from npm/pypi, cpan, whatnot, then your build is not reproducible; that will be hurting kittens." Instead, I am more in favour of packaging such external stuff as Buildroot packages, like we've been doing for erlang, lua, perl, and python, even if we have to add new infrastructures for thos (npm, I'm looking at you!) Besides, you're missing a big piece of potential non-reproducibility source: br2-external trees. If we ever were to have such a tainting mechanism, we should mark the build as tainted as soon as there is a br2-external tree. Ditto as soon as we have a FOO_OVERRIDE_SRCDIR set, which should mark the build as tainted. Except we already use FOO_OVERRIDE_SRCDIR when FOO_SITE = local... > This opens the door to include per language dependency > managers in buildroot. Instead, why not provide a helper script (or enhance the existing ones if need be), than generated buildroot packages for those external packge managers? Sorry, this is not the technical review you may have expected. Instead, it is a position statement on my part. Regards, Yann E. MORIN. > Angelo Compagnucci (3): > Makefile: add tainting support > docs/manual: adding infos about tainting > package/nodejs: taint the build on external modules > > Makefile | 10 ++++++++++ > docs/manual/adding-packages-generic.txt | 6 ++++++ > docs/manual/legal-notice.txt | 12 ++++++++++++ > package/nodejs/nodejs.mk | 1 + > package/pkg-generic.mk | 15 +++++++++++++++ > 5 files changed, 44 insertions(+) > > -- > 2.7.4 > > _______________________________________________ > buildroot mailing list > buildroot at busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot 2018-09-09 7:36 ` Yann E. MORIN @ 2018-09-09 12:10 ` Thomas Petazzoni 2018-09-09 12:25 ` Angelo Compagnucci 2018-09-09 13:27 ` Yann E. MORIN 0 siblings, 2 replies; 37+ messages in thread From: Thomas Petazzoni @ 2018-09-09 12:10 UTC (permalink / raw) To: buildroot Hello Yann, On Sun, 9 Sep 2018 09:36:16 +0200, Yann E. MORIN wrote: > So, I am not in favour of having support for such package managers in > Buildroot, to begin with. We already had this discussion in the first iteration of this patch series, and even before. I think everybody agrees that such package managers (npm and al.) are really not nice in the context of Linux distributions or build systems. However, from a practical point of view, some people want those features, and are ready to trade the reproducibility against "ease of use". In addition, specifically for nodejs/npm, the number of packages is so enormous that I am not sure it is really realistic to create one Buildroot package for each nodejs module, and keep things sane. Especially if only a few people are interested in nodejs modules. > As you say, the reproducibility of a build, as offered by Buildroot, is > really a very important point. In my opinion, reproducibility is even > paramount. Whatever gets in the way should be banned. I disagree on the "should be banned", and that's the whole point of this series: allow it, while making sure that the user is well aware of the fact that he has given up on the "reproducibility" by using one of those package managers. > > This patch adds a tainting mechanism in the form of a > > variable FOO_TAINTS that can be used to signal that > > a package harms the reproducibility or licensing under > > certain conditions. > > When I read that, it comes that the tainting mechanism serves two > purposes: first, mark (non-)reproducibility, and second, mark incorrect > and/or incomplete licensing information. > > This does not sound nice to me. > > For the licensing information, I would just rely on the existing > licensing infra, and be done with it, i.e. add : > > ifneq ($(FOO_NON_REPRODUCIBLE_EXTERNAL_DATA),) > FOO_LICENSE += Unknown (non reproducible external data) > endif That's true. > Now, what would be the purpose for this "tainted" flag? I understand > clearly what it provides, indeed, technically, but what would it serve > to the user? See above: it makes it absolutely clear to the user that he is using some feature that kills one key advantage of build system: reproducibility. Hiding this information in the manual or in a Config.in help text is IMO not enough: we really want the user to have a prominent warning at the end of the build. > If I were to use a non-reproduible set of data ffor;, say, nodejs, then > I know that this is not handled by Buildroot, and thus it is not > reproducible. I don't need to be told it, escept maybe as a note in the > manual: "If you use external data from npm/pypi, cpan, whatnot, then > your build is not reproducible; that will be hurting kittens." I disagree, newcomers are unlikely to realize this and read our manual end to end. > Instead, I am more in favour of packaging such external stuff as > Buildroot packages, like we've been doing for erlang, lua, perl, and > python, even if we have to add new infrastructures for thos (npm, I'm > looking at you!) Yes, that is the ideal situation. But that is not necessarily realistic for npm modules (due to their number), and having this option of using the provided package manager is reasonable, as long as the user is aware of the drawbacks. > Besides, you're missing a big piece of potential non-reproducibility > source: br2-external trees. If we ever were to have such a tainting > mechanism, we should mark the build as tainted as soon as there is a > br2-external tree. I disagree, a BR2_EXTERNAL does not make the build non-reproducible. As long as you have the same BR2_EXTERNAL + Buildroot, you can rebuild the same system. > Ditto as soon as we have a FOO_OVERRIDE_SRCDIR set, which should mark > the build as tainted. Except we already use FOO_OVERRIDE_SRCDIR when > FOO_SITE = local... FOO_OVERRIDE_SRCDIR is indeed a better example of thing that makes the build non-reproducible, we could potentially include this in the tainted mechanism. Though to me, this is somewhat less important: only advanced users are likely to use <pkg>_OVERRIDE_SRCDIR, and if they do, they will definitely realize the impact on reproducibility. While the newcomer toying around with nodejs/npm may not realize the impact of using a third-party package manager on reproducibility. So, yes, perhaps <pkg>_OVERRIDE_SRCDIR should ultimately taint the build, but I don't see handling that as a requirement to start introducing this "taint" flag. > Instead, why not provide a helper script (or enhance the existing ones > if need be), than generated buildroot packages for those external packge > managers? We really don't want thousands of npm packages I believe :-/ Best regards, Thomas -- Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot 2018-09-09 12:10 ` Thomas Petazzoni @ 2018-09-09 12:25 ` Angelo Compagnucci 2018-09-09 13:33 ` Yann E. MORIN 2018-09-09 13:27 ` Yann E. MORIN 1 sibling, 1 reply; 37+ messages in thread From: Angelo Compagnucci @ 2018-09-09 12:25 UTC (permalink / raw) To: buildroot Thomas, Yann, On Sun, Sep 9, 2018 at 1:10 PM Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote: > > Hello Yann, > > On Sun, 9 Sep 2018 09:36:16 +0200, Yann E. MORIN wrote: > > > So, I am not in favour of having support for such package managers in > > Buildroot, to begin with. > > We already had this discussion in the first iteration of this patch > series, and even before. I think everybody agrees that such package > managers (npm and al.) are really not nice in the context of Linux > distributions or build systems. > > However, from a practical point of view, some people want those > features, and are ready to trade the reproducibility against "ease of > use". In addition, specifically for nodejs/npm, the number of packages > is so enormous that I am not sure it is really realistic to create one > Buildroot package for each nodejs module, and keep things sane. > Especially if only a few people are interested in nodejs modules. > > > As you say, the reproducibility of a build, as offered by Buildroot, is > > really a very important point. In my opinion, reproducibility is even > > paramount. Whatever gets in the way should be banned. > > I disagree on the "should be banned", and that's the whole point of > this series: allow it, while making sure that the user is well aware of > the fact that he has given up on the "reproducibility" by using one of > those package managers. > > > > This patch adds a tainting mechanism in the form of a > > > variable FOO_TAINTS that can be used to signal that > > > a package harms the reproducibility or licensing under > > > certain conditions. > > > > When I read that, it comes that the tainting mechanism serves two > > purposes: first, mark (non-)reproducibility, and second, mark incorrect > > and/or incomplete licensing information. > > > > This does not sound nice to me. > > > > For the licensing information, I would just rely on the existing > > licensing infra, and be done with it, i.e. add : > > > > ifneq ($(FOO_NON_REPRODUCIBLE_EXTERNAL_DATA),) > > FOO_LICENSE += Unknown (non reproducible external data) > > endif > > That's true. > > > Now, what would be the purpose for this "tainted" flag? I understand > > clearly what it provides, indeed, technically, but what would it serve > > to the user? > > See above: it makes it absolutely clear to the user that he is using > some feature that kills one key advantage of build system: > reproducibility. Hiding this information in the manual or in a > Config.in help text is IMO not enough: we really want the user to have > a prominent warning at the end of the build. > > > If I were to use a non-reproduible set of data ffor;, say, nodejs, then > > I know that this is not handled by Buildroot, and thus it is not > > reproducible. I don't need to be told it, escept maybe as a note in the > > manual: "If you use external data from npm/pypi, cpan, whatnot, then > > your build is not reproducible; that will be hurting kittens." > > I disagree, newcomers are unlikely to realize this and read our manual > end to end. > > > Instead, I am more in favour of packaging such external stuff as > > Buildroot packages, like we've been doing for erlang, lua, perl, and > > python, even if we have to add new infrastructures for thos (npm, I'm > > looking at you!) > > Yes, that is the ideal situation. But that is not necessarily realistic > for npm modules (due to their number), and having this option of using > the provided package manager is reasonable, as long as the user is > aware of the drawbacks. > > > Besides, you're missing a big piece of potential non-reproducibility > > source: br2-external trees. If we ever were to have such a tainting > > mechanism, we should mark the build as tainted as soon as there is a > > br2-external tree. > > I disagree, a BR2_EXTERNAL does not make the build non-reproducible. As > long as you have the same BR2_EXTERNAL + Buildroot, you can rebuild the > same system. > > > Ditto as soon as we have a FOO_OVERRIDE_SRCDIR set, which should mark > > the build as tainted. Except we already use FOO_OVERRIDE_SRCDIR when > > FOO_SITE = local... > > FOO_OVERRIDE_SRCDIR is indeed a better example of thing that makes the > build non-reproducible, we could potentially include this in the > tainted mechanism. > > Though to me, this is somewhat less important: only advanced users are > likely to use <pkg>_OVERRIDE_SRCDIR, and if they do, they will > definitely realize the impact on reproducibility. While the newcomer > toying around with nodejs/npm may not realize the impact of using a > third-party package manager on reproducibility. > > So, yes, perhaps <pkg>_OVERRIDE_SRCDIR should ultimately taint the > build, but I don't see handling that as a requirement to start > introducing this "taint" flag. > > > Instead, why not provide a helper script (or enhance the existing ones > > if need be), than generated buildroot packages for those external packge > > managers? > > We really don't want thousands of npm packages I believe :-/ Thomas said evrithing exceptionally well, but I would like to underling another thing: automated building infrastructures like continuos integration. If a project hasa nedd of reproducibility, the countinous integration could check if a random developer introduced something not reproducible and mark the build as invalid. I think this is really a big plus of this solution. Sincerely, Angelo > > Best regards, > > Thomas > -- > Thomas Petazzoni, CTO, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com > _______________________________________________ > buildroot mailing list > buildroot at busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot 2018-09-09 12:25 ` Angelo Compagnucci @ 2018-09-09 13:33 ` Yann E. MORIN 2018-09-09 13:44 ` Angelo Compagnucci 0 siblings, 1 reply; 37+ messages in thread From: Yann E. MORIN @ 2018-09-09 13:33 UTC (permalink / raw) To: buildroot Angelo, Thomas, All, On 2018-09-09 13:25 +0100, Angelo Compagnucci spake thusly: > On Sun, Sep 9, 2018 at 1:10 PM Thomas Petazzoni > <thomas.petazzoni@bootlin.com> wrote: [--SNIP--] > Thomas said evrithing exceptionally well, but I would like to > underling another thing: automated building infrastructures like > continuos integration. > > If a project hasa nedd of reproducibility, the countinous integration > could check if a random developer introduced something not > reproducible and mark the build as invalid. I think this is really a > big plus of this solution. I do understand the concern, trust me, I do. What I am saying is that the solution you propose will not allow that, because there is no way to decide whether a specific .config is or is not reproducible, as per the examples I provided in the nodejs case. If a build is imprperly marked as tainted, then users will just disregard that information and never consult it, and just not use it in their automated buildsystemsd (jenkins, gitlab-ci, whatever). And even if they do have a job doing the check, that job can detect a change from "not tainted" to "tainted" because the job will always report "tainted". So, your solution does not even cover your automated builds use-case either. Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot 2018-09-09 13:33 ` Yann E. MORIN @ 2018-09-09 13:44 ` Angelo Compagnucci 2018-09-09 14:20 ` Yann E. MORIN 0 siblings, 1 reply; 37+ messages in thread From: Angelo Compagnucci @ 2018-09-09 13:44 UTC (permalink / raw) To: buildroot Yann, All, On Sun, Sep 9, 2018 at 2:33 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > Angelo, Thomas, All, > > On 2018-09-09 13:25 +0100, Angelo Compagnucci spake thusly: > > On Sun, Sep 9, 2018 at 1:10 PM Thomas Petazzoni > > <thomas.petazzoni@bootlin.com> wrote: > [--SNIP--] > > Thomas said evrithing exceptionally well, but I would like to > > underling another thing: automated building infrastructures like > > continuos integration. > > > > If a project hasa nedd of reproducibility, the countinous integration > > could check if a random developer introduced something not > > reproducible and mark the build as invalid. I think this is really a > > big plus of this solution. > > I do understand the concern, trust me, I do. > > What I am saying is that the solution you propose will not allow that, > because there is no way to decide whether a specific .config is or is > not reproducible, as per the examples I provided in the nodejs case. > > If a build is imprperly marked as tainted, then users will just > disregard that information and never consult it, and just not use it in > their automated buildsystemsd (jenkins, gitlab-ci, whatever). And even > if they do have a job doing the check, that job can detect a change from > "not tainted" to "tainted" because the job will always report "tainted". My concern here is that you start from a reproducible build, add your packages right and so maintain your build reproducible, buildroot will work as before. As soon you use a package manager tainting will be signaled. Taint is mean to signal that there is a potential problem, and if you don't want to slip into it, you can always do the right thing and package your software and packaging also it's dependencies. As soon as you do this, the taint disappear. I thin it could even be a deterrent to package the software randomly! Sincerely, Angelo > > So, your solution does not even cover your automated builds use-case > either. > > Regards, > Yann E. MORIN. > > -- > .-----------------.--------------------.------------------.--------------------. > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot 2018-09-09 13:44 ` Angelo Compagnucci @ 2018-09-09 14:20 ` Yann E. MORIN 2018-09-09 16:58 ` Angelo Compagnucci 0 siblings, 1 reply; 37+ messages in thread From: Yann E. MORIN @ 2018-09-09 14:20 UTC (permalink / raw) To: buildroot Angelo, All, On 2018-09-09 14:44 +0100, Angelo Compagnucci spake thusly: > On Sun, Sep 9, 2018 at 2:33 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > On 2018-09-09 13:25 +0100, Angelo Compagnucci spake thusly: > > > On Sun, Sep 9, 2018 at 1:10 PM Thomas Petazzoni > > > <thomas.petazzoni@bootlin.com> wrote: > > [--SNIP--] > > > Thomas said evrithing exceptionally well, but I would like to > > > underling another thing: automated building infrastructures like > > > continuos integration. > > > > > > If a project hasa nedd of reproducibility, the countinous integration > > > could check if a random developer introduced something not > > > reproducible and mark the build as invalid. I think this is really a > > > big plus of this solution. > > > > I do understand the concern, trust me, I do. > > > > What I am saying is that the solution you propose will not allow that, > > because there is no way to decide whether a specific .config is or is > > not reproducible, as per the examples I provided in the nodejs case. > > > > If a build is imprperly marked as tainted, then users will just > > disregard that information and never consult it, and just not use it in > > their automated buildsystemsd (jenkins, gitlab-ci, whatever). And even > > if they do have a job doing the check, that job can detect a change from > > "not tainted" to "tainted" because the job will always report "tainted". > > My concern here is that you start from a reproducible build, add your > packages right and so maintain your build reproducible, buildroot will > work as before. So you are, like I am, in fact arguing that we should have actual packages for such external modules? ;-) > As soon you use a package manager tainting will be > signaled. This is where I disagree. Using such package managers does not imply that the build is tainted. This is a false dichotomy. > Taint is mean to signal that there is a potential problem, and if you > don't want to slip into it, you can always do the right thing and > package your software and packaging also it's dependencies. And what I am saying is that the heuristic you suggest to decide whether a build should be considered tainted or not is incorrect. > As soon as you do this, the taint disappear. I thin it could even be a > deterrent to package the software randomly! Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot 2018-09-09 14:20 ` Yann E. MORIN @ 2018-09-09 16:58 ` Angelo Compagnucci 2018-09-09 18:55 ` Yann E. MORIN 0 siblings, 1 reply; 37+ messages in thread From: Angelo Compagnucci @ 2018-09-09 16:58 UTC (permalink / raw) To: buildroot Yann, All On Sun, Sep 9, 2018 at 3:20 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > Angelo, All, > > On 2018-09-09 14:44 +0100, Angelo Compagnucci spake thusly: > > On Sun, Sep 9, 2018 at 2:33 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > > On 2018-09-09 13:25 +0100, Angelo Compagnucci spake thusly: > > > > On Sun, Sep 9, 2018 at 1:10 PM Thomas Petazzoni > > > > <thomas.petazzoni@bootlin.com> wrote: > > > [--SNIP--] > > > > Thomas said evrithing exceptionally well, but I would like to > > > > underling another thing: automated building infrastructures like > > > > continuos integration. > > > > > > > > If a project hasa nedd of reproducibility, the countinous integration > > > > could check if a random developer introduced something not > > > > reproducible and mark the build as invalid. I think this is really a > > > > big plus of this solution. > > > > > > I do understand the concern, trust me, I do. > > > > > > What I am saying is that the solution you propose will not allow that, > > > because there is no way to decide whether a specific .config is or is > > > not reproducible, as per the examples I provided in the nodejs case. > > > > > > If a build is imprperly marked as tainted, then users will just > > > disregard that information and never consult it, and just not use it in > > > their automated buildsystemsd (jenkins, gitlab-ci, whatever). And even > > > if they do have a job doing the check, that job can detect a change from > > > "not tainted" to "tainted" because the job will always report "tainted". > > > > My concern here is that you start from a reproducible build, add your > > packages right and so maintain your build reproducible, buildroot will > > work as before. > > So you are, like I am, in fact arguing that we should have actual packages > for such external modules? ;-) I can't see any value on having hundreds of npm packages probably outdated at wich you should add hundreds of php composer packages, go modules, cargo packages and so on. >> > As soon you use a package manager tainting will be > > signaled. > > This is where I disagree. > > Using such package managers does not imply that the build is tainted. > This is a false dichotomy. Probably it is but my experience says not, anyway, if you are dealing with a complex php software for example you can: * call php composer manually on the modules you are sure being reproducible in your .mk and live happy * use the package host-composer distributed with buildroot asking it to do a "composer install" living with the fact that it could do anything and you build will be tainted. > > Taint is mean to signal that there is a potential problem, and if you > > don't want to slip into it, you can always do the right thing and > > package your software and packaging also it's dependencies. > > And what I am saying is that the heuristic you suggest to decide whether > a build should be considered tainted or not is incorrect. It's not an heuristic, it's a rule: you ask to a package manager to resolve your dependencies, your build _could_ be tainted. You want to be sure: you write your own rule. If you write a clean mk that does everything right you shouldn't add FOO_TAINTS, that's it. I can agree with you that some packages could be considered first class citizens, the others based on package managers could not be as good as first ones. But at least, you give the tools to developers who wants to add a non trivial web package or similar to buildroot. What I'm saying here is that we don't have such a rule we simply cannot add any time soon any software based on package managers. Probably there is some other smarter way to do it that I cannot see ... I'm open to suggestions! > > > As soon as you do this, the taint disappear. I thin it could even be a > > deterrent to package the software randomly! > > Regards, > Yann E. MORIN. > > -- > .-----------------.--------------------.------------------.--------------------. > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot 2018-09-09 16:58 ` Angelo Compagnucci @ 2018-09-09 18:55 ` Yann E. MORIN 2018-09-09 20:18 ` Angelo Compagnucci 0 siblings, 1 reply; 37+ messages in thread From: Yann E. MORIN @ 2018-09-09 18:55 UTC (permalink / raw) To: buildroot Angelo, All, This is the last email I'll be sending in this thread, because it looks like the both of us are re-hashing the same arguments again and again... Yet, let me reply to parts of your email... On 2018-09-09 17:58 +0100, Angelo Compagnucci spake thusly: > On Sun, Sep 9, 2018 at 3:20 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > > > On 2018-09-09 14:44 +0100, Angelo Compagnucci spake thusly: > > > On Sun, Sep 9, 2018 at 2:33 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > > > On 2018-09-09 13:25 +0100, Angelo Compagnucci spake thusly: [--SNIP--] > > > My concern here is that you start from a reproducible build, add your > > > packages right and so maintain your build reproducible, buildroot will > > > work as before. > > > > So you are, like I am, in fact arguing that we should have actual packages > > for such external modules? ;-) > > I can't see any value on having hundreds of npm packages probably > outdated at wich you should add hundreds of php composer packages, go > modules, cargo packages and so on. Why not? We do have hundreds (~270, which literaly makes that number 'hundreds') of python packages, some (a lot?) of them probably outdated. Worse, yet: we *do* have thousands of packages, a lot of them outdated. I don't see why we would single out npm packages at all. (Besides the fact that I don't like them; I don't like perl either, but I'm still quite happy to see perl packages.) > >> > As soon you use a package manager tainting will be > > > signaled. > > > > This is where I disagree. > > > > Using such package managers does not imply that the build is tainted. > > This is a false dichotomy. > > Probably it is but my experience says not, And in my experience, it can be. If you are using an internal repository of which you control the content, then the list of nodejs modules *is* reproducible. This is not a theoretical situation, by the way; this is a real situation. Your heursitic is indeed not a heuristic, but a policy. Your policy states "any external module taints the build". My policy states "use our internal repository of trusted and known modules, anything else taints the build". So, the code currently expreses your policy, and precludes using mine, which I believe is still superior because I do not even rely on external data which may disapear at the whim of an external entity; everything I need is already internal and stable. > anyway, if you are dealing > with a complex php software for example you can: > * call php composer manually on the modules you are sure being > reproducible in your .mk and live happy I have no idea what php-composer is. I guess it would download and install php files, not unlike npm does, and probably looks as ugly as npm. > * use the package host-composer distributed with buildroot asking it > to do a "composer install" living with the fact that it could do > anything and you build will be tainted. > > > > Taint is mean to signal that there is a potential problem, and if you > > > don't want to slip into it, you can always do the right thing and > > > package your software and packaging also it's dependencies. > > > > And what I am saying is that the heuristic you suggest to decide whether > > a build should be considered tainted or not is incorrect. > > It's not an heuristic, it's a rule: you ask to a package manager to > resolve your dependencies, your build _could_ be tainted. You want to > be sure: you write your own rule. So, because the build _could_ be tainted, you want to make all builds as tainted? That's insane... > If you write a clean mk that does everything right you shouldn't add > FOO_TAINTS, that's it. > > I can agree with you that some packages could be considered first > class citizens, the others based on package managers could not be as > good as first ones. But at least, you give the tools to developers who > wants to add a non trivial web package or similar to buildroot. Well, the tainted stuff is basically orthogonal to the package manager stuff, I think. Even if I despise the package managers, I can see that people are so keen on using them... What I am arguing in this thread, is that the tainted status can *not* be deduced from the configuratioe tainted stuff is basically orthogonal to the package manager stuff, I think. Even if I despise the package managers, I can see that people are so keen on using them... What I am arguing in this thread, is that the tainted status can *not* be deduced from the configuration. The way you are currently coding the tainted flag would break for the people that are already doing it right, and for whom this tainted flag would ultimately prove to be the most valuable. Regards, Yann E. MORIN. > What I'm saying here is that we don't have such a rule we simply > cannot add any time soon any software based on package managers. > > Probably there is some other smarter way to do it that I cannot see ... > > I'm open to suggestions! > > > > > > As soon as you do this, the taint disappear. I thin it could even be a > > > deterrent to package the software randomly! > > > > Regards, > > Yann E. MORIN. > > > > -- > > .-----------------.--------------------.------------------.--------------------. > > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > > '------------------------------^-------^------------------^--------------------' -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot 2018-09-09 18:55 ` Yann E. MORIN @ 2018-09-09 20:18 ` Angelo Compagnucci 2018-09-10 7:50 ` Angelo Compagnucci 0 siblings, 1 reply; 37+ messages in thread From: Angelo Compagnucci @ 2018-09-09 20:18 UTC (permalink / raw) To: buildroot Yann, All, On Sun, Sep 9, 2018 at 7:55 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > Angelo, All, > > This is the last email I'll be sending in this thread, because it looks > like the both of us are re-hashing the same arguments again and again... > Yet, let me reply to parts of your email... Last one, I promise! > On 2018-09-09 17:58 +0100, Angelo Compagnucci spake thusly: > > On Sun, Sep 9, 2018 at 3:20 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > > > > > On 2018-09-09 14:44 +0100, Angelo Compagnucci spake thusly: > > > > On Sun, Sep 9, 2018 at 2:33 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > > > > On 2018-09-09 13:25 +0100, Angelo Compagnucci spake thusly: > [--SNIP--] > > > > My concern here is that you start from a reproducible build, add your > > > > packages right and so maintain your build reproducible, buildroot will > > > > work as before. > > > > > > So you are, like I am, in fact arguing that we should have actual packages > > > for such external modules? ;-) > > > > I can't see any value on having hundreds of npm packages probably > > outdated at wich you should add hundreds of php composer packages, go > > modules, cargo packages and so on. > > Why not? We do have hundreds (~270, which literaly makes that number > 'hundreds') of python packages, some (a lot?) of them probably outdated. > > Worse, yet: we *do* have thousands of packages, a lot of them outdated. > > I don't see why we would single out npm packages at all. IMHO I think we should start do it differently. Even the major distributions started to not packaging everything anymore. Debian/Ubuntu have a really low number of npm packages or php ones. Owncloud homepage clearly states to not use the distributions packaged version cause it's outdated and insecure. Offering ton of outdated and insecure packages by default it's not on the best of interest of anyone. > (Besides the fact that I don't like them; I don't like perl either, but > I'm still quite happy to see perl packages.) > > > >> > As soon you use a package manager tainting will be > > > > signaled. > > > > > > This is where I disagree. > > > > > > Using such package managers does not imply that the build is tainted. > > > This is a false dichotomy. > > > > Probably it is but my experience says not, > > And in my experience, it can be. > > If you are using an internal repository of which you control the > content, then the list of nodejs modules *is* reproducible. This is > not a theoretical situation, by the way; this is a real situation. > > Your heursitic is indeed not a heuristic, but a policy. Your policy > states "any external module taints the build". My policy states "use > our internal repository of trusted and known modules, anything else > taints the build". > > So, the code currently expreses your policy, and precludes using mine, > which I believe is still superior because I do not even rely on external > data which may disapear at the whim of an external entity; everything I > need is already internal and stable. That's the point I can't understand. What precludes you to use the workflow you used until now to add a package in buildroot? Probably you don't use a package manager and after this series nothing changes for you or for all the people that uses an internal repository. If you have an internal repository with all of your dependencies mirrored you don't need to use a package manager, else you can call npm install or composer install or whatever else manually. > > anyway, if you are dealing > > with a complex php software for example you can: > > * call php composer manually on the modules you are sure being > > reproducible in your .mk and live happy > > I have no idea what php-composer is. I guess it would download and > install php files, not unlike npm does, and probably looks as ugly as > npm. That's the point. For each programming language there are at least 3/4 major package managers that do the same thing in a more or less different way. Or we cope with this, or we should state clearly in the manual that buildroot is not usable on that scenarios. > > > * use the package host-composer distributed with buildroot asking it > > to do a "composer install" living with the fact that it could do > > anything and you build will be tainted. > > > > > > Taint is mean to signal that there is a potential problem, and if you > > > > don't want to slip into it, you can always do the right thing and > > > > package your software and packaging also it's dependencies. > > > > > > And what I am saying is that the heuristic you suggest to decide whether > > > a build should be considered tainted or not is incorrect. > > > > It's not an heuristic, it's a rule: you ask to a package manager to > > resolve your dependencies, your build _could_ be tainted. You want to > > be sure: you write your own rule. > > So, because the build _could_ be tainted, you want to make all builds as > tainted? That's insane... All builds that uses a package manager that not ensure your dependency chain is clean. If it's clean, non need to taint. Taiting is an explicit property of a package after all. This doesn't look so insane to me after all ... > > > If you write a clean mk that does everything right you shouldn't add > > FOO_TAINTS, that's it. > > > > I can agree with you that some packages could be considered first > > class citizens, the others based on package managers could not be as > > good as first ones. But at least, you give the tools to developers who > > wants to add a non trivial web package or similar to buildroot. > > Well, the tainted stuff is basically orthogonal to the package manager > stuff, I think. Even if I despise the package managers, I can see that > people are so keen on using them... > What I am arguing in this thread, is that the tainted status can *not* > be deduced from the configuration. The way you are currently coding the > tainted flag would break for the people that are already doing it right, > and for whom this tainted flag would ultimately prove to be the most > valuable. Honestly I can't follow you here. If people are doing right why the heck the should be worried by the tainting? Npm package in buildroot is plain wrong right now. It should be decoupled from nodejs and made a host package. After that in your package you can do: * Call the host-npm on your reproducible modules and live happy (you current behavior is maintained) * Use the host-npm package on your non-reproducible modules and add a FOO_TAINTS to your package I have a host-glide package manager for golang that works this way and it is pure awesome! If not, what you propose? Denying package managers completely? Sincerely, Angelo > Regards, > Yann E. MORIN. > > > What I'm saying here is that we don't have such a rule we simply > > cannot add any time soon any software based on package managers. > > > > Probably there is some other smarter way to do it that I cannot see ... > > > > I'm open to suggestions! > > > > > > > > > As soon as you do this, the taint disappear. I thin it could even be a > > > > deterrent to package the software randomly! > > > > > > Regards, > > > Yann E. MORIN. > > > > > > -- > > > .-----------------.--------------------.------------------.--------------------. > > > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > > > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > > > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > > > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > > > '------------------------------^-------^------------------^--------------------' > > -- > .-----------------.--------------------.------------------.--------------------. > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot 2018-09-09 20:18 ` Angelo Compagnucci @ 2018-09-10 7:50 ` Angelo Compagnucci 2018-09-10 15:00 ` Yann E. MORIN 0 siblings, 1 reply; 37+ messages in thread From: Angelo Compagnucci @ 2018-09-10 7:50 UTC (permalink / raw) To: buildroot Yann, All, On Sun, Sep 9, 2018 at 10:18 PM Angelo Compagnucci <angelo@amarulasolutions.com> wrote: > > Yann, All, > On Sun, Sep 9, 2018 at 7:55 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > > > Angelo, All, > > > > This is the last email I'll be sending in this thread, because it looks > > like the both of us are re-hashing the same arguments again and again... > > Yet, let me reply to parts of your email... > > Last one, I promise! > > > On 2018-09-09 17:58 +0100, Angelo Compagnucci spake thusly: > > > On Sun, Sep 9, 2018 at 3:20 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > > > > > > > On 2018-09-09 14:44 +0100, Angelo Compagnucci spake thusly: > > > > > On Sun, Sep 9, 2018 at 2:33 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > > > > > On 2018-09-09 13:25 +0100, Angelo Compagnucci spake thusly: > > [--SNIP--] > > > > > My concern here is that you start from a reproducible build, add your > > > > > packages right and so maintain your build reproducible, buildroot will > > > > > work as before. > > > > > > > > So you are, like I am, in fact arguing that we should have actual packages > > > > for such external modules? ;-) > > > > > > I can't see any value on having hundreds of npm packages probably > > > outdated at wich you should add hundreds of php composer packages, go > > > modules, cargo packages and so on. > > > > Why not? We do have hundreds (~270, which literaly makes that number > > 'hundreds') of python packages, some (a lot?) of them probably outdated. > > > > Worse, yet: we *do* have thousands of packages, a lot of them outdated. > > > > I don't see why we would single out npm packages at all. > > IMHO I think we should start do it differently. Even the major > distributions started to not packaging everything anymore. > Debian/Ubuntu have a really low number of npm packages or php ones. > Owncloud homepage clearly states to not use the distributions packaged > version cause it's outdated and insecure. > > Offering ton of outdated and insecure packages by default it's not on > the best of interest of anyone. > > > (Besides the fact that I don't like them; I don't like perl either, but > > I'm still quite happy to see perl packages.) > > > > > >> > As soon you use a package manager tainting will be > > > > > signaled. > > > > > > > > This is where I disagree. > > > > > > > > Using such package managers does not imply that the build is tainted. > > > > This is a false dichotomy. > > > > > > Probably it is but my experience says not, > > > > And in my experience, it can be. > > > > If you are using an internal repository of which you control the > > content, then the list of nodejs modules *is* reproducible. This is > > not a theoretical situation, by the way; this is a real situation. > > > > Your heursitic is indeed not a heuristic, but a policy. Your policy > > states "any external module taints the build". My policy states "use > > our internal repository of trusted and known modules, anything else > > taints the build". > > > > So, the code currently expreses your policy, and precludes using mine, > > which I believe is still superior because I do not even rely on external > > data which may disapear at the whim of an external entity; everything I > > need is already internal and stable. > > That's the point I can't understand. What precludes you to use the > workflow you used until now to add a package in buildroot? > Probably you don't use a package manager and after this series nothing > changes for you or for all the people that uses an internal > repository. > If you have an internal repository with all of your dependencies > mirrored you don't need to use a package manager, else you can call > npm install or composer install or whatever else manually. > > > > anyway, if you are dealing > > > with a complex php software for example you can: > > > * call php composer manually on the modules you are sure being > > > reproducible in your .mk and live happy > > > > I have no idea what php-composer is. I guess it would download and > > install php files, not unlike npm does, and probably looks as ugly as > > npm. > > That's the point. For each programming language there are at least 3/4 > major package managers that do the same thing in a more or less > different way. Or we cope with this, or we should state clearly in the > manual that buildroot is not usable on that scenarios. > > > > > > * use the package host-composer distributed with buildroot asking it > > > to do a "composer install" living with the fact that it could do > > > anything and you build will be tainted. > > > > > > > > Taint is mean to signal that there is a potential problem, and if you > > > > > don't want to slip into it, you can always do the right thing and > > > > > package your software and packaging also it's dependencies. > > > > > > > > And what I am saying is that the heuristic you suggest to decide whether > > > > a build should be considered tainted or not is incorrect. > > > > > > It's not an heuristic, it's a rule: you ask to a package manager to > > > resolve your dependencies, your build _could_ be tainted. You want to > > > be sure: you write your own rule. > > > > So, because the build _could_ be tainted, you want to make all builds as > > tainted? That's insane... > > All builds that uses a package manager that not ensure your dependency > chain is clean. If it's clean, non need to taint. Taiting is an > explicit property of a package after all. > This doesn't look so insane to me after all ... > > > > > > If you write a clean mk that does everything right you shouldn't add > > > FOO_TAINTS, that's it. > > > > > > I can agree with you that some packages could be considered first > > > class citizens, the others based on package managers could not be as > > > good as first ones. But at least, you give the tools to developers who > > > wants to add a non trivial web package or similar to buildroot. > > > > Well, the tainted stuff is basically orthogonal to the package manager > > stuff, I think. Even if I despise the package managers, I can see that > > people are so keen on using them... > > > What I am arguing in this thread, is that the tainted status can *not* > > be deduced from the configuration. The way you are currently coding the > > tainted flag would break for the people that are already doing it right, > > and for whom this tainted flag would ultimately prove to be the most > > valuable. > > Honestly I can't follow you here. If people are doing right why the > heck the should be worried by the tainting? > > Npm package in buildroot is plain wrong right now. It should be > decoupled from nodejs and made a host package. > After that in your package you can do: > * Call the host-npm on your reproducible modules and live happy (you > current behavior is maintained) > * Use the host-npm package on your non-reproducible modules and add a > FOO_TAINTS to your package > > I have a host-glide package manager for golang that works this way and > it is pure awesome! I just pushed o POC on my local branch if you want to have a look. It's what I mean for tainting applied to a package with a more robust ad correct approach that the npm case: https://github.com/angeloc/buildroot/commits/watchtower Sincerely, Angelo > > If not, what you propose? Denying package managers completely? > > Sincerely, Angelo > > > Regards, > > Yann E. MORIN. > > > > > What I'm saying here is that we don't have such a rule we simply > > > cannot add any time soon any software based on package managers. > > > > > > Probably there is some other smarter way to do it that I cannot see ... > > > > > > I'm open to suggestions! > > > > > > > > > > > > As soon as you do this, the taint disappear. I thin it could even be a > > > > > deterrent to package the software randomly! > > > > > > > > Regards, > > > > Yann E. MORIN. > > > > > > > > -- > > > > .-----------------.--------------------.------------------.--------------------. > > > > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > > > > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > > > > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > > > > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > > > > '------------------------------^-------^------------------^--------------------' > > > > -- > > .-----------------.--------------------.------------------.--------------------. > > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > > '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot 2018-09-10 7:50 ` Angelo Compagnucci @ 2018-09-10 15:00 ` Yann E. MORIN 2018-09-10 15:37 ` Yann E. MORIN 2018-09-10 17:10 ` Angelo Compagnucci 0 siblings, 2 replies; 37+ messages in thread From: Yann E. MORIN @ 2018-09-10 15:00 UTC (permalink / raw) To: buildroot Angelo, All, On 2018-09-10 09:50 +0200, Angelo Compagnucci spake thusly: > > Last one, I promise! [--SNIP--] > I just pushed o POC on my local branch if you want to have a look. > It's what I mean for tainting applied to a package with a more robust > ad correct approach that the npm case: > https://github.com/angeloc/buildroot/commits/watchtower So, here is my last stance on the subject, in which I tried to summarise my position. Why would the "tainted" flag be set? - unknown licensing information: it is better to use the existing licensing infrastructure, which is made for this very purpose: FOO_LICENSE := $(FOO_LICENSE), Unknown (external data used) - non-reproducible packages in Buildroot: I don't think we should accept packages in Buildroot that would taint the build; and if we were, we could hide them behind !BR2_REPRODUCIBLE (with or without a comment stating "foo is not reproducible"); - packages that are in a br2-external, or in a private fork: I believe that people who do non-reproducible packages either don't care, have no choice, or both. If they did care, they would not create tainting packages; if they did care but still had no choice, they could also decide to hide them behind !BR2_REPRODUCIBLE; - packages with a list of external resources, like we have for nodejs/npm: we can't know if that list references reproducible resources or not. That last point is very important: there *are* people that do care about the reproduciblity of a build, and thus have already taken care that BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL *does* point to stable and reproducible set of nodejs modules [0]. Patch 3 in the series would mark for them that the build is tainted when it is not; since those people do care, the tainted flag would be most important to them, but by always marking their build as tainted, the flag becomes useless to the very people that do care... Yes, a lot of people will just use a non-stable list, and they even probably do not care either. I do not want to have to impose that limitation unto those who know what they are doing. So, I still conclude that we do not need to have a tainted flag. [0] For example, consider an explicit and complete list such as (spitted for readability): BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL=" http://internal-server/node/mod-1 http://internal-server/node/mod-2 http://internal-server/node/mod-3 http://internal-server/node/mod-4 " and that all the dependencies of those modules *are* present in that list, leaving npm no chance to download anything. Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot 2018-09-10 15:00 ` Yann E. MORIN @ 2018-09-10 15:37 ` Yann E. MORIN 2018-09-10 17:10 ` Angelo Compagnucci 1 sibling, 0 replies; 37+ messages in thread From: Yann E. MORIN @ 2018-09-10 15:37 UTC (permalink / raw) To: buildroot Angelo, All, On 2018-09-10 17:00 +0200, Yann E. MORIN spake thusly: > [0] For example, consider an explicit and complete list such as (spitted Woops... s/spitted/splitted/ Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot 2018-09-10 15:00 ` Yann E. MORIN 2018-09-10 15:37 ` Yann E. MORIN @ 2018-09-10 17:10 ` Angelo Compagnucci 2018-09-10 18:07 ` Yann E. MORIN 1 sibling, 1 reply; 37+ messages in thread From: Angelo Compagnucci @ 2018-09-10 17:10 UTC (permalink / raw) To: buildroot Yann, All, Il giorno lun 10 set 2018 alle ore 17:00 Yann E. MORIN <yann.morin.1998@free.fr> ha scritto: > > Angelo, All, > > On 2018-09-10 09:50 +0200, Angelo Compagnucci spake thusly: > > > Last one, I promise! > [--SNIP--] > > I just pushed o POC on my local branch if you want to have a look. > > It's what I mean for tainting applied to a package with a more robust > > ad correct approach that the npm case: > > https://github.com/angeloc/buildroot/commits/watchtower > > So, here is my last stance on the subject, in which I tried to summarise > my position. > > Why would the "tainted" flag be set? > > - unknown licensing information: it is better to use the existing > licensing infrastructure, which is made for this very purpose: > FOO_LICENSE := $(FOO_LICENSE), Unknown (external data used) > > - non-reproducible packages in Buildroot: I don't think we should > accept packages in Buildroot that would taint the build; and if we > were, we could hide them behind !BR2_REPRODUCIBLE (with or without > a comment stating "foo is not reproducible"); > > - packages that are in a br2-external, or in a private fork: I believe > that people who do non-reproducible packages either don't care, > have no choice, or both. If they did care, they would not create > tainting packages; if they did care but still had no choice, they > could also decide to hide them behind !BR2_REPRODUCIBLE; > > - packages with a list of external resources, like we have for > nodejs/npm: we can't know if that list references reproducible > resources or not. > > That last point is very important: there *are* people that do care about > the reproduciblity of a build, and thus have already taken care that > BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL *does* point to stable and > reproducible set of nodejs modules [0]. Patch 3 in the series would mark > for them that the build is tainted when it is not; since those people do > care, the tainted flag would be most important to them, but by always > marking their build as tainted, the flag becomes useless to the very > people that do care... > > Yes, a lot of people will just use a non-stable list, and they even > probably do not care either. I do not want to have to impose that > limitation unto those who know what they are doing. > > So, I still conclude that we do not need to have a tainted flag. I can agree with you, but what do you propose for handling package managers: Simply put a !BR2_REPRODUCIBLE for a package that needs dependencies resolved and cannot handle dependencies cleanly? > > [0] For example, consider an explicit and complete list such as (spitted > for readability): > BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL=" > http://internal-server/node/mod-1 > http://internal-server/node/mod-2 > http://internal-server/node/mod-3 > http://internal-server/node/mod-4 > " > and that all the dependencies of those modules *are* present in that > list, leaving npm no chance to download anything. As I said previously, hiding npm inside nodejs is wrong. What if you have a couple nodejs based software and for one you have a clear picture of dependencies but not for the other? Npm should decoupled and used each time a nodejs software needs it. This is what I did with watchtower: +################################################################################ +# +# watchtower +# +################################################################################ + +WATCHTOWER_VERSION = v0.3.0 +WATCHTOWER_SITE = $(call github,v2tec,watchtower,$(WATCHTOWER_VERSION)) +WATCHTOWER_LICENSE = Apache-2.0 +WATCHTOWER_LICENSE_FILES = LICENSE.md +WATCHTOWER_DEPENDENCIES = host-glide +WATCHTOWER_TAINTS = YES + +define WATCHTOWER_RESOLVE_DEPS +cd $(WATCHTOWER_SRC_PATH) && GOPATH="$(@D)/$(WATCHTOWER_WORKSPACE)" $(HOST_DIR)/usr/bin/glide install +endef + +WATCHTOWER_POST_CONFIGURE_HOOKS = WATCHTOWER_RESOLVE_DEPS + +$(eval $(golang-package)) This clearly mark watchtower as a tainting package, but it doesn't harm any other package that could use host-glide. > > Regards, > Yann E. MORIN. > > -- > .-----------------.--------------------.------------------.--------------------. > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > '------------------------------^-------^------------------^--------------------' > _______________________________________________ > buildroot mailing list > buildroot at busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot -- Profile: http://it.linkedin.com/in/compagnucciangelo ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot 2018-09-10 17:10 ` Angelo Compagnucci @ 2018-09-10 18:07 ` Yann E. MORIN 2018-09-10 19:17 ` Angelo Compagnucci 2018-09-10 19:37 ` Thomas Petazzoni 0 siblings, 2 replies; 37+ messages in thread From: Yann E. MORIN @ 2018-09-10 18:07 UTC (permalink / raw) To: buildroot Angelo, All, On 2018-09-10 19:10 +0200, Angelo Compagnucci spake thusly: > Il giorno lun 10 set 2018 alle ore 17:00 Yann E. MORIN > <yann.morin.1998@free.fr> ha scritto: > I can agree with you, but what do you propose for handling package > managers: Simply put a !BR2_REPRODUCIBLE for a package that needs > dependencies resolved and cannot handle dependencies cleanly? Yes, see below... > > [0] For example, consider an explicit and complete list such as (spitted > > for readability): > > BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL=" > > http://internal-server/node/mod-1 > > http://internal-server/node/mod-2 > > http://internal-server/node/mod-3 > > http://internal-server/node/mod-4 > > " > > and that all the dependencies of those modules *are* present in that > > list, leaving npm no chance to download anything. > > As I said previously, hiding npm inside nodejs is wrong. What if you > have a couple nodejs based software and for one you have a clear > picture of dependencies but not for the other? > Npm should decoupled and used each time a nodejs software needs it. npm (the utility) comes from nodejs (the source tree). So, the npm we use in Buildroot comes from host-nodejs (the Buildroot package). Thus, the package that calls npm (i.e. nodejs today) is already decorelated from the package that installs npm. And you can already use npm from a generic-package in Buildroot as it is today... There are two ways to install stuff with npm in Buildroot: - add a pacakge name (URL, name and version, or just name) in the existing BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL config option; - add a new generic-package that calls npm as part of its _INSTALL_CMDS. In either case you may or may not have reproducibility. In the former, Buildroot can not decide either way, because only a human can tell, so your patch 3 can't be applied, because it is wring. As for the latter case, the human was in charge of writing the .mk and Config.in. Whether they write TAINT=YES in the .mk, or depends on !BR2_REPRODUCIBLE in the Config.in has the same effect. As we already have BR2_REPRODUCIBLE, we don't need TAINT. Or I don't understand the problem... > This is what I did with watchtower: [--SNIP--] > +WATCHTOWER_TAINTS = YES If you are going so far as to explicitly write this line, just hide the package behing BR2_REPRODUCIBLE instead, yes. And even better yet, for packages we would include in Buildroot, provide a clean and reproducible list of packages for the dependencies for those packages. Although this would be the ideal situation, if that is not possible, then hide behind !BR2_REPRODUCIBLE. For packages that stay in a br2-external or a random fork elsewhere, I don't think we should care, as I explained previously. For a parallel: if I do a build without threads, then packages than need threads are hidden (with a comment). Similarly, if I want a reproducible build, the packages that can't be reproducible are hidden (with a comment). By the way, this has nothing to do with whether they are packages coming from a "package manager". Even "traditional" packages may want to do unclean downloads at build time; for example, some care had to be taken for asterisk, or it would download some sound stuff during the install step (although that was reproducible). > +cd $(WATCHTOWER_SRC_PATH) && GOPATH="$(@D)/$(WATCHTOWER_WORKSPACE)" $(HOST_DIR)/usr/bin/glide install What would prevent having a script like we have in utils/scancpan or utils/scanpypi to do the same processing for Go packages? Or npm modules? Yes, some people are afraid we get thousands of them. But we don't know how far we'll go. Today, we're pretty happy adding random python modules. Why would it be different for the other "package managers"? If we find that this is going "way too far", we can revisit that later. Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot 2018-09-10 18:07 ` Yann E. MORIN @ 2018-09-10 19:17 ` Angelo Compagnucci 2018-09-10 19:43 ` Yann E. MORIN 2018-09-10 19:37 ` Thomas Petazzoni 1 sibling, 1 reply; 37+ messages in thread From: Angelo Compagnucci @ 2018-09-10 19:17 UTC (permalink / raw) To: buildroot Yann, All, On Mon, Sep 10, 2018 at 7:07 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > Angelo, All, > > On 2018-09-10 19:10 +0200, Angelo Compagnucci spake thusly: > > Il giorno lun 10 set 2018 alle ore 17:00 Yann E. MORIN > > <yann.morin.1998@free.fr> ha scritto: > > I can agree with you, but what do you propose for handling package > > managers: Simply put a !BR2_REPRODUCIBLE for a package that needs > > dependencies resolved and cannot handle dependencies cleanly? > > Yes, see below... > > > > [0] For example, consider an explicit and complete list such as (spitted > > > for readability): > > > BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL=" > > > http://internal-server/node/mod-1 > > > http://internal-server/node/mod-2 > > > http://internal-server/node/mod-3 > > > http://internal-server/node/mod-4 > > > " > > > and that all the dependencies of those modules *are* present in that > > > list, leaving npm no chance to download anything. > > > > As I said previously, hiding npm inside nodejs is wrong. What if you > > have a couple nodejs based software and for one you have a clear > > picture of dependencies but not for the other? > > Npm should decoupled and used each time a nodejs software needs it. > > npm (the utility) comes from nodejs (the source tree). So, the npm we > use in Buildroot comes from host-nodejs (the Buildroot package). Thus, > the package that calls npm (i.e. nodejs today) is already decorelated > from the package that installs npm. > > And you can already use npm from a generic-package in Buildroot as it is > today... > > There are two ways to install stuff with npm in Buildroot: > > - add a pacakge name (URL, name and version, or just name) in the > existing BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL config option; > > - add a new generic-package that calls npm as part of its > _INSTALL_CMDS. > > In either case you may or may not have reproducibility. In the former, > Buildroot can not decide either way, because only a human can tell, so > your patch 3 can't be applied, because it is wring. > > As for the latter case, the human was in charge of writing the .mk and > Config.in. Whether they write TAINT=YES in the .mk, or depends on > !BR2_REPRODUCIBLE in the Config.in has the same effect. As we already > have BR2_REPRODUCIBLE, we don't need TAINT. > > Or I don't understand the problem... > > > This is what I did with watchtower: > [--SNIP--] > > +WATCHTOWER_TAINTS = YES > > If you are going so far as to explicitly write this line, just hide the > package behing BR2_REPRODUCIBLE instead, yes. Ok, I'm sold. I'm not interested on how we do it, it's way more important that we define a way to add packages based on package managers. > And even better yet, for packages we would include in Buildroot, provide > a clean and reproducible list of packages for the dependencies for those > packages. Although this would be the ideal situation, if that is not > possible, then hide behind !BR2_REPRODUCIBLE. Yes it could, only if the dependency chain is not nested and it easy to walk. That's not always true. > For packages that stay in a br2-external or a random fork elsewhere, I > don't think we should care, as I explained previously. > > For a parallel: if I do a build without threads, then packages than need > threads are hidden (with a comment). Similarly, if I want a reproducible > build, the packages that can't be reproducible are hidden (with a > comment). > > By the way, this has nothing to do with whether they are packages coming > from a "package manager". Even "traditional" packages may want to do > unclean downloads at build time; for example, some care had to be taken > for asterisk, or it would download some sound stuff during the install > step (although that was reproducible). > > > +cd $(WATCHTOWER_SRC_PATH) && GOPATH="$(@D)/$(WATCHTOWER_WORKSPACE)" $(HOST_DIR)/usr/bin/glide install > > What would prevent having a script like we have in utils/scancpan or > utils/scanpypi to do the same processing for Go packages? Or npm > modules? This is not possible imho, at least fo golang packages. Golang packages is basically git repositories, we could end up packaging a good part of github! > Yes, some people are afraid we get thousands of them. But we don't know > how far we'll go. Today, we're pretty happy adding random python modules. > Why would it be different for the other "package managers"? Well, the main difference is that python packages is self contained, npm packages for example not, they can depend on other modules in a very intricate way. You should write something that recursively walks all the dependency tree and outputs a package for everything. Probably we should only embrace the shift paradigm of having packages that downloads their own dependencies. There is also another reason why python packages are added frequently: we have no way to call a host-pip to resolve dependencies. I'm sure that there would be way less python packages if we had a host-pip package. Sincerely, Angelo > If we find that this is going "way too far", we can revisit that later. > > Regards, > Yann E. MORIN. > > -- > .-----------------.--------------------.------------------.--------------------. > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot 2018-09-10 19:17 ` Angelo Compagnucci @ 2018-09-10 19:43 ` Yann E. MORIN 2018-09-10 20:03 ` Angelo Compagnucci 0 siblings, 1 reply; 37+ messages in thread From: Yann E. MORIN @ 2018-09-10 19:43 UTC (permalink / raw) To: buildroot Angelo, All, On 2018-09-10 20:17 +0100, Angelo Compagnucci spake thusly: > On Mon, Sep 10, 2018 at 7:07 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: [--SNIP--] > > > +WATCHTOWER_TAINTS = YES > > If you are going so far as to explicitly write this line, just hide the > > package behing BR2_REPRODUCIBLE instead, yes. > Ok, I'm sold. OK, thanks. Should someone mark this series as rejected in patchwork, then? > > > +cd $(WATCHTOWER_SRC_PATH) && GOPATH="$(@D)/$(WATCHTOWER_WORKSPACE)" $(HOST_DIR)/usr/bin/glide install > > What would prevent having a script like we have in utils/scancpan or > > utils/scanpypi to do the same processing for Go packages? Or npm > > modules? > > This is not possible imho, at least fo golang packages. Golang > packages is basically git repositories, we could end up packaging a > good part of github! > > > Yes, some people are afraid we get thousands of them. But we don't know > > how far we'll go. Today, we're pretty happy adding random python modules. > > Why would it be different for the other "package managers"? > > Well, the main difference is that python packages is self contained, Hmm. Not so much: python packages do have dependencies onto other python packages. > npm packages for example not, they can depend on other modules in a > very intricate way. > You should write something that recursively walks all the dependency > tree and outputs a package for everything. Probably we should only > embrace the shift paradigm of having packages that downloads their own > dependencies. Now we're going somewhere, I believe. Would you expect that we could write something like (roughly): $ cat package/foo/Config.in config BR2_PACKAGE_FOO bool "foo" depends on !BR2_REPRODUCIBLE select SOME_PACKAGE $ cat package/foo/foo.mk FOO_VERSION = 1.2.3 FOO_DEPENDENCIES = some dependencies on other BR packages $(eval $(npm-package)) and that the npm-package infra would do the call and install whatever foo requires? Optionally, if the dependencies are already installed (e.g. because of a FOO_DEPENDENCIES or another previous npm-package already installed), then some dependencies may or may not be downloaded by npm. That would be fine by me. There is one gotcha though: this won't work for purely off-line builds. This can be a big limitation in some setups, where access to an external network is not always guaranteed, so we should carefully say so in the manual. > There is also another reason why python packages are added frequently: > we have no way to call a host-pip to resolve dependencies. I'm sure > that there would be way less python packages if we had a host-pip > package. Well, nothing prevents one from proposing a pip-package infra, like the npm-package I hinted at above. ;-) I may even offer some review of those infras. Not that I would be interested in using those infras, mind you... ;-] Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot 2018-09-10 19:43 ` Yann E. MORIN @ 2018-09-10 20:03 ` Angelo Compagnucci 2018-09-10 20:26 ` Yann E. MORIN 0 siblings, 1 reply; 37+ messages in thread From: Angelo Compagnucci @ 2018-09-10 20:03 UTC (permalink / raw) To: buildroot Yann, All, On Mon, Sep 10, 2018 at 8:44 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > Angelo, All, > > On 2018-09-10 20:17 +0100, Angelo Compagnucci spake thusly: > > On Mon, Sep 10, 2018 at 7:07 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > [--SNIP--] > > > > +WATCHTOWER_TAINTS = YES > > > If you are going so far as to explicitly write this line, just hide the > > > package behing BR2_REPRODUCIBLE instead, yes. > > Ok, I'm sold. > > OK, thanks. > > Should someone mark this series as rejected in patchwork, then? > > > > > +cd $(WATCHTOWER_SRC_PATH) && GOPATH="$(@D)/$(WATCHTOWER_WORKSPACE)" $(HOST_DIR)/usr/bin/glide install > > > What would prevent having a script like we have in utils/scancpan or > > > utils/scanpypi to do the same processing for Go packages? Or npm > > > modules? > > > > This is not possible imho, at least fo golang packages. Golang > > packages is basically git repositories, we could end up packaging a > > good part of github! > > > > > Yes, some people are afraid we get thousands of them. But we don't know > > > how far we'll go. Today, we're pretty happy adding random python modules. > > > Why would it be different for the other "package managers"? > > > > Well, the main difference is that python packages is self contained, > > Hmm. Not so much: python packages do have dependencies onto other python > packages. > > > npm packages for example not, they can depend on other modules in a > > very intricate way. > > You should write something that recursively walks all the dependency > > tree and outputs a package for everything. Probably we should only > > embrace the shift paradigm of having packages that downloads their own > > dependencies. > > Now we're going somewhere, I believe. > > Would you expect that we could write something like (roughly): > > $ cat package/foo/Config.in > config BR2_PACKAGE_FOO > bool "foo" > depends on !BR2_REPRODUCIBLE > select SOME_PACKAGE > > $ cat package/foo/foo.mk > FOO_VERSION = 1.2.3 > FOO_DEPENDENCIES = some dependencies on other BR packages > $(eval $(npm-package)) > > and that the npm-package infra would do the call and install whatever > foo requires? Optionally, if the dependencies are already installed > (e.g. because of a FOO_DEPENDENCIES or another previous npm-package > already installed), then some dependencies may or may not be downloaded > by npm. I'm not into this honestly. What I want to achieve is that a package can call a npm install or a glide install on whatever dependencies file the source code distributes. Without the need to open that file, understanding the dependency chain and reflect that in buildroot. I think that the original developer knows better the dependencies of his software and we should trust the building infrastructure he choose, even more if what he choose is the best practice for that programming language. Exactly As I did for the example I posted before. > That would be fine by me. > > There is one gotcha though: this won't work for purely off-line builds. > This can be a big limitation in some setups, where access to an external > network is not always guaranteed, so we should carefully say so in the > manual. Not true. A package can call the package manager in it's post download hook, so a make source should do an offline build without troubles. > > There is also another reason why python packages are added frequently: > > we have no way to call a host-pip to resolve dependencies. I'm sure > > that there would be way less python packages if we had a host-pip > > package. > > Well, nothing prevents one from proposing a pip-package infra, like the > npm-package I hinted at above. ;-) I may even offer some review of those > infras. Yes, if we can agree on a way to add package managers to buildroot! > Not that I would be interested in using those infras, mind you... ;-] Great! > > Regards, > Yann E. MORIN. > > -- > .-----------------.--------------------.------------------.--------------------. > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot 2018-09-10 20:03 ` Angelo Compagnucci @ 2018-09-10 20:26 ` Yann E. MORIN 2018-09-11 6:20 ` Angelo Compagnucci 0 siblings, 1 reply; 37+ messages in thread From: Yann E. MORIN @ 2018-09-10 20:26 UTC (permalink / raw) To: buildroot Angelo, All, On 2018-09-10 21:03 +0100, Angelo Compagnucci spake thusly: > On Mon, Sep 10, 2018 at 8:44 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > $ cat package/foo/Config.in > > config BR2_PACKAGE_FOO > > bool "foo" > > depends on !BR2_REPRODUCIBLE > > select SOME_PACKAGE > > > > $ cat package/foo/foo.mk > > FOO_VERSION = 1.2.3 > > FOO_DEPENDENCIES = some dependencies on other BR packages > > $(eval $(npm-package)) > > > > and that the npm-package infra would do the call and install whatever > > foo requires? Optionally, if the dependencies are already installed > > (e.g. because of a FOO_DEPENDENCIES or another previous npm-package > > already installed), then some dependencies may or may not be downloaded > > by npm. > > I'm not into this honestly. What I want to achieve is that a package > can call a npm install or a glide install on whatever dependencies > file the source code distributes. Without the need to open that file, > understanding the dependency chain and reflect that in buildroot. > > I think that the original developer knows better the dependencies of > his software and we should trust the building infrastructure he > choose, even more if what he choose is the best practice for that > programming language. > > Exactly As I did for the example I posted before. And this is exactly what the npm-package (or glide-package, or whatever) would end up doing: expand to FOO_BUILD_CMDS, FOO_INSTALL_CMDS et al. that do the job, instead of repeating it in every package, like the autotools-package infra allows to not repeat the .configure; make; make install dance in all the autotools-based package. For example, your watchtower would be writen as: WATCHTOWER_VERSION = v0.3.0 WATCHTOWER_LICENSE = Apache-2.0 WATCHTOWER_LICENSE_FILES = LICENSE.md $(eval $(glide-package)) And then the glide-pacakge infra would be defined sdmething like: define glide-pacakge-inner $(2)_DEPENDENCIES += host-glide define $(2)_GLIDE cd $$($(2)_SRC_PATH); GOPATH=$$($(@)_GOPATH) $$(HOST_DIR)/usr/bin/glide install endef $(2)_POST_CONFIIGURE_HOOKS += $(2)_GLIDE $(call golang-package,$(1),$(2),$(3),$(4)) endef Or something similar, like we do for the rebar-package infra, for example, which piggy-backs onto either the generic-package or the autotools-package infra as needed. > > That would be fine by me. > > > > There is one gotcha though: this won't work for purely off-line builds. > > This can be a big limitation in some setups, where access to an external > > network is not always guaranteed, so we should carefully say so in the > > manual. > > Not true. A package can call the package manager in it's post download > hook, so a make source should do an offline build without troubles. But then said package manager would do only the download in the download step? I.e. I want to be able to do on 'somemachine' which has acces to both the internal and external networks: $ export BR2_DL_DIR=/some/location $ make meh_defconfig $ make source $ scp -r /some/location someone at someserver:/somewhere And on 'someserver' which does not have access to the network at all (by a firewall that blocks out-going connections): $ export BR2_DL_DIR=/somewhere $ make meh_defconfig $ make This is what I call an off-line build. > Yes, if we can agree on a way to add package managers to buildroot! I don't see any technical limitation to that, given the example you provided. Of course, I guess it is just a trivial example, and that real life is little bit more tricky as usual, but there is nothing (besides my own stubbornness, that is) standing in the road, is there? Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot 2018-09-10 20:26 ` Yann E. MORIN @ 2018-09-11 6:20 ` Angelo Compagnucci 0 siblings, 0 replies; 37+ messages in thread From: Angelo Compagnucci @ 2018-09-11 6:20 UTC (permalink / raw) To: buildroot Yann, All, On Mon, Sep 10, 2018 at 10:26 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > Angelo, All, > > On 2018-09-10 21:03 +0100, Angelo Compagnucci spake thusly: > > On Mon, Sep 10, 2018 at 8:44 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > > $ cat package/foo/Config.in > > > config BR2_PACKAGE_FOO > > > bool "foo" > > > depends on !BR2_REPRODUCIBLE > > > select SOME_PACKAGE > > > > > > $ cat package/foo/foo.mk > > > FOO_VERSION = 1.2.3 > > > FOO_DEPENDENCIES = some dependencies on other BR packages > > > $(eval $(npm-package)) > > > > > > and that the npm-package infra would do the call and install whatever > > > foo requires? Optionally, if the dependencies are already installed > > > (e.g. because of a FOO_DEPENDENCIES or another previous npm-package > > > already installed), then some dependencies may or may not be downloaded > > > by npm. > > > > I'm not into this honestly. What I want to achieve is that a package > > can call a npm install or a glide install on whatever dependencies > > file the source code distributes. Without the need to open that file, > > understanding the dependency chain and reflect that in buildroot. > > > > I think that the original developer knows better the dependencies of > > his software and we should trust the building infrastructure he > > choose, even more if what he choose is the best practice for that > > programming language. > > > > Exactly As I did for the example I posted before. > > And this is exactly what the npm-package (or glide-package, or whatever) > would end up doing: expand to FOO_BUILD_CMDS, FOO_INSTALL_CMDS et al. > that do the job, instead of repeating it in every package, like the > autotools-package infra allows to not repeat the .configure; make; make > install dance in all the autotools-based package. > > For example, your watchtower would be writen as: > > WATCHTOWER_VERSION = v0.3.0 > WATCHTOWER_LICENSE = Apache-2.0 > WATCHTOWER_LICENSE_FILES = LICENSE.md > $(eval $(glide-package)) > > And then the glide-pacakge infra would be defined sdmething like: > > define glide-pacakge-inner > > $(2)_DEPENDENCIES += host-glide > > define $(2)_GLIDE > cd $$($(2)_SRC_PATH); GOPATH=$$($(@)_GOPATH) $$(HOST_DIR)/usr/bin/glide install > endef > $(2)_POST_CONFIIGURE_HOOKS += $(2)_GLIDE > > $(call golang-package,$(1),$(2),$(3),$(4)) > > endef > > Or something similar, like we do for the rebar-package infra, for > example, which piggy-backs onto either the generic-package or the > autotools-package infra as needed. I don't say it could not be done, and indeed it's a pretty clever solution. But for golang there are at least a couples of different packages managers I'm aware of, so it could end up having too many infras. > > > > That would be fine by me. > > > > > > There is one gotcha though: this won't work for purely off-line builds. > > > This can be a big limitation in some setups, where access to an external > > > network is not always guaranteed, so we should carefully say so in the > > > manual. > > > > Not true. A package can call the package manager in it's post download > > hook, so a make source should do an offline build without troubles. > > But then said package manager would do only the download in the download > step? > > I.e. I want to be able to do on 'somemachine' which has acces to both > the internal and external networks: > > $ export BR2_DL_DIR=/some/location > $ make meh_defconfig > $ make source > $ scp -r /some/location someone at someserver:/somewhere > > And on 'someserver' which does not have access to the network at all (by > a firewall that blocks out-going connections): > > $ export BR2_DL_DIR=/somewhere > $ make meh_defconfig > $ make > > This is what I call an off-line build. It could be tricky but yes, every package manager has a caching mechanism, so with a bit of hackery we can end up having the download step downloading also the dependencies and then you can cut your connection. > > > Yes, if we can agree on a way to add package managers to buildroot! > > I don't see any technical limitation to that, given the example you > provided. The real thing here is, we have a policy right now stating that package managers are not admitted. I asked this explicitly to Peter and Thomas at the Buidroot developers meeting. We have to find the bast way to have them included imho. > > Of course, I guess it is just a trivial example, and that real life is > little bit more tricky as usual, but there is nothing (besides my own > stubbornness, that is) standing in the road, is there? Not true, most of the web software I worked with needs only an npm install or glide install and nothing more. > > Regards, > Yann E. MORIN. > > -- > .-----------------.--------------------.------------------.--------------------. > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot 2018-09-10 18:07 ` Yann E. MORIN 2018-09-10 19:17 ` Angelo Compagnucci @ 2018-09-10 19:37 ` Thomas Petazzoni 2018-09-10 19:55 ` Angelo Compagnucci 2018-09-10 20:37 ` Yann E. MORIN 1 sibling, 2 replies; 37+ messages in thread From: Thomas Petazzoni @ 2018-09-10 19:37 UTC (permalink / raw) To: buildroot Hello, On Mon, 10 Sep 2018 20:07:05 +0200, Yann E. MORIN wrote: > If you are going so far as to explicitly write this line, just hide the > package behing BR2_REPRODUCIBLE instead, yes. I think BR2_REPRODUCIBLE is not the same thing. BR2_REPRODUCIBLE means "assuming we have the same source code and configuration, we guarantee that the binary results will be bit-to-bit identical". While the reproducibility issue we are talking about in this thread is about having the same source code each time the build is done. It's obviously a requirement for BR2_REPRODUCIBLE to work, but the fact that we build with the same source code each time the build is done is a property of Buildroot regardless of whether BR2_REPRODUCIBLE is enabled or not. So I'm sorry, but this suggestion to hide a package behind BR2_REPRODUCIBLE because the *source* is not guaranteed to be the same each time is not a good suggestion IMO, as it's confusing two different "reproducible" properties. Best regards, Thomas -- Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot 2018-09-10 19:37 ` Thomas Petazzoni @ 2018-09-10 19:55 ` Angelo Compagnucci 2018-09-10 20:37 ` Yann E. MORIN 1 sibling, 0 replies; 37+ messages in thread From: Angelo Compagnucci @ 2018-09-10 19:55 UTC (permalink / raw) To: buildroot Thomas, All, On Mon, Sep 10, 2018 at 8:38 PM Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote: > > Hello, > > On Mon, 10 Sep 2018 20:07:05 +0200, Yann E. MORIN wrote: > > > If you are going so far as to explicitly write this line, just hide the > > package behing BR2_REPRODUCIBLE instead, yes. > > I think BR2_REPRODUCIBLE is not the same thing. BR2_REPRODUCIBLE means > "assuming we have the same source code and configuration, we guarantee > that the binary results will be bit-to-bit identical". > > While the reproducibility issue we are talking about in this thread is > about having the same source code each time the build is done. It's > obviously a requirement for BR2_REPRODUCIBLE to work, but the fact that > we build with the same source code each time the build is done is a > property of Buildroot regardless of whether BR2_REPRODUCIBLE is enabled > or not. > > So I'm sorry, but this suggestion to hide a package behind > BR2_REPRODUCIBLE because the *source* is not guaranteed to be the same > each time is not a good suggestion IMO, as it's confusing two > different "reproducible" properties. I agree with you and indeed this patch series born after our discussion at the Buildroot developers meeting. I think we should find a common ground on this ... > > Best regards, > > Thomas > -- > Thomas Petazzoni, CTO, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot 2018-09-10 19:37 ` Thomas Petazzoni 2018-09-10 19:55 ` Angelo Compagnucci @ 2018-09-10 20:37 ` Yann E. MORIN 1 sibling, 0 replies; 37+ messages in thread From: Yann E. MORIN @ 2018-09-10 20:37 UTC (permalink / raw) To: buildroot Thomas, All, On 2018-09-10 21:37 +0200, Thomas Petazzoni spake thusly: > On Mon, 10 Sep 2018 20:07:05 +0200, Yann E. MORIN wrote: > > If you are going so far as to explicitly write this line, just hide the > > package behing BR2_REPRODUCIBLE instead, yes. > I think BR2_REPRODUCIBLE is not the same thing. BR2_REPRODUCIBLE means > "assuming we have the same source code and configuration, we guarantee > that the binary results will be bit-to-bit identical". > > While the reproducibility issue we are talking about in this thread is > about having the same source code each time the build is done. It's > obviously a requirement for BR2_REPRODUCIBLE to work, but the fact that > we build with the same source code each time the build is done is a > property of Buildroot regardless of whether BR2_REPRODUCIBLE is enabled > or not. > > So I'm sorry, but this suggestion to hide a package behind > BR2_REPRODUCIBLE because the *source* is not guaranteed to be the same > each time is not a good suggestion IMO, as it's confusing two > different "reproducible" properties. But if you select BR2_REPRODUCIBLE, do you accept that a package makes it so that the build is actually not reproducible? That is completely contradictory... But I'm really done this time on this topic, now. I'm pretty sure I clearly explained why a tainted flag is a bad idea... Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot 2018-09-09 12:10 ` Thomas Petazzoni 2018-09-09 12:25 ` Angelo Compagnucci @ 2018-09-09 13:27 ` Yann E. MORIN 1 sibling, 0 replies; 37+ messages in thread From: Yann E. MORIN @ 2018-09-09 13:27 UTC (permalink / raw) To: buildroot Thomas, Angelo, All, On 2018-09-09 14:10 +0200, Thomas Petazzoni spake thusly: > On Sun, 9 Sep 2018 09:36:16 +0200, Yann E. MORIN wrote: > > So, I am not in favour of having support for such package managers in > > Buildroot, to begin with. > > We already had this discussion in the first iteration of this patch > series, and even before. I think everybody agrees that such package > managers (npm and al.) are really not nice in the context of Linux > distributions or build systems. > > However, from a practical point of view, some people want those > features, and are ready to trade the reproducibility against "ease of > use". In addition, specifically for nodejs/npm, the number of packages > is so enormous that I am not sure it is really realistic to create one > Buildroot package for each nodejs module, and keep things sane. Well, we already have ~269 python packages, and that is already a lot. There are 151,504 (as displayed today on pypi.org) python packages. Yet, we have an infrastructure for them. > Especially if only a few people are interested in nodejs modules. > > > As you say, the reproducibility of a build, as offered by Buildroot, is > > really a very important point. In my opinion, reproducibility is even > > paramount. Whatever gets in the way should be banned. > > I disagree on the "should be banned", and that's the whole point of > this series: allow it, while making sure that the user is well aware of > the fact that he has given up on the "reproducibility" by using one of > those package managers. The taint status is not something that we can correctly come up with: we have no way to know whether a specific set of external data, packages, dependencies, stuff, whatever, will or won't taint the build, as outlined by the examples I provided in the nodejs/npm case. So, because we can not come up with a conclusive status of whether a build is or is not tainted, we should not have infra for reporting it. > > Now, what would be the purpose for this "tainted" flag? I understand > > clearly what it provides, indeed, technically, but what would it serve > > to the user? > > See above: it makes it absolutely clear to the user that he is using > some feature that kills one key advantage of build system: > reproducibility. And what I am saying is that we can't provide that status, because we don't know whether a build is tainted or is not. And if we report a build is tainted when it is not, then it becomes useless, as a real taint that is later introduced will go unnoticed. [--SNIP--] > > Instead, I am more in favour of packaging such external stuff as > > Buildroot packages, like we've been doing for erlang, lua, perl, and > > python, even if we have to add new infrastructures for thos (npm, I'm > > looking at you!) > > Yes, that is the ideal situation. But that is not necessarily realistic > for npm modules (due to their number), and having this option of using > the provided package manager is reasonable, as long as the user is > aware of the drawbacks. > > > Besides, you're missing a big piece of potential non-reproducibility > > source: br2-external trees. If we ever were to have such a tainting > > mechanism, we should mark the build as tainted as soon as there is a > > br2-external tree. > > I disagree, a BR2_EXTERNAL does not make the build non-reproducible. I said "potential non-reproducibility source". Because it is in a br2-external, the user is free to do so many hacks that we can't assess whether the build is reproducible or not. And again, the important piece of information is that: we _can_ _not_ assess the tainted status, either way. So, I am arguing that we should not have a way to report the tainted status to the userm because we don't know how to decide whther a build is tainted or not. > As > long as you have the same BR2_EXTERNAL + Buildroot, you can rebuild the > same system. Yeah, I know it very well, thank you! ;-) But how do you know that the combination is reproducible or is not? > > Ditto as soon as we have a FOO_OVERRIDE_SRCDIR set, which should mark > > the build as tainted. Except we already use FOO_OVERRIDE_SRCDIR when > > FOO_SITE = local... > > FOO_OVERRIDE_SRCDIR is indeed a better example of thing that makes the > build non-reproducible, we could potentially include this in the > tainted mechanism. But remember that the OVERRIDE_SRCDIR variable is also used when we use _SITE = local too, so if a package has its sources bundled besides the .mk (for example, al ot of packagers do that for very trivial packages), you would set that the build is tainted when it is not. > Though to me, this is somewhat less important: only advanced users are > likely to use <pkg>_OVERRIDE_SRCDIR, and if they do, they will > definitely realize the impact on reproducibility. I've seen people shipping into production, builds that were made locally with locally modified sources, and/or in uncomitted and/or un-pushed VCS trees. So, even "advanced users" do not always "realize" what they are doing. ;-) > While the newcomer > toying around with nodejs/npm may not realize the impact of using a > third-party package manager on reproducibility. > > So, yes, perhaps <pkg>_OVERRIDE_SRCDIR should ultimately taint the > build, but I don't see handling that as a requirement to start > introducing this "taint" flag. > > > Instead, why not provide a helper script (or enhance the existing ones > > if need be), than generated buildroot packages for those external packge > > managers? > > We really don't want thousands of npm packages I believe :-/ Yet we're ready to have thousands of python packages... Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot 2018-09-05 22:22 [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot Angelo Compagnucci ` (4 preceding siblings ...) 2018-09-09 7:36 ` Yann E. MORIN @ 2018-11-01 12:14 ` Arnout Vandecappelle 2018-11-01 12:25 ` Yann E. MORIN 5 siblings, 1 reply; 37+ messages in thread From: Arnout Vandecappelle @ 2018-11-01 12:14 UTC (permalink / raw) To: buildroot Hi Angelo, On 06/09/18 00:22, Angelo Compagnucci wrote: > Packages that need to resolve dependencies internally > and use a package manager would harm the reproducibility > of a build, moreover they escape the legal infrastructure > not giving enough informations on licensing. > > This patch adds a tainting mechanism in the form of a > variable FOO_TAINTS that can be used to signal that > a package harms the reproducibility or licensing under > certain conditions. We had a discussion about this at the BR developer meeting, and we decided that the taints mechanism is not worth it. As noted by Yann, it can only ever be used for a warning, not to block anything, because there are ways to do the right thing for e.g. BR2_REPRODUCIBLE. And if it is just a warning, then it can just be included in the help text of the Config.in option. Or, if we want to make it stronger, a conditional comment. > This opens the door to include per language dependency > managers in buildroot. To be honest (not discussed, so purely my personal opinion), maybe we should just relax our opposition against language package managers. Yes, there are those 7 points (which I still haven't added to the documentation, shame on me) but that doesn't mean it's entirely blocking. I hope I haven't opened a box of Pandora by saying this :-) Regards, Arnout ^ permalink raw reply [flat|nested] 37+ messages in thread
* [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot 2018-11-01 12:14 ` Arnout Vandecappelle @ 2018-11-01 12:25 ` Yann E. MORIN 0 siblings, 0 replies; 37+ messages in thread From: Yann E. MORIN @ 2018-11-01 12:25 UTC (permalink / raw) To: buildroot Arnout, Angelo, On 2018-11-01 13:14 +0100, Arnout Vandecappelle spake thusly: > On 06/09/18 00:22, Angelo Compagnucci wrote: > > Packages that need to resolve dependencies internally > > and use a package manager would harm the reproducibility > > of a build, moreover they escape the legal infrastructure > > not giving enough informations on licensing. > > > > This patch adds a tainting mechanism in the form of a > > variable FOO_TAINTS that can be used to signal that > > a package harms the reproducibility or licensing under > > certain conditions. > > We had a discussion about this at the BR developer meeting, and we decided that > the taints mechanism is not worth it. As noted by Yann, it can only ever be used > for a warning, not to block anything, because there are ways to do the right > thing for e.g. BR2_REPRODUCIBLE. > > And if it is just a warning, then it can just be included in the help text of > the Config.in option. Or, if we want to make it stronger, a conditional comment. > > > This opens the door to include per language dependency > > managers in buildroot. > > To be honest (not discussed, so purely my personal opinion), maybe we should > just relax our opposition against language package managers. Yes, there are > those 7 points (which I still haven't added to the documentation, shame on me) > but that doesn't mean it's entirely blocking. I thought we did discuss it, at least informally, and came to the conclusion that we should not block them, indeed. > I hope I haven't opened a box of Pandora by saying this :-) It was already opened! ;-) Regards, Yann E. MORIN. > Regards, > Arnout > _______________________________________________ > buildroot mailing list > buildroot at busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 37+ messages in thread
end of thread, other threads:[~2018-11-01 12:25 UTC | newest] Thread overview: 37+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-09-05 22:22 [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot Angelo Compagnucci 2018-09-05 22:22 ` [Buildroot] [PATCH v5 1/3] Makefile: add tainting support Angelo Compagnucci 2018-09-06 7:44 ` Thomas Petazzoni 2018-09-06 7:46 ` Angelo Compagnucci 2018-09-05 22:22 ` [Buildroot] [PATCH v5 2/3] docs/manual: adding infos about tainting Angelo Compagnucci 2018-09-09 8:00 ` Yann E. MORIN 2018-09-05 22:22 ` [Buildroot] [PATCH v5 3/3] package/nodejs: taint the build on external modules Angelo Compagnucci 2018-09-09 7:49 ` Yann E. MORIN 2018-09-09 12:17 ` Angelo Compagnucci 2018-09-09 13:01 ` Yann E. MORIN 2018-09-09 13:29 ` Angelo Compagnucci 2018-09-06 7:42 ` [Buildroot] [PATCH v5 0/3] Add tainting support to buildroot Thomas Petazzoni 2018-09-09 7:36 ` Yann E. MORIN 2018-09-09 12:10 ` Thomas Petazzoni 2018-09-09 12:25 ` Angelo Compagnucci 2018-09-09 13:33 ` Yann E. MORIN 2018-09-09 13:44 ` Angelo Compagnucci 2018-09-09 14:20 ` Yann E. MORIN 2018-09-09 16:58 ` Angelo Compagnucci 2018-09-09 18:55 ` Yann E. MORIN 2018-09-09 20:18 ` Angelo Compagnucci 2018-09-10 7:50 ` Angelo Compagnucci 2018-09-10 15:00 ` Yann E. MORIN 2018-09-10 15:37 ` Yann E. MORIN 2018-09-10 17:10 ` Angelo Compagnucci 2018-09-10 18:07 ` Yann E. MORIN 2018-09-10 19:17 ` Angelo Compagnucci 2018-09-10 19:43 ` Yann E. MORIN 2018-09-10 20:03 ` Angelo Compagnucci 2018-09-10 20:26 ` Yann E. MORIN 2018-09-11 6:20 ` Angelo Compagnucci 2018-09-10 19:37 ` Thomas Petazzoni 2018-09-10 19:55 ` Angelo Compagnucci 2018-09-10 20:37 ` Yann E. MORIN 2018-09-09 13:27 ` Yann E. MORIN 2018-11-01 12:14 ` Arnout Vandecappelle 2018-11-01 12:25 ` Yann E. MORIN
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.