From mboxrd@z Thu Jan 1 00:00:00 1970 From: Yann E. MORIN Date: Wed, 8 Apr 2020 21:27:53 +0200 Subject: [Buildroot] [PATCH] infra: add the transient download mechanism In-Reply-To: <20200408083903.547003f2@windsurf.home> References: <20200115203753.12110-1-yann.morin.1998@free.fr> <20200408083903.547003f2@windsurf.home> Message-ID: <20200408192753.GC29898@scaer> List-Id: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: buildroot@busybox.net Thomas, All, On 2020-04-08 08:39 +0200, Thomas Petazzoni spake thusly: > On Wed, 15 Jan 2020 21:37:53 +0100 > "Yann E. MORIN" wrote: > > This patch is a proof of concept for the transient download mechanism, > > that would help using branches as versions, and keep using the head of > > the branch each time a build started. > > So what is the status of this patch? I see Yann you marked it as to be > ignored by patchwork, so we don't have it in patchwork. > > I think there's generally been some positive feedback, with Arnout > saying: > > "" > Bottom line: even though repo is the better tool to manage the trees both for > development and release, the transient mechanism is a good quick win. > "" I will address all the questions and comments in the review, and will respin it for proper consideration. [--SNIP--] > > This stil has the drawback that two builds in parallel may get slightly > > different content for the same branch, and the first build could end up > > using the download of the second build: > > > > build-1 build-2 > > > > download > > | download > > | | > > save to dl-dir | > > [...] save to dl-dir > > extract > > That is not a drawback, that is exactly what you expect with this > feature. Sorry, the two builds are supposed to be running in parallel... For build 1, it would get the download from build 2, which is not what build 1 expects. But drawback is not the correct word. This is a pitfall. > > Furthermore, even with a single build, it might get what it expects: > > > > developer-1 developer-2 > > > > git push branch > > trigger CI git push branch > > [...] > > download > > > > In that case the build of delopper-1 would get the code of developper-2 > > who pushed on the same branch. > > Ditto here: this is exactly what we expect for a transient download > feature. If developer-1 wants to test only his own feature, then he has > to trigger CI on a branch that only he has control over. Yes, this is not a drawback, but a pitfall. > There is however one drawback: > > * You run a build (1), which downloads package A from a Git repo, and > places the tarball in global DL_DIR. > > * You run a different build (2), which downloads the same package A > from the Git repo. Due to the transient download mechanism being > enabled for package A, it re-downloads it, and places an updated > tarball in the global DL_DIR. > > * Back to build (1), you run "make legal-info", which picks up the > updated tarball, which was generated by build (2). This is not a drawback; this is another pitfall. > So, in build (1), your image uses the "previous" version of package A, > while your legal-info contains the "new" version of package A. > > To solve this, one solution would be to use a per-build download > directory for the transient packages, rather than the global DL_DIR. Meh... Or rather, make legal-info impossible for a package that is transient, like when it is 'local'. This is the best solution. > But I think this could be a separate improvement. Indeed, the transient > download mechanism is clearly oriented towards development/CI workflow, > where consistency of legal-info may not be a primary concern. For a > final release build, you would not use TRANSIENT_DOWNLOAD = YES. > > For example, custom packages that make use of TRANSIENT_DOWNLOAD could > be implemented like this: > > ifeq ($(BR2_DEVELOPER_MODE),y) > _TRANSIENT_DOWNLOAD = YES > _VERSION = some-branch > else > _VERSION = some-tag > endif The more I think about it, the less proud I am to have come up with this transient idea. Using unstable references is complete madness to begin with and is so flawed by design... We should not even provide such a feature. But we can't stop people from being... people... :-[ > So, while in development mode, users and CI use BR2_DEVELOPER_MODE=y, > but for final releases, BR2_DEVELOPER_MODE is disabled, which ensures > we have a fully reproducible build, and consistent legal-info output. > > Yann, do you think you could resubmit your patch, for actual merging ? Yes I will. > Of course, we don't need the busybox.mk change. We don't? How comne" ;-) > Best regards, > > Thomas > -- > Thomas Petazzoni, CTO, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'