[Buildroot] Allowing to build from Git branches

[Buildroot] Allowing to build from Git branches

[Thomas Petazzoni]

Hello,

I know some people are going to yell at me for bringing up this topic,
but let's do it anyway.

Currently, Buildroot says that using a Git branch as the VERSION for a
package is not supported. There are some patches that have been
proposed to explicitly prevent using branches, but in the current
Buildroot it "works" except that a cached tarball is preserved in
DL_DIR, which means that the branch is never fetched again unless the
tarball is manually deleted from DL_DIR. We have generally resisted
supporting Git branches with the (very valid) argument that it breaks
reproducibility.

However, over time, I have seen a number of users/companies complaining
about this, both on the mailing list and privately with customers or
while discussing with users at conferences.

The typical use case that they describe is the following: a company has
a large number of packages in Buildroot to build in-house software
components. Both for their CI runs, and for the developers actively
working on the system, they would like for those packages to point to a
branch, and always fetch the latest version. Yes, it is not
reproducible, but reproducibility is not what is desired at this point
of the project: what is wanted is precisely to fetch the latest and
greatest for those packages that point to a branch, to make sure the
most recent code gets tested and used by developers internally.

The only solution we offer for this today is <pkg>_OVERRIDE_SRCDIR or
<pkg>_SITE_METHOD = local, but neither of those are very practical: you
need to have fetched each of your software components outside of
Buildroot, and point Buildroot at them. It's a bit silly when Buildroot
is already perfectly capable of doing this.

Overall, I think it makes sense to realize that Buildroot is used in
different use-cases:

 - During active development, where we want to offer a lot of
   flexibility, possibly at the expense of reproducibility.

 - During release/generation of the final firmware that will be used on
   actual devices, where we want an absolute reproducibility, to the
   point where we're working on binary-identical results.

Buildroot has its roots in solving the second use-case. However, we
have already added some features, like OVERRIDE_SRCDIR and SITE_METHOD
= local, to provide the additional flexibility needed for the first use
case. In some sense, those features already kill reproducibility: what
you're building is some random source code from some random local
folder. Completely non-reproducible. Possibility worse that a Git
branch, where you can potentially at least trace back which commit was
built, and re-do that same build.

So I'd like to start the debate on whether it might makes sense to
relax our requirements on this, and be more pragmatic, and support
fetching from Git branches. Of course, this very likely needs to be
guarded by a Config.in option, so that we clearly set the boundary
between "you're in the safe zone where your build is reproducible" and
"you're in the zone where your build is non-reproducible".

And now, let's the flames start! :-)

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] Allowing to build from Git branches

[Vincent Fazio]

I can see the argument for this. I used to use a hacked up BR branch 
that added support for checking if the reference to the git branch has 
changed since the last download but this was only while images? were in 
development. I always switched to fixed hashes after the iterations were 
completed.

On 1/14/20 3:04 PM, Thomas Petazzoni wrote:
> Hello,
>
> I know some people are going to yell at me for bringing up this topic,
> but let's do it anyway.
>
> Currently, Buildroot says that using a Git branch as the VERSION for a
> package is not supported. There are some patches that have been
> proposed to explicitly prevent using branches, but in the current
> Buildroot it "works" except that a cached tarball is preserved in
> DL_DIR, which means that the branch is never fetched again unless the
> tarball is manually deleted from DL_DIR. We have generally resisted
> supporting Git branches with the (very valid) argument that it breaks
> reproducibility.
>
> However, over time, I have seen a number of users/companies complaining
> about this, both on the mailing list and privately with customers or
> while discussing with users at conferences.
>
> The typical use case that they describe is the following: a company has
> a large number of packages in Buildroot to build in-house software
> components. Both for their CI runs, and for the developers actively
> working on the system, they would like for those packages to point to a
> branch, and always fetch the latest version. Yes, it is not
> reproducible, but reproducibility is not what is desired at this point
> of the project: what is wanted is precisely to fetch the latest and
> greatest for those packages that point to a branch, to make sure the
> most recent code gets tested and used by developers internally.
>
> The only solution we offer for this today is <pkg>_OVERRIDE_SRCDIR or
> <pkg>_SITE_METHOD = local, but neither of those are very practical: you
> need to have fetched each of your software components outside of
> Buildroot, and point Buildroot at them. It's a bit silly when Buildroot
> is already perfectly capable of doing this.
>
> Overall, I think it makes sense to realize that Buildroot is used in
> different use-cases:
>
>   - During active development, where we want to offer a lot of
>     flexibility, possibly at the expense of reproducibility.
>
>   - During release/generation of the final firmware that will be used on
>     actual devices, where we want an absolute reproducibility, to the
>     point where we're working on binary-identical results.
>
> Buildroot has its roots in solving the second use-case. However, we
> have already added some features, like OVERRIDE_SRCDIR and SITE_METHOD
> = local, to provide the additional flexibility needed for the first use
> case. In some sense, those features already kill reproducibility: what
> you're building is some random source code from some random local
> folder. Completely non-reproducible. Possibility worse that a Git
> branch, where you can potentially at least trace back which commit was
> built, and re-do that same build.
>
> So I'd like to start the debate on whether it might makes sense to
> relax our requirements on this, and be more pragmatic, and support
> fetching from Git branches. Of course, this very likely needs to be
> guarded by a Config.in option, so that we clearly set the boundary
> between "you're in the safe zone where your build is reproducible" and
> "you're in the zone where your build is non-reproducible".
>
> And now, let's the flames start! :-)
>
> Thomas

-- 
Vincent Fazio
Embedded Software Engineer - Linux
Extreme Engineering Solutions, Inc
http://www.xes-inc.com

[Buildroot] Allowing to build from Git branches

[Michael Nazzareno Trimarchi]

Hi Thomas

On Tue, Jan 14, 2020 at 10:10 PM Vincent Fazio <vfazio@xes-inc.com> wrote:
>
> I can see the argument for this. I used to use a hacked up BR branch
> that added support for checking if the reference to the git branch has
> changed since the last download but this was only while images  were in
> development. I always switched to fixed hashes after the iterations were
> completed.
>
> On 1/14/20 3:04 PM, Thomas Petazzoni wrote:
> > Hello,
> >
> > I know some people are going to yell at me for bringing up this topic,
> > but let's do it anyway.
> >
> > Currently, Buildroot says that using a Git branch as the VERSION for a
> > package is not supported. There are some patches that have been
> > proposed to explicitly prevent using branches, but in the current
> > Buildroot it "works" except that a cached tarball is preserved in
> > DL_DIR, which means that the branch is never fetched again unless the
> > tarball is manually deleted from DL_DIR. We have generally resisted
> > supporting Git branches with the (very valid) argument that it breaks
> > reproducibility.
> >
> > However, over time, I have seen a number of users/companies complaining
> > about this, both on the mailing list and privately with customers or
> > while discussing with users at conferences.
> >
> > The typical use case that they describe is the following: a company has
> > a large number of packages in Buildroot to build in-house software
> > components. Both for their CI runs, and for the developers actively
> > working on the system, they would like for those packages to point to a
> > branch, and always fetch the latest version. Yes, it is not
> > reproducible, but reproducibility is not what is desired at this point
> > of the project: what is wanted is precisely to fetch the latest and
> > greatest for those packages that point to a branch, to make sure the
> > most recent code gets tested and used by developers internally.
> >
> > The only solution we offer for this today is <pkg>_OVERRIDE_SRCDIR or
> > <pkg>_SITE_METHOD = local, but neither of those are very practical: you
> > need to have fetched each of your software components outside of
> > Buildroot, and point Buildroot at them. It's a bit silly when Buildroot
> > is already perfectly capable of doing this.
> >

We use the SRC_OVERRIDE and repo tools. I think that buildroot change
is not required. We have couple of scripts to tag version easily and organize
project that are in continue development outside buildroot.

Michael

> > Overall, I think it makes sense to realize that Buildroot is used in
> > different use-cases:
> >
> >   - During active development, where we want to offer a lot of
> >     flexibility, possibly at the expense of reproducibility.
> >
> >   - During release/generation of the final firmware that will be used on
> >     actual devices, where we want an absolute reproducibility, to the
> >     point where we're working on binary-identical results.
> >
> > Buildroot has its roots in solving the second use-case. However, we
> > have already added some features, like OVERRIDE_SRCDIR and SITE_METHOD
> > = local, to provide the additional flexibility needed for the first use
> > case. In some sense, those features already kill reproducibility: what
> > you're building is some random source code from some random local
> > folder. Completely non-reproducible. Possibility worse that a Git
> > branch, where you can potentially at least trace back which commit was
> > built, and re-do that same build.
> >
> > So I'd like to start the debate on whether it might makes sense to
> > relax our requirements on this, and be more pragmatic, and support
> > fetching from Git branches. Of course, this very likely needs to be
> > guarded by a Config.in option, so that we clearly set the boundary
> > between "you're in the safe zone where your build is reproducible" and
> > "you're in the zone where your build is non-reproducible".
> >
> > And now, let's the flames start! :-)
> >
> > Thomas
>
> --
> Vincent Fazio
> Embedded Software Engineer - Linux
> Extreme Engineering Solutions, Inc
> http://www.xes-inc.com
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot



-- 
| Michael Nazzareno Trimarchi                     Amarula Solutions BV |
| COO  -  Founder                                      Cruquiuskade 47 |
| +31(0)851119172                                 Amsterdam 1018 AM NL |
|                  [`as] http://www.amarulasolutions.com               |

[Buildroot] Allowing to build from Git branches

[Chris Packham]

Hi Thomas,

On Wed, 15 Jan 2020, 10:04 AM Thomas Petazzoni, <
thomas.petazzoni@bootlin.com> wrote:

> Hello,
>
> I know some people are going to yell at me for bringing up this topic,
> but let's do it anyway.
>
> Currently, Buildroot says that using a Git branch as the VERSION for a
> package is not supported. There are some patches that have been
> proposed to explicitly prevent using branches, but in the current
> Buildroot it "works" except that a cached tarball is preserved in
> DL_DIR, which means that the branch is never fetched again unless the
> tarball is manually deleted from DL_DIR. We have generally resisted
> supporting Git branches with the (very valid) argument that it breaks
> reproducibility.
>
> However, over time, I have seen a number of users/companies complaining
> about this, both on the mailing list and privately with customers or
> while discussing with users at conferences.
>

Yep I'm one of those users.

The typical use case that they describe is the following: a company has
>
a large number of packages in Buildroot to build in-house software
> components. Both for their CI runs, and for the developers actively
> working on the system, they would like for those packages to point to a
> branch, and always fetch the latest version. Yes, it is not
> reproducible, but reproducibility is not what is desired at this point
> of the project: what is wanted is precisely to fetch the latest and
> greatest for those packages that point to a branch, to make sure the
> most recent code gets tested and used by developers internally.
>
> The only solution we offer for this today is <pkg>_OVERRIDE_SRCDIR or
> <pkg>_SITE_METHOD = local, but neither of those are very practical: you
> need to have fetched each of your software components outside of
> Buildroot, and point Buildroot at them. It's a bit silly when Buildroot
> is already perfectly capable of doing this.
>
> Overall, I think it makes sense to realize that Buildroot is used in
> different use-cases:
>
>  - During active development, where we want to offer a lot of
>    flexibility, possibly at the expense of reproducibility.
>

I'd qualify that reproducibility is good but we also want correct builds.

A frustration for us (bear in mind this was ~2010) was that it was often
hit and miss between whether buildroot would pick up the fact that the
source had changed so we were constantly second-guessing the validity of
tests which led to a lot of make <package>-dirclean.

Ultimately it may not be possible to fix that issue when sticking with make
as the underlying tool (we ended up rolling our own system so we could hash
stuff instead of relying on timestamps).

 - During release/generation of the final firmware that will be used on
>    actual devices, where we want an absolute reproducibility, to the
>    point where we're working on binary-identical results.
>
> Buildroot has its roots in solving the second use-case. However, we
> have already added some features, like OVERRIDE_SRCDIR and SITE_METHOD
> = local, to provide the additional flexibility needed for the first use
> case. In some sense, those features already kill reproducibility: what
> you're building is some random source code from some random local
> folder. Completely non-reproducible. Possibility worse that a Git
> branch, where you can potentially at least trace back which commit was
> built, and re-do that same build.
>
> So I'd like to start the debate on whether it might makes sense to
> relax our requirements on this, and be more pragmatic, and support
> fetching from Git branches. Of course, this very likely needs to be
> guarded by a Config.in option, so that we clearly set the boundary
> between "you're in the safe zone where your build is reproducible" and
> "you're in the zone where your build is non-reproducible".
>

As you've hinted at it should be possible to at least track the git
describe --always --dirty (or hg equivalent) to provide some level of
traceability even if it's not complete binary equvialence.


> And now, let's the flames start! :-)
>
> Thomas
> --
> Thomas Petazzoni, CTO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20200115/94cd88c4/attachment.html>

[Buildroot] Allowing to build from Git branches

[Thomas Petazzoni]

Hello Chris,

On Wed, 15 Jan 2020 20:07:29 +1300
Chris Packham <judge.packham@gmail.com> wrote:

> > However, over time, I have seen a number of users/companies complaining
> > about this, both on the mailing list and privately with customers or
> > while discussing with users at conferences.
> 
> Yep I'm one of those users.

And you're not the only one :-)

> I'd qualify that reproducibility is good but we also want correct builds.
> 
> A frustration for us (bear in mind this was ~2010) was that it was often
> hit and miss between whether buildroot would pick up the fact that the
> source had changed so we were constantly second-guessing the validity of
> tests which led to a lot of make <package>-dirclean.

I don't really understand how it can be "hit and miss". The way
Buildroot downloads from the upstream location for each package is
completely deterministic.

In addition, if you were doing <package>-dirclean to avoid the problem,
then we're not talking about the same issue, because <package>-dirclean
has zero impact on the contents of DL_DIR, and therefore zero impact on
whether Buildroot will re-download things or not.

Perhaps your issue was broader in the sense that you wanted an existing
Buildroot build (where all packages have already been built) to
automatically refetch some of the packages from Git, without having to
do a "make clean". And that is why you had to selectively "make
<package>-dirclean" those packages.

However, what I am proposing is not going that far. All what I'm
proposing is that if you're doing a clean build, if a package points to
a Git branch as an upstream source, then Buildroot would re-fetch from
that Git branch rather than using a locally cached tarball in DL_DIR
potentially corresponding to an older version of that Git branch.

> > So I'd like to start the debate on whether it might makes sense to
> > relax our requirements on this, and be more pragmatic, and support
> > fetching from Git branches. Of course, this very likely needs to be
> > guarded by a Config.in option, so that we clearly set the boundary
> > between "you're in the safe zone where your build is reproducible" and
> > "you're in the zone where your build is non-reproducible".
> 
> As you've hinted at it should be possible to at least track the git
> describe --always --dirty (or hg equivalent) to provide some level of
> traceability even if it's not complete binary equvialence.

Yes, absolutely. In fact, automatically refetching branches is in some
sense more reproducible than when OVERRIDE_SRCDIR / SITE_METHOD = local
are used.

Indeed, when a package points to a branch, you can at least save the
Git commit which was pointed by the branch at the moment of the build,
and save that somewhere, allowing you to reproduce the same build later
if you need to. However a source directory used with OVERRIDE_SRCDIR /
SITE_METHOD = local can contain any random changes that Buildroot can't
track at all.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] Allowing to build from Git branches

[Chris Packham]

On Wed, 15 Jan 2020, 9:00 PM Thomas Petazzoni, <thomas.petazzoni@bootlin.com>
wrote:

> Hello Chris,
>
> On Wed, 15 Jan 2020 20:07:29 +1300
> Chris Packham <judge.packham@gmail.com> wrote:
>
> > > However, over time, I have seen a number of users/companies complaining
> > > about this, both on the mailing list and privately with customers or
> > > while discussing with users at conferences.
> >
> > Yep I'm one of those users.
>
> And you're not the only one :-)
>
> > I'd qualify that reproducibility is good but we also want correct builds.
> >
> > A frustration for us (bear in mind this was ~2010) was that it was often
> > hit and miss between whether buildroot would pick up the fact that the
> > source had changed so we were constantly second-guessing the validity of
> > tests which led to a lot of make <package>-dirclean.
>
> I don't really understand how it can be "hit and miss". The way
> Buildroot downloads from the upstream location for each package is
> completely deterministic.
>
> In addition, if you were doing <package>-dirclean to avoid the problem,
> then we're not talking about the same issue, because <package>-dirclean
> has zero impact on the contents of DL_DIR, and therefore zero impact on
> whether Buildroot will re-download things or not.
>
> Perhaps your issue was broader in the sense that you wanted an existing
> Buildroot build (where all packages have already been built) to
> automatically refetch some of the packages from Git, without having to
> do a "make clean". And that is why you had to selectively "make
> <package>-dirclean" those packages.
>

Yes it was the re-build case that was problematic for us.

The hit and miss was probably our attempt to make buildroot build packages
using local sources. This was before SITE_METHOD=local was a thing.

However, what I am proposing is not going that far. All what I'm
> proposing is that if you're doing a clean build, if a package points to
> a Git branch as an upstream source, then Buildroot would re-fetch from
> that Git branch rather than using a locally cached tarball in DL_DIR
> potentially corresponding to an older version of that Git branch.
>
> > > So I'd like to start the debate on whether it might makes sense to
> > > relax our requirements on this, and be more pragmatic, and support
> > > fetching from Git branches. Of course, this very likely needs to be
> > > guarded by a Config.in option, so that we clearly set the boundary
> > > between "you're in the safe zone where your build is reproducible" and
> > > "you're in the zone where your build is non-reproducible".
> >
> > As you've hinted at it should be possible to at least track the git
> > describe --always --dirty (or hg equivalent) to provide some level of
> > traceability even if it's not complete binary equvialence.
>
> Yes, absolutely. In fact, automatically refetching branches is in some
> sense more reproducible than when OVERRIDE_SRCDIR / SITE_METHOD = local
> are used.
>
> Indeed, when a package points to a branch, you can at least save the
> Git commit which was pointed by the branch at the moment of the build,
> and save that somewhere, allowing you to reproduce the same build later
> if you need to. However a source directory used with OVERRIDE_SRCDIR /
> SITE_METHOD = local can contain any random changes that Buildroot can't
> track at all.
>
> Best regards,
>
> Thomas
> --
> Thomas Petazzoni, CTO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20200115/82c81501/attachment.html>

[Buildroot] Allowing to build from Git branches

[Christophe Blaess]

Hello Thomas,

> However, what I am proposing is not going that far. All what I'm
> proposing is that if you're doing a clean build, if a package points to
> a Git branch as an upstream source, then Buildroot would re-fetch from
> that Git branch rather than using a locally cached tarball in DL_DIR
> potentially corresponding to an older version of that Git branch.

Would it be possible to have an option on a per-package basis to always 
by-pass the DL_DIR mechanism and re-fetch the sources?

-- 

Christophe Blaess

https://www.logilin.fr

[Buildroot] Allowing to build from Git branches

[Nicolas Carrier]

Hi Everyone,

As an integrator I'm directly concerned by this discussion.

The landscape you've described Thomas, is really accurate, with the
conflicting worflows, one for developers (who want to build what's more
recent) and one for the integrators (who want to know exactly what's
going into a release.

At Orolia, I've solved it using a conjunction of SRC_OVERRIDE and repo
(like Michael apparently) by having different sets of manifests for the
different workflows, that is one set following the branches, for
development and another set on sha1s for releases.

The implicit contract being that, for a given buildroot revision, the
build of the packages it provides has to be reproducible.
Because of the partial support of build on branches, this is not
guaranteed and I've been bitten by that recently, when a branch was
used (due to a broken patch of ours on buildroot) when a tag should
have been used instead.


So AFAIC, I don't need to be able to build from a branch but I'd really
need to _detect_ that a branch is being used by a package handled
directly by buildroot (that is, not in the EXTERNAL dir) and ideally,
to fail the build.
This could be controlled by an option, to keep the compatibility with
the current behavior...


On Wed, 2020-01-15 at 21:30 +1300, Chris Packham wrote:
> 
> 
> On Wed, 15 Jan 2020, 9:00 PM Thomas Petazzoni, <
> thomas.petazzoni at bootlin.com> wrote:
> > Hello Chris,
> > 
> > On Wed, 15 Jan 2020 20:07:29 +1300
> > Chris Packham <judge.packham@gmail.com> wrote:
> > 
> > > > However, over time, I have seen a number of users/companies
> > complaining
> > > > about this, both on the mailing list and privately with
> > customers or
> > > > while discussing with users at conferences.
> > > 
> > > Yep I'm one of those users.
> > 
> > And you're not the only one :-)
> > 
> > > I'd qualify that reproducibility is good but we also want correct
> > builds.
> > > 
> > > A frustration for us (bear in mind this was ~2010) was that it
> > was often
> > > hit and miss between whether buildroot would pick up the fact
> > that the
> > > source had changed so we were constantly second-guessing the
> > validity of
> > > tests which led to a lot of make <package>-dirclean.
> > 
> > I don't really understand how it can be "hit and miss". The way
> > Buildroot downloads from the upstream location for each package is
> > completely deterministic.
> > 
> > In addition, if you were doing <package>-dirclean to avoid the
> > problem,
> > then we're not talking about the same issue, because <package>-
> > dirclean
> > has zero impact on the contents of DL_DIR, and therefore zero
> > impact on
> > whether Buildroot will re-download things or not.
> > 
> > Perhaps your issue was broader in the sense that you wanted an
> > existing
> > Buildroot build (where all packages have already been built) to
> > automatically refetch some of the packages from Git, without having
> > to
> > do a "make clean". And that is why you had to selectively "make
> > <package>-dirclean" those packages.
> > 
> 
> Yes it was the re-build case that was problematic for us.
> 
> The hit and miss was probably our attempt to make buildroot build
> packages using local sources. This was before SITE_METHOD=local was a
> thing.
> 
> > However, what I am proposing is not going that far. All what I'm
> > proposing is that if you're doing a clean build, if a package
> > points to
> > a Git branch as an upstream source, then Buildroot would re-fetch
> > from
> > that Git branch rather than using a locally cached tarball in
> > DL_DIR
> > potentially corresponding to an older version of that Git branch.
> > 
> > > > So I'd like to start the debate on whether it might makes sense
> > to
> > > > relax our requirements on this, and be more pragmatic, and
> > support
> > > > fetching from Git branches. Of course, this very likely needs
> > to be
> > > > guarded by a Config.in option, so that we clearly set the
> > boundary
> > > > between "you're in the safe zone where your build is
> > reproducible" and
> > > > "you're in the zone where your build is non-reproducible".
> > > 
> > > As you've hinted at it should be possible to at least track the
> > git
> > > describe --always --dirty (or hg equivalent) to provide some
> > level of
> > > traceability even if it's not complete binary equvialence.
> > 
> > Yes, absolutely. In fact, automatically refetching branches is in
> > some
> > sense more reproducible than when OVERRIDE_SRCDIR / SITE_METHOD =
> > local
> > are used.
> > 
> > Indeed, when a package points to a branch, you can at least save
> > the
> > Git commit which was pointed by the branch at the moment of the
> > build,
> > and save that somewhere, allowing you to reproduce the same build
> > later
> > if you need to. However a source directory used with
> > OVERRIDE_SRCDIR /
> > SITE_METHOD = local can contain any random changes that Buildroot
> > can't
> > track at all.
> > 
> > Best regards,
> > 
> > Thomas
> > _______________________________________________
> > buildroot mailing list
> > buildroot at busybox.net
> > http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] Allowing to build from Git branches

[Thomas Petazzoni]

On Wed, 15 Jan 2020 09:54:16 +0100
Christophe Blaess <mailist@logilin.fr> wrote:

> > However, what I am proposing is not going that far. All what I'm
> > proposing is that if you're doing a clean build, if a package points to
> > a Git branch as an upstream source, then Buildroot would re-fetch from
> > that Git branch rather than using a locally cached tarball in DL_DIR
> > potentially corresponding to an older version of that Git branch.  
> 
> Would it be possible to have an option on a per-package basis to always 
> by-pass the DL_DIR mechanism and re-fetch the sources?

Yes, that would definitely be an idea. There was some discussion
yesterday evening about this on IRC with Yann E. Morin, and indeed one
of the idea that came up is to have something like
<pkg>_DOWNLOAD_TRANSIENT = YES, which would tell the download
infrastructure to refetch from upstream everytime the package is built
instead of using a locally available tarball.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] Allowing to build from Git branches

[Adam Duskett]

Hello;

On Wed, Jan 15, 2020 at 1:32 AM Christophe Blaess <mailist@logilin.fr> wrote:
>
> Hello Thomas,
>
> > However, what I am proposing is not going that far. All what I'm
> > proposing is that if you're doing a clean build, if a package points to
> > a Git branch as an upstream source, then Buildroot would re-fetch from
> > that Git branch rather than using a locally cached tarball in DL_DIR
> > potentially corresponding to an older version of that Git branch.
>
> Would it be possible to have an option on a per-package basis to always
> by-pass the DL_DIR mechanism and re-fetch the sources?
>
This is what I am thinking.
Something like: <PKG>_GIT_METHOD=branch


> --
>
> Christophe Blaess
>
> https://www.logilin.fr
>
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] Allowing to build from Git branches

[Yann E. MORIN]

Thomas, All,

On 2020-01-15 14:40 +0100, Thomas Petazzoni spake thusly:
> On Wed, 15 Jan 2020 09:54:16 +0100
> Christophe Blaess <mailist@logilin.fr> wrote:
> > Would it be possible to have an option on a per-package basis to always 
> > by-pass the DL_DIR mechanism and re-fetch the sources?
> 
> Yes, that would definitely be an idea. There was some discussion
> yesterday evening about this on IRC with Yann E. Morin, and indeed one
> of the idea that came up is to have something like
> <pkg>_DOWNLOAD_TRANSIENT = YES, which would tell the download
> infrastructure to refetch from upstream everytime the package is built
> instead of using a locally available tarball.

I've jsut sent a proof-of-concept of this 'transient' mchanism, which
turned out to be relatively easy to come up with; there is even far more
boilerplate and documentation that actual code to implement the feature.

    http://lists.busybox.net/pipermail/buildroot/2020-January/271660.html

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'