[Buildroot] cached downloads for scm branches

[Buildroot] cached downloads for scm branches

[Michael Walle]

Hi,

If I have a package which downloads it's source from a SCM branch, the 
download will be cached in DL_DIR/<pkgname>-<branchname>.tar.gz. This 
makes sense if the version is a tag or a (git) hash. But if used with 
branches, the package will never be downloaded again until I remove it 
from the DL_DIR. This may be the intended behaviour, but as far as I can 
tell, I have no chance to change it.

Do I miss something? Does it make sense to add a new package variable to 
force a download?

-michael

[Buildroot] cached downloads for scm branches

[Thomas Petazzoni]

Hello,

On Thu, 20 Oct 2016 19:27:19 +0200, Michael Walle wrote:

> If I have a package which downloads it's source from a SCM branch, the 
> download will be cached in DL_DIR/<pkgname>-<branchname>.tar.gz. This 
> makes sense if the version is a tag or a (git) hash. But if used with 
> branches, the package will never be downloaded again until I remove it 
> from the DL_DIR. This may be the intended behaviour, but as far as I can 
> tell, I have no chance to change it.
> 
> Do I miss something? Does it make sense to add a new package variable to 
> force a download?

This behavior is intended and adding a new package variable to force a
download is not what we want to do. Indeed, Buildroot aims at doing
*reproducible* builds. If a package points to a branch, then you have
no idea what you are building, this what you're building depends on the
moment you start the build (as it decides when you will fetch, and
therefore what you will fetch).

*However*, we have a mechanism called <pkg>_OVERRIDE_SRCDIR, which
allows you to tell Buildroot: "For this specific package, please skip
the download/extract/patch steps, and instead take the source from
<this> local directory". Thanks to that, you can checkout/clone your
project source code in <this> folder, and ask Buildroot to fetch it
from there. This is what we recommend during active development on a
given component.

See
http://free-electrons.com/doc/training/buildroot/buildroot-slides.pdf
slides 261 and following for more details.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] cached downloads for scm branches

[Yann E. MORIN]

Michael, All,

On 2016-10-20 19:27 +0200, Michael Walle spake thusly:
> Hi,
> 
> If I have a package which downloads it's source from a SCM branch, the
> download will be cached in DL_DIR/<pkgname>-<branchname>.tar.gz. This makes
> sense if the version is a tag or a (git) hash. But if used with branches,
> the package will never be downloaded again until I remove it from the
> DL_DIR. This may be the intended behaviour, but as far as I can tell, I have
> no chance to change it.

Yes, this is the intended behaviour: you should not use a branch names
or other unstable references Using a branch name makes the build
non-reproducible, and that is not good.

Only use stable references, like a tag or a sha1 or ...

If however you are actively working on the package, then you should use
the source-override mechanism instead:

  - create a local.mk file (in the directory where you ran 'make
    menuconfig')

  - in that file, add this line

        MY_PACKAGE_OVERRIDE_SRCDIR = /path/to/my-package

Then you can manage that location "by hand" (like pulling new changes...)
and have it automatically used by Buildroot:

    make my-package-rebuild

(and other such commands, see "make help" and the manual for the
complete set.)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] cached downloads for scm branches

[Michael Walle]

hi, 

thanks. yeah i thought so. Just to give you some background regarding my question. We have some kind of CI, actually it's rather a daily build. we build a root filesystem with some fixed packages, but also with multiple packages of our own and we want to track the tip of a branch for a build (because it is CI ;)). I don't think the src override method really suit the CI case. I guess one way could be that the CI defines the commit that should be build. Or simply delete the download. mhh ;) 

-michael

sorry for the tofu. i'm on my mobile. 


Am 20. Oktober 2016 21:38:41 MESZ, schrieb Thomas Petazzoni <thomas.petazzoni@free-electrons.com>:
>Hello,
>
>On Thu, 20 Oct 2016 19:27:19 +0200, Michael Walle wrote:
>
>> If I have a package which downloads it's source from a SCM branch,
>the 
>> download will be cached in DL_DIR/<pkgname>-<branchname>.tar.gz. This
>
>> makes sense if the version is a tag or a (git) hash. But if used with
>
>> branches, the package will never be downloaded again until I remove
>it 
>> from the DL_DIR. This may be the intended behaviour, but as far as I
>can 
>> tell, I have no chance to change it.
>> 
>> Do I miss something? Does it make sense to add a new package variable
>to 
>> force a download?
>
>This behavior is intended and adding a new package variable to force a
>download is not what we want to do. Indeed, Buildroot aims at doing
>*reproducible* builds. If a package points to a branch, then you have
>no idea what you are building, this what you're building depends on the
>moment you start the build (as it decides when you will fetch, and
>therefore what you will fetch).
>
>*However*, we have a mechanism called <pkg>_OVERRIDE_SRCDIR, which
>allows you to tell Buildroot: "For this specific package, please skip
>the download/extract/patch steps, and instead take the source from
><this> local directory". Thanks to that, you can checkout/clone your
>project source code in <this> folder, and ask Buildroot to fetch it
>from there. This is what we recommend during active development on a
>given component.
>
>See
>http://free-electrons.com/doc/training/buildroot/buildroot-slides.pdf
>slides 261 and following for more details.
>
>Best regards,
>
>Thomas
>-- 
>Thomas Petazzoni, CTO, Free Electrons
>Embedded Linux and Kernel engineering
>http://free-electrons.com
>_______________________________________________
>buildroot mailing list
>buildroot at busybox.net
>http://lists.busybox.net/mailman/listinfo/buildroot
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20161020/41bb724b/attachment.html>

[Buildroot] cached downloads for scm branches

[Jérôme Pouiller]

Hello Michael,

On Thursday 20 October 2016 22:29:15 Michael Walle wrote:
> hi,
> 
> thanks. yeah i thought so. Just to give you some background regarding
> my question. We have some kind of CI, actually it's rather a daily
> build. we build a root filesystem with some fixed packages, but also
> with multiple packages of our own and we want to track the tip of a
> branch for a build (because it is CI ;)). I don't think the src
> override method really suit the CI case. I guess one way could be
> that the CI defines the commit that should be build. Or simply delete
> the download. mhh ;)

I suggest to pass package version to command line:

  make MY_PKG_VERSION=$(git ls-remote http://repo.git/path master) all

You may also force versions using local.mk:

  override MY_PKG_VERSION = $(shell git ls-remote http://repo.git/path 
master)

However, these ways are not officially supported. Consider them
as hacks.

BR,

-- 
J?r?me Pouiller

[Buildroot] cached downloads for scm branches

[Arnout Vandecappelle]

On 20-10-16 22:29, Michael Walle wrote:
> thanks. yeah i thought so. Just to give you some background regarding my
> question. We have some kind of CI, actually it's rather a daily build. we build
> a root filesystem with some fixed packages, but also with multiple packages of
> our own and we want to track the tip of a branch for a build (because it is CI
> ;)). I don't think the src override method really suit the CI case. I guess one
> way could be that the CI defines the commit that should be build. Or simply
> delete the download. mhh ;)

 Actually, the src override method fits the CI case perfectly because CI can
watch the repositories to be overridden and check them out.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF