[Buildroot] Buildroot defconfigs now being built on Travis CI

[Buildroot] Buildroot defconfigs now being built on Travis CI

[Thomas Petazzoni]

Hello,

I used to build the Buildroot defconfigs with a Jenkins instance
provided by Free Electrons, but with the increasing number of
defconfigs, it started to take too much time on our build server (and
therefore less CPU time was available for autobuild.b.o testing).

So, I've moved the testing of the Buildroot defconfigs to Travis CI,
which provides essentially free CPU time to allow open-source projects
to do continuous integration.

You can see the results at:

  https://travis-ci.org/buildroot/buildroot-defconfig-testing

The last build has been fully successful, with all 95 defconfigs
building fine. I have scheduled to rebuild all defconfigs every two
days, of course only if commits have been made to Buildroot.

For the moment, notifications of build working fine or failing are just
sent to some testing IRC channel. Once the mechanism has proven to work
well for a week or two, I'll adjust the notifications so that they are
sent to the official #buildroot IRC channel, and possibly by e-mail as
well (to the mailing list or directly to interested people).
Suggestions on this are welcome.

Now, if you want the gory details of how this is implemented:

In Travis, projects are directly connected to a Github repository. The
buildroot-defconfig-testing project is not connected directly to the
Buildroot Github repository. Instead, I have created a small
intermediate project, at
https://github.com/buildroot/buildroot-defconfig-testing, which
contains the .travis.yml file (for those who don't know Travis, this is
where you describe what your continuous integration tests should do).

In this repository, a shell script called update.sh pulls the latest
Buildrooot Git repository, and updates the .travis.yml to account for
changes in the list of available defconfigs, and to adjust the
Buildroot commit to be tested. It then pushed the result, which
triggers the Travis build. This script is executed every two days,
which is how the builds get triggered.

Do not hesitate to ask if you have any questions about this.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Buildroot defconfigs now being built on Travis CI

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

Hi,

 > Hello,
 > I used to build the Buildroot defconfigs with a Jenkins instance
 > provided by Free Electrons, but with the increasing number of
 > defconfigs, it started to take too much time on our build server (and
 > therefore less CPU time was available for autobuild.b.o testing).

 > So, I've moved the testing of the Buildroot defconfigs to Travis CI,
 > which provides essentially free CPU time to allow open-source projects
 > to do continuous integration.

 > You can see the results at:

 >   https://travis-ci.org/buildroot/buildroot-defconfig-testing

Cool, great!

 > The last build has been fully successful, with all 95 defconfigs
 > building fine. I have scheduled to rebuild all defconfigs every two
 > days, of course only if commits have been made to Buildroot.

 > For the moment, notifications of build working fine or failing are just
 > sent to some testing IRC channel. Once the mechanism has proven to work
 > well for a week or two, I'll adjust the notifications so that they are
 > sent to the official #buildroot IRC channel, and possibly by e-mail as
 > well (to the mailing list or directly to interested people).
 > Suggestions on this are welcome.

I wouldn't mind getting a notification per email if something fails.


 > Now, if you want the gory details of how this is implemented:

Is that XFS issue something we want to fix before 2015.11? What is that
stupidpid thing about? Does travis stop if it doesn't get any output?

I see that results are copied to the autobuilder server? How are they
visualized? Just together with the other autobuild results?

Thanks for setting it up!

-- 
Bye, Peter Korsgaard

[Buildroot] Buildroot defconfigs now being built on Travis CI

[Thomas Petazzoni]

Peter,

On Mon, 23 Nov 2015 22:51:20 +0100, Peter Korsgaard wrote:

>  > The last build has been fully successful, with all 95 defconfigs
>  > building fine. I have scheduled to rebuild all defconfigs every two
>  > days, of course only if commits have been made to Buildroot.
> 
>  > For the moment, notifications of build working fine or failing are just
>  > sent to some testing IRC channel. Once the mechanism has proven to work
>  > well for a week or two, I'll adjust the notifications so that they are
>  > sent to the official #buildroot IRC channel, and possibly by e-mail as
>  > well (to the mailing list or directly to interested people).
>  > Suggestions on this are welcome.
> 
> I wouldn't mind getting a notification per email if something fails.

Sure, I'll add that soon.

>  > Now, if you want the gory details of how this is implemented:
> 
> Is that XFS issue something we want to fix before 2015.11?

I don't think we can fix it before 2015.11. The hack I've done in
Travis is just what it is: a crude hack. The underlying problem is that
we ask the host filesystem how many blocks are needed to store
output/target/. It works fine if your host filesystem == target
filesystem == ext2/3/4. And it probably works for most target
filesystems, by chance. But it is clearly not correct, since the
filesystem on your host may be different than the one used on your
target.

Specifically, XFS stores small files and symlinks directly inside the
inode instead of allocating a block for them. So the number of blocks
needed to store output/target on the host is much smaller than what is
needed on the target ext2 filesystem.

> What is that
> stupidpid thing about? Does travis stop if it doesn't get any output?

Travis is somewhat annoying in his handling of logs:

 * Originally, I was just calling "make", so Travis was seeing the
   entire make output. But Travis doesn't like to have more than 4 MB
   of text displayed on the standard output, so it was killing the
   builds when they reached 4 MB.

 * So, I pipped the output into tee to keep the log, and then grep to
   only show the ">>>" messages on stdout. But then, Travis is not
   happy because stdout is sometimes quiet for "too long".

So, I ended up starting a stupid background process that just says
"Still building" every minute. A bit silly, but works.

> I see that results are copied to the autobuilder server? How are they
> visualized? Just together with the other autobuild results?

Right now they are not publicly available. However, they are indeed on
the machine that hosts autobuild.buildroot.org, so it's just a matter
of doing a bit of webserver configuration to provide access to them. My
plan is to add a message at the end of the build that tells the URL at
which the full log has been saved.

However, the logs are quite big, so I don't think I will be able to
keep them forever (just like I don't keep the full logs of the random
tests on autobuild.b.o, I only keep build-end.log).

Isn't the last 1000 lines of log already displayed in the Travis
console sufficient to debug most problems anyway?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Buildroot defconfigs now being built on Travis CI

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

Hi,

>> Is that XFS issue something we want to fix before 2015.11?

 > I don't think we can fix it before 2015.11. The hack I've done in
 > Travis is just what it is: a crude hack. The underlying problem is that
 > we ask the host filesystem how many blocks are needed to store
 > output/target/. It works fine if your host filesystem == target
 > filesystem == ext2/3/4. And it probably works for most target
 > filesystems, by chance. But it is clearly not correct, since the
 > filesystem on your host may be different than the one used on your
 > target.

 > Specifically, XFS stores small files and symlinks directly inside the
 > inode instead of allocating a block for them. So the number of blocks
 > needed to store output/target on the host is much smaller than what is
 > needed on the target ext2 filesystem.

Ok :/


 >> What is that
 >> stupidpid thing about? Does travis stop if it doesn't get any output?

 > Travis is somewhat annoying in his handling of logs:

 >  * Originally, I was just calling "make", so Travis was seeing the
 >    entire make output. But Travis doesn't like to have more than 4 MB
 >    of text displayed on the standard output, so it was killing the
 >    builds when they reached 4 MB.

 >  * So, I pipped the output into tee to keep the log, and then grep to
 >    only show the ">>>" messages on stdout. But then, Travis is not
 >    happy because stdout is sometimes quiet for "too long".

 > So, I ended up starting a stupid background process that just says
 > "Still building" every minute. A bit silly, but works.

Gaah, silly - But ok, if it works..


 >> I see that results are copied to the autobuilder server? How are they
 >> visualized? Just together with the other autobuild results?

 > Right now they are not publicly available. However, they are indeed on
 > the machine that hosts autobuild.buildroot.org, so it's just a matter
 > of doing a bit of webserver configuration to provide access to them. My
 > plan is to add a message at the end of the build that tells the URL at
 > which the full log has been saved.

 > However, the logs are quite big, so I don't think I will be able to
 > keep them forever (just like I don't keep the full logs of the random
 > tests on autobuild.b.o, I only keep build-end.log).

 > Isn't the last 1000 lines of log already displayed in the Travis
 > console sufficient to debug most problems anyway?

Probably it is, I just saw that you were copying them to the
autobuilders.

Thanks!

-- 
Venlig hilsen,
Peter Korsgaard 

[Buildroot] Buildroot defconfigs now being built on Travis CI

[Yann E. MORIN]

Peter, All,

On 2015-11-24 21:53 +0100, Peter Korsgaard spake thusly:
> >>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:
> >> Is that XFS issue something we want to fix before 2015.11?
>  > I don't think we can fix it before 2015.11. The hack I've done in
>  > Travis is just what it is: a crude hack. The underlying problem is that
>  > we ask the host filesystem how many blocks are needed to store
>  > output/target/. It works fine if your host filesystem == target
>  > filesystem == ext2/3/4. And it probably works for most target
>  > filesystems, by chance. But it is clearly not correct, since the
>  > filesystem on your host may be different than the one used on your
>  > target.
> 
>  > Specifically, XFS stores small files and symlinks directly inside the
>  > inode instead of allocating a block for them. So the number of blocks
>  > needed to store output/target on the host is much smaller than what is
>  > needed on the target ext2 filesystem.
> 
> Ok :/

It's been tricky to track down...

[--SNIP--]
>  > Isn't the last 1000 lines of log already displayed in the Travis
>  > console sufficient to debug most problems anyway?
> 
> Probably it is, I just saw that you were copying them to the
> autobuilders.

Thomas did that so we could try to reproduce locally. Since it was
working on our machines, we hunted a difference on the host. And
suddenly, XFS... :-/

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Buildroot defconfigs now being built on Travis CI

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

Hi,

>> Probably it is, I just saw that you were copying them to the
 >> autobuilders.

 > Thomas did that so we could try to reproduce locally. Since it was
 > working on our machines, we hunted a difference on the host. And
 > suddenly, XFS... :-/

Heh ;) Any ideas about how to fix it for real? I think I saw a reference
on IRC to something in e2fsprogs to generate images?

-- 
Venlig hilsen,
Peter Korsgaard 

[Buildroot] Buildroot defconfigs now being built on Travis CI

[Yann E. MORIN]

Peter, All,

On 2015-11-24 23:42 +0100, Peter Korsgaard spake thusly:
> >>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:
> >> Probably it is, I just saw that you were copying them to the
>  >> autobuilders.
>  > Thomas did that so we could try to reproduce locally. Since it was
>  > working on our machines, we hunted a difference on the host. And
>  > suddenly, XFS... :-/
> Heh ;) Any ideas about how to fix it for real? I think I saw a reference
> on IRC to something in e2fsprogs to generate images?

The ideal solution would be that e2fsprogs do a new major release: there
is no in their master branch suport for mkfs.ext2 to behave like
genext2fs and populate the filesystem with the content of a diectory.

But it's only in their master, and there no release with that yet, even
though they've had that for 1.5 years now...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'