[Buildroot] [autobuild.buildroot.net] Build results for 2014-06-14

[Buildroot] [autobuild.buildroot.net] Build results for 2014-06-14

[Thomas Petazzoni]

Build statistics for 2014-06-14
===============================

        success : 74 
       failures : 23 
       timeouts : 2  
          TOTAL : 99 

Classification of failures by reason
====================================

           dialog-1.2-20140219 | 10
make: Leaving directory `/h... | 2 
make: *** [core-dependencie... | 1 
                 cairo-1.12.10 | 1 
               audiofile-0.3.6 | 1 
                oprofile-0.9.9 | 1 
       gst1-plugins-good-1.2.4 | 1 
             host-mysql-5.1.73 | 1 
                  zeromq-4.0.4 | 1 
            wpa_supplicant-2.2 | 1 
                  snmppp-3.3.4 | 1 
                 xenomai-2.6.3 | 1 
            eigen-ffa86ffb5570 | 1 
 make[1]: *** [dialog] Error 1 | 1 
 make[2]: *** [opt] Terminated | 1 

Detail of failures
===================

   powerpc |                audiofile-0.3.6 | NOK | http://autobuild.buildroot.net/results/b62615a2d933f2b69c8b402d0134c72675a7e527/
      i686 |                  cairo-1.12.10 | TIM | http://autobuild.buildroot.net/results/211fbc0aee997139a7a9d65b4776e02a5b294b4e/
microblazeel |            dialog-1.2-20140219 | NOK | http://autobuild.buildroot.net/results/5f9957a76381890a5efa5f52900592bf915550ce/
    x86_64 |            dialog-1.2-20140219 | NOK | http://autobuild.buildroot.net/results/273539eaf9e930d0230a8c4b11e295733ab4f871/
       arm |            dialog-1.2-20140219 | NOK | http://autobuild.buildroot.net/results/128abc06acbf7d4872bbebffd99f97c073237303/
       arm |            dialog-1.2-20140219 | NOK | http://autobuild.buildroot.net/results/1bf4342d4aa1d49294a278f25322a00ee0d188ea/
       arm |            dialog-1.2-20140219 | NOK | http://autobuild.buildroot.net/results/411f6171e972eab4486143dedbfd078136886ab0/
       arm |            dialog-1.2-20140219 | NOK | http://autobuild.buildroot.net/results/365e6d1ef2faef546dd41b5c0a6ab04212b519dd/
      i686 |            dialog-1.2-20140219 | NOK | http://autobuild.buildroot.net/results/072cb54fdbc246a97c6cfbf854a6e648779e9755/
    xtensa |            dialog-1.2-20140219 | NOK | http://autobuild.buildroot.net/results/875ae36dff9fdd84f71f1177349d7c80adfb3f27/
   powerpc |            dialog-1.2-20140219 | NOK | http://autobuild.buildroot.net/results/de6000fbe76fe5e8bd2832f090ddf81c09c9d8d8/
       arm |            dialog-1.2-20140219 | NOK | http://autobuild.buildroot.net/results/0dd652d3caa212089caab0f962213d9938d9745e/
   powerpc |             eigen-ffa86ffb5570 | NOK | http://autobuild.buildroot.net/results/d7daa7ac359c6bef85ea0c65c5318f3068159010/
   powerpc |        gst1-plugins-good-1.2.4 | NOK | http://autobuild.buildroot.net/results/4f746cb9d8266abea2671718ad0f292763263873/
  mips64el |              host-mysql-5.1.73 | NOK | http://autobuild.buildroot.net/results/cc99d4b8d0d83705071034bc72dd4e2efbb07acf/
microblazeel | make: *** [core-dependencie... | NOK | http://autobuild.buildroot.net/results/30af5cd7f39fb556925321f9ce5c733a6688bc81/
  mips64el | make: Leaving directory `/h... | NOK | http://autobuild.buildroot.net/results/16c9d9baf11082e6b7db8bc5076ee7c54ddbd164/
      i486 | make: Leaving directory `/h... | NOK | http://autobuild.buildroot.net/results/67f79482f8a80e63aca366df6d3ea232336c9ce1/
       arm |  make[1]: *** [dialog] Error 1 | NOK | http://autobuild.buildroot.net/results/dae1d1cfff19f52ebdbf558b884cbe0a45cf2564/
   powerpc |  make[2]: *** [opt] Terminated | TIM | http://autobuild.buildroot.net/results/b9f67e5b23032eb81be30107fda8104952bf775f/
       arm |                 oprofile-0.9.9 | NOK | http://autobuild.buildroot.net/results/549ef23ea1c9daeba8337b45ffabb254321c72e3/
      bfin |                   snmppp-3.3.4 | NOK | http://autobuild.buildroot.net/results/ec2fa986493cf278681e9aa85d80356f9f3f15d1/
     avr32 |             wpa_supplicant-2.2 | NOK | http://autobuild.buildroot.net/results/20908f479b33c1e2952622f5e8ad6b60d58af693/
      bfin |                  xenomai-2.6.3 | NOK | http://autobuild.buildroot.net/results/db4f913a0dc9a0d8875cf3879e350837a73e26aa/
      bfin |                   zeromq-4.0.4 | NOK | http://autobuild.buildroot.net/results/3af3d41ba5b60f30e3c9e7bcea33f52dc0c89cf2/


-- 
http://autobuild.buildroot.net

[Buildroot] Analysis of build failures

[Thomas Petazzoni]

Hello all,

Over the next few days, I'll try to do a quick analysis of the build
failures, to let you know which build failures are "real", and which
build failures are caused by issues in the autobuilder infrastructure.

On Sun, 15 Jun 2014 08:30:14 +0200 (CEST), Thomas Petazzoni wrote:

>    powerpc |                audiofile-0.3.6 | NOK | http://autobuild.buildroot.net/results/b62615a2d933f2b69c8b402d0134c72675a7e527/

Still the libstdc++ static linking issue.

>       i686 |                  cairo-1.12.10 | TIM | http://autobuild.buildroot.net/results/211fbc0aee997139a7a9d65b4776e02a5b294b4e/

Timeout on my private server. It seems like this server is by an order
of magnitude slower than the Free Electrons build server, and that
therefore the timeout  of 4 hours is not appropriate for builds that
can have up to 30% of the packages enabled. For example, this private
server needs 20 minutes to build libglib2 (which seems very long, not
sure what's going on there).

I'm looking for suggestions on this, because I'm sure other people
testing the autobuild-run script will have similar problems, as they
will not all have 8 cores monsters with 32 GB of RAM and fast I/O.
There are two possible directions:

 * Make the timeout configurable, so that people can adjust it to their
   configuration to avoid false positives as much as possible.

 * Make the timeout a per-package timeout. However, that requires
   having a process running in parallel to the build to monitor the
   progress of the build. Not impossible, but requires a bit of
   additional logic in autobuild-run.

 * Make the KCONFIG_PROBABILITY configurable. My statistics lessons are
   way too much in the past, but I'm wondering if having different
   KCONFIG_PROBABILITY in the various builders is not going to make the
   statistic of failed vs. success builds a bit meaningless. If one
   machine runs with a low KCONFIG_PROBABILITY, this machine will do a
   lot of small builds, and small builds have a much higher chance of
   being successful than bigger builds. Therefore, such a machine could
   easily reach a very high success rate of builds, compared to a
   machine running with a higher KCONFIG_PROBABILITY value. Thoughts on
   this? Or people better in statistics than me to confirm or infirm my
   reasoning?

> microblazeel |            dialog-1.2-20140219 | NOK | http://autobuild.buildroot.net/results/5f9957a76381890a5efa5f52900592bf915550ce/
>     x86_64 |            dialog-1.2-20140219 | NOK | http://autobuild.buildroot.net/results/273539eaf9e930d0230a8c4b11e295733ab4f871/
>        arm |            dialog-1.2-20140219 | NOK | http://autobuild.buildroot.net/results/128abc06acbf7d4872bbebffd99f97c073237303/
>        arm |            dialog-1.2-20140219 | NOK | http://autobuild.buildroot.net/results/1bf4342d4aa1d49294a278f25322a00ee0d188ea/
>        arm |            dialog-1.2-20140219 | NOK | http://autobuild.buildroot.net/results/411f6171e972eab4486143dedbfd078136886ab0/
>        arm |            dialog-1.2-20140219 | NOK | http://autobuild.buildroot.net/results/365e6d1ef2faef546dd41b5c0a6ab04212b519dd/
>       i686 |            dialog-1.2-20140219 | NOK | http://autobuild.buildroot.net/results/072cb54fdbc246a97c6cfbf854a6e648779e9755/
>     xtensa |            dialog-1.2-20140219 | NOK | http://autobuild.buildroot.net/results/875ae36dff9fdd84f71f1177349d7c80adfb3f27/
>    powerpc |            dialog-1.2-20140219 | NOK | http://autobuild.buildroot.net/results/de6000fbe76fe5e8bd2832f090ddf81c09c9d8d8/
>        arm |            dialog-1.2-20140219 | NOK | http://autobuild.buildroot.net/results/0dd652d3caa212089caab0f962213d9938d9745e/

This is all fixed by
http://git.buildroot.net/buildroot/commit/?id=9c080f34076cdf48bcf72e1a1fbe2024b47a2694.

>    powerpc |             eigen-ffa86ffb5570 | NOK | http://autobuild.buildroot.net/results/d7daa7ac359c6bef85ea0c65c5318f3068159010/

Temporary failure of upstream Mercurial server + lack of the tarball on
sources.buildroot.net. I just checked, the upstream Mercurial server
works fine now.

>    powerpc |        gst1-plugins-good-1.2.4 | NOK | http://autobuild.buildroot.net/results/4f746cb9d8266abea2671718ad0f292763263873/

  CC       libgstvideo4linux2_la-gstv4l2sink.lo
gstv4l2object.c: In function 'gst_v4l2_object_set_format':
gstv4l2object.c:2384:23: error: 'union <anonymous>' has no member named 'pix_mp'
gstv4l2object.c:2385:24: error: 'union <anonymous>' has no member named 'pix_mp'

pix_mp was added in 2.6.39, and this toolchain uses older kernel
headers. So I propose to package this package depend on kernel headers
greater than 3.0. Even though that's not exactly correct, it's good enough, and
we decided to not have Config.in options for each of the 2.6 kernels. Is that OK ?

>   mips64el |              host-mysql-5.1.73 | NOK | http://autobuild.buildroot.net/results/cc99d4b8d0d83705071034bc72dd4e2efbb07acf/

Seems like it could be solved by
http://patchwork.ozlabs.org/patch/326425/. I'll try to reproduce in the
autobuilders to be sure.

> microblazeel | make: *** [core-dependencie... | NOK | http://autobuild.buildroot.net/results/30af5cd7f39fb556925321f9ce5c733a6688bc81/

That's an issue with the autobuilder infrastructure, ignore.

>   mips64el | make: Leaving directory `/h... | NOK | http://autobuild.buildroot.net/results/16c9d9baf11082e6b7db8bc5076ee7c54ddbd164/

Same, ignore.

>       i486 | make: Leaving directory `/h... | NOK | http://autobuild.buildroot.net/results/67f79482f8a80e63aca366df6d3ea232336c9ce1/

That's the same dialog issue, which is already fixed. The "reason" was
mis-detected due to slight variations in the format that caused the
logic on http://autobuild.buildroot.org to not recognized the correct
reason in the log file.

>        arm |  make[1]: *** [dialog] Error 1 | NOK | http://autobuild.buildroot.net/results/dae1d1cfff19f52ebdbf558b884cbe0a45cf2564/

Same.

>    powerpc |  make[2]: *** [opt] Terminated | TIM | http://autobuild.buildroot.net/results/b9f67e5b23032eb81be30107fda8104952bf775f/

A timeout on the private server.

>        arm |                 oprofile-0.9.9 | NOK | http://autobuild.buildroot.net/results/549ef23ea1c9daeba8337b45ffabb254321c72e3/

../libutil++/libutil++.a(bfd_support.o): In function `bfd_info::get_synth_symbols()':
bfd_support.cpp:(.text+0x190c): undefined reference to `bfd_elf64_powerpc_vec'
bfd_support.cpp:(.text+0x1910): undefined reference to `bfd_elf64_powerpcle_vec'
collect2: error: ld returned 1 exit status

Looks weird. PowerPC symbols, but we're building for ARM. Why?

>       bfin |                   snmppp-3.3.4 | NOK | http://autobuild.buildroot.net/results/ec2fa986493cf278681e9aa85d80356f9f3f15d1/

I investigated that: it builds fine with the gcc 4.3 Blackfin toolchain
(called stable), but not with the gcc 4.5 Blackfin toolchain (called
experimental). Maybe we should switch to use the gcc 4.3 Blackfin
toolchain by default. But on the other hand, that doesn't allow us to
detect issues and report them to ADI. That being said, when I look at
the toolchain bug tracker of ADI, it looks more like /dev/null that
anything really useful.

Suggestions?

>      avr32 |             wpa_supplicant-2.2 | NOK | http://autobuild.buildroot.net/results/20908f479b33c1e2952622f5e8ad6b60d58af693/

Baruch has already sent a patch disabling wpa_supplicant to avoid this
problem.

>       bfin |                  xenomai-2.6.3 | NOK | http://autobuild.buildroot.net/results/db4f913a0dc9a0d8875cf3879e350837a73e26aa/

Many "error: inline function 'pthread_atfork' cannot be declared weak"
and similar messages for other functions.

>       bfin |                   zeromq-4.0.4 | NOK | http://autobuild.buildroot.net/results/3af3d41ba5b60f30e3c9e7bcea33f52dc0c89cf2/

Exact same failure as snmpp.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Analysis of build failures

[Thomas De Schampheleire]

Thomas Petazzoni <thomas.petazzoni@free-electrons.com> schreef:
>Hello all,
>
>Over the next few days, I'll try to do a quick analysis of the build
>failures, to let you know which build failures are "real", and which
>build failures are caused by issues in the autobuilder infrastructure.
>
>On Sun, 15 Jun 2014 08:30:14 +0200 (CEST), Thomas Petazzoni wrote:
>
>>    powerpc |                audiofile-0.3.6 | NOK | http://autobuild.buildroot.net/results/b62615a2d933f2b69c8b402d0134c72675a7e527/
>
>Still the libstdc++ static linking issue.
>
>>       i686 |                  cairo-1.12.10 | TIM | http://autobuild.buildroot.net/results/211fbc0aee997139a7a9d65b4776e02a5b294b4e/
>
>Timeout on my private server. It seems like this server is by an order
>of magnitude slower than the Free Electrons build server, and that
>therefore the timeout  of 4 hours is not appropriate for builds that
>can have up to 30% of the packages enabled. For example, this private
>server needs 20 minutes to build libglib2 (which seems very long, not
>sure what's going on there).
>
>I'm looking for suggestions on this, because I'm sure other people
>testing the autobuild-run script will have similar problems, as they
>will not all have 8 cores monsters with 32 GB of RAM and fast I/O.
>There are two possible directions:
>
> * Make the timeout configurable, so that people can adjust it to their
>   configuration to avoid false positives as much as possible.
>
> * Make the timeout a per-package timeout. However, that requires
>   having a process running in parallel to the build to monitor the
>   progress of the build. Not impossible, but requires a bit of
>   additional logic in autobuild-run.
>
> * Make the KCONFIG_PROBABILITY configurable. My statistics lessons are
>   way too much in the past, but I'm wondering if having different
>   KCONFIG_PROBABILITY in the various builders is not going to make the
>   statistic of failed vs. success builds a bit meaningless. If one
>   machine runs with a low KCONFIG_PROBABILITY, this machine will do a
>   lot of small builds, and small builds have a much higher chance of
>   being successful than bigger builds. Therefore, such a machine could
>   easily reach a very high success rate of builds, compared to a
>   machine running with a higher KCONFIG_PROBABILITY value. Thoughts on
>   this? Or people better in statistics than me to confirm or infirm my
>   reasoning?
>

Could you clarify why we need a timeout in the first place? Have there been occurrences of builds that get stuck (in a loop or otherwise)?
According to me, it doesn't matter that a build takes ten hours for a given configuration, as long as it progresses and doesn't get stuck...

Best regards,
Thomas

[Buildroot] Analysis of build failures

[Thomas Petazzoni]

Dear Thomas De Schampheleire,

On Sun, 15 Jun 2014 12:36:44 +0200, Thomas De Schampheleire wrote:

> Could you clarify why we need a timeout in the first place? Have
> there been occurrences of builds that get stuck (in a loop or
> otherwise)? According to me, it doesn't matter that a build takes ten
> hours for a given configuration, as long as it progresses and doesn't
> get stuck...

The reason a timeout was introduced is because there used to be an old
PowerPC toolchain in which 'ld' had a bug, and this bug caused ld to
enter an infinite loop, consuming 100% of the CPU forever, when linking
a specific piece of code. There have been occurrences where my build
server has remained stuck for several days in this infinite loop before
I realized that the builds were no longer occurring, and figured out
what was going on.

I don't think we still have this toolchain tested in the current
configurations, but the timeout mechanism has remained in place, and I
believe it's still possible to have similar issues in the future.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Analysis of build failures

[Thomas De Schampheleire]

Hi Thomas,

On Sun, Jun 15, 2014 at 12:42 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Dear Thomas De Schampheleire,
>
> On Sun, 15 Jun 2014 12:36:44 +0200, Thomas De Schampheleire wrote:
>
>> Could you clarify why we need a timeout in the first place? Have
>> there been occurrences of builds that get stuck (in a loop or
>> otherwise)? According to me, it doesn't matter that a build takes ten
>> hours for a given configuration, as long as it progresses and doesn't
>> get stuck...
>
> The reason a timeout was introduced is because there used to be an old
> PowerPC toolchain in which 'ld' had a bug, and this bug caused ld to
> enter an infinite loop, consuming 100% of the CPU forever, when linking
> a specific piece of code. There have been occurrences where my build
> server has remained stuck for several days in this infinite loop before
> I realized that the builds were no longer occurring, and figured out
> what was going on.
>
> I don't think we still have this toolchain tested in the current
> configurations, but the timeout mechanism has remained in place, and I
> believe it's still possible to have similar issues in the future.
>

If we can agree that the situation is rare, than we could keep things
simple and make the timeout relatively large, say 10 hours. This
reduces the false positives due to big configurations, but still
prevent endless builds. In the good cases, no timeout is effectively
present and all builds can run to completion. In the exceptional bad
case of an endless loop, there is a large penalty of 10 hours (or
whatever the timeout value) but since this should only happen
exceptionally it is acceptable. These situations should be
investigated in detail anyway and resolved shortly after (ideally).

The other solutions you proposed have their value too, but make things
more complex while not really necessary IMO.

Best regards,
Thomas