[Buildroot] [PATCH 0/4] support: script to build-test packages

[Buildroot] [PATCH 0/4] support: script to build-test packages

[Yann E. MORIN]

Hello All!

As Thomas Petazzoni initially wrote (and slightly edited to match the
new script):

This has been often asked, and yesterday I needed it, so I did a quick
and dirty hack. The following script takes as argument a defconfig
fragment that defines a bunch of Buildroot configuration options, and
will built this fragment against all the toolchain configurations
tested in the autobuilders.

Example:

    $ cat template
    BR2_PACKAGE_JACK2=y
    BR2_PACKAGE_JACK2_LEGACY=y
    BR2_PACKAGE_JACK2_DBUS=y
    $ ./support/scripts/test-pkg -c template -d ~/br-test-jack2 -p jack2
             armv5-ctng-linux-gnueabi : download config, olddefconfig, dirclean, build: OK
           armv7-ctng-linux-gnueabihf : download config, olddefconfig, dirclean, build: OK
                     br-aarch64-glibc : download config, olddefconfig, dirclean, build: OK
                        br-arcle-hs38 : download config, olddefconfig, dirclean, build: OK
                         br-arm-basic : download config, olddefconfig, SKIP
               br-arm-cortex-a9-glibc : download config, olddefconfig, dirclean, build: OK
                br-arm-cortex-a9-musl : download config, olddefconfig, dirclean, build: OK
                br-arm-cortex-m4-full : download config, olddefconfig, SKIP
                          br-arm-full : download config, olddefconfig, dirclean, build: OK
                 br-arm-full-nothread : download config, olddefconfig, SKIP
                   br-arm-full-static : download config, olddefconfig, SKIP
                         br-bfin-full : download config, olddefconfig, SKIP
                br-i386-pentium4-full : download config, olddefconfig, dirclean, build: OK
             br-i386-pentium-mmx-musl : download config, olddefconfig, dirclean, build: OK
                    br-m68k-5208-full : download config, olddefconfig, SKIP
                   br-m68k-68040-full : download config, olddefconfig, dirclean, build: OK
                 br-microblazeel-full : download config, olddefconfig, dirclean, build: OK
                   br-mips64-n64-full : download config, olddefconfig, dirclean, build: OK
              br-mips32r6-el-hf-glibc : download config, olddefconfig, dirclean, build: OK
              br-mips64r6-el-hf-glibc : download config, olddefconfig, dirclean, build: OK
                   br-mipsel-o32-full : download config, olddefconfig, dirclean, build: OK
                       br-nios2-glibc : download config, olddefconfig, dirclean, build: OK
            br-powerpc-603e-basic-cpp : download config, olddefconfig, SKIP
          br-powerpc64le-power8-glibc : download config, olddefconfig, dirclean, build: OK
            br-powerpc64-power7-glibc : download config, olddefconfig, dirclean, build: OK
               br-powerpc-e500mc-full : download config, olddefconfig, dirclean, build: OK
                          br-sh4-full : download config, olddefconfig, dirclean, build: OK
                      br-sparc-uclibc : download config, olddefconfig, SKIP
                     br-sparc64-glibc : download config, olddefconfig, dirclean, build: OK
                 br-x86-64-core2-full : download config, olddefconfig, dirclean, build: OK
                       br-x86-64-musl : download config, olddefconfig, dirclean, build: OK
                       br-xtensa-full : download config, olddefconfig, dirclean, build: OK
                  i686-ctng-linux-gnu : download config, olddefconfig, dirclean, build: OK
                       linaro-aarch64 : download config, olddefconfig, dirclean, build: FAIL
                           linaro-arm : download config, olddefconfig, dirclean, build: FAIL
          mips64el-ctng_n32-linux-gnu : download config, olddefconfig, dirclean, build: OK
          mips64el-ctng_n64-linux-gnu : download config, olddefconfig, dirclean, build: FAIL
     powerpc-ctng_e500v2-linux-gnuspe : download config, olddefconfig, dirclean, build: OK
                  sourcery-arm-armv4t : download config, olddefconfig, dirclean, build: OK
                         sourcery-arm : download config, olddefconfig, dirclean, build: OK
                  sourcery-arm-thumb2 : download config, olddefconfig, dirclean, build: OK
                      sourcery-mips64 : download config, olddefconfig, dirclean, build: OK
                        sourcery-mips : download config, olddefconfig, dirclean, build: OK
                       sourcery-nios2 : download config, olddefconfig, dirclean, build: OK
                          sourcery-sh : download config, olddefconfig, dirclean, build: FAIL
                      sourcery-x86-64 : download config, olddefconfig, dirclean, build: OK
                         sourcery-x86 : download config, olddefconfig, dirclean, build: OK
        x86_64-ctng_locales-linux-gnu : download config, olddefconfig, dirclean, build: OK
    8 configurations were skipped
    4 configurations would not build

So, the jack2 package, with both its "legacy" and "dbus" backends has
been built in all those configurations.

It says "OK" when the build was successful, "FAIL" when it failed and
"SKIP" when for some reason the options could not be enabled for the
current toolchain configuration (for example if this package is
disabled on this architecture, or if a toolchain feature is missing,
etc.).

Note that the script excludes the "internal toolchain" configurations,
because they take too long to build.

One nice trick is that you can re-run the script, and it will simply
restart the build. So after fixing your package, if you want to test it
again, you don't have to clean everything and restart from scratch:
just remove the build directory of your package in all output
directories, and restart the script.


Regards,
Yann E. MORIN.


The following changes since commit c03f112b63327409c6ebb301d2fa8d873233e0da

  python-service-identity: bump version to 16.0.0 (2017-02-07 14:25:23 +0100)


are available in the git repository at:

  git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to 3a2dfc3a5dcf4389d2b65b91de95124385241a18

  supprt/test-pkg: add option to limit the number of tests (2017-02-07 14:41:50 +0100)


----------------------------------------------------------------
Yann E. MORIN (4):
      support/scripts: add script to test a package
      support/test-pkg: store lines missing from resulting configuraiton
      support/test-pkg: report number and types of failures
      supprt/test-pkg: add option to limit the number of tests

 support/scripts/test-pkg | 214 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 214 insertions(+)
 create mode 100755 support/scripts/test-pkg

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/4] support/scripts: add script to test a package

[Yann E. MORIN]

This script helps in testing that a pacakge builds fine on a wide range
of architectures and toolchains: BE/LE, 32/64-bit, musl/glibc/uclibc...

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
[yann.morin.1998 at free.fr:
 - completely rewrite the script from Thomas, with help from Luca
]
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
---
 support/scripts/test-pkg | 164 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 164 insertions(+)
 create mode 100755 support/scripts/test-pkg

diff --git a/support/scripts/test-pkg b/support/scripts/test-pkg
new file mode 100755
index 0000000..daebe62
--- /dev/null
+++ b/support/scripts/test-pkg
@@ -0,0 +1,164 @@
+#!/bin/bash
+set -e
+
+TOOLCHAINS_BASE_URL='http://autobuild.buildroot.org/toolchains/configs'
+
+main() {
+    local o O opts
+    local cfg dir pkg toolchain
+    local -a toolchains
+
+    o='hc:d:p:'
+    O='help,config-snippet:build-dir:package:'
+    opts="$( getopt -n "${my_name}" -o "${o}" -l "${O}" -- "${@}"  )"
+    eval set -- "${opts}"
+
+    while [ ${#} -gt 0 ]; do
+        case "${1}" in
+        (-h|--help)
+            help; exit 0
+            ;;
+        (-c|--config-snippet)
+            cfg="${2}"; shift 2
+            ;;
+        (-d|--build-dir)
+            dir="${2}"; shift 2
+            ;;
+        (-p|--package)
+            pkg="${2}"; shift 2
+            ;;
+        (--)
+            shift; break
+            ;;
+        esac
+    done
+    if [ -z "${cfg}" ]; then
+        printf "error: no config snippet specified\n" >&2; exit 1
+    fi
+    if [ -z "${dir}" ]; then
+        dir="${HOME}/br-test-pkg"
+    fi
+
+    # Extract the names of the toolchains
+    # http://server/path/to/name.config,arch,libc  -->  name
+    toolchains=( $( curl -s "${TOOLCHAINS_BASE_URL}/toolchain-configs.csv" \
+                    |sed -r -e 's/,.*//; s:.*/(.*)\.config:\1:; /internal/d;'
+                  )
+               )
+
+    if [ ${#toolchains[@]} -eq 0 ]; then
+        printf "error: no toolchain found (networking issue?)\n" >&2; exit 1
+    fi
+
+    for toolchain in "${toolchains[@]}"; do
+        build_one "${dir}" "${toolchain}" "${cfg}" "${pkg}"
+    done
+}
+
+build_one() {
+    local dir="${1}"
+    local toolchain="${2}"
+    local cfg="${3}"
+    local pkg="${4}"
+    local line
+
+    printf "%40s: " "${toolchain}"
+
+    dir="${dir}/${toolchain}"
+    mkdir -p "${dir}"
+
+    printf "download config"
+    if ! curl -s "${TOOLCHAINS_BASE_URL}/${toolchain}.config" >"${dir}/.config"; then
+        printf ": FAILED\n"
+        return
+    fi
+
+    cat >>"${dir}/.config" <<-_EOF_
+	BR2_INIT_NONE=y
+	BR2_SYSTEM_BIN_SH_NONE=y
+	# BR2_PACKAGE_BUSYBOX is not set
+	# BR2_TARGET_ROOTFS_TAR is not set
+	_EOF_
+    cat "${cfg}" >>"${dir}/.config"
+
+    printf ", olddefconfig"
+    if ! make O="${dir}" olddefconfig >/dev/null 2>&1; then
+        printf ": FAILED\n"
+        return
+    fi
+    # We want all the options from the snippet to be present as-is (set
+    # or not set) in the actual .config; if one of them is not, it means
+    # some dependency from the toolchain or arch is not available, in
+    # which case this config is untestable and we skip it.
+    while read line; do
+        if ! grep "^${line}\$" "${dir}/.config" >/dev/null 2>&1; then
+            printf ", SKIPPED\n"
+            return
+        fi
+    done <"${cfg}"
+
+    if [ -n "${pkg}" ]; then
+        printf ", dirclean"
+        if ! make O="${dir}" "${pkg}-dirclean" >> "${dir}/logfile" 2>&1; then
+            printf ": FAILED\n"
+            return
+        fi
+    fi
+
+    printf ", build"
+    # shellcheck disable=SC2086
+    if ! make O="${dir}" ${pkg} >> "${dir}/logfile" 2>&1; then
+        printf ": FAILED\n"
+        return
+    fi
+
+    printf ": OK\n"
+}
+
+help() {
+    cat <<_EOF_
+test-pkg: test-build a package against various toolchains and architectures
+
+The supplied config snippet is appended to each toolchain config, the
+resulting configuration is checked that it still contains all options
+specified in the snippet; if any is missing, the build is skipped, on the
+assumption that the package under test requires a toolchain or architecture
+feature that is missing.
+
+The list of toolchains is retrieved from Buildroot autobuilders, available
+at ${TOOLCHAINS_BASE_URL}.
+
+In case failures are noticed, you can fix the package and just re-run the
+same command again; it will re-run the test where it failed. If you did
+specify a package (with -p), the package build dir will be removed first.
+
+Options
+
+    -h, --help
+        Print this help.
+
+    -c CFG, --config-snippet CFG
+        Use the CFG file as the source for the config snippet. This file
+        should contain all the config options required to build a package.
+
+    -d DIR, --build-dir DIR
+        Do the builds in directory DIR, one sub-dir per toolchain.
+
+    -p PKG, --package PKG
+        Test-build the package PKG, by running 'make PKG'; if not specified,
+        just runs 'make'.
+
+Example:
+
+    Testing libcec would require a config snippet that contains:
+        BR2_PACKAGE_LIBCEC=y
+
+    Testing libcurl with openSSL support would require a snippet such as:
+        BR2_PACKAGE_OPENSSL=y
+        BR2_PACKAGE_LIBCURL=y
+
+_EOF_
+}
+
+my_name="${0##*/}"
+main "${@}"
-- 
2.7.4

[Buildroot] [PATCH 2/4] support/test-pkg: store lines missing from resulting configuraiton

[Yann E. MORIN]

When a build is skipped, store the lines from the config snippet, that
are missing in the resulting configuration, in a fie in the build
directory, for the user to inspect.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
---
 support/scripts/test-pkg | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/support/scripts/test-pkg b/support/scripts/test-pkg
index daebe62..67f73b3 100755
--- a/support/scripts/test-pkg
+++ b/support/scripts/test-pkg
@@ -60,7 +60,8 @@ build_one() {
     local toolchain="${2}"
     local cfg="${3}"
     local pkg="${4}"
-    local line
+    local line skip
+    local -a missing_lines
 
     printf "%40s: " "${toolchain}"
 
@@ -90,12 +91,20 @@ build_one() {
     # or not set) in the actual .config; if one of them is not, it means
     # some dependency from the toolchain or arch is not available, in
     # which case this config is untestable and we skip it.
+    skip=false
     while read line; do
         if ! grep "^${line}\$" "${dir}/.config" >/dev/null 2>&1; then
-            printf ", SKIPPED\n"
-            return
+            missing_lines+=( "${line}" )
+            skip=true
         fi
     done <"${cfg}"
+    if ${skip}; then
+        for line in "${missing_lines[@]}"; do
+            printf "%s\n" "${line}"
+        done >"${dir}/missing.config"
+        printf ", SKIPPED\n"
+        return
+    fi
 
     if [ -n "${pkg}" ]; then
         printf ", dirclean"
-- 
2.7.4

[Buildroot] [PATCH 3/4] support/test-pkg: report number and types of failures

[Yann E. MORIN]

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
---
 support/scripts/test-pkg | 42 ++++++++++++++++++++++++++++++++++++------
 1 file changed, 36 insertions(+), 6 deletions(-)

diff --git a/support/scripts/test-pkg b/support/scripts/test-pkg
index 67f73b3..604a759 100755
--- a/support/scripts/test-pkg
+++ b/support/scripts/test-pkg
@@ -6,6 +6,7 @@ TOOLCHAINS_BASE_URL='http://autobuild.buildroot.org/toolchains/configs'
 main() {
     local o O opts
     local cfg dir pkg toolchain
+    local ret nb_dl nb_cfg nb_skip nb_clean nb_build
     local -a toolchains
 
     o='hc:d:p:'
@@ -50,9 +51,38 @@ main() {
         printf "error: no toolchain found (networking issue?)\n" >&2; exit 1
     fi
 
+    nb_dl=0
+    nb_cfg=0
+    nb_skip=0
+    nb_clean=0
+    nb_build=0
     for toolchain in "${toolchains[@]}"; do
-        build_one "${dir}" "${toolchain}" "${cfg}" "${pkg}"
+        build_one "${dir}" "${toolchain}" "${cfg}" "${pkg}" && ret=0 || ret=${?}
+        case ${ret} in
+        (0) ;;
+        (1) : $((nb_dl++));;
+        (2) : $((nb_cfg++));;
+        (3) : $((nb_skip++));;
+        (4) : $((nb_clean++));;
+        (5) : $((nb_build++));;
+        esac
     done
+
+    if [ ${nb_dl} -ne 0 ]; then
+        printf "%d configurations could not be downloaded\n" ${nb_dl}
+    fi
+    if [ ${nb_cfg} -ne 0 ]; then
+        printf "%d configurations could not be applied\n" ${nb_cfg}
+    fi
+    if [ ${nb_skip} -ne 0 ]; then
+        printf "%d configurations were skipped\n" ${nb_skip}
+    fi
+    if [ ${nb_clean} -ne 0 ]; then
+        printf "%d configurations could not be dircleaned\n" ${nb_clean}
+    fi
+    if [ ${nb_build} -ne 0 ]; then
+        printf "%d configurations would not build\n" ${nb_build}
+    fi
 }
 
 build_one() {
@@ -71,7 +101,7 @@ build_one() {
     printf "download config"
     if ! curl -s "${TOOLCHAINS_BASE_URL}/${toolchain}.config" >"${dir}/.config"; then
         printf ": FAILED\n"
-        return
+        return 1
     fi
 
     cat >>"${dir}/.config" <<-_EOF_
@@ -85,7 +115,7 @@ build_one() {
     printf ", olddefconfig"
     if ! make O="${dir}" olddefconfig >/dev/null 2>&1; then
         printf ": FAILED\n"
-        return
+        return 2
     fi
     # We want all the options from the snippet to be present as-is (set
     # or not set) in the actual .config; if one of them is not, it means
@@ -103,14 +133,14 @@ build_one() {
             printf "%s\n" "${line}"
         done >"${dir}/missing.config"
         printf ", SKIPPED\n"
-        return
+        return 3
     fi
 
     if [ -n "${pkg}" ]; then
         printf ", dirclean"
         if ! make O="${dir}" "${pkg}-dirclean" >> "${dir}/logfile" 2>&1; then
             printf ": FAILED\n"
-            return
+            return 4
         fi
     fi
 
@@ -118,7 +148,7 @@ build_one() {
     # shellcheck disable=SC2086
     if ! make O="${dir}" ${pkg} >> "${dir}/logfile" 2>&1; then
         printf ": FAILED\n"
-        return
+        return 5
     fi
 
     printf ": OK\n"
-- 
2.7.4

[Buildroot] [PATCH 4/4] supprt/test-pkg: add option to limit the number of tests

[Yann E. MORIN]

Sometimes, it interesting to have a global overview of whether the
package builds at all or not, rather than test on all toolchains.

Add an option that allows testing on a limited set of randomly choosen
toolchains.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
---
 support/scripts/test-pkg | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)

diff --git a/support/scripts/test-pkg b/support/scripts/test-pkg
index 604a759..0b4615d 100755
--- a/support/scripts/test-pkg
+++ b/support/scripts/test-pkg
@@ -5,15 +5,16 @@ TOOLCHAINS_BASE_URL='http://autobuild.buildroot.org/toolchains/configs'
 
 main() {
     local o O opts
-    local cfg dir pkg toolchain
+    local cfg dir pkg random toolchain
     local ret nb_dl nb_cfg nb_skip nb_clean nb_build
     local -a toolchains
 
-    o='hc:d:p:'
-    O='help,config-snippet:build-dir:package:'
+    o='hc:d:p:r:'
+    O='help,config-snippet:build-dir:package:,random::'
     opts="$( getopt -n "${my_name}" -o "${o}" -l "${O}" -- "${@}"  )"
     eval set -- "${opts}"
 
+    random=0
     while [ ${#} -gt 0 ]; do
         case "${1}" in
         (-h|--help)
@@ -28,6 +29,9 @@ main() {
         (-p|--package)
             pkg="${2}"; shift 2
             ;;
+        (-r|--random)
+            random="${2}"; shift 2
+            ;;
         (--)
             shift; break
             ;;
@@ -43,7 +47,10 @@ main() {
     # Extract the names of the toolchains
     # http://server/path/to/name.config,arch,libc  -->  name
     toolchains=( $( curl -s "${TOOLCHAINS_BASE_URL}/toolchain-configs.csv" \
-                    |sed -r -e 's/,.*//; s:.*/(.*)\.config:\1:; /internal/d;'
+                    |sed -r -e 's/,.*//; s:.*/(.*)\.config:\1:; /internal/d;' \
+                    |if [ ${random} -gt 0 ]; then \
+                        sort -R |head -n ${random} |sort
+                     fi
                   )
                )
 
@@ -187,6 +194,10 @@ Options
         Test-build the package PKG, by running 'make PKG'; if not specified,
         just runs 'make'.
 
+    -r N, --random N
+        Limit the tests to the N randomly selected toolchains, instead of
+        building all.
+
 Example:
 
     Testing libcec would require a config snippet that contains:
-- 
2.7.4

[Buildroot] [PATCH 4/4] supprt/test-pkg: add option to limit the number of tests

[Yann E. MORIN]

All,

On 2017-02-07 14:53 +0100, Yann E. MORIN spake thusly:
> Sometimes, it interesting to have a global overview of whether the
> package builds at all or not, rather than test on all toolchains.
> 
> Add an option that allows testing on a limited set of randomly choosen
> toolchains.

Forget that patch, there is an issue when the option is not provided. :-(

Regards,
Yann E. MORIN.

> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Luca Ceresoli <luca@lucaceresoli.net>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> ---
>  support/scripts/test-pkg | 19 +++++++++++++++----
>  1 file changed, 15 insertions(+), 4 deletions(-)
> 
> diff --git a/support/scripts/test-pkg b/support/scripts/test-pkg
> index 604a759..0b4615d 100755
> --- a/support/scripts/test-pkg
> +++ b/support/scripts/test-pkg
> @@ -5,15 +5,16 @@ TOOLCHAINS_BASE_URL='http://autobuild.buildroot.org/toolchains/configs'
>  
>  main() {
>      local o O opts
> -    local cfg dir pkg toolchain
> +    local cfg dir pkg random toolchain
>      local ret nb_dl nb_cfg nb_skip nb_clean nb_build
>      local -a toolchains
>  
> -    o='hc:d:p:'
> -    O='help,config-snippet:build-dir:package:'
> +    o='hc:d:p:r:'
> +    O='help,config-snippet:build-dir:package:,random::'
>      opts="$( getopt -n "${my_name}" -o "${o}" -l "${O}" -- "${@}"  )"
>      eval set -- "${opts}"
>  
> +    random=0
>      while [ ${#} -gt 0 ]; do
>          case "${1}" in
>          (-h|--help)
> @@ -28,6 +29,9 @@ main() {
>          (-p|--package)
>              pkg="${2}"; shift 2
>              ;;
> +        (-r|--random)
> +            random="${2}"; shift 2
> +            ;;
>          (--)
>              shift; break
>              ;;
> @@ -43,7 +47,10 @@ main() {
>      # Extract the names of the toolchains
>      # http://server/path/to/name.config,arch,libc  -->  name
>      toolchains=( $( curl -s "${TOOLCHAINS_BASE_URL}/toolchain-configs.csv" \
> -                    |sed -r -e 's/,.*//; s:.*/(.*)\.config:\1:; /internal/d;'
> +                    |sed -r -e 's/,.*//; s:.*/(.*)\.config:\1:; /internal/d;' \
> +                    |if [ ${random} -gt 0 ]; then \
> +                        sort -R |head -n ${random} |sort
> +                     fi
>                    )
>                 )
>  
> @@ -187,6 +194,10 @@ Options
>          Test-build the package PKG, by running 'make PKG'; if not specified,
>          just runs 'make'.
>  
> +    -r N, --random N
> +        Limit the tests to the N randomly selected toolchains, instead of
> +        building all.
> +
>  Example:
>  
>      Testing libcec would require a config snippet that contains:
> -- 
> 2.7.4
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/4] support/scripts: add script to test a package

[Luca Ceresoli]

Hi,

On 07/02/2017 14:52, Yann E. MORIN wrote:
> This script helps in testing that a pacakge builds fine on a wide range
> of architectures and toolchains: BE/LE, 32/64-bit, musl/glibc/uclibc...
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> [yann.morin.1998 at free.fr:
>  - completely rewrite the script from Thomas, with help from Luca
> ]
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Luca Ceresoli <luca@lucaceresoli.net>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>

Just two very minor nits about wording. For the rest the diff from the
version I already reviewed is very limited anyway.

[...]

> +help() {
> +    cat <<_EOF_
> +test-pkg: test-build a package against various toolchains and architectures
> +
> +The supplied config snippet is appended to each toolchain config, the
> +resulting configuration is checked that it still contains all options

"is checked that it still contains" seems wrong to me.
I'd reword it as "is checked to ensure it still contains".

> +specified in the snippet; if any is missing, the build is skipped, on the
> +assumption that the package under test requires a toolchain or architecture
> +feature that is missing.
> +
> +The list of toolchains is retrieved from Buildroot autobuilders, available

from Buildroot autobuilders -> from the Buildroot autobuilders

With these fixed:
Acked-by: Luca Ceresoli <luca@lucaceresoli.net>

-- 
Luca

[Buildroot] [PATCH 2/4] support/test-pkg: store lines missing from resulting configuraiton

[Luca Ceresoli]

Hi,

On 07/02/2017 14:52, Yann E. MORIN wrote:
> When a build is skipped, store the lines from the config snippet, that
> are missing in the resulting configuration, in a fie in the build

fie -> file

> directory, for the user to inspect.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Luca Ceresoli <luca@lucaceresoli.net>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> ---
>  support/scripts/test-pkg | 15 ++++++++++++---
>  1 file changed, 12 insertions(+), 3 deletions(-)
> 
> diff --git a/support/scripts/test-pkg b/support/scripts/test-pkg
> index daebe62..67f73b3 100755
> --- a/support/scripts/test-pkg
> +++ b/support/scripts/test-pkg
> @@ -60,7 +60,8 @@ build_one() {
>      local toolchain="${2}"
>      local cfg="${3}"
>      local pkg="${4}"
> -    local line
> +    local line skip
> +    local -a missing_lines
>  
>      printf "%40s: " "${toolchain}"
>  
> @@ -90,12 +91,20 @@ build_one() {
>      # or not set) in the actual .config; if one of them is not, it means
>      # some dependency from the toolchain or arch is not available, in
>      # which case this config is untestable and we skip it.
> +    skip=false
>      while read line; do
>          if ! grep "^${line}\$" "${dir}/.config" >/dev/null 2>&1; then
> -            printf ", SKIPPED\n"
> -            return
> +            missing_lines+=( "${line}" )
> +            skip=true
>          fi
>      done <"${cfg}"
> +    if ${skip}; then
> +        for line in "${missing_lines[@]}"; do
> +            printf "%s\n" "${line}"
> +        done >"${dir}/missing.config"
> +        printf ", SKIPPED\n"
> +        return
> +    fi

Nice feature, but the array-based implementation seems a more complex
than needed.

What about:

+    skip=false
     rm -f "${dir}/missing.config"
     while read line; do
         if ! grep "^${line}\$" "${dir}/.config" >/dev/null 2>&1; then
-            printf ", SKIPPED\n"
-            return
+            echo "${line}" >>"${dir}/missing.config"
+            skip=true
         fi
     done <"${cfg}"
+    if ${skip}; then
+        printf ", SKIPPED\n"
+        return
+    fi

This is more concise, does not use arrays (so it's easier to understand
to shell newbies). It also has the effect of removing the missing.config
file@the next iteration, so you won't find it there when the config
step succeeded.

-- 
Luca

[Buildroot] [PATCH 3/4] support/test-pkg: report number and types of failures

[Luca Ceresoli]

Hi,

On 07/02/2017 14:52, Yann E. MORIN wrote:
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Luca Ceresoli <luca@lucaceresoli.net>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> ---
>  support/scripts/test-pkg | 42 ++++++++++++++++++++++++++++++++++++------
>  1 file changed, 36 insertions(+), 6 deletions(-)
> 
> diff --git a/support/scripts/test-pkg b/support/scripts/test-pkg
> index 67f73b3..604a759 100755
> --- a/support/scripts/test-pkg
> +++ b/support/scripts/test-pkg
> @@ -6,6 +6,7 @@ TOOLCHAINS_BASE_URL='http://autobuild.buildroot.org/toolchains/configs'
>  main() {
>      local o O opts
>      local cfg dir pkg toolchain
> +    local ret nb_dl nb_cfg nb_skip nb_clean nb_build
>      local -a toolchains
>  
>      o='hc:d:p:'
> @@ -50,9 +51,38 @@ main() {
>          printf "error: no toolchain found (networking issue?)\n" >&2; exit 1
>      fi
>  
> +    nb_dl=0
> +    nb_cfg=0
> +    nb_skip=0
> +    nb_clean=0
> +    nb_build=0
>      for toolchain in "${toolchains[@]}"; do
> -        build_one "${dir}" "${toolchain}" "${cfg}" "${pkg}"
> +        build_one "${dir}" "${toolchain}" "${cfg}" "${pkg}" && ret=0 || ret=${?}
> +        case ${ret} in
> +        (0) ;;
> +        (1) : $((nb_dl++));;
> +        (2) : $((nb_cfg++));;
> +        (3) : $((nb_skip++));;
> +        (4) : $((nb_clean++));;
> +        (5) : $((nb_build++));;
> +        esac
>      done
> +
> +    if [ ${nb_dl} -ne 0 ]; then
> +        printf "%d configurations could not be downloaded\n" ${nb_dl}
> +    fi
> +    if [ ${nb_cfg} -ne 0 ]; then
> +        printf "%d configurations could not be applied\n" ${nb_cfg}
> +    fi
> +    if [ ${nb_skip} -ne 0 ]; then
> +        printf "%d configurations were skipped\n" ${nb_skip}
> +    fi
> +    if [ ${nb_clean} -ne 0 ]; then
> +        printf "%d configurations could not be dircleaned\n" ${nb_clean}
> +    fi
> +    if [ ${nb_build} -ne 0 ]; then
> +        printf "%d configurations would not build\n" ${nb_build}
> +    fi

The script gets pretty verbose, so it would be nice if we had a way to
avoid the lines that are almost equal. Maybe an associative array, heh.
But in the end it would probably result in poor readability, so:

Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>

-- 
Luca

[Buildroot] [PATCH 1/4] support/scripts: add script to test a package

[Thomas De Schampheleire]

On Tue, Feb 7, 2017 at 4:18 PM, Luca Ceresoli <lucaceresoli77@gmail.com> wrote:
> Hi,
>
> On 07/02/2017 14:52, Yann E. MORIN wrote:
>> This script helps in testing that a pacakge builds fine on a wide range
>> of architectures and toolchains: BE/LE, 32/64-bit, musl/glibc/uclibc...
>>
>> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
>> [yann.morin.1998 at free.fr:
>>  - completely rewrite the script from Thomas, with help from Luca
>> ]
>> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
>> Cc: Luca Ceresoli <luca@lucaceresoli.net>
>> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
>
> Just two very minor nits about wording. For the rest the diff from the
> version I already reviewed is very limited anyway.
>
> [...]
>
>> +help() {
>> +    cat <<_EOF_
>> +test-pkg: test-build a package against various toolchains and architectures
>> +
>> +The supplied config snippet is appended to each toolchain config, the
>> +resulting configuration is checked that it still contains all options
>
> "is checked that it still contains" seems wrong to me.
> I'd reword it as "is checked to ensure it still contains".
>
>> +specified in the snippet; if any is missing, the build is skipped, on the
>> +assumption that the package under test requires a toolchain or architecture
>> +feature that is missing.
>> +
>> +The list of toolchains is retrieved from Buildroot autobuilders, available
>
> from Buildroot autobuilders -> from the Buildroot autobuilders
>
> With these fixed:
> Acked-by: Luca Ceresoli <luca@lucaceresoli.net>
>


With suggestions from Luca fixed:
Reviewed-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>