[Buildroot] [PATCH 1/3] .gitlab-ci.yml: run our runtime tests

[Buildroot] [PATCH 1/3] .gitlab-ci.yml: run our runtime tests

[Arnout Vandecappelle]

From: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

This commit improves our .gitlab-ci.yml logic to execute our runtime
tests located in support/testing/. To do so, this commit:

 - Adds more Debian packages to be installed, namely the nose2 and
   pexpect packages needed by the runtime testing infrastructure, as
   well as the necessary Qemu emulators

 - The description of how to run the runtime tests. Each test is
   executed as a separate Gitlab CI job, so that the status of each
   test is easily visible in the Gitlab CI web interface.

 - The Makefile is improved to auto-generate .gitlab-ci.yml from
   .gitlab-ci.yml.in, like we're doing for defconfigs. Since the
   dependencies of .gitlab-ci.yml are no longer correct, we abandon
   them and instead make it a PHONY target.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
[Arnout: simplify .gitlab-ci.in a little, removing redundant stuff;
         make .gitlab-ci.yml a PHONY target]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 .gitlab-ci.yml    | 33 +++++++++++++++++++++++++++++++++
 .gitlab-ci.yml.in |  8 ++++++++
 Makefile          |  7 +++++--
 3 files changed, 46 insertions(+), 2 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index eda8af4997..7acf4b620e 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -13,6 +13,7 @@ before_script:
     - apt-get install -y -qq --no-install-recommends
         build-essential locales bc ca-certificates file rsync gcc-multilib
         git bzr cvs mercurial subversion libc6:i386 unzip wget cpio
+        python-nose2 python-pexpect qemu-system-arm qemu-system-x86
     # To be able to generate a toolchain with locales, enable one UTF-8 locale
     - sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen
     - /usr/sbin/locale-gen
@@ -57,6 +58,13 @@ check-DEVELOPERS:
             - output/build/build-time.log
             - output/build/packages-file-list.txt
 
+.runtime_test: &runtime_test
+    script: ./support/testing/run-tests -o test-output/ -d test-dl/ ${CI_BUILD_NAME}
+    artifacts:
+        when: always
+        expire_in: 2 weeks
+        paths:
+            - test-output/*.log
 acmesystems_aria_g25_128mb_defconfig: *defconfig
 acmesystems_aria_g25_256mb_defconfig: *defconfig
 acmesystems_arietta_g25_128mb_defconfig: *defconfig
@@ -209,3 +217,28 @@ zynq_microzed_defconfig: *defconfig
 zynq_zc706_defconfig: *defconfig
 zynq_zed_defconfig: *defconfig
 zynq_zybo_defconfig: *defconfig
+tests.core.test_post_scripts.TestPostScripts: *runtime_test
+tests.core.test_rootfs_overlay.TestRootfsOverlay: *runtime_test
+tests.core.test_timezone.TestGlibcAllTimezone: *runtime_test
+tests.core.test_timezone.TestGlibcNonDefaultLimitedTimezone: *runtime_test
+tests.core.test_timezone.TestNoTimezone: *runtime_test
+tests.fs.test_ext.TestExt2: *runtime_test
+tests.fs.test_ext.TestExt2r1: *runtime_test
+tests.fs.test_ext.TestExt3: *runtime_test
+tests.fs.test_ext.TestExt4: *runtime_test
+tests.fs.test_iso9660.TestIso9660Grub2External: *runtime_test
+tests.fs.test_iso9660.TestIso9660Grub2Internal: *runtime_test
+tests.fs.test_iso9660.TestIso9660GrubExternal: *runtime_test
+tests.fs.test_iso9660.TestIso9660GrubInternal: *runtime_test
+tests.fs.test_iso9660.TestIso9660SyslinuxExternal: *runtime_test
+tests.fs.test_iso9660.TestIso9660SyslinuxInternal: *runtime_test
+tests.fs.test_jffs2.TestJffs2: *runtime_test
+tests.fs.test_squashfs.TestSquashfs: *runtime_test
+tests.fs.test_ubi.TestUbi: *runtime_test
+tests.fs.test_yaffs2.TestYaffs2: *runtime_test
+tests.package.test_dropbear.TestDropbear: *runtime_test
+tests.package.test_python.TestPythonBase: *runtime_test
+tests.toolchain.test_external.TestExternalToolchainLinaroArm: *runtime_test
+tests.toolchain.test_external.TestExternalToolchainSourceryArmv4: *runtime_test
+tests.toolchain.test_external.TestExternalToolchainSourceryArmv5: *runtime_test
+tests.toolchain.test_external.TestExternalToolchainSourceryArmv7: *runtime_test
diff --git a/.gitlab-ci.yml.in b/.gitlab-ci.yml.in
index 24a1888c1e..c69f9de99c 100644
--- a/.gitlab-ci.yml.in
+++ b/.gitlab-ci.yml.in
@@ -13,6 +13,7 @@ before_script:
     - apt-get install -y -qq --no-install-recommends
         build-essential locales bc ca-certificates file rsync gcc-multilib
         git bzr cvs mercurial subversion libc6:i386 unzip wget cpio
+        python-nose2 python-pexpect qemu-system-arm qemu-system-x86
     # To be able to generate a toolchain with locales, enable one UTF-8 locale
     - sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen
     - /usr/sbin/locale-gen
@@ -57,3 +58,10 @@ check-DEVELOPERS:
             - output/build/build-time.log
             - output/build/packages-file-list.txt
 
+.runtime_test: &runtime_test
+    script: ./support/testing/run-tests -o test-output/ -d test-dl/ ${CI_BUILD_NAME}
+    artifacts:
+        when: always
+        expire_in: 2 weeks
+        paths:
+            - test-output/*.log
diff --git a/Makefile b/Makefile
index 811158295a..36231fa9d6 100644
--- a/Makefile
+++ b/Makefile
@@ -1105,8 +1105,11 @@ release:
 print-version:
 	@echo $(BR2_VERSION_FULL)
 
-.gitlab-ci.yml: .gitlab-ci.yml.in configs/*_defconfig
-	(cd configs; LC_ALL=C ls -1 *_defconfig) | sed 's/$$/: *defconfig/' | cat $< - > $@
+.PHONY: .gitlab-ci.yml
+.gitlab-ci.yml: .gitlab-ci.yml.in
+	cp $< $@
+	(cd configs; LC_ALL=C ls -1 *_defconfig) | sed 's/$$/: *defconfig/' >> $@
+	./support/testing/run-tests -l 2>&1 | sed -r -e '/^test_run \((.*)\).*/!d; s//\1: *runtime_test/' | LC_ALL=C sort >> $@
 
 include docs/manual/manual.mk
 -include $(foreach dir,$(BR2_EXTERNAL_DIRS),$(dir)/docs/*/*.mk)
-- 
2.13.2

[Buildroot] [PATCH 2/3] Add Dockerfile for CI

[Arnout Vandecappelle]

For Gitlab-CI, we want to avoid re-generating the minimal install to
be able to run tests all the time. So let's create a docker image that
we can post on Docker Hub and then pull.

For the time being, this is just what we need for running our CI. Later
we can produce something that is also useful for users.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 support/docker/Dockerfile | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
 create mode 100644 support/docker/Dockerfile

diff --git a/support/docker/Dockerfile b/support/docker/Dockerfile
new file mode 100644
index 0000000000..d45b23dc23
--- /dev/null
+++ b/support/docker/Dockerfile
@@ -0,0 +1,30 @@
+# This Dockerfile generates the docker image that gets used by Gitlab CI
+# To build it:
+#   sudo docker build -t buildroot/base support/docker
+#   sudo docker push buildroot/base
+
+FROM debian:stable
+LABEL maintainer="Buildroot mailing list <buildroot@buildroot.org>" \
+      vendor="Buildroot" \
+description="Container with everything needed to run Buildroot"
+
+# Setup environment
+ENV DEBIAN_FRONTEND noninteractive
+
+RUN dpkg --add-architecture i386
+# The container has no package lists, so need to update first
+RUN apt-get update -y -qq
+RUN apt-get install -y -qq --no-install-recommends \
+    build-essential cmake libc6:i386 gcc-multilib \
+    bc ca-certificates file locales rsync \
+    cvs bzr git mercurial subversion wget \
+    cpio unzip \
+    libncurses5-dev \
+    python-nose2 python-pexpect qemu-system-arm qemu-system-x86
+RUN apt-get -q -y autoremove
+RUN apt-get -q -y clean
+
+# To be able to generate a toolchain with locales, enable one UTF-8 locale
+RUN sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen
+RUN /usr/sbin/locale-gen
+
-- 
2.13.2

[Buildroot] [PATCH 3/3] .gitlab-ci.yml: use the Buildroot CI image published on Docker Hub

[Arnout Vandecappelle]

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 .gitlab-ci.yml    | 14 +-------------
 .gitlab-ci.yml.in | 14 +-------------
 2 files changed, 2 insertions(+), 26 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 7acf4b620e..3539b7e439 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -4,19 +4,7 @@
 # It needs to be regenerated every time a defconfig is added, using
 # "make .gitlab-ci.yml".
 
-image: debian:stable
-
-before_script:
-    - dpkg --add-architecture i386
-    # The container has no package lists, so need to update first
-    - apt-get update -qq
-    - apt-get install -y -qq --no-install-recommends
-        build-essential locales bc ca-certificates file rsync gcc-multilib
-        git bzr cvs mercurial subversion libc6:i386 unzip wget cpio
-        python-nose2 python-pexpect qemu-system-arm qemu-system-x86
-    # To be able to generate a toolchain with locales, enable one UTF-8 locale
-    - sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen
-    - /usr/sbin/locale-gen
+image: buildroot/ci
 
 .defconfig_script: &defconfig_script
     - echo 'Configure Buildroot'
diff --git a/.gitlab-ci.yml.in b/.gitlab-ci.yml.in
index c69f9de99c..3088677e67 100644
--- a/.gitlab-ci.yml.in
+++ b/.gitlab-ci.yml.in
@@ -4,19 +4,7 @@
 # It needs to be regenerated every time a defconfig is added, using
 # "make .gitlab-ci.yml".
 
-image: debian:stable
-
-before_script:
-    - dpkg --add-architecture i386
-    # The container has no package lists, so need to update first
-    - apt-get update -qq
-    - apt-get install -y -qq --no-install-recommends
-        build-essential locales bc ca-certificates file rsync gcc-multilib
-        git bzr cvs mercurial subversion libc6:i386 unzip wget cpio
-        python-nose2 python-pexpect qemu-system-arm qemu-system-x86
-    # To be able to generate a toolchain with locales, enable one UTF-8 locale
-    - sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen
-    - /usr/sbin/locale-gen
+image: buildroot/base
 
 .defconfig_script: &defconfig_script
     - echo 'Configure Buildroot'
-- 
2.13.2

[Buildroot] [PATCH v2] .gitlab-ci.yml: use the Buildroot CI image published on Docker Hub

[Arnout Vandecappelle]

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
v2: Forgot to update .gitlab-ci.yml after the final changes
---
 .gitlab-ci.yml    | 14 +-------------
 .gitlab-ci.yml.in | 14 +-------------
 2 files changed, 2 insertions(+), 26 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 7acf4b620e..3ae7e5db64 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -4,19 +4,7 @@
 # It needs to be regenerated every time a defconfig is added, using
 # "make .gitlab-ci.yml".
 
-image: debian:stable
-
-before_script:
-    - dpkg --add-architecture i386
-    # The container has no package lists, so need to update first
-    - apt-get update -qq
-    - apt-get install -y -qq --no-install-recommends
-        build-essential locales bc ca-certificates file rsync gcc-multilib
-        git bzr cvs mercurial subversion libc6:i386 unzip wget cpio
-        python-nose2 python-pexpect qemu-system-arm qemu-system-x86
-    # To be able to generate a toolchain with locales, enable one UTF-8 locale
-    - sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen
-    - /usr/sbin/locale-gen
+image: buildroot/base
 
 .defconfig_script: &defconfig_script
     - echo 'Configure Buildroot'
diff --git a/.gitlab-ci.yml.in b/.gitlab-ci.yml.in
index c69f9de99c..3088677e67 100644
--- a/.gitlab-ci.yml.in
+++ b/.gitlab-ci.yml.in
@@ -4,19 +4,7 @@
 # It needs to be regenerated every time a defconfig is added, using
 # "make .gitlab-ci.yml".
 
-image: debian:stable
-
-before_script:
-    - dpkg --add-architecture i386
-    # The container has no package lists, so need to update first
-    - apt-get update -qq
-    - apt-get install -y -qq --no-install-recommends
-        build-essential locales bc ca-certificates file rsync gcc-multilib
-        git bzr cvs mercurial subversion libc6:i386 unzip wget cpio
-        python-nose2 python-pexpect qemu-system-arm qemu-system-x86
-    # To be able to generate a toolchain with locales, enable one UTF-8 locale
-    - sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen
-    - /usr/sbin/locale-gen
+image: buildroot/base
 
 .defconfig_script: &defconfig_script
     - echo 'Configure Buildroot'
-- 
2.13.2

[Buildroot] [PATCH 1/3] .gitlab-ci.yml: run our runtime tests

[Thomas Petazzoni]

Hello,

On Sun, 2 Jul 2017 18:13:22 +0200, Arnout Vandecappelle
(Essensium/Mind) wrote:
> From: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> 
> This commit improves our .gitlab-ci.yml logic to execute our runtime
> tests located in support/testing/. To do so, this commit:
> 
>  - Adds more Debian packages to be installed, namely the nose2 and
>    pexpect packages needed by the runtime testing infrastructure, as
>    well as the necessary Qemu emulators
> 
>  - The description of how to run the runtime tests. Each test is
>    executed as a separate Gitlab CI job, so that the status of each
>    test is easily visible in the Gitlab CI web interface.
> 
>  - The Makefile is improved to auto-generate .gitlab-ci.yml from
>    .gitlab-ci.yml.in, like we're doing for defconfigs. Since the
>    dependencies of .gitlab-ci.yml are no longer correct, we abandon
>    them and instead make it a PHONY target.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> [Arnout: simplify .gitlab-ci.in a little, removing redundant stuff;
>          make .gitlab-ci.yml a PHONY target]
> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> ---
>  .gitlab-ci.yml    | 33 +++++++++++++++++++++++++++++++++
>  .gitlab-ci.yml.in |  8 ++++++++
>  Makefile          |  7 +++++--
>  3 files changed, 46 insertions(+), 2 deletions(-)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 2/3] Add Dockerfile for CI

[Thomas Petazzoni]

Hello,

On Sun, 2 Jul 2017 18:13:23 +0200, Arnout Vandecappelle
(Essensium/Mind) wrote:
> For Gitlab-CI, we want to avoid re-generating the minimal install to
> be able to run tests all the time. So let's create a docker image that
> we can post on Docker Hub and then pull.
> 
> For the time being, this is just what we need for running our CI. Later
> we can produce something that is also useful for users.
> 
> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> ---
>  support/docker/Dockerfile | 30 ++++++++++++++++++++++++++++++
>  1 file changed, 30 insertions(+)
>  create mode 100644 support/docker/Dockerfile

Applied to master, thanks. Will you provide an update to the Buildroot
manual, documenting this next to the Vagrant stuff?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v2] .gitlab-ci.yml: use the Buildroot CI image published on Docker Hub

[Thomas Petazzoni]

Hello,

On Sun, 2 Jul 2017 18:24:59 +0200, Arnout Vandecappelle
(Essensium/Mind) wrote:
> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> ---
> v2: Forgot to update .gitlab-ci.yml after the final changes
> ---
>  .gitlab-ci.yml    | 14 +-------------
>  .gitlab-ci.yml.in | 14 +-------------
>  2 files changed, 2 insertions(+), 26 deletions(-)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com