[Buildroot] [PATCH] .gitlab-ci.yml: do runtime tests only on explicit trigger

[Buildroot] [PATCH] .gitlab-ci.yml: do runtime tests only on explicit trigger

[Arnout Vandecappelle]

When the runtime tests were first introduced, they still ran pretty
quickly. Nowadays, however, there are a lot of runtime tests, and some
of them take a really long time. So running them on every push is
really too much.

Just like we do for the defconfigs, run them on explicit trigger only.

The explicit trigger is now done every week, but it can be increased
to e.g. twice or three times per week.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 .gitlab-ci.yml    | 73 ++++-------------------------------------------
 .gitlab-ci.yml.in |  5 ++++
 2 files changed, 10 insertions(+), 68 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index d6a2319ba0..d621692ce1 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -62,6 +62,11 @@ check-package:
             - output/build/packages-file-list.txt
 
 .runtime_test: &runtime_test
+    # Running the runtime tests for every push is too much, so limit to
+    # explicit triggers through the API.
+    only:
+        - triggers
+        - tags
     # Keep build directories so the rootfs can be an artifact of the job. The
     # runner will clean up those files for us.
     # Multiply every emulator timeout by 10 to avoid sporadic failures in
@@ -261,71 +266,3 @@ zynq_zc706_defconfig: *defconfig
 zynq_zed_defconfig: *defconfig
 zynq_zybo_defconfig: *defconfig
 zynqmp_zcu106_defconfig: *defconfig
-tests.boot.test_atf.TestATFAllwinner: *runtime_test
-tests.boot.test_atf.TestATFMarvell: *runtime_test
-tests.boot.test_atf.TestATFVexpress: *runtime_test
-tests.core.test_file_capabilities.TestFileCapabilities: *runtime_test
-tests.core.test_hardening.TestFortifyConserv: *runtime_test
-tests.core.test_hardening.TestFortifyNone: *runtime_test
-tests.core.test_hardening.TestRelro: *runtime_test
-tests.core.test_hardening.TestRelroPartial: *runtime_test
-tests.core.test_hardening.TestSspNone: *runtime_test
-tests.core.test_hardening.TestSspStrong: *runtime_test
-tests.core.test_post_scripts.TestPostScripts: *runtime_test
-tests.core.test_rootfs_overlay.TestRootfsOverlay: *runtime_test
-tests.core.test_timezone.TestGlibcAllTimezone: *runtime_test
-tests.core.test_timezone.TestGlibcNonDefaultLimitedTimezone: *runtime_test
-tests.core.test_timezone.TestNoTimezone: *runtime_test
-tests.fs.test_ext.TestExt2: *runtime_test
-tests.fs.test_ext.TestExt2r1: *runtime_test
-tests.fs.test_ext.TestExt3: *runtime_test
-tests.fs.test_ext.TestExt4: *runtime_test
-tests.fs.test_iso9660.TestIso9660Grub2External: *runtime_test
-tests.fs.test_iso9660.TestIso9660Grub2ExternalCompress: *runtime_test
-tests.fs.test_iso9660.TestIso9660Grub2Internal: *runtime_test
-tests.fs.test_iso9660.TestIso9660SyslinuxExternal: *runtime_test
-tests.fs.test_iso9660.TestIso9660SyslinuxExternalCompress: *runtime_test
-tests.fs.test_iso9660.TestIso9660SyslinuxInternal: *runtime_test
-tests.fs.test_jffs2.TestJffs2: *runtime_test
-tests.fs.test_squashfs.TestSquashfs: *runtime_test
-tests.fs.test_ubi.TestUbi: *runtime_test
-tests.fs.test_yaffs2.TestYaffs2: *runtime_test
-tests.init.test_busybox.TestInitSystemBusyboxRo: *runtime_test
-tests.init.test_busybox.TestInitSystemBusyboxRoNet: *runtime_test
-tests.init.test_busybox.TestInitSystemBusyboxRw: *runtime_test
-tests.init.test_busybox.TestInitSystemBusyboxRwNet: *runtime_test
-tests.init.test_none.TestInitSystemNone: *runtime_test
-tests.init.test_systemd.TestInitSystemSystemdRoFull: *runtime_test
-tests.init.test_systemd.TestInitSystemSystemdRoIfupdown: *runtime_test
-tests.init.test_systemd.TestInitSystemSystemdRoNetworkd: *runtime_test
-tests.init.test_systemd.TestInitSystemSystemdRwFull: *runtime_test
-tests.init.test_systemd.TestInitSystemSystemdRwIfupdown: *runtime_test
-tests.init.test_systemd.TestInitSystemSystemdRwNetworkd: *runtime_test
-tests.package.test_dropbear.TestDropbear: *runtime_test
-tests.package.test_ipython.TestIPythonPy2: *runtime_test
-tests.package.test_ipython.TestIPythonPy3: *runtime_test
-tests.package.test_python.TestPython2: *runtime_test
-tests.package.test_python.TestPython3: *runtime_test
-tests.package.test_python_autobahn.TestPythonPy2Autobahn: *runtime_test
-tests.package.test_python_autobahn.TestPythonPy3Autobahn: *runtime_test
-tests.package.test_python_cryptography.TestPythonPy2Cryptography: *runtime_test
-tests.package.test_python_cryptography.TestPythonPy3Cryptography: *runtime_test
-tests.package.test_python_incremental.TestPythonPy2Incremental: *runtime_test
-tests.package.test_python_incremental.TestPythonPy3Incremental: *runtime_test
-tests.package.test_python_twisted.TestPythonPy2Twisted: *runtime_test
-tests.package.test_python_twisted.TestPythonPy3Twisted: *runtime_test
-tests.package.test_python_txaio.TestPythonPy2Txaio: *runtime_test
-tests.package.test_python_txaio.TestPythonPy3Txaio: *runtime_test
-tests.package.test_python_txtorcon.TestPythonPy2Txtorcon: *runtime_test
-tests.package.test_python_txtorcon.TestPythonPy3Txtorcon: *runtime_test
-tests.package.test_rust.TestRust: *runtime_test
-tests.package.test_rust.TestRustBin: *runtime_test
-tests.package.test_syslog_ng.TestSyslogNg: *runtime_test
-tests.toolchain.test_external.TestExternalToolchainBuildrootMusl: *runtime_test
-tests.toolchain.test_external.TestExternalToolchainBuildrootuClibc: *runtime_test
-tests.toolchain.test_external.TestExternalToolchainCCache: *runtime_test
-tests.toolchain.test_external.TestExternalToolchainCtngMusl: *runtime_test
-tests.toolchain.test_external.TestExternalToolchainLinaroArm: *runtime_test
-tests.toolchain.test_external.TestExternalToolchainSourceryArmv4: *runtime_test
-tests.toolchain.test_external.TestExternalToolchainSourceryArmv5: *runtime_test
-tests.toolchain.test_external.TestExternalToolchainSourceryArmv7: *runtime_test
diff --git a/.gitlab-ci.yml.in b/.gitlab-ci.yml.in
index 38e24c8d01..db526c4b5a 100644
--- a/.gitlab-ci.yml.in
+++ b/.gitlab-ci.yml.in
@@ -62,6 +62,11 @@ check-package:
             - output/build/packages-file-list.txt
 
 .runtime_test: &runtime_test
+    # Running the runtime tests for every push is too much, so limit to
+    # explicit triggers through the API.
+    only:
+        - triggers
+        - tags
     # Keep build directories so the rootfs can be an artifact of the job. The
     # runner will clean up those files for us.
     # Multiply every emulator timeout by 10 to avoid sporadic failures in
-- 
2.17.2

[Buildroot] [PATCH] .gitlab-ci.yml: do runtime tests only on explicit trigger

[Ricardo Martincoski]

Hello,

On Sat, Oct 20, 2018 at 07:42 PM, Arnout Vandecappelle (Essensium/Mind) wrote:

[snip]
> +++ b/.gitlab-ci.yml
> @@ -62,6 +62,11 @@ check-package:
>              - output/build/packages-file-list.txt
>  
>  .runtime_test: &runtime_test
> +    # Running the runtime tests for every push is too much, so limit to
> +    # explicit triggers through the API.
> +    only:
> +        - triggers
> +        - tags
>      # Keep build directories so the rootfs can be an artifact of the job. The
>      # runner will clean up those files for us.
>      # Multiply every emulator timeout by 10 to avoid sporadic failures in
> @@ -261,71 +266,3 @@ zynq_zc706_defconfig: *defconfig
>  zynq_zed_defconfig: *defconfig
>  zynq_zybo_defconfig: *defconfig
>  zynqmp_zcu106_defconfig: *defconfig
> -tests.boot.test_atf.TestATFAllwinner: *runtime_test
> -tests.boot.test_atf.TestATFMarvell: *runtime_test
> -tests.boot.test_atf.TestATFVexpress: *runtime_test
[snip]

You forgot to re-run 'make .gitlab-ci.yml'.
We don't want to remove the test list.

> +++ b/.gitlab-ci.yml.in
> @@ -62,6 +62,11 @@ check-package:
>              - output/build/packages-file-list.txt
>  
>  .runtime_test: &runtime_test
> +    # Running the runtime tests for every push is too much, so limit to
> +    # explicit triggers through the API.
> +    only:
> +        - triggers
> +        - tags

We could also allow a regexp on the branch name to trigger it:
         - /^.*-runtime_test$/

Using the same commit, I pushed 2 branches:
branch name: gitlabci-triggers, #jobs = 4
https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33723520
branch name: gitlabci-triggers-runtime_test, #jobs = 72
https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33723541

>      # Keep build directories so the rootfs can be an artifact of the job. The
>      # runner will clean up those files for us.
>      # Multiply every emulator timeout by 10 to avoid sporadic failures in
> -- 

Regards,
Ricardo

[Buildroot] [PATCH] .gitlab-ci.yml: do runtime tests only on explicit trigger

[Arnout Vandecappelle]

On 21/10/2018 03:54, Ricardo Martincoski wrote:
> Hello,
> 
> On Sat, Oct 20, 2018 at 07:42 PM, Arnout Vandecappelle (Essensium/Mind) wrote:
> 
> [snip]
>> +++ b/.gitlab-ci.yml
>> @@ -62,6 +62,11 @@ check-package:
>>              - output/build/packages-file-list.txt
>>  
>>  .runtime_test: &runtime_test
>> +    # Running the runtime tests for every push is too much, so limit to
>> +    # explicit triggers through the API.
>> +    only:
>> +        - triggers
>> +        - tags
>>      # Keep build directories so the rootfs can be an artifact of the job. The
>>      # runner will clean up those files for us.
>>      # Multiply every emulator timeout by 10 to avoid sporadic failures in
>> @@ -261,71 +266,3 @@ zynq_zc706_defconfig: *defconfig
>>  zynq_zed_defconfig: *defconfig
>>  zynq_zybo_defconfig: *defconfig
>>  zynqmp_zcu106_defconfig: *defconfig
>> -tests.boot.test_atf.TestATFAllwinner: *runtime_test
>> -tests.boot.test_atf.TestATFMarvell: *runtime_test
>> -tests.boot.test_atf.TestATFVexpress: *runtime_test
> [snip]
> 
> You forgot to re-run 'make .gitlab-ci.yml'.
> We don't want to remove the test list.

 Oops. Actually I didn't forget, but it failed (silently) to generate correctly
because nsoe2 wasn't installed. So I've submitted a patch that errors out if the
listing fails.

> 
>> +++ b/.gitlab-ci.yml.in
>> @@ -62,6 +62,11 @@ check-package:
>>              - output/build/packages-file-list.txt
>>  
>>  .runtime_test: &runtime_test
>> +    # Running the runtime tests for every push is too much, so limit to
>> +    # explicit triggers through the API.
>> +    only:
>> +        - triggers
>> +        - tags
> 
> We could also allow a regexp on the branch name to trigger it:
>          - /^.*-runtime_test$/
> 
> Using the same commit, I pushed 2 branches:
> branch name: gitlabci-triggers, #jobs = 4
> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33723520
> branch name: gitlabci-triggers-runtime_test, #jobs = 72
> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33723541

 Looks nice, but that's really material for a separate patch (and probably
should be done for defconfigs too then).

 Would it be possible to also define branches for a specific test or defconfig?

 Could you take care of this?

 Regards,
 Arnout

[Buildroot] [PATCH] .gitlab-ci.yml: do runtime tests only on explicit trigger

[Ricardo Martincoski]

Hello,

On Sun, Oct 21, 2018 at 08:23 AM, Arnout Vandecappelle wrote:

[snip]
>>>  .runtime_test: &runtime_test
>>> +    # Running the runtime tests for every push is too much, so limit to
>>> +    # explicit triggers through the API.
>>> +    only:
>>> +        - triggers
>>> +        - tags
>> 
>> We could also allow a regexp on the branch name to trigger it:
>>          - /^.*-runtime_test$/
>> 
>> Using the same commit, I pushed 2 branches:
>> branch name: gitlabci-triggers, #jobs = 4
>> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33723520
>> branch name: gitlabci-triggers-runtime_test, #jobs = 72
>> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33723541
> 
>  Looks nice, but that's really material for a separate patch (and probably

Indeed.

> should be done for defconfigs too then).

OK.

> 
>  Would it be possible to also define branches for a specific test or defconfig?

Yes.

Using the same commit, I pushed 3 branches and a tag:
git push gitlab HEAD:test2               #jobs = 4
https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765315
git push gitlab HEAD:test2-TestSyslogNg  #jobs = 5
https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765342
git push gitlab HEAD:test2-runtime_test  #jobs = 72
https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765317
git push gitlab test2-tag                #jobs = 260
https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765321

We will need some extra 'sed' trickery to create a regexp for each test case:
-tests.package.test_syslog_ng.TestSyslogNg: *runtime_test
+tests.package.test_syslog_ng.TestSyslogNg:
+    <<: *runtime_test
+    only:
+        - triggers
+        - tags
+        - /^.*-runtime_test$/
+        - /^.*-TestSyslogNg$/

Unfortunately the inheritance using the anchor runtime_test does not allow to
merge two 'only' keywords, the 'only' declared on the test case key overrides
entirely the 'only' declared on the hidden key '.runtime_test' (or the other way
around if << is placed@the end), so we would need to repeat those lines for
every test case listed.
But maybe we don't care that .gitlab-ci.yml grows because those lines would be
automatically generated by 'make .gitlab-ci.yml'.

> 
>  Could you take care of this?

Sure.

Regards,
Ricardo

[Buildroot] [PATCH] .gitlab-ci.yml: do runtime tests only on explicit trigger

[Arnout Vandecappelle]

On 10/22/18 3:35 AM, Ricardo Martincoski wrote:
> Hello,
> 
> On Sun, Oct 21, 2018 at 08:23 AM, Arnout Vandecappelle wrote:
> 
> [snip]
>>>>  .runtime_test: &runtime_test
>>>> +    # Running the runtime tests for every push is too much, so limit to
>>>> +    # explicit triggers through the API.
>>>> +    only:
>>>> +        - triggers
>>>> +        - tags
>>>
>>> We could also allow a regexp on the branch name to trigger it:
>>>          - /^.*-runtime_test$/
>>>
>>> Using the same commit, I pushed 2 branches:
>>> branch name: gitlabci-triggers, #jobs = 4
>>> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33723520
>>> branch name: gitlabci-triggers-runtime_test, #jobs = 72
>>> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33723541
>>
>>  Looks nice, but that's really material for a separate patch (and probably
> 
> Indeed.
> 
>> should be done for defconfigs too then).
> 
> OK.
> 
>>
>>  Would it be possible to also define branches for a specific test or defconfig?
> 
> Yes.
> 
> Using the same commit, I pushed 3 branches and a tag:
> git push gitlab HEAD:test2               #jobs = 4
> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765315
> git push gitlab HEAD:test2-TestSyslogNg  #jobs = 5
> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765342
> git push gitlab HEAD:test2-runtime_test  #jobs = 72
> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765317
> git push gitlab test2-tag                #jobs = 260
> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765321

 Nice! I suppose test2-tag is actually a tag, not a branch, so it matches the
'tags' trigger?

> 
> We will need some extra 'sed' trickery to create a regexp for each test case:
> -tests.package.test_syslog_ng.TestSyslogNg: *runtime_test
> +tests.package.test_syslog_ng.TestSyslogNg:
> +    <<: *runtime_test
> +    only:
> +        - triggers
> +        - tags
> +        - /^.*-runtime_test$/
> +        - /^.*-TestSyslogNg$/

 Maybe you can use variables in there? So, in the only specification of
.runtime_test:

    only:
        - triggers
        - tags
        - /^.*-runtime_test$/
        - /^.*-${CI_JOB_NAME}$/

 Hm, unlikely, otherwise the other $ would have to be quoted.

 Apparently nowadays there is also support for more complex only specifications
using variables. Perhaps you could take a look at that? I'm afraid though that
it's always an AND of the conditions you specify, so no way to specify that it's
for tags OR for things matching the CI_JOB_NAME.

 Regards,
 Arnout

> 
> Unfortunately the inheritance using the anchor runtime_test does not allow to
> merge two 'only' keywords, the 'only' declared on the test case key overrides
> entirely the 'only' declared on the hidden key '.runtime_test' (or the other way
> around if << is placed@the end), so we would need to repeat those lines for
> every test case listed.
> But maybe we don't care that .gitlab-ci.yml grows because those lines would be
> automatically generated by 'make .gitlab-ci.yml'.
> 
>>
>>  Could you take care of this?
> 
> Sure.
> 
> Regards,
> Ricardo
> 

[Buildroot] [PATCH] .gitlab-ci.yml: do runtime tests only on explicit trigger

[Ricardo Martincoski]

Hello,

On Mon, Oct 22, 2018 at 08:08 PM, Arnout Vandecappelle wrote:

> On 10/22/18 3:35 AM, Ricardo Martincoski wrote:
>> 
>> On Sun, Oct 21, 2018 at 08:23 AM, Arnout Vandecappelle wrote:
[snip]
>>>  Would it be possible to also define branches for a specific test or defconfig?
>> 
>> Yes.
>> 
>> Using the same commit, I pushed 3 branches and a tag:
>> git push gitlab HEAD:test2               #jobs = 4
>> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765315
>> git push gitlab HEAD:test2-TestSyslogNg  #jobs = 5
>> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765342
>> git push gitlab HEAD:test2-runtime_test  #jobs = 72
>> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765317
>> git push gitlab test2-tag                #jobs = 260
>> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765321
> 
>  Nice! I suppose test2-tag is actually a tag, not a branch, so it matches the
> 'tags' trigger?

Yes. It is a tag.

> 
>> 
>> We will need some extra 'sed' trickery to create a regexp for each test case:
>> -tests.package.test_syslog_ng.TestSyslogNg: *runtime_test
>> +tests.package.test_syslog_ng.TestSyslogNg:
>> +    <<: *runtime_test
>> +    only:
>> +        - triggers
>> +        - tags
>> +        - /^.*-runtime_test$/
>> +        - /^.*-TestSyslogNg$/
> 
>  Maybe you can use variables in there? So, in the only specification of
> .runtime_test:
> 
>     only:
>         - triggers
>         - tags
>         - /^.*-runtime_test$/
>         - /^.*-${CI_JOB_NAME}$/
> 
>  Hm, unlikely, otherwise the other $ would have to be quoted.

Maybe using %variable%. I didn't tested yet.

> 
>  Apparently nowadays there is also support for more complex only specifications
> using variables. Perhaps you could take a look at that? I'm afraid though that
> it's always an AND of the conditions you specify, so no way to specify that it's
> for tags OR for things matching the CI_JOB_NAME.

I will take a look.


Regards,
Ricardo