* [Buildroot] [PATCH] .gitlab-ci.yml: do runtime tests only on explicit trigger
@ 2018-10-20 22:42 Arnout Vandecappelle
2018-10-21 2:54 ` Ricardo Martincoski
0 siblings, 1 reply; 6+ messages in thread
From: Arnout Vandecappelle @ 2018-10-20 22:42 UTC (permalink / raw)
To: buildroot
When the runtime tests were first introduced, they still ran pretty
quickly. Nowadays, however, there are a lot of runtime tests, and some
of them take a really long time. So running them on every push is
really too much.
Just like we do for the defconfigs, run them on explicit trigger only.
The explicit trigger is now done every week, but it can be increased
to e.g. twice or three times per week.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
.gitlab-ci.yml | 73 ++++-------------------------------------------
.gitlab-ci.yml.in | 5 ++++
2 files changed, 10 insertions(+), 68 deletions(-)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index d6a2319ba0..d621692ce1 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -62,6 +62,11 @@ check-package:
- output/build/packages-file-list.txt
.runtime_test: &runtime_test
+ # Running the runtime tests for every push is too much, so limit to
+ # explicit triggers through the API.
+ only:
+ - triggers
+ - tags
# Keep build directories so the rootfs can be an artifact of the job. The
# runner will clean up those files for us.
# Multiply every emulator timeout by 10 to avoid sporadic failures in
@@ -261,71 +266,3 @@ zynq_zc706_defconfig: *defconfig
zynq_zed_defconfig: *defconfig
zynq_zybo_defconfig: *defconfig
zynqmp_zcu106_defconfig: *defconfig
-tests.boot.test_atf.TestATFAllwinner: *runtime_test
-tests.boot.test_atf.TestATFMarvell: *runtime_test
-tests.boot.test_atf.TestATFVexpress: *runtime_test
-tests.core.test_file_capabilities.TestFileCapabilities: *runtime_test
-tests.core.test_hardening.TestFortifyConserv: *runtime_test
-tests.core.test_hardening.TestFortifyNone: *runtime_test
-tests.core.test_hardening.TestRelro: *runtime_test
-tests.core.test_hardening.TestRelroPartial: *runtime_test
-tests.core.test_hardening.TestSspNone: *runtime_test
-tests.core.test_hardening.TestSspStrong: *runtime_test
-tests.core.test_post_scripts.TestPostScripts: *runtime_test
-tests.core.test_rootfs_overlay.TestRootfsOverlay: *runtime_test
-tests.core.test_timezone.TestGlibcAllTimezone: *runtime_test
-tests.core.test_timezone.TestGlibcNonDefaultLimitedTimezone: *runtime_test
-tests.core.test_timezone.TestNoTimezone: *runtime_test
-tests.fs.test_ext.TestExt2: *runtime_test
-tests.fs.test_ext.TestExt2r1: *runtime_test
-tests.fs.test_ext.TestExt3: *runtime_test
-tests.fs.test_ext.TestExt4: *runtime_test
-tests.fs.test_iso9660.TestIso9660Grub2External: *runtime_test
-tests.fs.test_iso9660.TestIso9660Grub2ExternalCompress: *runtime_test
-tests.fs.test_iso9660.TestIso9660Grub2Internal: *runtime_test
-tests.fs.test_iso9660.TestIso9660SyslinuxExternal: *runtime_test
-tests.fs.test_iso9660.TestIso9660SyslinuxExternalCompress: *runtime_test
-tests.fs.test_iso9660.TestIso9660SyslinuxInternal: *runtime_test
-tests.fs.test_jffs2.TestJffs2: *runtime_test
-tests.fs.test_squashfs.TestSquashfs: *runtime_test
-tests.fs.test_ubi.TestUbi: *runtime_test
-tests.fs.test_yaffs2.TestYaffs2: *runtime_test
-tests.init.test_busybox.TestInitSystemBusyboxRo: *runtime_test
-tests.init.test_busybox.TestInitSystemBusyboxRoNet: *runtime_test
-tests.init.test_busybox.TestInitSystemBusyboxRw: *runtime_test
-tests.init.test_busybox.TestInitSystemBusyboxRwNet: *runtime_test
-tests.init.test_none.TestInitSystemNone: *runtime_test
-tests.init.test_systemd.TestInitSystemSystemdRoFull: *runtime_test
-tests.init.test_systemd.TestInitSystemSystemdRoIfupdown: *runtime_test
-tests.init.test_systemd.TestInitSystemSystemdRoNetworkd: *runtime_test
-tests.init.test_systemd.TestInitSystemSystemdRwFull: *runtime_test
-tests.init.test_systemd.TestInitSystemSystemdRwIfupdown: *runtime_test
-tests.init.test_systemd.TestInitSystemSystemdRwNetworkd: *runtime_test
-tests.package.test_dropbear.TestDropbear: *runtime_test
-tests.package.test_ipython.TestIPythonPy2: *runtime_test
-tests.package.test_ipython.TestIPythonPy3: *runtime_test
-tests.package.test_python.TestPython2: *runtime_test
-tests.package.test_python.TestPython3: *runtime_test
-tests.package.test_python_autobahn.TestPythonPy2Autobahn: *runtime_test
-tests.package.test_python_autobahn.TestPythonPy3Autobahn: *runtime_test
-tests.package.test_python_cryptography.TestPythonPy2Cryptography: *runtime_test
-tests.package.test_python_cryptography.TestPythonPy3Cryptography: *runtime_test
-tests.package.test_python_incremental.TestPythonPy2Incremental: *runtime_test
-tests.package.test_python_incremental.TestPythonPy3Incremental: *runtime_test
-tests.package.test_python_twisted.TestPythonPy2Twisted: *runtime_test
-tests.package.test_python_twisted.TestPythonPy3Twisted: *runtime_test
-tests.package.test_python_txaio.TestPythonPy2Txaio: *runtime_test
-tests.package.test_python_txaio.TestPythonPy3Txaio: *runtime_test
-tests.package.test_python_txtorcon.TestPythonPy2Txtorcon: *runtime_test
-tests.package.test_python_txtorcon.TestPythonPy3Txtorcon: *runtime_test
-tests.package.test_rust.TestRust: *runtime_test
-tests.package.test_rust.TestRustBin: *runtime_test
-tests.package.test_syslog_ng.TestSyslogNg: *runtime_test
-tests.toolchain.test_external.TestExternalToolchainBuildrootMusl: *runtime_test
-tests.toolchain.test_external.TestExternalToolchainBuildrootuClibc: *runtime_test
-tests.toolchain.test_external.TestExternalToolchainCCache: *runtime_test
-tests.toolchain.test_external.TestExternalToolchainCtngMusl: *runtime_test
-tests.toolchain.test_external.TestExternalToolchainLinaroArm: *runtime_test
-tests.toolchain.test_external.TestExternalToolchainSourceryArmv4: *runtime_test
-tests.toolchain.test_external.TestExternalToolchainSourceryArmv5: *runtime_test
-tests.toolchain.test_external.TestExternalToolchainSourceryArmv7: *runtime_test
diff --git a/.gitlab-ci.yml.in b/.gitlab-ci.yml.in
index 38e24c8d01..db526c4b5a 100644
--- a/.gitlab-ci.yml.in
+++ b/.gitlab-ci.yml.in
@@ -62,6 +62,11 @@ check-package:
- output/build/packages-file-list.txt
.runtime_test: &runtime_test
+ # Running the runtime tests for every push is too much, so limit to
+ # explicit triggers through the API.
+ only:
+ - triggers
+ - tags
# Keep build directories so the rootfs can be an artifact of the job. The
# runner will clean up those files for us.
# Multiply every emulator timeout by 10 to avoid sporadic failures in
--
2.17.2
^ permalink raw reply related [flat|nested] 6+ messages in thread* [Buildroot] [PATCH] .gitlab-ci.yml: do runtime tests only on explicit trigger
2018-10-20 22:42 [Buildroot] [PATCH] .gitlab-ci.yml: do runtime tests only on explicit trigger Arnout Vandecappelle
@ 2018-10-21 2:54 ` Ricardo Martincoski
2018-10-21 11:23 ` Arnout Vandecappelle
0 siblings, 1 reply; 6+ messages in thread
From: Ricardo Martincoski @ 2018-10-21 2:54 UTC (permalink / raw)
To: buildroot
Hello,
On Sat, Oct 20, 2018 at 07:42 PM, Arnout Vandecappelle (Essensium/Mind) wrote:
[snip]
> +++ b/.gitlab-ci.yml
> @@ -62,6 +62,11 @@ check-package:
> - output/build/packages-file-list.txt
>
> .runtime_test: &runtime_test
> + # Running the runtime tests for every push is too much, so limit to
> + # explicit triggers through the API.
> + only:
> + - triggers
> + - tags
> # Keep build directories so the rootfs can be an artifact of the job. The
> # runner will clean up those files for us.
> # Multiply every emulator timeout by 10 to avoid sporadic failures in
> @@ -261,71 +266,3 @@ zynq_zc706_defconfig: *defconfig
> zynq_zed_defconfig: *defconfig
> zynq_zybo_defconfig: *defconfig
> zynqmp_zcu106_defconfig: *defconfig
> -tests.boot.test_atf.TestATFAllwinner: *runtime_test
> -tests.boot.test_atf.TestATFMarvell: *runtime_test
> -tests.boot.test_atf.TestATFVexpress: *runtime_test
[snip]
You forgot to re-run 'make .gitlab-ci.yml'.
We don't want to remove the test list.
> +++ b/.gitlab-ci.yml.in
> @@ -62,6 +62,11 @@ check-package:
> - output/build/packages-file-list.txt
>
> .runtime_test: &runtime_test
> + # Running the runtime tests for every push is too much, so limit to
> + # explicit triggers through the API.
> + only:
> + - triggers
> + - tags
We could also allow a regexp on the branch name to trigger it:
- /^.*-runtime_test$/
Using the same commit, I pushed 2 branches:
branch name: gitlabci-triggers, #jobs = 4
https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33723520
branch name: gitlabci-triggers-runtime_test, #jobs = 72
https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33723541
> # Keep build directories so the rootfs can be an artifact of the job. The
> # runner will clean up those files for us.
> # Multiply every emulator timeout by 10 to avoid sporadic failures in
> --
Regards,
Ricardo
^ permalink raw reply [flat|nested] 6+ messages in thread* [Buildroot] [PATCH] .gitlab-ci.yml: do runtime tests only on explicit trigger
2018-10-21 2:54 ` Ricardo Martincoski
@ 2018-10-21 11:23 ` Arnout Vandecappelle
2018-10-22 2:35 ` Ricardo Martincoski
0 siblings, 1 reply; 6+ messages in thread
From: Arnout Vandecappelle @ 2018-10-21 11:23 UTC (permalink / raw)
To: buildroot
On 21/10/2018 03:54, Ricardo Martincoski wrote:
> Hello,
>
> On Sat, Oct 20, 2018 at 07:42 PM, Arnout Vandecappelle (Essensium/Mind) wrote:
>
> [snip]
>> +++ b/.gitlab-ci.yml
>> @@ -62,6 +62,11 @@ check-package:
>> - output/build/packages-file-list.txt
>>
>> .runtime_test: &runtime_test
>> + # Running the runtime tests for every push is too much, so limit to
>> + # explicit triggers through the API.
>> + only:
>> + - triggers
>> + - tags
>> # Keep build directories so the rootfs can be an artifact of the job. The
>> # runner will clean up those files for us.
>> # Multiply every emulator timeout by 10 to avoid sporadic failures in
>> @@ -261,71 +266,3 @@ zynq_zc706_defconfig: *defconfig
>> zynq_zed_defconfig: *defconfig
>> zynq_zybo_defconfig: *defconfig
>> zynqmp_zcu106_defconfig: *defconfig
>> -tests.boot.test_atf.TestATFAllwinner: *runtime_test
>> -tests.boot.test_atf.TestATFMarvell: *runtime_test
>> -tests.boot.test_atf.TestATFVexpress: *runtime_test
> [snip]
>
> You forgot to re-run 'make .gitlab-ci.yml'.
> We don't want to remove the test list.
Oops. Actually I didn't forget, but it failed (silently) to generate correctly
because nsoe2 wasn't installed. So I've submitted a patch that errors out if the
listing fails.
>
>> +++ b/.gitlab-ci.yml.in
>> @@ -62,6 +62,11 @@ check-package:
>> - output/build/packages-file-list.txt
>>
>> .runtime_test: &runtime_test
>> + # Running the runtime tests for every push is too much, so limit to
>> + # explicit triggers through the API.
>> + only:
>> + - triggers
>> + - tags
>
> We could also allow a regexp on the branch name to trigger it:
> - /^.*-runtime_test$/
>
> Using the same commit, I pushed 2 branches:
> branch name: gitlabci-triggers, #jobs = 4
> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33723520
> branch name: gitlabci-triggers-runtime_test, #jobs = 72
> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33723541
Looks nice, but that's really material for a separate patch (and probably
should be done for defconfigs too then).
Would it be possible to also define branches for a specific test or defconfig?
Could you take care of this?
Regards,
Arnout
^ permalink raw reply [flat|nested] 6+ messages in thread
* [Buildroot] [PATCH] .gitlab-ci.yml: do runtime tests only on explicit trigger
2018-10-21 11:23 ` Arnout Vandecappelle
@ 2018-10-22 2:35 ` Ricardo Martincoski
2018-10-22 23:08 ` Arnout Vandecappelle
0 siblings, 1 reply; 6+ messages in thread
From: Ricardo Martincoski @ 2018-10-22 2:35 UTC (permalink / raw)
To: buildroot
Hello,
On Sun, Oct 21, 2018 at 08:23 AM, Arnout Vandecappelle wrote:
[snip]
>>> .runtime_test: &runtime_test
>>> + # Running the runtime tests for every push is too much, so limit to
>>> + # explicit triggers through the API.
>>> + only:
>>> + - triggers
>>> + - tags
>>
>> We could also allow a regexp on the branch name to trigger it:
>> - /^.*-runtime_test$/
>>
>> Using the same commit, I pushed 2 branches:
>> branch name: gitlabci-triggers, #jobs = 4
>> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33723520
>> branch name: gitlabci-triggers-runtime_test, #jobs = 72
>> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33723541
>
> Looks nice, but that's really material for a separate patch (and probably
Indeed.
> should be done for defconfigs too then).
OK.
>
> Would it be possible to also define branches for a specific test or defconfig?
Yes.
Using the same commit, I pushed 3 branches and a tag:
git push gitlab HEAD:test2 #jobs = 4
https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765315
git push gitlab HEAD:test2-TestSyslogNg #jobs = 5
https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765342
git push gitlab HEAD:test2-runtime_test #jobs = 72
https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765317
git push gitlab test2-tag #jobs = 260
https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765321
We will need some extra 'sed' trickery to create a regexp for each test case:
-tests.package.test_syslog_ng.TestSyslogNg: *runtime_test
+tests.package.test_syslog_ng.TestSyslogNg:
+ <<: *runtime_test
+ only:
+ - triggers
+ - tags
+ - /^.*-runtime_test$/
+ - /^.*-TestSyslogNg$/
Unfortunately the inheritance using the anchor runtime_test does not allow to
merge two 'only' keywords, the 'only' declared on the test case key overrides
entirely the 'only' declared on the hidden key '.runtime_test' (or the other way
around if << is placed@the end), so we would need to repeat those lines for
every test case listed.
But maybe we don't care that .gitlab-ci.yml grows because those lines would be
automatically generated by 'make .gitlab-ci.yml'.
>
> Could you take care of this?
Sure.
Regards,
Ricardo
^ permalink raw reply [flat|nested] 6+ messages in thread
* [Buildroot] [PATCH] .gitlab-ci.yml: do runtime tests only on explicit trigger
2018-10-22 2:35 ` Ricardo Martincoski
@ 2018-10-22 23:08 ` Arnout Vandecappelle
2018-10-23 3:13 ` Ricardo Martincoski
0 siblings, 1 reply; 6+ messages in thread
From: Arnout Vandecappelle @ 2018-10-22 23:08 UTC (permalink / raw)
To: buildroot
On 10/22/18 3:35 AM, Ricardo Martincoski wrote:
> Hello,
>
> On Sun, Oct 21, 2018 at 08:23 AM, Arnout Vandecappelle wrote:
>
> [snip]
>>>> .runtime_test: &runtime_test
>>>> + # Running the runtime tests for every push is too much, so limit to
>>>> + # explicit triggers through the API.
>>>> + only:
>>>> + - triggers
>>>> + - tags
>>>
>>> We could also allow a regexp on the branch name to trigger it:
>>> - /^.*-runtime_test$/
>>>
>>> Using the same commit, I pushed 2 branches:
>>> branch name: gitlabci-triggers, #jobs = 4
>>> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33723520
>>> branch name: gitlabci-triggers-runtime_test, #jobs = 72
>>> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33723541
>>
>> Looks nice, but that's really material for a separate patch (and probably
>
> Indeed.
>
>> should be done for defconfigs too then).
>
> OK.
>
>>
>> Would it be possible to also define branches for a specific test or defconfig?
>
> Yes.
>
> Using the same commit, I pushed 3 branches and a tag:
> git push gitlab HEAD:test2 #jobs = 4
> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765315
> git push gitlab HEAD:test2-TestSyslogNg #jobs = 5
> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765342
> git push gitlab HEAD:test2-runtime_test #jobs = 72
> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765317
> git push gitlab test2-tag #jobs = 260
> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765321
Nice! I suppose test2-tag is actually a tag, not a branch, so it matches the
'tags' trigger?
>
> We will need some extra 'sed' trickery to create a regexp for each test case:
> -tests.package.test_syslog_ng.TestSyslogNg: *runtime_test
> +tests.package.test_syslog_ng.TestSyslogNg:
> + <<: *runtime_test
> + only:
> + - triggers
> + - tags
> + - /^.*-runtime_test$/
> + - /^.*-TestSyslogNg$/
Maybe you can use variables in there? So, in the only specification of
.runtime_test:
only:
- triggers
- tags
- /^.*-runtime_test$/
- /^.*-${CI_JOB_NAME}$/
Hm, unlikely, otherwise the other $ would have to be quoted.
Apparently nowadays there is also support for more complex only specifications
using variables. Perhaps you could take a look at that? I'm afraid though that
it's always an AND of the conditions you specify, so no way to specify that it's
for tags OR for things matching the CI_JOB_NAME.
Regards,
Arnout
>
> Unfortunately the inheritance using the anchor runtime_test does not allow to
> merge two 'only' keywords, the 'only' declared on the test case key overrides
> entirely the 'only' declared on the hidden key '.runtime_test' (or the other way
> around if << is placed@the end), so we would need to repeat those lines for
> every test case listed.
> But maybe we don't care that .gitlab-ci.yml grows because those lines would be
> automatically generated by 'make .gitlab-ci.yml'.
>
>>
>> Could you take care of this?
>
> Sure.
>
> Regards,
> Ricardo
>
^ permalink raw reply [flat|nested] 6+ messages in thread* [Buildroot] [PATCH] .gitlab-ci.yml: do runtime tests only on explicit trigger
2018-10-22 23:08 ` Arnout Vandecappelle
@ 2018-10-23 3:13 ` Ricardo Martincoski
0 siblings, 0 replies; 6+ messages in thread
From: Ricardo Martincoski @ 2018-10-23 3:13 UTC (permalink / raw)
To: buildroot
Hello,
On Mon, Oct 22, 2018 at 08:08 PM, Arnout Vandecappelle wrote:
> On 10/22/18 3:35 AM, Ricardo Martincoski wrote:
>>
>> On Sun, Oct 21, 2018 at 08:23 AM, Arnout Vandecappelle wrote:
[snip]
>>> Would it be possible to also define branches for a specific test or defconfig?
>>
>> Yes.
>>
>> Using the same commit, I pushed 3 branches and a tag:
>> git push gitlab HEAD:test2 #jobs = 4
>> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765315
>> git push gitlab HEAD:test2-TestSyslogNg #jobs = 5
>> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765342
>> git push gitlab HEAD:test2-runtime_test #jobs = 72
>> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765317
>> git push gitlab test2-tag #jobs = 260
>> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765321
>
> Nice! I suppose test2-tag is actually a tag, not a branch, so it matches the
> 'tags' trigger?
Yes. It is a tag.
>
>>
>> We will need some extra 'sed' trickery to create a regexp for each test case:
>> -tests.package.test_syslog_ng.TestSyslogNg: *runtime_test
>> +tests.package.test_syslog_ng.TestSyslogNg:
>> + <<: *runtime_test
>> + only:
>> + - triggers
>> + - tags
>> + - /^.*-runtime_test$/
>> + - /^.*-TestSyslogNg$/
>
> Maybe you can use variables in there? So, in the only specification of
> .runtime_test:
>
> only:
> - triggers
> - tags
> - /^.*-runtime_test$/
> - /^.*-${CI_JOB_NAME}$/
>
> Hm, unlikely, otherwise the other $ would have to be quoted.
Maybe using %variable%. I didn't tested yet.
>
> Apparently nowadays there is also support for more complex only specifications
> using variables. Perhaps you could take a look at that? I'm afraid though that
> it's always an AND of the conditions you specify, so no way to specify that it's
> for tags OR for things matching the CI_JOB_NAME.
I will take a look.
Regards,
Ricardo
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2018-10-23 3:13 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-10-20 22:42 [Buildroot] [PATCH] .gitlab-ci.yml: do runtime tests only on explicit trigger Arnout Vandecappelle
2018-10-21 2:54 ` Ricardo Martincoski
2018-10-21 11:23 ` Arnout Vandecappelle
2018-10-22 2:35 ` Ricardo Martincoski
2018-10-22 23:08 ` Arnout Vandecappelle
2018-10-23 3:13 ` Ricardo Martincoski
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.