* [Buildroot] [RFC PATCH 1/4] .gitlab-ci.yml: save rootfs as artifact for runtime tests @ 2017-07-30 4:49 Ricardo Martincoski 2017-07-30 4:49 ` [Buildroot] [RFC PATCH 2/4] support/testing: allow to use extra large timeouts Ricardo Martincoski ` (3 more replies) 0 siblings, 4 replies; 12+ messages in thread From: Ricardo Martincoski @ 2017-07-30 4:49 UTC (permalink / raw) To: buildroot Sometimes when a test fails in a gitlab pipeline the reason of the failure cannot be determined using only the logfile. Add the modified rootfs as an artifact of the job to improve troubleshooting. To accomplish this, always use -k option from the test infra, unconditionally add the resulting images to the artifacts, and let the runner do the cleanup for us. These artifacts can also be useful when a test fails locally but pass at gitlab runners. When the test does not generate a image, this message is displayed in the runner log: WARNING: test-output/*/images/*: no matching files Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> --- See also http://lists.busybox.net/pipermail/buildroot/2017-July/199332.html --- .gitlab-ci.yml | 5 ++++- .gitlab-ci.yml.in | 5 ++++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 42707ad1a6..f9e5b1fa6b 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -47,12 +47,15 @@ check-DEVELOPERS: - output/build/packages-file-list.txt .runtime_test: &runtime_test - script: ./support/testing/run-tests -o test-output/ -d test-dl/ ${CI_BUILD_NAME} + # Keep build directories so the rootfs can be an artifact of the job. The + # runner will clean up those files for us. + script: ./support/testing/run-tests -o test-output/ -d test-dl/ -k ${CI_BUILD_NAME} artifacts: when: always expire_in: 2 weeks paths: - test-output/*.log + - test-output/*/images/* acmesystems_aria_g25_128mb_defconfig: *defconfig acmesystems_aria_g25_256mb_defconfig: *defconfig acmesystems_arietta_g25_128mb_defconfig: *defconfig diff --git a/.gitlab-ci.yml.in b/.gitlab-ci.yml.in index 3088677e67..b8fce9ef96 100644 --- a/.gitlab-ci.yml.in +++ b/.gitlab-ci.yml.in @@ -47,9 +47,12 @@ check-DEVELOPERS: - output/build/packages-file-list.txt .runtime_test: &runtime_test - script: ./support/testing/run-tests -o test-output/ -d test-dl/ ${CI_BUILD_NAME} + # Keep build directories so the rootfs can be an artifact of the job. The + # runner will clean up those files for us. + script: ./support/testing/run-tests -o test-output/ -d test-dl/ -k ${CI_BUILD_NAME} artifacts: when: always expire_in: 2 weeks paths: - test-output/*.log + - test-output/*/images/* -- 2.13.0 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [Buildroot] [RFC PATCH 2/4] support/testing: allow to use extra large timeouts 2017-07-30 4:49 [Buildroot] [RFC PATCH 1/4] .gitlab-ci.yml: save rootfs as artifact for runtime tests Ricardo Martincoski @ 2017-07-30 4:49 ` Ricardo Martincoski 2017-07-31 19:25 ` Thomas Petazzoni 2017-08-05 2:05 ` [Buildroot] [PATCH v2 1/2] support/testing: allow to use a multiplier for timeouts Ricardo Martincoski 2017-07-30 4:49 ` [Buildroot] [RFC PATCH 3/4] .gitlab-ci.yml: use large timeouts for runtime tests Ricardo Martincoski ` (2 subsequent siblings) 3 siblings, 2 replies; 12+ messages in thread From: Ricardo Martincoski @ 2017-07-30 4:49 UTC (permalink / raw) To: buildroot Add a parameter to run-tests to act as a multiplier for all timeouts of emulator. It can be used to avoid sporadic failures on slow host machines as well in elastic runners on the cloud. Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> --- See also http://lists.busybox.net/pipermail/buildroot/2017-July/199329.html --- support/testing/infra/basetest.py | 4 +++- support/testing/infra/emulator.py | 13 ++++++++++--- support/testing/run-tests | 9 +++++++++ 3 files changed, 22 insertions(+), 4 deletions(-) diff --git a/support/testing/infra/basetest.py b/support/testing/infra/basetest.py index 431605b23f..1ef76a988b 100644 --- a/support/testing/infra/basetest.py +++ b/support/testing/infra/basetest.py @@ -35,6 +35,7 @@ class BRTest(unittest.TestCase): logtofile = True keepbuilds = False jlevel = 0 + elastic_timeout = 1 def __init__(self, names): super(BRTest, self).__init__(names) @@ -60,7 +61,8 @@ class BRTest(unittest.TestCase): self.b.build() self.show_msg("Building done") - self.emulator = Emulator(self.builddir, self.downloaddir, self.logtofile) + self.emulator = Emulator(self.builddir, self.downloaddir, + self.logtofile, self.elastic_timeout) def tearDown(self): self.show_msg("Cleaning up") diff --git a/support/testing/infra/emulator.py b/support/testing/infra/emulator.py index 9b079cbf23..7f02a01553 100644 --- a/support/testing/infra/emulator.py +++ b/support/testing/infra/emulator.py @@ -5,10 +5,14 @@ import infra class Emulator(object): - def __init__(self, builddir, downloaddir, logtofile): + def __init__(self, builddir, downloaddir, logtofile, multiplier): self.qemu = None self.downloaddir = downloaddir self.logfile = infra.open_log_file(builddir, "run", logtofile) + # We use elastic runners on the cloud to runs our tests. Those runners + # can take a long time to run the emulator. Use a timeout multiplier + # when running the tests to avoid sporadic failures. + self.multiplier = multiplier # Start Qemu to boot the system # @@ -65,7 +69,8 @@ class Emulator(object): qemu_cmd += ["-append", " ".join(kernel_cmdline)] self.logfile.write("> starting qemu with '%s'\n" % " ".join(qemu_cmd)) - self.qemu = pexpect.spawn(qemu_cmd[0], qemu_cmd[1:], timeout=5) + self.qemu = pexpect.spawn(qemu_cmd[0], qemu_cmd[1:], + timeout=5 * self.multiplier) # We want only stdout into the log to avoid double echo self.qemu.logfile_read = self.logfile @@ -75,7 +80,7 @@ class Emulator(object): # The login prompt can take some time to appear when running multiple # instances in parallel, so set the timeout to a large value index = self.qemu.expect(["buildroot login:", pexpect.TIMEOUT], - timeout=60) + timeout=60 * self.multiplier) if index != 0: self.logfile.write("==> System does not boot") raise SystemError("System does not boot") @@ -93,6 +98,8 @@ class Emulator(object): # return a tuple (output, exit_code) def run(self, cmd, timeout=-1): self.qemu.sendline(cmd) + if timeout != -1: + timeout *= self.multiplier self.qemu.expect("# ", timeout=timeout) # Remove double carriage return from qemu stdout so str.splitlines() # works as expected. diff --git a/support/testing/run-tests b/support/testing/run-tests index 0cb673c61f..33ac60d6b2 100755 --- a/support/testing/run-tests +++ b/support/testing/run-tests @@ -28,6 +28,8 @@ def main(): help='number of testcases to run simultaneously') parser.add_argument('-j', '--jlevel', type=int, help='BR2_JLEVEL to use for each testcase') + parser.add_argument('-e', '--elastic-timeout', type=int, default=1, + help='increase timeouts (useful for slow machines)') args = parser.parse_args() @@ -97,6 +99,13 @@ def main(): # the user can override the auto calculated value BRTest.jlevel = args.jlevel + if args.elastic_timeout < 1: + print "Invalid multiplier for timeout values" + print "" + parser.print_help() + return 1 + BRTest.elastic_timeout = args.elastic_timeout + nose2_args = ["-v", "-N", str(args.testcases), "-s", "support/testing", -- 2.13.0 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [Buildroot] [RFC PATCH 2/4] support/testing: allow to use extra large timeouts 2017-07-30 4:49 ` [Buildroot] [RFC PATCH 2/4] support/testing: allow to use extra large timeouts Ricardo Martincoski @ 2017-07-31 19:25 ` Thomas Petazzoni 2017-07-31 23:27 ` Ricardo Martincoski 2017-08-05 2:05 ` [Buildroot] [PATCH v2 1/2] support/testing: allow to use a multiplier for timeouts Ricardo Martincoski 1 sibling, 1 reply; 12+ messages in thread From: Thomas Petazzoni @ 2017-07-31 19:25 UTC (permalink / raw) To: buildroot Hello, On Sun, 30 Jul 2017 01:49:44 -0300, Ricardo Martincoski wrote: > diff --git a/support/testing/run-tests b/support/testing/run-tests > index 0cb673c61f..33ac60d6b2 100755 > --- a/support/testing/run-tests > +++ b/support/testing/run-tests > @@ -28,6 +28,8 @@ def main(): > help='number of testcases to run simultaneously') > parser.add_argument('-j', '--jlevel', type=int, > help='BR2_JLEVEL to use for each testcase') > + parser.add_argument('-e', '--elastic-timeout', type=int, default=1, > + help='increase timeouts (useful for slow machines)') I find the choice of the "elastic timeout" a bit weird. To me, it feels like a timeout that will reduce and extend like an elastic, not a timeout multiplier. Perhaps we should call this --timeout-multiplier or something like that ? Thomas -- Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com ^ permalink raw reply [flat|nested] 12+ messages in thread
* [Buildroot] [RFC PATCH 2/4] support/testing: allow to use extra large timeouts 2017-07-31 19:25 ` Thomas Petazzoni @ 2017-07-31 23:27 ` Ricardo Martincoski 2017-08-01 6:01 ` Thomas Petazzoni 0 siblings, 1 reply; 12+ messages in thread From: Ricardo Martincoski @ 2017-07-31 23:27 UTC (permalink / raw) To: buildroot Hello, On Mon, Jul 31, 2017 at 04:25 PM, Thomas Petazzoni wrote: > On Sun, 30 Jul 2017 01:49:44 -0300, Ricardo Martincoski wrote: [snip] >> + parser.add_argument('-e', '--elastic-timeout', type=int, default=1, >> + help='increase timeouts (useful for slow machines)') > > I find the choice of the "elastic timeout" a bit weird. To me, it feels > like a timeout that will reduce and extend like an elastic, not a > timeout multiplier. > > Perhaps we should call this --timeout-multiplier or something like > that ? Sure. I will fix and resend. I will use -m as short name because -t is already in use. Regards, Ricardo ^ permalink raw reply [flat|nested] 12+ messages in thread
* [Buildroot] [RFC PATCH 2/4] support/testing: allow to use extra large timeouts 2017-07-31 23:27 ` Ricardo Martincoski @ 2017-08-01 6:01 ` Thomas Petazzoni 0 siblings, 0 replies; 12+ messages in thread From: Thomas Petazzoni @ 2017-08-01 6:01 UTC (permalink / raw) To: buildroot Hello, On Mon, 31 Jul 2017 20:27:59 -0300, Ricardo Martincoski wrote: > > Perhaps we should call this --timeout-multiplier or something like > > that ? > > Sure. I will fix and resend. > I will use -m as short name because -t is already in use. Or perhaps this option is "specialized" enough that it doesn't need a short name, so that we avoid "polluting" the namespace of short names, which as you've seen, already has some collisions :) Best regards, Thomas -- Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com ^ permalink raw reply [flat|nested] 12+ messages in thread
* [Buildroot] [PATCH v2 1/2] support/testing: allow to use a multiplier for timeouts 2017-07-30 4:49 ` [Buildroot] [RFC PATCH 2/4] support/testing: allow to use extra large timeouts Ricardo Martincoski 2017-07-31 19:25 ` Thomas Petazzoni @ 2017-08-05 2:05 ` Ricardo Martincoski 2017-08-05 2:05 ` [Buildroot] [PATCH v2 2/2] .gitlab-ci.yml: use large timeouts for runtime tests Ricardo Martincoski 2017-08-10 9:18 ` [Buildroot] [PATCH v2 1/2] support/testing: allow to use a multiplier for timeouts Arnout Vandecappelle 1 sibling, 2 replies; 12+ messages in thread From: Ricardo Martincoski @ 2017-08-05 2:05 UTC (permalink / raw) To: buildroot Add a parameter to run-tests to act as a multiplier for all timeouts of emulator. It can be used to avoid sporadic failures on slow host machines as well in elastic runners on the cloud. Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> --- Changes v1 -> v2: - use clearer parameter name (elastic-timeout -> timeout-multiplier) (Thomas); - adapt commit message; --- support/testing/infra/basetest.py | 4 +++- support/testing/infra/emulator.py | 13 ++++++++++--- support/testing/run-tests | 9 +++++++++ 3 files changed, 22 insertions(+), 4 deletions(-) diff --git a/support/testing/infra/basetest.py b/support/testing/infra/basetest.py index 431605b23f..493dea5125 100644 --- a/support/testing/infra/basetest.py +++ b/support/testing/infra/basetest.py @@ -35,6 +35,7 @@ class BRTest(unittest.TestCase): logtofile = True keepbuilds = False jlevel = 0 + timeout_multiplier = 1 def __init__(self, names): super(BRTest, self).__init__(names) @@ -60,7 +61,8 @@ class BRTest(unittest.TestCase): self.b.build() self.show_msg("Building done") - self.emulator = Emulator(self.builddir, self.downloaddir, self.logtofile) + self.emulator = Emulator(self.builddir, self.downloaddir, + self.logtofile, self.timeout_multiplier) def tearDown(self): self.show_msg("Cleaning up") diff --git a/support/testing/infra/emulator.py b/support/testing/infra/emulator.py index 588a92f6a2..214a1067e9 100644 --- a/support/testing/infra/emulator.py +++ b/support/testing/infra/emulator.py @@ -5,10 +5,14 @@ import infra class Emulator(object): - def __init__(self, builddir, downloaddir, logtofile): + def __init__(self, builddir, downloaddir, logtofile, multiplier): self.qemu = None self.downloaddir = downloaddir self.logfile = infra.open_log_file(builddir, "run", logtofile) + # We use elastic runners on the cloud to runs our tests. Those runners + # can take a long time to run the emulator. Use a timeout multiplier + # when running the tests to avoid sporadic failures. + self.multiplier = multiplier # Start Qemu to boot the system # @@ -65,7 +69,8 @@ class Emulator(object): qemu_cmd += ["-append", " ".join(kernel_cmdline)] self.logfile.write("> starting qemu with '%s'\n" % " ".join(qemu_cmd)) - self.qemu = pexpect.spawn(qemu_cmd[0], qemu_cmd[1:], timeout=5, + self.qemu = pexpect.spawn(qemu_cmd[0], qemu_cmd[1:], + timeout=5 * self.multiplier, env={"QEMU_AUDIO_DRV": "none"}) # We want only stdout into the log to avoid double echo self.qemu.logfile_read = self.logfile @@ -76,7 +81,7 @@ class Emulator(object): # The login prompt can take some time to appear when running multiple # instances in parallel, so set the timeout to a large value index = self.qemu.expect(["buildroot login:", pexpect.TIMEOUT], - timeout=60) + timeout=60 * self.multiplier) if index != 0: self.logfile.write("==> System does not boot") raise SystemError("System does not boot") @@ -94,6 +99,8 @@ class Emulator(object): # return a tuple (output, exit_code) def run(self, cmd, timeout=-1): self.qemu.sendline(cmd) + if timeout != -1: + timeout *= self.multiplier self.qemu.expect("# ", timeout=timeout) # Remove double carriage return from qemu stdout so str.splitlines() # works as expected. diff --git a/support/testing/run-tests b/support/testing/run-tests index 0cb673c61f..95c1565f72 100755 --- a/support/testing/run-tests +++ b/support/testing/run-tests @@ -28,6 +28,8 @@ def main(): help='number of testcases to run simultaneously') parser.add_argument('-j', '--jlevel', type=int, help='BR2_JLEVEL to use for each testcase') + parser.add_argument('--timeout-multiplier', type=int, default=1, + help='increase timeouts (useful for slow machines)') args = parser.parse_args() @@ -97,6 +99,13 @@ def main(): # the user can override the auto calculated value BRTest.jlevel = args.jlevel + if args.timeout_multiplier < 1: + print "Invalid multiplier for timeout values" + print "" + parser.print_help() + return 1 + BRTest.timeout_multiplier = args.timeout_multiplier + nose2_args = ["-v", "-N", str(args.testcases), "-s", "support/testing", -- 2.13.0 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [Buildroot] [PATCH v2 2/2] .gitlab-ci.yml: use large timeouts for runtime tests 2017-08-05 2:05 ` [Buildroot] [PATCH v2 1/2] support/testing: allow to use a multiplier for timeouts Ricardo Martincoski @ 2017-08-05 2:05 ` Ricardo Martincoski 2017-08-10 9:18 ` [Buildroot] [PATCH v2 1/2] support/testing: allow to use a multiplier for timeouts Arnout Vandecappelle 1 sibling, 0 replies; 12+ messages in thread From: Ricardo Martincoski @ 2017-08-05 2:05 UTC (permalink / raw) To: buildroot Multiply the timeouts for emulator in the gitlab runners by 10 to avoid sporadic failures in elastic runners. Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> --- Changes v1 -> v2: - use --timeout-multiplier instead of -e, as it changed in v2 of the previous patch; Sample pipeline using only gitlab runners: https://gitlab.com/RicardoMartincoski/buildroot/pipelines/10605884 PS: for this other sporadic error: ERROR: Job failed: execution took longer than 1h0m0s seconds https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/28241402 see https://docs.gitlab.com/ee/user/project/pipelines/settings.html#timeout --- .gitlab-ci.yml | 4 +++- .gitlab-ci.yml.in | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 90efd81e50..19335d7bfd 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -49,7 +49,9 @@ check-DEVELOPERS: .runtime_test: &runtime_test # Keep build directories so the rootfs can be an artifact of the job. The # runner will clean up those files for us. - script: ./support/testing/run-tests -o test-output/ -d test-dl/ -k ${CI_BUILD_NAME} + # Multiply every emulator timeout by 10 to avoid sporadic failures in + # elastic runners. + script: ./support/testing/run-tests -o test-output/ -d test-dl/ -k --timeout-multiplier 10 ${CI_BUILD_NAME} artifacts: when: always expire_in: 2 weeks diff --git a/.gitlab-ci.yml.in b/.gitlab-ci.yml.in index b8fce9ef96..3abf7d5313 100644 --- a/.gitlab-ci.yml.in +++ b/.gitlab-ci.yml.in @@ -49,7 +49,9 @@ check-DEVELOPERS: .runtime_test: &runtime_test # Keep build directories so the rootfs can be an artifact of the job. The # runner will clean up those files for us. - script: ./support/testing/run-tests -o test-output/ -d test-dl/ -k ${CI_BUILD_NAME} + # Multiply every emulator timeout by 10 to avoid sporadic failures in + # elastic runners. + script: ./support/testing/run-tests -o test-output/ -d test-dl/ -k --timeout-multiplier 10 ${CI_BUILD_NAME} artifacts: when: always expire_in: 2 weeks -- 2.13.0 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [Buildroot] [PATCH v2 1/2] support/testing: allow to use a multiplier for timeouts 2017-08-05 2:05 ` [Buildroot] [PATCH v2 1/2] support/testing: allow to use a multiplier for timeouts Ricardo Martincoski 2017-08-05 2:05 ` [Buildroot] [PATCH v2 2/2] .gitlab-ci.yml: use large timeouts for runtime tests Ricardo Martincoski @ 2017-08-10 9:18 ` Arnout Vandecappelle 1 sibling, 0 replies; 12+ messages in thread From: Arnout Vandecappelle @ 2017-08-10 9:18 UTC (permalink / raw) To: buildroot On 05-08-17 04:05, Ricardo Martincoski wrote: > Add a parameter to run-tests to act as a multiplier for all timeouts of > emulator. > It can be used to avoid sporadic failures on slow host machines as well > in elastic runners on the cloud. > > Cc: Arnout Vandecappelle <arnout@mind.be> > Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> > Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> [snip] > - def __init__(self, builddir, downloaddir, logtofile): > + def __init__(self, builddir, downloaddir, logtofile, multiplier): For consistency, it would be better to use timeout_multiplier here as well. I've fixed that up and applied both to master, thanks. Regards, Arnout [snip] -- Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF ^ permalink raw reply [flat|nested] 12+ messages in thread
* [Buildroot] [RFC PATCH 3/4] .gitlab-ci.yml: use large timeouts for runtime tests 2017-07-30 4:49 [Buildroot] [RFC PATCH 1/4] .gitlab-ci.yml: save rootfs as artifact for runtime tests Ricardo Martincoski 2017-07-30 4:49 ` [Buildroot] [RFC PATCH 2/4] support/testing: allow to use extra large timeouts Ricardo Martincoski @ 2017-07-30 4:49 ` Ricardo Martincoski 2017-07-30 4:49 ` [Buildroot] [RFC PATCH 4/4] testing/infra/emulator: remove qemu warnings about audio Ricardo Martincoski 2017-07-31 19:24 ` [Buildroot] [RFC PATCH 1/4] .gitlab-ci.yml: save rootfs as artifact for runtime tests Thomas Petazzoni 3 siblings, 0 replies; 12+ messages in thread From: Ricardo Martincoski @ 2017-07-30 4:49 UTC (permalink / raw) To: buildroot Multiply the timeouts for emulator in the gitlab runners by 10 to avoid sporadic failures in elastic runners. Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> --- See also http://lists.busybox.net/pipermail/buildroot/2017-July/199329.html --- .gitlab-ci.yml | 4 +++- .gitlab-ci.yml.in | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index f9e5b1fa6b..b1a1cd31a4 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -49,7 +49,9 @@ check-DEVELOPERS: .runtime_test: &runtime_test # Keep build directories so the rootfs can be an artifact of the job. The # runner will clean up those files for us. - script: ./support/testing/run-tests -o test-output/ -d test-dl/ -k ${CI_BUILD_NAME} + # Multiply every emulator timeout by 10 to avoid sporadic failures in + # elastic runners. + script: ./support/testing/run-tests -o test-output/ -d test-dl/ -k -e 10 ${CI_BUILD_NAME} artifacts: when: always expire_in: 2 weeks diff --git a/.gitlab-ci.yml.in b/.gitlab-ci.yml.in index b8fce9ef96..9d3ddfec3c 100644 --- a/.gitlab-ci.yml.in +++ b/.gitlab-ci.yml.in @@ -49,7 +49,9 @@ check-DEVELOPERS: .runtime_test: &runtime_test # Keep build directories so the rootfs can be an artifact of the job. The # runner will clean up those files for us. - script: ./support/testing/run-tests -o test-output/ -d test-dl/ -k ${CI_BUILD_NAME} + # Multiply every emulator timeout by 10 to avoid sporadic failures in + # elastic runners. + script: ./support/testing/run-tests -o test-output/ -d test-dl/ -k -e 10 ${CI_BUILD_NAME} artifacts: when: always expire_in: 2 weeks -- 2.13.0 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [Buildroot] [RFC PATCH 4/4] testing/infra/emulator: remove qemu warnings about audio 2017-07-30 4:49 [Buildroot] [RFC PATCH 1/4] .gitlab-ci.yml: save rootfs as artifact for runtime tests Ricardo Martincoski 2017-07-30 4:49 ` [Buildroot] [RFC PATCH 2/4] support/testing: allow to use extra large timeouts Ricardo Martincoski 2017-07-30 4:49 ` [Buildroot] [RFC PATCH 3/4] .gitlab-ci.yml: use large timeouts for runtime tests Ricardo Martincoski @ 2017-07-30 4:49 ` Ricardo Martincoski 2017-07-31 19:26 ` Thomas Petazzoni 2017-07-31 19:24 ` [Buildroot] [RFC PATCH 1/4] .gitlab-ci.yml: save rootfs as artifact for runtime tests Thomas Petazzoni 3 siblings, 1 reply; 12+ messages in thread From: Ricardo Martincoski @ 2017-07-30 4:49 UTC (permalink / raw) To: buildroot The default audio backend for qemu is configured at compile time. It generates annoying warning messages to qemu's stderr when running our tests, like these: pulseaudio: set_sink_input_volume() failed pulseaudio: Reason: Invalid argument pulseaudio: set_sink_input_mute() failed pulseaudio: Reason: Invalid argument Explicitly set the audio backend to "none" at runtime to remove those messages from our logs. There is no command line argument for this, so use an environment variable when starting qemu. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> --- support/testing/infra/emulator.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/support/testing/infra/emulator.py b/support/testing/infra/emulator.py index 7f02a01553..fe3da869e7 100644 --- a/support/testing/infra/emulator.py +++ b/support/testing/infra/emulator.py @@ -70,7 +70,9 @@ class Emulator(object): self.logfile.write("> starting qemu with '%s'\n" % " ".join(qemu_cmd)) self.qemu = pexpect.spawn(qemu_cmd[0], qemu_cmd[1:], - timeout=5 * self.multiplier) + timeout=5 * self.multiplier, + env={"QEMU_AUDIO_DRV": "none"}) + # We want only stdout into the log to avoid double echo self.qemu.logfile_read = self.logfile -- 2.13.0 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [Buildroot] [RFC PATCH 4/4] testing/infra/emulator: remove qemu warnings about audio 2017-07-30 4:49 ` [Buildroot] [RFC PATCH 4/4] testing/infra/emulator: remove qemu warnings about audio Ricardo Martincoski @ 2017-07-31 19:26 ` Thomas Petazzoni 0 siblings, 0 replies; 12+ messages in thread From: Thomas Petazzoni @ 2017-07-31 19:26 UTC (permalink / raw) To: buildroot Hello, On Sun, 30 Jul 2017 01:49:46 -0300, Ricardo Martincoski wrote: > The default audio backend for qemu is configured at compile time. It > generates annoying warning messages to qemu's stderr when running our > tests, like these: > pulseaudio: set_sink_input_volume() failed > pulseaudio: Reason: Invalid argument > pulseaudio: set_sink_input_mute() failed > pulseaudio: Reason: Invalid argument > > Explicitly set the audio backend to "none" at runtime to remove those > messages from our logs. There is no command line argument for this, so > use an environment variable when starting qemu. > > Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> > --- > support/testing/infra/emulator.py | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) Applied to master, thanks. Thomas -- Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com ^ permalink raw reply [flat|nested] 12+ messages in thread
* [Buildroot] [RFC PATCH 1/4] .gitlab-ci.yml: save rootfs as artifact for runtime tests 2017-07-30 4:49 [Buildroot] [RFC PATCH 1/4] .gitlab-ci.yml: save rootfs as artifact for runtime tests Ricardo Martincoski ` (2 preceding siblings ...) 2017-07-30 4:49 ` [Buildroot] [RFC PATCH 4/4] testing/infra/emulator: remove qemu warnings about audio Ricardo Martincoski @ 2017-07-31 19:24 ` Thomas Petazzoni 3 siblings, 0 replies; 12+ messages in thread From: Thomas Petazzoni @ 2017-07-31 19:24 UTC (permalink / raw) To: buildroot Hello, On Sun, 30 Jul 2017 01:49:43 -0300, Ricardo Martincoski wrote: > Sometimes when a test fails in a gitlab pipeline the reason of the > failure cannot be determined using only the logfile. > > Add the modified rootfs as an artifact of the job to improve > troubleshooting. To accomplish this, always use -k option from the test > infra, unconditionally add the resulting images to the artifacts, and > let the runner do the cleanup for us. > > These artifacts can also be useful when a test fails locally but pass at > gitlab runners. > > When the test does not generate a image, this message is displayed in > the runner log: > WARNING: test-output/*/images/*: no matching files > > Cc: Arnout Vandecappelle <arnout@mind.be> > Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> > Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> > --- > See also > http://lists.busybox.net/pipermail/buildroot/2017-July/199332.html > --- > .gitlab-ci.yml | 5 ++++- > .gitlab-ci.yml.in | 5 ++++- > 2 files changed, 8 insertions(+), 2 deletions(-) Applied to master, thanks. Thomas -- Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2017-08-10 9:18 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-07-30 4:49 [Buildroot] [RFC PATCH 1/4] .gitlab-ci.yml: save rootfs as artifact for runtime tests Ricardo Martincoski 2017-07-30 4:49 ` [Buildroot] [RFC PATCH 2/4] support/testing: allow to use extra large timeouts Ricardo Martincoski 2017-07-31 19:25 ` Thomas Petazzoni 2017-07-31 23:27 ` Ricardo Martincoski 2017-08-01 6:01 ` Thomas Petazzoni 2017-08-05 2:05 ` [Buildroot] [PATCH v2 1/2] support/testing: allow to use a multiplier for timeouts Ricardo Martincoski 2017-08-05 2:05 ` [Buildroot] [PATCH v2 2/2] .gitlab-ci.yml: use large timeouts for runtime tests Ricardo Martincoski 2017-08-10 9:18 ` [Buildroot] [PATCH v2 1/2] support/testing: allow to use a multiplier for timeouts Arnout Vandecappelle 2017-07-30 4:49 ` [Buildroot] [RFC PATCH 3/4] .gitlab-ci.yml: use large timeouts for runtime tests Ricardo Martincoski 2017-07-30 4:49 ` [Buildroot] [RFC PATCH 4/4] testing/infra/emulator: remove qemu warnings about audio Ricardo Martincoski 2017-07-31 19:26 ` Thomas Petazzoni 2017-07-31 19:24 ` [Buildroot] [RFC PATCH 1/4] .gitlab-ci.yml: save rootfs as artifact for runtime tests Thomas Petazzoni
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.