* [Buildroot] [PATCH 0/3] support/testing: misc improvements (branch yem/runtime-test-ppd) @ 2022-12-24 9:18 Yann E. MORIN 2022-12-24 9:18 ` [Buildroot] [PATCH 1/3] support/tests: print failed command and output on assertRunOK error Yann E. MORIN ` (2 more replies) 0 siblings, 3 replies; 10+ messages in thread From: Yann E. MORIN @ 2022-12-24 9:18 UTC (permalink / raw) To: buildroot; +Cc: Yann E . MORIN, Thomas Petazzoni Hello All! This small three-patch series brings in two improvements to the runtime testing infrastructure: - enable top-level parallel build (TLPB) for tests that have per-package directories (PPD) enabled; - display the failed commands and its output on assertRunOk(). Finally, it enables PPD, and thus TLPB, for systemd test cases. Regards, Yann E. MORIN. ---------------------------------------------------------------- Yann E. MORIN (3): support/tests: print failed command and output on assertRunOK error support/tests: allow top-level parallel builds support/tests: enable PPD, and thus TLPB, for systemd tests support/testing/infra/basetest.py | 13 +++++++++---- support/testing/infra/builder.py | 5 ++++- support/testing/tests/init/test_systemd.py | 1 + 3 files changed, 14 insertions(+), 5 deletions(-) -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 10+ messages in thread
* [Buildroot] [PATCH 1/3] support/tests: print failed command and output on assertRunOK error 2022-12-24 9:18 [Buildroot] [PATCH 0/3] support/testing: misc improvements (branch yem/runtime-test-ppd) Yann E. MORIN @ 2022-12-24 9:18 ` Yann E. MORIN 2022-12-27 20:42 ` Thomas Petazzoni via buildroot 2022-12-24 9:18 ` [Buildroot] [PATCH 2/3] support/tests: allow top-level parallel builds Yann E. MORIN 2022-12-24 9:18 ` [Buildroot] [PATCH 3/3] support/tests: enable PPD, and thus TLPB, for systemd tests Yann E. MORIN 2 siblings, 1 reply; 10+ messages in thread From: Yann E. MORIN @ 2022-12-24 9:18 UTC (permalink / raw) To: buildroot; +Cc: Yann E. MORIN, Thomas Petazzoni Currently, when asserting that a command succeeded, we just capture the return code of the command. If that is not zero, the assertion fails, but the error message is not very splicit: AssertionError: 1 != 0 Replace the error message with an explicit message that dumps the failed command, the error code, and the resulting output. Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> --- support/testing/infra/basetest.py | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/support/testing/infra/basetest.py b/support/testing/infra/basetest.py index 96c6848dfc..45bcd4c2e2 100644 --- a/support/testing/infra/basetest.py +++ b/support/testing/infra/basetest.py @@ -88,7 +88,12 @@ class BRTest(BRConfigTest): super(BRTest, self).tearDown() # Run the given 'cmd' with a 'timeout' on the target and - # assert that the command succeeded + # assert that the command succeeded; on error, print the + # faulty command and its output def assertRunOk(self, cmd, timeout=-1): - _, exit_code = self.emulator.run(cmd, timeout) - self.assertEqual(exit_code, 0) + out, exit_code = self.emulator.run(cmd, timeout) + self.assertEqual( + exit_code, + 0, + "\nFailed to run: {}\noutput was:\n{}".format(cmd, ' '+'\n '.join(out)) + ) -- 2.25.1 _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [Buildroot] [PATCH 1/3] support/tests: print failed command and output on assertRunOK error 2022-12-24 9:18 ` [Buildroot] [PATCH 1/3] support/tests: print failed command and output on assertRunOK error Yann E. MORIN @ 2022-12-27 20:42 ` Thomas Petazzoni via buildroot 0 siblings, 0 replies; 10+ messages in thread From: Thomas Petazzoni via buildroot @ 2022-12-27 20:42 UTC (permalink / raw) To: Yann E. MORIN; +Cc: buildroot On Sat, 24 Dec 2022 10:18:11 +0100 "Yann E. MORIN" <yann.morin.1998@free.fr> wrote: > Currently, when asserting that a command succeeded, we just capture the > return code of the command. If that is not zero, the assertion fails, > but the error message is not very splicit: > AssertionError: 1 != 0 > > Replace the error message with an explicit message that dumps the failed > command, the error code, and the resulting output. > > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> > Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> > --- > support/testing/infra/basetest.py | 11 ++++++++--- > 1 file changed, 8 insertions(+), 3 deletions(-) Applied to master, thanks. Thomas -- Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 10+ messages in thread
* [Buildroot] [PATCH 2/3] support/tests: allow top-level parallel builds 2022-12-24 9:18 [Buildroot] [PATCH 0/3] support/testing: misc improvements (branch yem/runtime-test-ppd) Yann E. MORIN 2022-12-24 9:18 ` [Buildroot] [PATCH 1/3] support/tests: print failed command and output on assertRunOK error Yann E. MORIN @ 2022-12-24 9:18 ` Yann E. MORIN 2022-12-27 20:45 ` Thomas Petazzoni via buildroot 2023-02-07 8:39 ` Thomas Petazzoni via buildroot 2022-12-24 9:18 ` [Buildroot] [PATCH 3/3] support/tests: enable PPD, and thus TLPB, for systemd tests Yann E. MORIN 2 siblings, 2 replies; 10+ messages in thread From: Yann E. MORIN @ 2022-12-24 9:18 UTC (permalink / raw) To: buildroot; +Cc: Yann E. MORIN, Thomas Petazzoni Running tests with top-level parallel builds can speed up running some tests, expecially those that have a lot of packages like the systemd init tests. Trigger TLPB when the configuration enables per-package directories. Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> --- support/testing/infra/basetest.py | 2 +- support/testing/infra/builder.py | 5 ++++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/support/testing/infra/basetest.py b/support/testing/infra/basetest.py index 45bcd4c2e2..670c7213d6 100644 --- a/support/testing/infra/basetest.py +++ b/support/testing/infra/basetest.py @@ -52,7 +52,7 @@ class BRConfigTest(unittest.TestCase): def setUp(self): self.show_msg("Starting") - self.b = Builder(self.config, self.builddir, self.logtofile) + self.b = Builder(self.config, self.builddir, self.logtofile, self.jlevel) if not self.keepbuilds: self.b.delete() diff --git a/support/testing/infra/builder.py b/support/testing/infra/builder.py index 922a707220..a2abb9ed89 100644 --- a/support/testing/infra/builder.py +++ b/support/testing/infra/builder.py @@ -6,11 +6,12 @@ import infra class Builder(object): - def __init__(self, config, builddir, logtofile): + def __init__(self, config, builddir, logtofile, jlevel=None): self.config = '\n'.join([line.lstrip() for line in config.splitlines()]) + '\n' self.builddir = builddir self.logfile = infra.open_log_file(builddir, "build", logtofile) + self.jlevel = jlevel def is_defconfig_valid(self, configfile, defconfig): """Check if the .config is contains all lines present in the defconfig.""" @@ -87,6 +88,8 @@ class Builder(object): env.update(make_extra_env) cmd = ["make", "-C", self.builddir] + if "BR2_PER_PACKAGE_DIRECTORIES=y" in self.config.splitlines() and self.jlevel: + cmd.append(f"-j{self.jlevel}") cmd += make_extra_opts ret = subprocess.call(cmd, stdout=self.logfile, stderr=self.logfile, -- 2.25.1 _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [Buildroot] [PATCH 2/3] support/tests: allow top-level parallel builds 2022-12-24 9:18 ` [Buildroot] [PATCH 2/3] support/tests: allow top-level parallel builds Yann E. MORIN @ 2022-12-27 20:45 ` Thomas Petazzoni via buildroot 2022-12-27 20:53 ` Yann E. MORIN 2023-02-07 8:39 ` Thomas Petazzoni via buildroot 1 sibling, 1 reply; 10+ messages in thread From: Thomas Petazzoni via buildroot @ 2022-12-27 20:45 UTC (permalink / raw) To: Yann E. MORIN; +Cc: buildroot On Sat, 24 Dec 2022 10:18:12 +0100 "Yann E. MORIN" <yann.morin.1998@free.fr> wrote: > Running tests with top-level parallel builds can speed up running some > tests, expecially those that have a lot of packages like the systemd > init tests. > > Trigger TLPB when the configuration enables per-package directories. > > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> > Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> I certainly support the idea of running some tests with TLPB. However, this implementation makes a confusion between two different settings: - The existing BRConfigTest.jlevel, which is set by run-tests -j, and used to define BR2_JLEVEL in the Buildroot configuration of each test case. This determines the number of parallel jobs used to build each independent package. - Your new work, which uses "make -j" to do TLPB... but relies on the same above value, even though it's a completely different setting. Is this expected? Best regards, Thomas -- Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training https://bootlin.com _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Buildroot] [PATCH 2/3] support/tests: allow top-level parallel builds 2022-12-27 20:45 ` Thomas Petazzoni via buildroot @ 2022-12-27 20:53 ` Yann E. MORIN 2022-12-27 20:55 ` Yann E. MORIN 0 siblings, 1 reply; 10+ messages in thread From: Yann E. MORIN @ 2022-12-27 20:53 UTC (permalink / raw) To: Thomas Petazzoni; +Cc: buildroot Thomas, All, On 2022-12-27 21:45 +0100, Thomas Petazzoni via buildroot spake thusly: > On Sat, 24 Dec 2022 10:18:12 +0100 > "Yann E. MORIN" <yann.morin.1998@free.fr> wrote: > > Running tests with top-level parallel builds can speed up running some > > tests, expecially those that have a lot of packages like the systemd > > init tests. > > > > Trigger TLPB when the configuration enables per-package directories. > > > > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> > > Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> > > I certainly support the idea of running some tests with TLPB. However, > this implementation makes a confusion between two different settings: > > - The existing BRConfigTest.jlevel, which is set by run-tests -j, and > used to define BR2_JLEVEL in the Buildroot configuration of each test > case. This determines the number of parallel jobs used to build each > independent package. > > - Your new work, which uses "make -j" to do TLPB... but relies on the > same above value, even though it's a completely different setting. > > Is this expected? Yes, this is the intended behaviour, which I was explicitly seeking. So, if one runs with PPD and TLPB (outside the run-time infra), one would do something like: $ make -jN This spawns a top-level make process that is parallel. In turn, in rules, when we call $(MAKE), this is the magic that tells make that it is recursive, but that it should use the jobserver from the calling process. So, in this case, the BR2_JLEVEL is unused by whatever uses the make jobserver; only the number of jobs in the top-level jobserver is meaningful, i.e. whatever we pass as -jN. The exception, of course, is whatever uses BR2_JLEVEL but does not talk to the jobserver, but this is mostly a few packages (scons, waf et al.). Even ninja packages do talk to the top-level jobserver, now that we use the ninja fork that knows to talk to it. So, yes, using top-level -jN with the same value as BR2_JLEVEL is exactly what I intended to do. Unless I totally missed something... Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Buildroot] [PATCH 2/3] support/tests: allow top-level parallel builds 2022-12-27 20:53 ` Yann E. MORIN @ 2022-12-27 20:55 ` Yann E. MORIN 0 siblings, 0 replies; 10+ messages in thread From: Yann E. MORIN @ 2022-12-27 20:55 UTC (permalink / raw) To: Thomas Petazzoni; +Cc: buildroot Thomas, All, On 2022-12-27 21:53 +0100, Yann E. MORIN spake thusly: > On 2022-12-27 21:45 +0100, Thomas Petazzoni via buildroot spake thusly: > > On Sat, 24 Dec 2022 10:18:12 +0100 > > "Yann E. MORIN" <yann.morin.1998@free.fr> wrote: > > > Running tests with top-level parallel builds can speed up running some > > > tests, expecially those that have a lot of packages like the systemd > > > init tests. > > > > > > Trigger TLPB when the configuration enables per-package directories. > > > > > > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> > > > Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> > > > > I certainly support the idea of running some tests with TLPB. However, > > this implementation makes a confusion between two different settings: > > > > - The existing BRConfigTest.jlevel, which is set by run-tests -j, and > > used to define BR2_JLEVEL in the Buildroot configuration of each test > > case. This determines the number of parallel jobs used to build each > > independent package. > > > > - Your new work, which uses "make -j" to do TLPB... but relies on the > > same above value, even though it's a completely different setting. > > > > Is this expected? > > Yes, this is the intended behaviour, which I was explicitly seeking. > > So, if one runs with PPD and TLPB (outside the run-time infra), one > would do something like: > > $ make -jN > > This spawns a top-level make process that is parallel. In turn, in > rules, when we call $(MAKE), this is the magic that tells make that > it is recursive, but that it should use the jobserver from the calling > process. > > So, in this case, the BR2_JLEVEL is unused by whatever uses the make > jobserver; only the number of jobs in the top-level jobserver is > meaningful, i.e. whatever we pass as -jN. Slight correction: BR2_JLEVEL does have an actual effect, but only if its value is lower than the one we pass as -jN. So, in practice, we want to have BR2_JLEVEL equal to the top-level -jN. Regards, Yann E. MORIN. > The exception, of course, is whatever uses BR2_JLEVEL but does not talk > to the jobserver, but this is mostly a few packages (scons, waf et al.). > Even ninja packages do talk to the top-level jobserver, now that we use > the ninja fork that knows to talk to it. > > So, yes, using top-level -jN with the same value as BR2_JLEVEL is > exactly what I intended to do. > > Unless I totally missed something... > > Regards, > Yann E. MORIN. > > -- > .-----------------.--------------------.------------------.--------------------. > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > '------------------------------^-------^------------------^--------------------' > _______________________________________________ > buildroot mailing list > buildroot@buildroot.org > https://lists.buildroot.org/mailman/listinfo/buildroot -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Buildroot] [PATCH 2/3] support/tests: allow top-level parallel builds 2022-12-24 9:18 ` [Buildroot] [PATCH 2/3] support/tests: allow top-level parallel builds Yann E. MORIN 2022-12-27 20:45 ` Thomas Petazzoni via buildroot @ 2023-02-07 8:39 ` Thomas Petazzoni via buildroot 1 sibling, 0 replies; 10+ messages in thread From: Thomas Petazzoni via buildroot @ 2023-02-07 8:39 UTC (permalink / raw) To: Yann E. MORIN; +Cc: buildroot On Sat, 24 Dec 2022 10:18:12 +0100 "Yann E. MORIN" <yann.morin.1998@free.fr> wrote: > Running tests with top-level parallel builds can speed up running some > tests, expecially those that have a lot of packages like the systemd > init tests. > > Trigger TLPB when the configuration enables per-package directories. > > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> > Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> > --- > support/testing/infra/basetest.py | 2 +- > support/testing/infra/builder.py | 5 ++++- > 2 files changed, 5 insertions(+), 2 deletions(-) Thanks to the additional explanation you provided, I understood better the idea, and therefore applied the patch. I updated the commit log a little bit to summarize the explanation you gave, and applied. Thanks! Thomas -- Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 10+ messages in thread
* [Buildroot] [PATCH 3/3] support/tests: enable PPD, and thus TLPB, for systemd tests 2022-12-24 9:18 [Buildroot] [PATCH 0/3] support/testing: misc improvements (branch yem/runtime-test-ppd) Yann E. MORIN 2022-12-24 9:18 ` [Buildroot] [PATCH 1/3] support/tests: print failed command and output on assertRunOK error Yann E. MORIN 2022-12-24 9:18 ` [Buildroot] [PATCH 2/3] support/tests: allow top-level parallel builds Yann E. MORIN @ 2022-12-24 9:18 ` Yann E. MORIN 2023-02-07 8:39 ` Thomas Petazzoni via buildroot 2 siblings, 1 reply; 10+ messages in thread From: Yann E. MORIN @ 2022-12-24 9:18 UTC (permalink / raw) To: buildroot; +Cc: Yann E. MORIN, Thomas Petazzoni Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> --- support/testing/tests/init/test_systemd.py | 1 + 1 file changed, 1 insertion(+) diff --git a/support/testing/tests/init/test_systemd.py b/support/testing/tests/init/test_systemd.py index 06c42736ae..ddc32b0838 100644 --- a/support/testing/tests/init/test_systemd.py +++ b/support/testing/tests/init/test_systemd.py @@ -13,6 +13,7 @@ class InitSystemSystemdBase(InitSystemBase): BR2_INIT_SYSTEMD=y BR2_TARGET_GENERIC_GETTY_PORT="ttyAMA0" # BR2_TARGET_ROOTFS_TAR is not set + BR2_PER_PACKAGE_DIRECTORIES=y """ def check_systemd(self, fs): -- 2.25.1 _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [Buildroot] [PATCH 3/3] support/tests: enable PPD, and thus TLPB, for systemd tests 2022-12-24 9:18 ` [Buildroot] [PATCH 3/3] support/tests: enable PPD, and thus TLPB, for systemd tests Yann E. MORIN @ 2023-02-07 8:39 ` Thomas Petazzoni via buildroot 0 siblings, 0 replies; 10+ messages in thread From: Thomas Petazzoni via buildroot @ 2023-02-07 8:39 UTC (permalink / raw) To: Yann E. MORIN; +Cc: buildroot On Sat, 24 Dec 2022 10:18:13 +0100 "Yann E. MORIN" <yann.morin.1998@free.fr> wrote: > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> > Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> > --- > support/testing/tests/init/test_systemd.py | 1 + > 1 file changed, 1 insertion(+) Applied to master, thanks. Thomas -- Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2023-02-07 8:39 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-12-24 9:18 [Buildroot] [PATCH 0/3] support/testing: misc improvements (branch yem/runtime-test-ppd) Yann E. MORIN 2022-12-24 9:18 ` [Buildroot] [PATCH 1/3] support/tests: print failed command and output on assertRunOK error Yann E. MORIN 2022-12-27 20:42 ` Thomas Petazzoni via buildroot 2022-12-24 9:18 ` [Buildroot] [PATCH 2/3] support/tests: allow top-level parallel builds Yann E. MORIN 2022-12-27 20:45 ` Thomas Petazzoni via buildroot 2022-12-27 20:53 ` Yann E. MORIN 2022-12-27 20:55 ` Yann E. MORIN 2023-02-07 8:39 ` Thomas Petazzoni via buildroot 2022-12-24 9:18 ` [Buildroot] [PATCH 3/3] support/tests: enable PPD, and thus TLPB, for systemd tests Yann E. MORIN 2023-02-07 8:39 ` Thomas Petazzoni via buildroot
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.