[PATCH 0/2] Add check-native and check-nonnative as make targets

[PATCH 0/2] Add check-native and check-nonnative as make targets

[Glenn Washburn]

Tests can be put into two categories, native (tests that run on the build
system) and non-native (tests run in QEMU). For any two targets (even of
completely different architectures), the native tests will be running the
same binary code (because they will be compiled for and run on the build
machine), and thus will have the same result. So when building and
running tests for multiple targets on a build machine, the native tests on
need be run once. This can decrease the runtime of a multi-target test run
significantly (like hours). This patch series makes it possible to run only
run the non-native tests (skipping the native tests) by partitioning the set
of tests into native and non-native based on whether QEMU is used by the test.
The first patch is not necessary, but makes things look cleaner.

Phcoder, could you confirm that the logic above is sound?

Glenn

Glenn Washburn (2):
  conf/Makefile.common: Order alphabetically variables
  tests: Add check-native and check-nonnative make targets

 Makefile.am          |   9 +++
 Makefile.util.def    | 164 +++++++++++++++++++++----------------------
 conf/Makefile.common |  22 +++---
 gentpl.py            |   6 +-
 4 files changed, 106 insertions(+), 95 deletions(-)

-- 
2.27.0

[PATCH 1/2] conf/Makefile.common: Order alphabetically variables

[Glenn Washburn]

Signed-off-by: Glenn Washburn <development@efficientek.com>
---
 conf/Makefile.common | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/conf/Makefile.common b/conf/Makefile.common
index 2a1a886f6..f0bb6e160 100644
--- a/conf/Makefile.common
+++ b/conf/Makefile.common
@@ -102,24 +102,24 @@ MODULE_FILES =
 MARKER_FILES =
 KERNEL_HEADER_FILES =
 
-man_MANS =
-noinst_DATA =
-pkgdata_DATA =
 bin_SCRIPTS =
-sbin_SCRIPTS =
 bin_PROGRAMS =
-platform_DATA =
-sbin_PROGRAMS =
 check_SCRIPTS =
-dist_grubconf_DATA =
 check_PROGRAMS =
+dist_grubconf_DATA =
+dist_noinst_DATA =
+grubconf_SCRIPTS =
+man_MANS =
+noinst_DATA =
 noinst_SCRIPTS =
 noinst_PROGRAMS =
-grubconf_SCRIPTS =
 noinst_LIBRARIES =
-dist_noinst_DATA =
+pkgdata_DATA =
+platform_DATA =
 platform_SCRIPTS =
 platform_PROGRAMS =
+sbin_SCRIPTS =
+sbin_PROGRAMS =
 
 TESTS =
 EXTRA_DIST =
-- 
2.27.0

[PATCH 2/2] tests: Add check-native and check-nonnative make targets

[Glenn Washburn]

This allows for testing only tests that run directly on the build machine or
only tests that run in a virtualized environment. When testing multiple
targets on the same build machine the native tests only need to be run once
for all targets. Whereas, the nonnative tests must be run for each target
because the test is potentially compiled differently for each target.

Signed-off-by: Glenn Washburn <development@efficientek.com>
---
 Makefile.am          |   9 +++
 Makefile.util.def    | 164 +++++++++++++++++++++----------------------
 conf/Makefile.common |   4 ++
 gentpl.py            |   6 +-
 4 files changed, 97 insertions(+), 86 deletions(-)

diff --git a/Makefile.am b/Makefile.am
index bf9c1ba64..10faf670b 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -24,6 +24,15 @@ CCASFLAGS_PROGRAM += $(CCASFLAGS_GNULIB)
 
 include $(srcdir)/Makefile.util.am
 
+check_SCRIPTS = $(check_SCRIPTS_native) $(check_SCRIPTS_nonnative)
+check_PROGRAMS = $(check_PROGRAMS_native) $(check_PROGRAMS_nonnative)
+TESTS = $(check_SCRIPTS) $(check_PROGRAMS)
+
+check-native:
+	$(MAKE) TESTS="$(check_PROGRAMS_native) $(check_SCRIPTS_native)" check
+check-nonnative:
+	$(MAKE) TESTS="$(check_PROGRAMS_nonnative) $(check_SCRIPTS_nonnative)" check
+
 # XXX Use Automake's LEX & YACC support
 grub_script.tab.h: $(top_srcdir)/grub-core/script/parser.y
 	$(YACC) -d -p grub_script_yy -b grub_script $(top_srcdir)/grub-core/script/parser.y
diff --git a/Makefile.util.def b/Makefile.util.def
index f8b356cc1..b098d5bba 100644
--- a/Makefile.util.def
+++ b/Makefile.util.def
@@ -749,470 +749,470 @@ script = {
 };
 
 script = {
-  testcase;
+  testcase = native;
   name = ext234_test;
   common = tests/ext234_test.in;
 };
 
 script = {
-  testcase;
+  testcase = native;
   name = squashfs_test;
   common = tests/squashfs_test.in;
 };
 
 script = {
-  testcase;
+  testcase = native;
   name = iso9660_test;
   common = tests/iso9660_test.in;
 };
 
 script = {
-  testcase;
+  testcase = native;
   name = hfsplus_test;
   common = tests/hfsplus_test.in;
 };
 
 script = {
-  testcase;
+  testcase = native;
   name = ntfs_test;
   common = tests/ntfs_test.in;
 };
 
 script = {
-  testcase;
+  testcase = native;
   name = reiserfs_test;
   common = tests/reiserfs_test.in;
 };
 
 script = {
-  testcase;
+  testcase = native;
   name = fat_test;
   common = tests/fat_test.in;
 };
 
 script = {
-  testcase;
+  testcase = native;
   name = minixfs_test;
   common = tests/minixfs_test.in;
 };
 
 script = {
-  testcase;
+  testcase = native;
   name = xfs_test;
   common = tests/xfs_test.in;
 };
 
 script = {
-  testcase;
+  testcase = native;
   name = f2fs_test;
   common = tests/f2fs_test.in;
 };
 
 script = {
-  testcase;
+  testcase = native;
   name = nilfs2_test;
   common = tests/nilfs2_test.in;
 };
 
 script = {
-  testcase;
+  testcase = native;
   name = romfs_test;
   common = tests/romfs_test.in;
 };
 
 script = {
-  testcase;
+  testcase = native;
   name = exfat_test;
   common = tests/exfat_test.in;
 };
 
 script = {
-  testcase;
+  testcase = native;
   name = tar_test;
   common = tests/tar_test.in;
 };
 
 script = {
-  testcase;
+  testcase = native;
   name = udf_test;
   common = tests/udf_test.in;
 };
 
 script = {
-  testcase;
+  testcase = native;
   name = hfs_test;
   common = tests/hfs_test.in;
 };
 
 script = {
-  testcase;
+  testcase = native;
   name = jfs_test;
   common = tests/jfs_test.in;
 };
 
 script = {
-  testcase;
+  testcase = native;
   name = btrfs_test;
   common = tests/btrfs_test.in;
 };
 
 script = {
-  testcase;
+  testcase = native;
   name = zfs_test;
   common = tests/zfs_test.in;
 };
 
 script = {
-  testcase;
+  testcase = native;
   name = cpio_test;
   common = tests/cpio_test.in;
 };
 
 script = {
-  testcase;
+  testcase = native;
   name = example_scripted_test;
   common = tests/example_scripted_test.in;
 };
 
 script = {
-  testcase;
+  testcase = native;
   name = gettext_strings_test;
   common = tests/gettext_strings_test.in;
   extra_dist = po/exclude.pot;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = pata_test;
   common = tests/pata_test.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = ahci_test;
   common = tests/ahci_test.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = uhci_test;
   common = tests/uhci_test.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = ohci_test;
   common = tests/ohci_test.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = ehci_test;
   common = tests/ehci_test.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = example_grub_script_test;
   common = tests/example_grub_script_test.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_script_eval;
   common = tests/grub_script_eval.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_script_test;
   common = tests/grub_script_test.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_script_echo1;
   common = tests/grub_script_echo1.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_script_leading_whitespace;
   common = tests/grub_script_leading_whitespace.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_script_echo_keywords;
   common = tests/grub_script_echo_keywords.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_script_vars1;
   common = tests/grub_script_vars1.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_script_for1;
   common = tests/grub_script_for1.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_script_while1;
   common = tests/grub_script_while1.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_script_if;
   common = tests/grub_script_if.in;
 };
 
 script = {
-  testcase;
+  testcase = native;
   name = grub_script_blanklines;
   common = tests/grub_script_blanklines.in;
 };
 
 script = {
-  testcase;
+  testcase = native;
   name = grub_script_final_semicolon;
   common = tests/grub_script_final_semicolon.in;
 };
 
 script = {
-  testcase;
+  testcase = native;
   name = grub_script_dollar;
   common = tests/grub_script_dollar.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_script_comments;
   common = tests/grub_script_comments.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_script_functions;
   common = tests/grub_script_functions.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_script_break;
   common = tests/grub_script_break.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_script_continue;
   common = tests/grub_script_continue.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_script_shift;
   common = tests/grub_script_shift.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_script_blockarg;
   common = tests/grub_script_blockarg.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_script_setparams;
   common = tests/grub_script_setparams.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_script_return;
   common = tests/grub_script_return.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_cmd_regexp;
   common = tests/grub_cmd_regexp.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_cmd_date;
   common = tests/grub_cmd_date.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_cmd_set_date;
   common = tests/grub_cmd_set_date.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_cmd_sleep;
   common = tests/grub_cmd_sleep.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_script_expansion;
   common = tests/grub_script_expansion.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_script_not;
   common = tests/grub_script_not.in;
 };
 
 script = {
-  testcase;
+  testcase = native;
   name = grub_script_no_commands;
   common = tests/grub_script_no_commands.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = partmap_test;
   common = tests/partmap_test.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = hddboot_test;
   common = tests/hddboot_test.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = fddboot_test;
   common = tests/fddboot_test.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = cdboot_test;
   common = tests/cdboot_test.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = netboot_test;
   common = tests/netboot_test.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = pseries_test;
   common = tests/pseries_test.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = core_compress_test;
   common = tests/core_compress_test.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = xzcompress_test;
   common = tests/xzcompress_test.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = gzcompress_test;
   common = tests/gzcompress_test.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = lzocompress_test;
   common = tests/lzocompress_test.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_cmd_echo;
   common = tests/grub_cmd_echo.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = help_test;
   common = tests/help_test.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_script_gettext;
   common = tests/grub_script_gettext.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_script_escape_comma;
   common = tests/grub_script_escape_comma.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_script_strcmp;
   common = tests/grub_script_strcmp.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = test_sha512sum;
   common = tests/test_sha512sum.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = test_unset;
   common = tests/test_unset.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_func_test;
   common = tests/grub_func_test.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_cmd_tr;
   common = tests/grub_cmd_tr.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = file_filter_test;
   common = tests/file_filter_test.in;
 };
 
 script = {
-  testcase;
+  testcase = nonnative;
   name = grub_cmd_test;
   common = tests/grub_cmd_test.in;
 };
 
 script = {
-  testcase;
+  testcase = native;
   name = syslinux_test;
   common = tests/syslinux_test.in;
 };
 
 program = {
-  testcase;
+  testcase = native;
   name = example_unit_test;
   common = tests/example_unit_test.c;
   common = tests/lib/unit_test.c;
@@ -1227,7 +1227,7 @@ program = {
 };
 
 program = {
-  testcase;
+  testcase = native;
   name = printf_test;
   common = tests/printf_unit_test.c;
   common = tests/lib/unit_test.c;
@@ -1242,7 +1242,7 @@ program = {
 };
 
 program = {
-  testcase;
+  testcase = native;
   name = date_test;
   common = tests/date_unit_test.c;
   common = tests/lib/unit_test.c;
@@ -1257,7 +1257,7 @@ program = {
 };
 
 program = {
-  testcase;
+  testcase = native;
   name = priority_queue_unit_test;
   common = tests/priority_queue_unit_test.cc;
   common = tests/lib/unit_test.c;
@@ -1274,7 +1274,7 @@ program = {
 };
 
 program = {
-  testcase;
+  testcase = native;
   name = cmp_test;
   common = tests/cmp_unit_test.c;
   common = tests/lib/unit_test.c;
diff --git a/conf/Makefile.common b/conf/Makefile.common
index f0bb6e160..02107898d 100644
--- a/conf/Makefile.common
+++ b/conf/Makefile.common
@@ -105,7 +105,11 @@ KERNEL_HEADER_FILES =
 bin_SCRIPTS =
 bin_PROGRAMS =
 check_SCRIPTS =
+check_SCRIPTS_native =
+check_SCRIPTS_nonnative =
 check_PROGRAMS =
+check_PROGRAMS_native =
+check_PROGRAMS_nonnative =
 dist_grubconf_DATA =
 dist_noinst_DATA =
 grubconf_SCRIPTS =
diff --git a/gentpl.py b/gentpl.py
index c86550d4f..28ec24209 100644
--- a/gentpl.py
+++ b/gentpl.py
@@ -817,8 +817,7 @@ def program(defn, platform, test=False):
     set_canonical_name_suffix("")
 
     if 'testcase' in defn:
-        gvar_add("check_PROGRAMS", name)
-        gvar_add("TESTS", name)
+        gvar_add("check_PROGRAMS_" + defn['testcase'], name)
     else:
         var_add(installdir(defn) + "_PROGRAMS", name)
         if 'mansection' in defn:
@@ -859,8 +858,7 @@ def script(defn, platform):
     name = defn['name']
 
     if 'testcase' in defn:
-        gvar_add("check_SCRIPTS", name)
-        gvar_add ("TESTS", name)
+        gvar_add("check_SCRIPTS_" + defn['testcase'], name)
     else:
         var_add(installdir(defn) + "_SCRIPTS", name)
         if 'mansection' in defn:
-- 
2.27.0

Re: [PATCH 0/2] Add check-native and check-nonnative as make targets

[Daniel Kiper]

On Wed, Jan 12, 2022 at 08:19:02PM -0600, Glenn Washburn wrote:
> Tests can be put into two categories, native (tests that run on the build
> system) and non-native (tests run in QEMU). For any two targets (even of
> completely different architectures), the native tests will be running the
> same binary code (because they will be compiled for and run on the build
> machine), and thus will have the same result. So when building and
> running tests for multiple targets on a build machine, the native tests on
> need be run once. This can decrease the runtime of a multi-target test run
> significantly (like hours). This patch series makes it possible to run only
> run the non-native tests (skipping the native tests) by partitioning the set
> of tests into native and non-native based on whether QEMU is used by the test.
> The first patch is not necessary, but makes things look cleaner.

For both Reviewed-by: Daniel Kiper <daniel.kiper@oracle.com>...

Daniel

Re: [PATCH 2/2] tests: Add check-native and check-nonnative make targets

[Daniel Kiper]

On Wed, Jan 12, 2022 at 08:19:04PM -0600, Glenn Washburn wrote:
> This allows for testing only tests that run directly on the build machine or
> only tests that run in a virtualized environment. When testing multiple
> targets on the same build machine the native tests only need to be run once
> for all targets. Whereas, the nonnative tests must be run for each target
> because the test is potentially compiled differently for each target.
>
> Signed-off-by: Glenn Washburn <development@efficientek.com>

This patch makes bootstrap unhappy:
  Makefile.am:27: warning: check_SCRIPTS multiply defined in condition TRUE ...
  conf/Makefile.common:107: ... 'check_SCRIPTS' previously defined here
  Makefile.am:10:   'conf/Makefile.common' included from here
  Makefile.am:28: warning: check_PROGRAMS multiply defined in condition TRUE ...
  conf/Makefile.common:110: ... 'check_PROGRAMS' previously defined here
  Makefile.am:10:   'conf/Makefile.common' included from here
  Makefile.am:29: warning: TESTS multiply defined in condition TRUE ...
  conf/Makefile.common:128: ... 'TESTS' previously defined here
  Makefile.am:10:   'conf/Makefile.common' included from here

I have just taken patch #1.

Daniel

Re: [PATCH 2/2] tests: Add check-native and check-nonnative make targets

[Glenn Washburn]

On Tue, 8 Feb 2022 17:10:32 +0100
Daniel Kiper <dkiper@net-space.pl> wrote:

> On Wed, Jan 12, 2022 at 08:19:04PM -0600, Glenn Washburn wrote:
> > This allows for testing only tests that run directly on the build machine or
> > only tests that run in a virtualized environment. When testing multiple
> > targets on the same build machine the native tests only need to be run once
> > for all targets. Whereas, the nonnative tests must be run for each target
> > because the test is potentially compiled differently for each target.
> >
> > Signed-off-by: Glenn Washburn <development@efficientek.com>
> 
> This patch makes bootstrap unhappy:
>   Makefile.am:27: warning: check_SCRIPTS multiply defined in condition TRUE ...
>   conf/Makefile.common:107: ... 'check_SCRIPTS' previously defined here
>   Makefile.am:10:   'conf/Makefile.common' included from here
>   Makefile.am:28: warning: check_PROGRAMS multiply defined in condition TRUE ...
>   conf/Makefile.common:110: ... 'check_PROGRAMS' previously defined here
>   Makefile.am:10:   'conf/Makefile.common' included from here
>   Makefile.am:29: warning: TESTS multiply defined in condition TRUE ...
>   conf/Makefile.common:128: ... 'TESTS' previously defined here
>   Makefile.am:10:   'conf/Makefile.common' included from here
> 
> I have just taken patch #1.

I didn't notice this as it seems to have no effect on building or
testing. I'll send an updated patch removing these warnings.

Glenn