* [kvm-unit-tests PATCH 0/3] Add support for external tests and litmus7 documentation @ 2021-03-24 17:13 Nikos Nikoleris 2021-03-24 17:14 ` [kvm-unit-tests PATCH 1/3] arm/arm64: Avoid wildcard in the arm_clean recipe of the Makefile Nikos Nikoleris ` (3 more replies) 0 siblings, 4 replies; 14+ messages in thread From: Nikos Nikoleris @ 2021-03-24 17:13 UTC (permalink / raw) To: kvm; +Cc: pbonzini, drjones, alexandru.elisei, Nikos Nikoleris This set of patches makes small changes to the build system to allow easy integration of tests not included in the repository. To this end, it adds a parameter to the configuration script `--ext-dir=DIR` which will instruct the build system to include the Makefile in DIR/Makefile. The external Makefile can then add extra tests, link object files and modify/extend flags. In addition, to demonstrate how we can use this functionality, a README file explains how to use litmus7 to generate the C code for litmus tests and link with kvm-unit-tests to produce flat files. Note that currently, litmus7 produces its own independent Makefile as an intermediate step. Once this set of changes is committed, litmus7 will be modifed to make use hook to specify external tests and leverage the build system to build the external tests (https://github.com/relokin/herdtools7/commit/8f23eb39d25931c2c34f4effa096df58547a3bb4). Nikos Nikoleris (3): arm/arm64: Avoid wildcard in the arm_clean recipe of the Makefile arm/arm64: Add a way to specify an external directory with tests README: Add a guide of how to run tests with litmus7 configure | 7 +++ arm/Makefile.common | 11 +++- README.litmus7.md | 125 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 141 insertions(+), 2 deletions(-) create mode 100644 README.litmus7.md -- 2.25.1 ^ permalink raw reply [flat|nested] 14+ messages in thread
* [kvm-unit-tests PATCH 1/3] arm/arm64: Avoid wildcard in the arm_clean recipe of the Makefile 2021-03-24 17:13 [kvm-unit-tests PATCH 0/3] Add support for external tests and litmus7 documentation Nikos Nikoleris @ 2021-03-24 17:14 ` Nikos Nikoleris 2021-03-24 17:14 ` [kvm-unit-tests PATCH 2/3] arm/arm64: Add a way to specify an external directory with tests Nikos Nikoleris ` (2 subsequent siblings) 3 siblings, 0 replies; 14+ messages in thread From: Nikos Nikoleris @ 2021-03-24 17:14 UTC (permalink / raw) To: kvm; +Cc: pbonzini, drjones, alexandru.elisei, Nikos Nikoleris This change adds recipes to discover the generated .o and .d files and removes assumptions about their locations in the arm_clean recipe. Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com> --- arm/Makefile.common | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/arm/Makefile.common b/arm/Makefile.common index a123e85..19db50d 100644 --- a/arm/Makefile.common +++ b/arm/Makefile.common @@ -78,9 +78,12 @@ FLATLIBS = $(libcflat) $(LIBFDT_archive) $(libgcc) $(libeabi) $(libeabi): $(eabiobjs) $(AR) rcs $@ $^ +generated-o = $(cflatobjs) $(eabiobjs) $(cstart.o) $(tests-all:.flat=.o) +generated-d = $(join $(dir $(generated-o)), $(addprefix ., $(notdir $(generated-o:.o=.d)))) + arm_clean: libfdt_clean asm_offsets_clean - $(RM) $(TEST_DIR)/*.{o,flat,elf} $(libeabi) $(eabiobjs) \ - $(TEST_DIR)/.*.d lib/arm/.*.d + $(RM) $(generated-d) + $(RM) $(generated-o) $(tests-all:.flat=.elf) $(tests-all) $(libeabi) generated-files = $(asm-offsets) $(tests-all:.flat=.o) $(cstart.o) $(cflatobjs): $(generated-files) -- 2.25.1 ^ permalink raw reply related [flat|nested] 14+ messages in thread
* [kvm-unit-tests PATCH 2/3] arm/arm64: Add a way to specify an external directory with tests 2021-03-24 17:13 [kvm-unit-tests PATCH 0/3] Add support for external tests and litmus7 documentation Nikos Nikoleris 2021-03-24 17:14 ` [kvm-unit-tests PATCH 1/3] arm/arm64: Avoid wildcard in the arm_clean recipe of the Makefile Nikos Nikoleris @ 2021-03-24 17:14 ` Nikos Nikoleris 2021-03-24 17:14 ` [kvm-unit-tests PATCH 3/3] README: Add a guide of how to run tests with litmus7 Nikos Nikoleris 2021-04-13 16:52 ` [kvm-unit-tests PATCH 0/3] Add support for external tests and litmus7 documentation Nikos Nikoleris 3 siblings, 0 replies; 14+ messages in thread From: Nikos Nikoleris @ 2021-03-24 17:14 UTC (permalink / raw) To: kvm; +Cc: pbonzini, drjones, alexandru.elisei, Nikos Nikoleris This change adds an argument to the configure script which allows a user to specify an external directory with extra tests. When specified, the build system will include the Makefile in that directory allowing a user to add extra tests. For example: For example, DIR contains a test in test.c which depends on symbols defined in obj.c and the Makefile: tests += $(EXT_DIR)/test.flat cflatobjs += $(EXT_DIR)/obj.o With this change, we can add DIR to the build process and generate the test using $> ./configure --ext-dir=DIR $> make Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com> --- configure | 7 +++++++ arm/Makefile.common | 4 ++++ 2 files changed, 11 insertions(+) diff --git a/configure b/configure index cdcd34e..e734b9d 100755 --- a/configure +++ b/configure @@ -26,6 +26,7 @@ errata_force=0 erratatxt="$srcdir/errata.txt" host_key_document= page_size= +ext_dir= usage() { cat <<-EOF @@ -54,6 +55,8 @@ usage() { --page-size=PAGE_SIZE Specify the page size (translation granule) (4k, 16k or 64k, default is 64k, arm64 only) + --ext-dir=DIR specify an additional location with more tests; when enabled + DIR/Makefile is included to the build system (arm/arm64 only) EOF exit 1 } @@ -112,6 +115,9 @@ while [[ "$1" = -* ]]; do --page-size) page_size="$arg" ;; + --ext-dir) + ext_dir="$arg" + ;; --help) usage ;; @@ -264,6 +270,7 @@ U32_LONG_FMT=$u32_long WA_DIVIDE=$wa_divide GENPROTIMG=${GENPROTIMG-genprotimg} HOST_KEY_DOCUMENT=$host_key_document +EXT_DIR=$ext_dir EOF cat <<EOF > lib/config.h diff --git a/arm/Makefile.common b/arm/Makefile.common index 19db50d..ffe1a49 100644 --- a/arm/Makefile.common +++ b/arm/Makefile.common @@ -13,6 +13,10 @@ tests-common += $(TEST_DIR)/psci.flat tests-common += $(TEST_DIR)/sieve.flat tests-common += $(TEST_DIR)/pl031.flat +ifdef EXT_DIR +include $(EXT_DIR)/Makefile +endif + tests-all = $(tests-common) $(tests) all: directories $(tests-all) -- 2.25.1 ^ permalink raw reply related [flat|nested] 14+ messages in thread
* [kvm-unit-tests PATCH 3/3] README: Add a guide of how to run tests with litmus7 2021-03-24 17:13 [kvm-unit-tests PATCH 0/3] Add support for external tests and litmus7 documentation Nikos Nikoleris 2021-03-24 17:14 ` [kvm-unit-tests PATCH 1/3] arm/arm64: Avoid wildcard in the arm_clean recipe of the Makefile Nikos Nikoleris 2021-03-24 17:14 ` [kvm-unit-tests PATCH 2/3] arm/arm64: Add a way to specify an external directory with tests Nikos Nikoleris @ 2021-03-24 17:14 ` Nikos Nikoleris 2021-03-24 18:27 ` Nikos Nikoleris 2021-04-13 16:52 ` [kvm-unit-tests PATCH 0/3] Add support for external tests and litmus7 documentation Nikos Nikoleris 3 siblings, 1 reply; 14+ messages in thread From: Nikos Nikoleris @ 2021-03-24 17:14 UTC (permalink / raw) To: kvm; +Cc: pbonzini, drjones, alexandru.elisei, Nikos Nikoleris litmus7 (http://diy.inria.fr/doc/litmus.html) is a tool that takes as an input a litmus test specification and generates a program that can execute on hardware. Using kvm-unit-tests, litmus7 can generate tests that run on KVM. This change adds some documentation to introduce this functionality along with a basic example of how to build and run such tests. Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com> --- README.litmus7.md | 125 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 125 insertions(+) create mode 100644 README.litmus7.md diff --git a/README.litmus7.md b/README.litmus7.md new file mode 100644 index 0000000..4653488 --- /dev/null +++ b/README.litmus7.md @@ -0,0 +1,125 @@ +# Memory model litmus tests as kvm-unit-tests + +In this guide, we explain how to use kvm-unit-tests in combination +with litmus7 to generate and run litmus tests as tiny operating +systems. + +## Background + +### Memory model litmus tests + +A litmus test is a small program designed to exercise a certain +behavior. Traditionally, litmus tests have been used to demonstrate +and exercise the memory model of parallel computing systems. Litmus +tests are often described in assembly or pseudo-assembly code and +require tools like [litmus7](http://diy.inria.fr/doc/litmus.html) to +genererate executable programs that we can then run on real hardware. + +### Why kvm-unit-tests + +litmus7 uses kvm-unit-tests to encapsulate a litmus tests and generate +executables in the form of tiny operating systems. Inside these tiny +operating systems, the litmus tests can control parameters of the +execution that a user space application cannot. For example, control +virtual address translation and handle exceptions. + +## Building and running litmus kvm-unit-tests + +litmus7 is a tool that given a litmus test will generate C source +code. The generated C source code is compiled and linked with +kvm-unit-tests to generate the binary. + +## Prerequisites + +litmus7 is part of the herdtools7 toolsuite. See +https://github.com/herd/herdtools7/blob/master/INSTALL.md for +instructions of how to build and install herdtools7 from source. + +In addition to herdtools7, this guide assumes that you have a copy of +kvm-unit-tests and you have already configured and built the included +tests. + +## Building and running a litmus test + +In this guide, we use `MP` (Message Passing), a popular litmus test +which demonstrates the communication pattern between a sender (P0) and +a receiver (P1_ process of a message `x`. There is also variable `y` +which is a flag, the sender sets it after setting the message `x` and +the receiver reads it before reading the message `x`. The test also +specifies a condition after the execution of the test which is +validated when P1 reads the initial value of message `x` (0) and the +new value of flag `y` (1). + +``` +AArch64 MP +"PodWW Rfe PodRR Fre" +Generator=diyone7 (version 7.56) +Prefetch=0:x=F,0:y=W,1:y=F,1:x=T +Com=Rf Fr +Orig=PodWW Rfe PodRR Fre +{ +0:X1=x; 0:X3=y; +1:X1=y; 1:X3=x; +} + P0 | P1 ; + MOV W0,#1 | LDR W0,[X1] ; + STR W0,[X1] | LDR W2,[X3] ; + MOV W2,#1 | ; + STR W2,[X3] | ; +exists (1:X0=1 /\ 1:X2=0) +``` + +Note: [diy7](http://diy.inria.fr/doc/gen.html) and +[diyone7]](http://diy.inria.fr/doc/gen.html) from the herdtools7 +toolsuite can systematically generate litmus tests. For example to +generate the MP litmus test: + + diyone7 -arch AArch64 PodWW Rfe PodRR Fre -name MP + +Assuming the file `MP.litmus` contains the test and KUT_DIR is the top +directory of kvm-unit-tests, we can use litmus7 to generate +the C sources: + + litmus7 -mach kvm-armv8.1 -variant precise -a 4 -o ${KUT_DIR}/litmus_tests MP.litmus + +To build the test: + + cd litmus-tests && make && cd - + +This will build the test in the file `litmus-tests/MP.flat` which you +can run like any other test: + + ./arm/run litmus-tests/MP.flat -smp 4 + +The test will print whether the condition of the test was observed +(`Allowed`), stats about the observed outcomes and metadata (e.g., +hash). + +``` +Test MP Allowed +Histogram (4 states) +19865 :>1:X0=0; 1:X2=1; +20885 *>1:X0=1; 1:X2=0; +975348:>1:X0=0; 1:X2=0; +983902:>1:X0=1; 1:X2=1; +Ok + +Witnesses +Positive: 20885, Negative: 1979115 +Condition exists (1:X0=1 /\ 1:X2=0) is validated +Hash=211d5b298572012a0869d4ded6a40b7f +Cycle=Rfe PodRR Fre PodWW +Generator=diycross7 (version 7.54+01(dev)) +Com=Rf Fr +Orig=PodWW Rfe PodRR Fre +Observation MP Sometimes 20885 1979115 +Time MP 3.19 +``` + +## References + +https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/expanding-memory-model-tools-system-level-architecture +https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/running-litmus-tests-on-hardware-litmus7 +http://diy.inria.fr/doc/litmus.html +https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/generate-litmus-tests-automatically-diy7-tool +http://diy.inria.fr/doc/gen.html -- 2.25.1 ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [kvm-unit-tests PATCH 3/3] README: Add a guide of how to run tests with litmus7 2021-03-24 17:14 ` [kvm-unit-tests PATCH 3/3] README: Add a guide of how to run tests with litmus7 Nikos Nikoleris @ 2021-03-24 18:27 ` Nikos Nikoleris 0 siblings, 0 replies; 14+ messages in thread From: Nikos Nikoleris @ 2021-03-24 18:27 UTC (permalink / raw) To: kvm; +Cc: pbonzini, drjones, alexandru.elisei, Jade Alglave, maranget On 24/03/2021 17:14, Nikos Nikoleris wrote: > litmus7 (http://diy.inria.fr/doc/litmus.html) is a tool that takes as > an input a litmus test specification and generates a program that can > execute on hardware. Using kvm-unit-tests, litmus7 can generate tests > that run on KVM. This change adds some documentation to introduce this > functionality along with a basic example of how to build and run such > tests. > > Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com> +cc: Jade and Luc Apologies for the spam, I should have done that in my first email. Thanks, Nikos > --- > README.litmus7.md | 125 ++++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 125 insertions(+) > create mode 100644 README.litmus7.md > > diff --git a/README.litmus7.md b/README.litmus7.md > new file mode 100644 > index 0000000..4653488 > --- /dev/null > +++ b/README.litmus7.md > @@ -0,0 +1,125 @@ > +# Memory model litmus tests as kvm-unit-tests > + > +In this guide, we explain how to use kvm-unit-tests in combination > +with litmus7 to generate and run litmus tests as tiny operating > +systems. > + > +## Background > + > +### Memory model litmus tests > + > +A litmus test is a small program designed to exercise a certain > +behavior. Traditionally, litmus tests have been used to demonstrate > +and exercise the memory model of parallel computing systems. Litmus > +tests are often described in assembly or pseudo-assembly code and > +require tools like [litmus7](http://diy.inria.fr/doc/litmus.html) to > +genererate executable programs that we can then run on real hardware. > + > +### Why kvm-unit-tests > + > +litmus7 uses kvm-unit-tests to encapsulate a litmus tests and generate > +executables in the form of tiny operating systems. Inside these tiny > +operating systems, the litmus tests can control parameters of the > +execution that a user space application cannot. For example, control > +virtual address translation and handle exceptions. > + > +## Building and running litmus kvm-unit-tests > + > +litmus7 is a tool that given a litmus test will generate C source > +code. The generated C source code is compiled and linked with > +kvm-unit-tests to generate the binary. > + > +## Prerequisites > + > +litmus7 is part of the herdtools7 toolsuite. See > +https://github.com/herd/herdtools7/blob/master/INSTALL.md for > +instructions of how to build and install herdtools7 from source. > + > +In addition to herdtools7, this guide assumes that you have a copy of > +kvm-unit-tests and you have already configured and built the included > +tests. > + > +## Building and running a litmus test > + > +In this guide, we use `MP` (Message Passing), a popular litmus test > +which demonstrates the communication pattern between a sender (P0) and > +a receiver (P1_ process of a message `x`. There is also variable `y` > +which is a flag, the sender sets it after setting the message `x` and > +the receiver reads it before reading the message `x`. The test also > +specifies a condition after the execution of the test which is > +validated when P1 reads the initial value of message `x` (0) and the > +new value of flag `y` (1). > + > +``` > +AArch64 MP > +"PodWW Rfe PodRR Fre" > +Generator=diyone7 (version 7.56) > +Prefetch=0:x=F,0:y=W,1:y=F,1:x=T > +Com=Rf Fr > +Orig=PodWW Rfe PodRR Fre > +{ > +0:X1=x; 0:X3=y; > +1:X1=y; 1:X3=x; > +} > + P0 | P1 ; > + MOV W0,#1 | LDR W0,[X1] ; > + STR W0,[X1] | LDR W2,[X3] ; > + MOV W2,#1 | ; > + STR W2,[X3] | ; > +exists (1:X0=1 /\ 1:X2=0) > +``` > + > +Note: [diy7](http://diy.inria.fr/doc/gen.html) and > +[diyone7]](http://diy.inria.fr/doc/gen.html) from the herdtools7 > +toolsuite can systematically generate litmus tests. For example to > +generate the MP litmus test: > + > + diyone7 -arch AArch64 PodWW Rfe PodRR Fre -name MP > + > +Assuming the file `MP.litmus` contains the test and KUT_DIR is the top > +directory of kvm-unit-tests, we can use litmus7 to generate > +the C sources: > + > + litmus7 -mach kvm-armv8.1 -variant precise -a 4 -o ${KUT_DIR}/litmus_tests MP.litmus > + > +To build the test: > + > + cd litmus-tests && make && cd - > + > +This will build the test in the file `litmus-tests/MP.flat` which you > +can run like any other test: > + > + ./arm/run litmus-tests/MP.flat -smp 4 > + > +The test will print whether the condition of the test was observed > +(`Allowed`), stats about the observed outcomes and metadata (e.g., > +hash). > + > +``` > +Test MP Allowed > +Histogram (4 states) > +19865 :>1:X0=0; 1:X2=1; > +20885 *>1:X0=1; 1:X2=0; > +975348:>1:X0=0; 1:X2=0; > +983902:>1:X0=1; 1:X2=1; > +Ok > + > +Witnesses > +Positive: 20885, Negative: 1979115 > +Condition exists (1:X0=1 /\ 1:X2=0) is validated > +Hash=211d5b298572012a0869d4ded6a40b7f > +Cycle=Rfe PodRR Fre PodWW > +Generator=diycross7 (version 7.54+01(dev)) > +Com=Rf Fr > +Orig=PodWW Rfe PodRR Fre > +Observation MP Sometimes 20885 1979115 > +Time MP 3.19 > +``` > + > +## References > + > +https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/expanding-memory-model-tools-system-level-architecture > +https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/running-litmus-tests-on-hardware-litmus7 > +http://diy.inria.fr/doc/litmus.html > +https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/generate-litmus-tests-automatically-diy7-tool > +http://diy.inria.fr/doc/gen.html > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [kvm-unit-tests PATCH 0/3] Add support for external tests and litmus7 documentation 2021-03-24 17:13 [kvm-unit-tests PATCH 0/3] Add support for external tests and litmus7 documentation Nikos Nikoleris ` (2 preceding siblings ...) 2021-03-24 17:14 ` [kvm-unit-tests PATCH 3/3] README: Add a guide of how to run tests with litmus7 Nikos Nikoleris @ 2021-04-13 16:52 ` Nikos Nikoleris 2021-04-14 8:42 ` Andrew Jones 3 siblings, 1 reply; 14+ messages in thread From: Nikos Nikoleris @ 2021-04-13 16:52 UTC (permalink / raw) To: kvm; +Cc: pbonzini, drjones, alexandru.elisei, Jade Alglave, maranget On 24/03/2021 17:13, Nikos Nikoleris wrote: > This set of patches makes small changes to the build system to allow > easy integration of tests not included in the repository. To this end, > it adds a parameter to the configuration script `--ext-dir=DIR` which > will instruct the build system to include the Makefile in > DIR/Makefile. The external Makefile can then add extra tests, > link object files and modify/extend flags. > > In addition, to demonstrate how we can use this functionality, a > README file explains how to use litmus7 to generate the C code for > litmus tests and link with kvm-unit-tests to produce flat files. > > Note that currently, litmus7 produces its own independent Makefile as > an intermediate step. Once this set of changes is committed, litmus7 > will be modifed to make use hook to specify external tests and > leverage the build system to build the external tests > (https://github.com/relokin/herdtools7/commit/8f23eb39d25931c2c34f4effa096df58547a3bb4). > Just wanted to add that if anyone's interested in trying out this series with litmus7 I am very happy to help. Any feedback on this series or the way we use kvm-unit-tests would be very welcome! Thanks, Nikos > Nikos Nikoleris (3): > arm/arm64: Avoid wildcard in the arm_clean recipe of the Makefile > arm/arm64: Add a way to specify an external directory with tests > README: Add a guide of how to run tests with litmus7 > > configure | 7 +++ > arm/Makefile.common | 11 +++- > README.litmus7.md | 125 ++++++++++++++++++++++++++++++++++++++++++++ > 3 files changed, 141 insertions(+), 2 deletions(-) > create mode 100644 README.litmus7.md > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [kvm-unit-tests PATCH 0/3] Add support for external tests and litmus7 documentation 2021-04-13 16:52 ` [kvm-unit-tests PATCH 0/3] Add support for external tests and litmus7 documentation Nikos Nikoleris @ 2021-04-14 8:42 ` Andrew Jones 2021-04-14 8:50 ` Nikos Nikoleris 2021-06-29 13:32 ` Nikos Nikoleris 0 siblings, 2 replies; 14+ messages in thread From: Andrew Jones @ 2021-04-14 8:42 UTC (permalink / raw) To: Nikos Nikoleris; +Cc: kvm, pbonzini, alexandru.elisei, Jade Alglave, maranget On Tue, Apr 13, 2021 at 05:52:37PM +0100, Nikos Nikoleris wrote: > On 24/03/2021 17:13, Nikos Nikoleris wrote: > > This set of patches makes small changes to the build system to allow > > easy integration of tests not included in the repository. To this end, > > it adds a parameter to the configuration script `--ext-dir=DIR` which > > will instruct the build system to include the Makefile in > > DIR/Makefile. The external Makefile can then add extra tests, > > link object files and modify/extend flags. > > > > In addition, to demonstrate how we can use this functionality, a > > README file explains how to use litmus7 to generate the C code for > > litmus tests and link with kvm-unit-tests to produce flat files. > > > > Note that currently, litmus7 produces its own independent Makefile as > > an intermediate step. Once this set of changes is committed, litmus7 > > will be modifed to make use hook to specify external tests and > > leverage the build system to build the external tests > > (https://github.com/relokin/herdtools7/commit/8f23eb39d25931c2c34f4effa096df58547a3bb4). > > > > Just wanted to add that if anyone's interested in trying out this series > with litmus7 I am very happy to help. Any feedback on this series or the way > we use kvm-unit-tests would be very welcome! Hi Nikos, It's on my TODO to play with this. I just haven't had a chance yet. I'm particularly slow right now because I'm in the process of handling a switch of my email server from one type to another, requiring rewrites of filters, new mail synchronization methods, and, in general, lots of pain... Hopefully by the end of this week all will be done. Then, I can start ignoring emails on purpose again, instead of due to the fact that I can't find them :-) Thanks, drew > > Thanks, > > Nikos > > > Nikos Nikoleris (3): > > arm/arm64: Avoid wildcard in the arm_clean recipe of the Makefile > > arm/arm64: Add a way to specify an external directory with tests > > README: Add a guide of how to run tests with litmus7 > > > > configure | 7 +++ > > arm/Makefile.common | 11 +++- > > README.litmus7.md | 125 ++++++++++++++++++++++++++++++++++++++++++++ > > 3 files changed, 141 insertions(+), 2 deletions(-) > > create mode 100644 README.litmus7.md > > > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [kvm-unit-tests PATCH 0/3] Add support for external tests and litmus7 documentation 2021-04-14 8:42 ` Andrew Jones @ 2021-04-14 8:50 ` Nikos Nikoleris 2021-06-29 13:32 ` Nikos Nikoleris 1 sibling, 0 replies; 14+ messages in thread From: Nikos Nikoleris @ 2021-04-14 8:50 UTC (permalink / raw) To: Andrew Jones; +Cc: kvm, pbonzini, alexandru.elisei, Jade Alglave, maranget On 14/04/2021 09:42, Andrew Jones wrote: > On Tue, Apr 13, 2021 at 05:52:37PM +0100, Nikos Nikoleris wrote: >> On 24/03/2021 17:13, Nikos Nikoleris wrote: >>> This set of patches makes small changes to the build system to allow >>> easy integration of tests not included in the repository. To this end, >>> it adds a parameter to the configuration script `--ext-dir=DIR` which >>> will instruct the build system to include the Makefile in >>> DIR/Makefile. The external Makefile can then add extra tests, >>> link object files and modify/extend flags. >>> >>> In addition, to demonstrate how we can use this functionality, a >>> README file explains how to use litmus7 to generate the C code for >>> litmus tests and link with kvm-unit-tests to produce flat files. >>> >>> Note that currently, litmus7 produces its own independent Makefile as >>> an intermediate step. Once this set of changes is committed, litmus7 >>> will be modifed to make use hook to specify external tests and >>> leverage the build system to build the external tests >>> (https://github.com/relokin/herdtools7/commit/8f23eb39d25931c2c34f4effa096df58547a3bb4). >>> >> >> Just wanted to add that if anyone's interested in trying out this series >> with litmus7 I am very happy to help. Any feedback on this series or the way >> we use kvm-unit-tests would be very welcome! > > Hi Nikos, > > It's on my TODO to play with this. I just haven't had a chance yet. I'm > particularly slow right now because I'm in the process of handling a > switch of my email server from one type to another, requiring rewrites > of filters, new mail synchronization methods, and, in general, lots of > pain... Hopefully by the end of this week all will be done. Then, I can > start ignoring emails on purpose again, instead of due to the fact that > I can't find them :-) > No problem at all. If you run into problems with litmus7 please let me know! Thanks, Nikos > Thanks, > drew > >> >> Thanks, >> >> Nikos >> >>> Nikos Nikoleris (3): >>> arm/arm64: Avoid wildcard in the arm_clean recipe of the Makefile >>> arm/arm64: Add a way to specify an external directory with tests >>> README: Add a guide of how to run tests with litmus7 >>> >>> configure | 7 +++ >>> arm/Makefile.common | 11 +++- >>> README.litmus7.md | 125 ++++++++++++++++++++++++++++++++++++++++++++ >>> 3 files changed, 141 insertions(+), 2 deletions(-) >>> create mode 100644 README.litmus7.md >>> >> > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [kvm-unit-tests PATCH 0/3] Add support for external tests and litmus7 documentation 2021-04-14 8:42 ` Andrew Jones 2021-04-14 8:50 ` Nikos Nikoleris @ 2021-06-29 13:32 ` Nikos Nikoleris 2021-06-29 16:13 ` Andrew Jones 1 sibling, 1 reply; 14+ messages in thread From: Nikos Nikoleris @ 2021-06-29 13:32 UTC (permalink / raw) To: Andrew Jones; +Cc: kvm, pbonzini, alexandru.elisei, Jade Alglave, maranget Hi all, On 14/04/2021 11:42, Andrew Jones wrote: > On Tue, Apr 13, 2021 at 05:52:37PM +0100, Nikos Nikoleris wrote: >> On 24/03/2021 17:13, Nikos Nikoleris wrote: >>> This set of patches makes small changes to the build system to allow >>> easy integration of tests not included in the repository. To this end, >>> it adds a parameter to the configuration script `--ext-dir=DIR` which >>> will instruct the build system to include the Makefile in >>> DIR/Makefile. The external Makefile can then add extra tests, >>> link object files and modify/extend flags. >>> >>> In addition, to demonstrate how we can use this functionality, a >>> README file explains how to use litmus7 to generate the C code for >>> litmus tests and link with kvm-unit-tests to produce flat files. >>> >>> Note that currently, litmus7 produces its own independent Makefile as >>> an intermediate step. Once this set of changes is committed, litmus7 >>> will be modifed to make use hook to specify external tests and >>> leverage the build system to build the external tests >>> (https://github.com/relokin/herdtools7/commit/8f23eb39d25931c2c34f4effa096df58547a3bb4). >>> >> >> Just wanted to add that if anyone's interested in trying out this series >> with litmus7 I am very happy to help. Any feedback on this series or the way >> we use kvm-unit-tests would be very welcome! > > Hi Nikos, > > It's on my TODO to play with this. I just haven't had a chance yet. I'm > particularly slow right now because I'm in the process of handling a > switch of my email server from one type to another, requiring rewrites > of filters, new mail synchronization methods, and, in general, lots of > pain... Hopefully by the end of this week all will be done. Then, I can > start ignoring emails on purpose again, instead of due to the fact that > I can't find them :-) > > Thanks, > drew > Just wanted to revive the discussion on this. In particular there are two fairly small changes to the build system that allow us to add external tests (in our case, generated using litmus7) to the list of tests we build. This is specific to arm builds but I am happy to look into generalizing it to include all archs. Thanks, Nikos >> >> Thanks, >> >> Nikos >> >>> Nikos Nikoleris (3): >>> arm/arm64: Avoid wildcard in the arm_clean recipe of the Makefile >>> arm/arm64: Add a way to specify an external directory with tests >>> README: Add a guide of how to run tests with litmus7 >>> >>> configure | 7 +++ >>> arm/Makefile.common | 11 +++- >>> README.litmus7.md | 125 ++++++++++++++++++++++++++++++++++++++++++++ >>> 3 files changed, 141 insertions(+), 2 deletions(-) >>> create mode 100644 README.litmus7.md >>> >> > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [kvm-unit-tests PATCH 0/3] Add support for external tests and litmus7 documentation 2021-06-29 13:32 ` Nikos Nikoleris @ 2021-06-29 16:13 ` Andrew Jones 2021-06-29 16:49 ` Nikos Nikoleris 0 siblings, 1 reply; 14+ messages in thread From: Andrew Jones @ 2021-06-29 16:13 UTC (permalink / raw) To: Nikos Nikoleris; +Cc: kvm, pbonzini, alexandru.elisei, Jade Alglave, maranget On Tue, Jun 29, 2021 at 04:32:36PM +0300, Nikos Nikoleris wrote: > Hi all, > > On 14/04/2021 11:42, Andrew Jones wrote: > > On Tue, Apr 13, 2021 at 05:52:37PM +0100, Nikos Nikoleris wrote: > > > On 24/03/2021 17:13, Nikos Nikoleris wrote: > > > > This set of patches makes small changes to the build system to allow > > > > easy integration of tests not included in the repository. To this end, > > > > it adds a parameter to the configuration script `--ext-dir=DIR` which > > > > will instruct the build system to include the Makefile in > > > > DIR/Makefile. The external Makefile can then add extra tests, > > > > link object files and modify/extend flags. > > > > > > > > In addition, to demonstrate how we can use this functionality, a > > > > README file explains how to use litmus7 to generate the C code for > > > > litmus tests and link with kvm-unit-tests to produce flat files. > > > > > > > > Note that currently, litmus7 produces its own independent Makefile as > > > > an intermediate step. Once this set of changes is committed, litmus7 > > > > will be modifed to make use hook to specify external tests and > > > > leverage the build system to build the external tests > > > > (https://github.com/relokin/herdtools7/commit/8f23eb39d25931c2c34f4effa096df58547a3bb4). > > > > > > > > > > Just wanted to add that if anyone's interested in trying out this series > > > with litmus7 I am very happy to help. Any feedback on this series or the way > > > we use kvm-unit-tests would be very welcome! > > > > Hi Nikos, > > > > It's on my TODO to play with this. I just haven't had a chance yet. I'm > > particularly slow right now because I'm in the process of handling a > > switch of my email server from one type to another, requiring rewrites > > of filters, new mail synchronization methods, and, in general, lots of > > pain... Hopefully by the end of this week all will be done. Then, I can > > start ignoring emails on purpose again, instead of due to the fact that > > I can't find them :-) > > > > Thanks, > > drew > > > > Just wanted to revive the discussion on this. In particular there are two > fairly small changes to the build system that allow us to add external tests > (in our case, generated using litmus7) to the list of tests we build. This > is specific to arm builds but I am happy to look into generalizing it to > include all archs. > Hi Nikos, I just spent a few minutes playing around with litmus7. I see the litmus/libdir/kvm-*.cfg files in herdtools7[1] are very kvm-unit-tests specific. They appear to absorb much of the kvm-unit-tests Makefile paths, flags, cross compiler prefixes, etc. Are these .cfg files the only kvm-unit-tests specific files in herdtools7? Here's a half-baked proposal that I'd like your input on: 1) Generate the kvm-unit-tests specific .cfg files in kvm-unit-tests when configured with a new --litmus7 configure switch. This will ensure that the paths, flags, etc. will be up to date in the .cfg file. 2) Add kvm-unit-tests as a git submodule to [1] to get access to the generated .cfg files and to build the litmus tests for kvm-unit-tests. A litmus7 command will invoke the kvm-unit-tests build (using make -C). 3) Create an additional unittests.cfg file (e.g. litmus7-tests.cfg) for kvm-unit-tests that allows easily running all the litmus7 tests. (That should also allow 'make standalone' to work for litmus7 tests.) 4) Like patch 3/3 already does, document the litmus7 stuff in kvm-unit-tests, so people understand the purpose of the --litmus7 configure switch and also to inform them of the ability to run additional tests and how (by using [1]). [1] https://github.com/herd/herdtools7.git Thanks, drew ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [kvm-unit-tests PATCH 0/3] Add support for external tests and litmus7 documentation 2021-06-29 16:13 ` Andrew Jones @ 2021-06-29 16:49 ` Nikos Nikoleris 2021-06-30 12:23 ` Andrew Jones 0 siblings, 1 reply; 14+ messages in thread From: Nikos Nikoleris @ 2021-06-29 16:49 UTC (permalink / raw) To: Andrew Jones; +Cc: kvm, pbonzini, alexandru.elisei, Jade Alglave, maranget On 29/06/2021 19:13, Andrew Jones wrote: > On Tue, Jun 29, 2021 at 04:32:36PM +0300, Nikos Nikoleris wrote: >> Hi all, >> >> On 14/04/2021 11:42, Andrew Jones wrote: >>> On Tue, Apr 13, 2021 at 05:52:37PM +0100, Nikos Nikoleris wrote: >>>> On 24/03/2021 17:13, Nikos Nikoleris wrote: >>>>> This set of patches makes small changes to the build system to allow >>>>> easy integration of tests not included in the repository. To this end, >>>>> it adds a parameter to the configuration script `--ext-dir=DIR` which >>>>> will instruct the build system to include the Makefile in >>>>> DIR/Makefile. The external Makefile can then add extra tests, >>>>> link object files and modify/extend flags. >>>>> >>>>> In addition, to demonstrate how we can use this functionality, a >>>>> README file explains how to use litmus7 to generate the C code for >>>>> litmus tests and link with kvm-unit-tests to produce flat files. >>>>> >>>>> Note that currently, litmus7 produces its own independent Makefile as >>>>> an intermediate step. Once this set of changes is committed, litmus7 >>>>> will be modifed to make use hook to specify external tests and >>>>> leverage the build system to build the external tests >>>>> (https://github.com/relokin/herdtools7/commit/8f23eb39d25931c2c34f4effa096df58547a3bb4). >>>>> >>>> >>>> Just wanted to add that if anyone's interested in trying out this series >>>> with litmus7 I am very happy to help. Any feedback on this series or the way >>>> we use kvm-unit-tests would be very welcome! >>> >>> Hi Nikos, >>> >>> It's on my TODO to play with this. I just haven't had a chance yet. I'm >>> particularly slow right now because I'm in the process of handling a >>> switch of my email server from one type to another, requiring rewrites >>> of filters, new mail synchronization methods, and, in general, lots of >>> pain... Hopefully by the end of this week all will be done. Then, I can >>> start ignoring emails on purpose again, instead of due to the fact that >>> I can't find them :-) >>> >>> Thanks, >>> drew >>> >> >> Just wanted to revive the discussion on this. In particular there are two >> fairly small changes to the build system that allow us to add external tests >> (in our case, generated using litmus7) to the list of tests we build. This >> is specific to arm builds but I am happy to look into generalizing it to >> include all archs. >> > > Hi Nikos, > Hi Drew, Thanks for having a look! > I just spent a few minutes playing around with litmus7. I see the > litmus/libdir/kvm-*.cfg files in herdtools7[1] are very kvm-unit-tests > specific. They appear to absorb much of the kvm-unit-tests Makefile > paths, flags, cross compiler prefixes, etc. Are these .cfg files the > only kvm-unit-tests specific files in herdtools7? > Indeed these kvm-*.cfg files redefine much of the same variables we have in kvm-unit-tests make files. litmus7 uses these cfg files (and litmus/libdir/_aarch64/kvm.rules for the rules) to generate a standalone Makefile. When we call make, we compile the generated sources and link with kvm-unit-tests object files. The generated Makefile redefines much of the build system in kvm-unit-tests which is not great. If we make any change to the build system in kvm-unit-tests (e.g., add support for efi) we have to port this change to the standalone Makefile we generate using litmus7. > Here's a half-baked proposal that I'd like your input on: > > 1) Generate the kvm-unit-tests specific .cfg files in kvm-unit-tests when > configured with a new --litmus7 configure switch. This will ensure > that the paths, flags, etc. will be up to date in the .cfg file. This wouldn't be enough we would also need some sort of minimal Makefile too (something like litmus/libdir/_aarch64/kvm.rules). > 2) Add kvm-unit-tests as a git submodule to [1] to get access to the > generated .cfg files and to build the litmus tests for kvm-unit-tests. > A litmus7 command will invoke the kvm-unit-tests build (using > make -C). That's possible but it doesn't solve the biggest problem which is figuring out what is the command(s) we need to run to link an elf and subsequently generate a flat file. > 3) Create an additional unittests.cfg file (e.g. litmus7-tests.cfg) for > kvm-unit-tests that allows easily running all the litmus7 tests. > (That should also allow 'make standalone' to work for litmus7 tests.) This is a good point I can have a look at how we could add this. > 4) Like patch 3/3 already does, document the litmus7 stuff in > kvm-unit-tests, so people understand the purpose of the --litmus7 > configure switch and also to inform them of the ability to run > additional tests and how (by using [1]). > Overall it would be great if we could piggyback on the build system of kvm-unit-tests rather than try to re-generate (part of) it. This is what the patch 2/3 tries to do. This is not solving the problem in a way that is specific to litmus7 and allows for adding more source files to the all-tests list. If 2/3 was accepted then we would do something like [1]. And the generated Makefile for the litmus7 tests turns into something very simple: CFLAGS += -march=armv8.1-a tests += $(EXT_DIR)/MP.flat cflatobjs += $(EXT_DIR)/utils.o cflatobjs += $(EXT_DIR)/kvm_timeofday.o [1]: https://github.com/relokin/herdtools7/commit/6fa5ec06856c8263a0823ad21e097a39c97cabc1 Thanks, Nikos > [1] https://github.com/herd/herdtools7.git > > Thanks, > drew > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [kvm-unit-tests PATCH 0/3] Add support for external tests and litmus7 documentation 2021-06-29 16:49 ` Nikos Nikoleris @ 2021-06-30 12:23 ` Andrew Jones 2021-06-30 14:19 ` Nikos Nikoleris 0 siblings, 1 reply; 14+ messages in thread From: Andrew Jones @ 2021-06-30 12:23 UTC (permalink / raw) To: Nikos Nikoleris; +Cc: kvm, pbonzini, alexandru.elisei, Jade Alglave, maranget On Tue, Jun 29, 2021 at 07:49:37PM +0300, Nikos Nikoleris wrote: > > > On 29/06/2021 19:13, Andrew Jones wrote: > > On Tue, Jun 29, 2021 at 04:32:36PM +0300, Nikos Nikoleris wrote: > > > Hi all, > > > > > > On 14/04/2021 11:42, Andrew Jones wrote: > > > > On Tue, Apr 13, 2021 at 05:52:37PM +0100, Nikos Nikoleris wrote: > > > > > On 24/03/2021 17:13, Nikos Nikoleris wrote: > > > > > > This set of patches makes small changes to the build system to allow > > > > > > easy integration of tests not included in the repository. To this end, > > > > > > it adds a parameter to the configuration script `--ext-dir=DIR` which > > > > > > will instruct the build system to include the Makefile in > > > > > > DIR/Makefile. The external Makefile can then add extra tests, > > > > > > link object files and modify/extend flags. > > > > > > > > > > > > In addition, to demonstrate how we can use this functionality, a > > > > > > README file explains how to use litmus7 to generate the C code for > > > > > > litmus tests and link with kvm-unit-tests to produce flat files. > > > > > > > > > > > > Note that currently, litmus7 produces its own independent Makefile as > > > > > > an intermediate step. Once this set of changes is committed, litmus7 > > > > > > will be modifed to make use hook to specify external tests and > > > > > > leverage the build system to build the external tests > > > > > > (https://github.com/relokin/herdtools7/commit/8f23eb39d25931c2c34f4effa096df58547a3bb4). > > > > > > > > > > > > > > > > Just wanted to add that if anyone's interested in trying out this series > > > > > with litmus7 I am very happy to help. Any feedback on this series or the way > > > > > we use kvm-unit-tests would be very welcome! > > > > > > > > Hi Nikos, > > > > > > > > It's on my TODO to play with this. I just haven't had a chance yet. I'm > > > > particularly slow right now because I'm in the process of handling a > > > > switch of my email server from one type to another, requiring rewrites > > > > of filters, new mail synchronization methods, and, in general, lots of > > > > pain... Hopefully by the end of this week all will be done. Then, I can > > > > start ignoring emails on purpose again, instead of due to the fact that > > > > I can't find them :-) > > > > > > > > Thanks, > > > > drew > > > > > > > > > > Just wanted to revive the discussion on this. In particular there are two > > > fairly small changes to the build system that allow us to add external tests > > > (in our case, generated using litmus7) to the list of tests we build. This > > > is specific to arm builds but I am happy to look into generalizing it to > > > include all archs. > > > > > > > Hi Nikos, > > > > Hi Drew, > > Thanks for having a look! > > > I just spent a few minutes playing around with litmus7. I see the > > litmus/libdir/kvm-*.cfg files in herdtools7[1] are very kvm-unit-tests > > specific. They appear to absorb much of the kvm-unit-tests Makefile > > paths, flags, cross compiler prefixes, etc. Are these .cfg files the > > only kvm-unit-tests specific files in herdtools7? > > > > Indeed these kvm-*.cfg files redefine much of the same variables we have in > kvm-unit-tests make files. litmus7 uses these cfg files (and > litmus/libdir/_aarch64/kvm.rules for the rules) to generate a standalone > Makefile. When we call make, we compile the generated sources and link with > kvm-unit-tests object files. The generated Makefile redefines much of the > build system in kvm-unit-tests which is not great. If we make any change to > the build system in kvm-unit-tests (e.g., add support for efi) we have to > port this change to the standalone Makefile we generate using litmus7. Right, that's why I'm suggesting that kvm-unit-tests be brought into the litmus7 build as a submodule. > > > Here's a half-baked proposal that I'd like your input on: > > > > 1) Generate the kvm-unit-tests specific .cfg files in kvm-unit-tests when > > configured with a new --litmus7 configure switch. This will ensure > > that the paths, flags, etc. will be up to date in the .cfg file. > > This wouldn't be enough we would also need some sort of minimal Makefile too > (something like litmus/libdir/_aarch64/kvm.rules). It also looks derived from kvm-unit-tests:arm/Makefile.common. So why not just modify that instead? Possibly creating a new make target in order to accommodate any differences. > > > 2) Add kvm-unit-tests as a git submodule to [1] to get access to the > > generated .cfg files and to build the litmus tests for kvm-unit-tests. > > A litmus7 command will invoke the kvm-unit-tests build (using > > make -C). > > That's possible but it doesn't solve the biggest problem which is figuring > out what is the command(s) we need to run to link an elf and subsequently > generate a flat file. Shouldn't all those commands be in the Makefiles that will be used as part of the 'make -C <kut-submodule-dir>' call? > > > 3) Create an additional unittests.cfg file (e.g. litmus7-tests.cfg) for > > kvm-unit-tests that allows easily running all the litmus7 tests. > > (That should also allow 'make standalone' to work for litmus7 tests.) > > This is a good point I can have a look at how we could add this. > > > 4) Like patch 3/3 already does, document the litmus7 stuff in > > kvm-unit-tests, so people understand the purpose of the --litmus7 > > configure switch and also to inform them of the ability to run > > additional tests and how (by using [1]). > > > > Overall it would be great if we could piggyback on the build system of > kvm-unit-tests rather than try to re-generate (part of) it. This is what the > patch 2/3 tries to do. This is not solving the problem in a way that is > specific to litmus7 and allows for adding more source files to the all-tests > list. Patch 2/3 only adds the ability to add a new dir to look at. It leaves everything else up to litmus7 build code to duplicate what it needs and also manual commands to populate that new directory. I'd rather we have a more coherent solution. We want to build litmus7 tests as kvm-unit-tests. We can look at this two ways: 1) we want to add litmus7 tests to kvm-unit-tests or 2) we want to build litmus7 tests for kvm-unit-tests. This patch series is going for (1), but without actually committing the tests to kvm-unit-tests. I'm arguing we should do (2), especially since the litmus7 build code already appears to need to know about kvm-unit-tests. I think we should only need to modify the build scripts of litmus7 to use/build kvm-unit-tests as a submodule and to somehow build litmus7 tests with it. Also, litmus7 test running could be done from the litmus7 build repo or kvm-unit-tests standalone tests could built for litmus7 tests and installed elsewhere. A final note on patch 2/3. Why not just override the TEST_DIR config variable with a different directory? (If it doesn't work for some reason, then we could hopefully fix that.) Thanks, drew > > If 2/3 was accepted then we would do something like [1]. And the generated > Makefile for the litmus7 tests turns into something very simple: > > CFLAGS += -march=armv8.1-a > > tests += $(EXT_DIR)/MP.flat > > cflatobjs += $(EXT_DIR)/utils.o > cflatobjs += $(EXT_DIR)/kvm_timeofday.o > > [1]: https://github.com/relokin/herdtools7/commit/6fa5ec06856c8263a0823ad21e097a39c97cabc1 > > Thanks, > > Nikos > > > [1] https://github.com/herd/herdtools7.git > > > > Thanks, > > drew > > > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [kvm-unit-tests PATCH 0/3] Add support for external tests and litmus7 documentation 2021-06-30 12:23 ` Andrew Jones @ 2021-06-30 14:19 ` Nikos Nikoleris 2021-06-30 16:03 ` Andrew Jones 0 siblings, 1 reply; 14+ messages in thread From: Nikos Nikoleris @ 2021-06-30 14:19 UTC (permalink / raw) To: Andrew Jones; +Cc: kvm, pbonzini, alexandru.elisei, Jade Alglave, maranget On 30/06/2021 15:23, Andrew Jones wrote: > On Tue, Jun 29, 2021 at 07:49:37PM +0300, Nikos Nikoleris wrote: >> >> >> On 29/06/2021 19:13, Andrew Jones wrote: >>> On Tue, Jun 29, 2021 at 04:32:36PM +0300, Nikos Nikoleris wrote: >>>> Hi all, >>>> >>>> On 14/04/2021 11:42, Andrew Jones wrote: >>>>> On Tue, Apr 13, 2021 at 05:52:37PM +0100, Nikos Nikoleris wrote: >>>>>> On 24/03/2021 17:13, Nikos Nikoleris wrote: >>>>>>> This set of patches makes small changes to the build system to allow >>>>>>> easy integration of tests not included in the repository. To this end, >>>>>>> it adds a parameter to the configuration script `--ext-dir=DIR` which >>>>>>> will instruct the build system to include the Makefile in >>>>>>> DIR/Makefile. The external Makefile can then add extra tests, >>>>>>> link object files and modify/extend flags. >>>>>>> >>>>>>> In addition, to demonstrate how we can use this functionality, a >>>>>>> README file explains how to use litmus7 to generate the C code for >>>>>>> litmus tests and link with kvm-unit-tests to produce flat files. >>>>>>> >>>>>>> Note that currently, litmus7 produces its own independent Makefile as >>>>>>> an intermediate step. Once this set of changes is committed, litmus7 >>>>>>> will be modifed to make use hook to specify external tests and >>>>>>> leverage the build system to build the external tests >>>>>>> (https://github.com/relokin/herdtools7/commit/8f23eb39d25931c2c34f4effa096df58547a3bb4). >>>>>>> >>>>>> >>>>>> Just wanted to add that if anyone's interested in trying out this series >>>>>> with litmus7 I am very happy to help. Any feedback on this series or the way >>>>>> we use kvm-unit-tests would be very welcome! >>>>> >>>>> Hi Nikos, >>>>> >>>>> It's on my TODO to play with this. I just haven't had a chance yet. I'm >>>>> particularly slow right now because I'm in the process of handling a >>>>> switch of my email server from one type to another, requiring rewrites >>>>> of filters, new mail synchronization methods, and, in general, lots of >>>>> pain... Hopefully by the end of this week all will be done. Then, I can >>>>> start ignoring emails on purpose again, instead of due to the fact that >>>>> I can't find them :-) >>>>> >>>>> Thanks, >>>>> drew >>>>> >>>> >>>> Just wanted to revive the discussion on this. In particular there are two >>>> fairly small changes to the build system that allow us to add external tests >>>> (in our case, generated using litmus7) to the list of tests we build. This >>>> is specific to arm builds but I am happy to look into generalizing it to >>>> include all archs. >>>> >>> >>> Hi Nikos, >>> >> >> Hi Drew, >> >> Thanks for having a look! >> >>> I just spent a few minutes playing around with litmus7. I see the >>> litmus/libdir/kvm-*.cfg files in herdtools7[1] are very kvm-unit-tests >>> specific. They appear to absorb much of the kvm-unit-tests Makefile >>> paths, flags, cross compiler prefixes, etc. Are these .cfg files the >>> only kvm-unit-tests specific files in herdtools7? >>> >> >> Indeed these kvm-*.cfg files redefine much of the same variables we have in >> kvm-unit-tests make files. litmus7 uses these cfg files (and >> litmus/libdir/_aarch64/kvm.rules for the rules) to generate a standalone >> Makefile. When we call make, we compile the generated sources and link with >> kvm-unit-tests object files. The generated Makefile redefines much of the >> build system in kvm-unit-tests which is not great. If we make any change to >> the build system in kvm-unit-tests (e.g., add support for efi) we have to >> port this change to the standalone Makefile we generate using litmus7. > > Right, that's why I'm suggesting that kvm-unit-tests be brought into the > litmus7 build as a submodule. > Hi Drew, Making kvm-unit-tests a submodule of herdtools will make sure: * we know where kvm-unit-tests is * that we're using a compatible version. We could definitely do that. >> >>> Here's a half-baked proposal that I'd like your input on: >>> >>> 1) Generate the kvm-unit-tests specific .cfg files in kvm-unit-tests when >>> configured with a new --litmus7 configure switch. This will ensure >>> that the paths, flags, etc. will be up to date in the .cfg file. >> >> This wouldn't be enough we would also need some sort of minimal Makefile too >> (something like litmus/libdir/_aarch64/kvm.rules). > > It also looks derived from kvm-unit-tests:arm/Makefile.common. So why not > just modify that instead? Possibly creating a new make target in order to > accommodate any differences. > I was trying to completely get rid of herdtools:litmus/libdir/_aarch64/kvm.rules and the cfg files because they are replicating much of kvm-unit-tests:arm/Makefile.common and kvm-unit-tests:Makefile. If I understand correctly in your proposal we will generate them automatically? >> >>> 2) Add kvm-unit-tests as a git submodule to [1] to get access to the >>> generated .cfg files and to build the litmus tests for kvm-unit-tests. >>> A litmus7 command will invoke the kvm-unit-tests build (using >>> make -C). >> >> That's possible but it doesn't solve the biggest problem which is figuring >> out what is the command(s) we need to run to link an elf and subsequently >> generate a flat file. > > Shouldn't all those commands be in the Makefiles that will be used as part > of the 'make -C <kut-submodule-dir>' call? > Without making any changes to the existing kvm-unit-tests build system 'make -C <kut-submodule-dir>' will build all objects and flat files for the existing tests. Or am I missing something? >> >>> 3) Create an additional unittests.cfg file (e.g. litmus7-tests.cfg) for >>> kvm-unit-tests that allows easily running all the litmus7 tests. >>> (That should also allow 'make standalone' to work for litmus7 tests.) >> >> This is a good point I can have a look at how we could add this. >> >>> 4) Like patch 3/3 already does, document the litmus7 stuff in >>> kvm-unit-tests, so people understand the purpose of the --litmus7 >>> configure switch and also to inform them of the ability to run >>> additional tests and how (by using [1]). >>> >> >> Overall it would be great if we could piggyback on the build system of >> kvm-unit-tests rather than try to re-generate (part of) it. This is what the >> patch 2/3 tries to do. This is not solving the problem in a way that is >> specific to litmus7 and allows for adding more source files to the all-tests >> list. > > Patch 2/3 only adds the ability to add a new dir to look at. It leaves > everything else up to litmus7 build code to duplicate what it needs and > also manual commands to populate that new directory. I'd rather we have a > more coherent solution. > > We want to build litmus7 tests as kvm-unit-tests. We can look at this two > ways: 1) we want to add litmus7 tests to kvm-unit-tests or 2) we want to > build litmus7 tests for kvm-unit-tests. > > This patch series is going for (1), but without actually committing the > tests to kvm-unit-tests. I'm arguing we should do (2), especially since > the litmus7 build code already appears to need to know about > kvm-unit-tests. > I was more looking to add support for external tests similar to the idea of having external modules to the linux kernel. The implementation might not be ideal; I didn't want the changes to be very intrusive but if that's the problem then I am happy to make changes. As for assumptions, the external dir needs to have at least a .c file for the code of the test, a minimal Makefile specifying tests and object files and as you pointed out a unittests.cfg. If we ignore litmus7 for a bit, the general question is, I have a test (e.g., kvm-unit-test:arm/selftest.c) which is not part of the tree and not in the build system, how can I build the corresponding flat or efi file based on it? I think your proposal, is more in line with what we do now. At least, we would automatically generate the config files. kvm-unit-tests generates config files, which litmus7 would use as input to generate a Makefile. I wanted to move away from that but we can live with it. Also note that if we added a --litmus7 configure switch, it would be very specific to way we do things in litmus7. > I think we should only need to modify the build scripts of litmus7 to > use/build kvm-unit-tests as a submodule and to somehow build litmus7 > tests with it. Also, litmus7 test running could be done from the > litmus7 build repo or kvm-unit-tests standalone tests could built for > litmus7 tests and installed elsewhere. > > A final note on patch 2/3. Why not just override the TEST_DIR config > variable with a different directory? (If it doesn't work for some > reason, then we could hopefully fix that.) > External (litmus7) tests target an arch (could be arm64, could be x86), and there are many variables inside $(TEST_DIR)/ that we would need. If I am building for arm64, I need arm/flat.lds and information that currently resides in arm/Makefile.arm64: bits = 64 ldarch = elf64-littleaarch64 arch_LDFLAGS = -pie -n CFLAGS += -mstrict-align and if I change TEST_DIR to point to ~/litmus7_tests I will need to replicate all that somewhere else. I am not saying that it would be impossible to override TEST_DIR but it doesn't make things simple. Thanks, Nikos > Thanks, > drew > >> >> If 2/3 was accepted then we would do something like [1]. And the generated >> Makefile for the litmus7 tests turns into something very simple: >> >> CFLAGS += -march=armv8.1-a >> >> tests += $(EXT_DIR)/MP.flat >> >> cflatobjs += $(EXT_DIR)/utils.o >> cflatobjs += $(EXT_DIR)/kvm_timeofday.o >> >> [1]: https://github.com/relokin/herdtools7/commit/6fa5ec06856c8263a0823ad21e097a39c97cabc1 >> >> Thanks, >> >> Nikos >> >>> [1] https://github.com/herd/herdtools7.git >>> >>> Thanks, >>> drew >>> >> > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [kvm-unit-tests PATCH 0/3] Add support for external tests and litmus7 documentation 2021-06-30 14:19 ` Nikos Nikoleris @ 2021-06-30 16:03 ` Andrew Jones 0 siblings, 0 replies; 14+ messages in thread From: Andrew Jones @ 2021-06-30 16:03 UTC (permalink / raw) To: Nikos Nikoleris; +Cc: kvm, pbonzini, alexandru.elisei, Jade Alglave, maranget On Wed, Jun 30, 2021 at 05:19:57PM +0300, Nikos Nikoleris wrote: > On 30/06/2021 15:23, Andrew Jones wrote: ... > > It also looks derived from kvm-unit-tests:arm/Makefile.common. So why not > > just modify that instead? Possibly creating a new make target in order to > > accommodate any differences. > > > > I was trying to completely get rid of > herdtools:litmus/libdir/_aarch64/kvm.rules and the cfg files because they > are replicating much of kvm-unit-tests:arm/Makefile.common and > kvm-unit-tests:Makefile. If I understand correctly in your proposal we will > generate them automatically? Either generate or add lines to kvm-unit-tests Makefiles which can be used when building a litmus7 specific build target, e.g. 'make litmus7' > > > > > > > > 2) Add kvm-unit-tests as a git submodule to [1] to get access to the > > > > generated .cfg files and to build the litmus tests for kvm-unit-tests. > > > > A litmus7 command will invoke the kvm-unit-tests build (using > > > > make -C). > > > > > > That's possible but it doesn't solve the biggest problem which is figuring > > > out what is the command(s) we need to run to link an elf and subsequently > > > generate a flat file. > > > > Shouldn't all those commands be in the Makefiles that will be used as part > > of the 'make -C <kut-submodule-dir>' call? > > > > Without making any changes to the existing kvm-unit-tests build system 'make > -C <kut-submodule-dir>' will build all objects and flat files for the > existing tests. Or am I missing something? You can modify kvm-unit-tests Makefiles if necessary. And/or add a new one, e.g. arm/Makefile.litmus7 > > > > > > > > 3) Create an additional unittests.cfg file (e.g. litmus7-tests.cfg) for > > > > kvm-unit-tests that allows easily running all the litmus7 tests. > > > > (That should also allow 'make standalone' to work for litmus7 tests.) > > > > > > This is a good point I can have a look at how we could add this. > > > > > > > 4) Like patch 3/3 already does, document the litmus7 stuff in > > > > kvm-unit-tests, so people understand the purpose of the --litmus7 > > > > configure switch and also to inform them of the ability to run > > > > additional tests and how (by using [1]). > > > > > > > > > > Overall it would be great if we could piggyback on the build system of > > > kvm-unit-tests rather than try to re-generate (part of) it. This is what the > > > patch 2/3 tries to do. This is not solving the problem in a way that is > > > specific to litmus7 and allows for adding more source files to the all-tests > > > list. > > > > Patch 2/3 only adds the ability to add a new dir to look at. It leaves > > everything else up to litmus7 build code to duplicate what it needs and > > also manual commands to populate that new directory. I'd rather we have a > > more coherent solution. > > > > We want to build litmus7 tests as kvm-unit-tests. We can look at this two > > ways: 1) we want to add litmus7 tests to kvm-unit-tests or 2) we want to > > build litmus7 tests for kvm-unit-tests. > > > This patch series is going for (1), but without actually committing the > > tests to kvm-unit-tests. I'm arguing we should do (2), especially since > > the litmus7 build code already appears to need to know about > > kvm-unit-tests. > > > > I was more looking to add support for external tests similar to the idea of > having external modules to the linux kernel. The implementation might not be > ideal; I didn't want the changes to be very intrusive but if that's the > problem then I am happy to make changes. I don't expect the changes to be too intrusive and they can probably be mostly isolated into their own Makefile.litmus7 file if necessary. > > As for assumptions, the external dir needs to have at least a .c file for > the code of the test, a minimal Makefile specifying tests and object files > and as you pointed out a unittests.cfg. > > If we ignore litmus7 for a bit, the general question is, I have a test > (e.g., kvm-unit-test:arm/selftest.c) which is not part of the tree and not > in the build system, how can I build the corresponding flat or efi file > based on it? > > I think your proposal, is more in line with what we do now. At least, we > would automatically generate the config files. kvm-unit-tests generates > config files, which litmus7 would use as input to generate a Makefile. I > wanted to move away from that but we can live with it. Also note that if we > added a --litmus7 configure switch, it would be very specific to way we do > things in litmus7. Let's see what it looks like. I'm having trouble imagining how this repo combination will work, but I'm optimistic that it's doable without too many changes to either repo. And most the specific to litmus7 stuff should still be in the litmus7 repo, even if we need kvm-unit-tests to be aware of some build differences and therefore provide the configure switch. > > > I think we should only need to modify the build scripts of litmus7 to > > use/build kvm-unit-tests as a submodule and to somehow build litmus7 > > tests with it. Also, litmus7 test running could be done from the > > litmus7 build repo or kvm-unit-tests standalone tests could built for > > litmus7 tests and installed elsewhere. > > > > A final note on patch 2/3. Why not just override the TEST_DIR config > > variable with a different directory? (If it doesn't work for some > > reason, then we could hopefully fix that.) > > > > External (litmus7) tests target an arch (could be arm64, could be x86), and > there are many variables inside $(TEST_DIR)/ that we would need. If I am > building for arm64, I need arm/flat.lds and information that currently > resides in arm/Makefile.arm64: > > bits = 64 > ldarch = elf64-littleaarch64 > > arch_LDFLAGS = -pie -n > CFLAGS += -mstrict-align > > and if I change TEST_DIR to point to ~/litmus7_tests I will need to > replicate all that somewhere else. > > I am not saying that it would be impossible to override TEST_DIR but it > doesn't make things simple. It'd probably be safe to change the build scripts to use something like $ARCH/Makefile.$ARCH and $ARCH/flat.lds, but we'll need a arm64 -> arm link. Overriding TEST_DIR may not be easier, but I think it's worth experimenting with it, since I feel like it would be cleaner than the "add an arbitrary directory" configure switch. Thanks, drew ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2021-06-30 16:03 UTC | newest] Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-03-24 17:13 [kvm-unit-tests PATCH 0/3] Add support for external tests and litmus7 documentation Nikos Nikoleris 2021-03-24 17:14 ` [kvm-unit-tests PATCH 1/3] arm/arm64: Avoid wildcard in the arm_clean recipe of the Makefile Nikos Nikoleris 2021-03-24 17:14 ` [kvm-unit-tests PATCH 2/3] arm/arm64: Add a way to specify an external directory with tests Nikos Nikoleris 2021-03-24 17:14 ` [kvm-unit-tests PATCH 3/3] README: Add a guide of how to run tests with litmus7 Nikos Nikoleris 2021-03-24 18:27 ` Nikos Nikoleris 2021-04-13 16:52 ` [kvm-unit-tests PATCH 0/3] Add support for external tests and litmus7 documentation Nikos Nikoleris 2021-04-14 8:42 ` Andrew Jones 2021-04-14 8:50 ` Nikos Nikoleris 2021-06-29 13:32 ` Nikos Nikoleris 2021-06-29 16:13 ` Andrew Jones 2021-06-29 16:49 ` Nikos Nikoleris 2021-06-30 12:23 ` Andrew Jones 2021-06-30 14:19 ` Nikos Nikoleris 2021-06-30 16:03 ` Andrew Jones
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).