[PATCH v3 0/2] Wrap EFL binaries into ISO images

[PATCH v3 0/2] Wrap EFL binaries into ISO images

[Yi Sun]

This patch set make use of tool 'grub-mkrescue' to wrap ELF binaries 
into bootable ISO images.

Cases in kvm-unit-tests can be run with QEMU. But the problem is that
some newer VMMs such as Crosvm/Cloud-hyperviosr does NOT support 
multiboot protocol with which QEMU loads and executes those testing 
binaries correctly. This patch set can wrap each kvm-unit-tests EFL 
binaries into a bootable ISO image aiming to adapt it to more usage 
scenarios. As we know, all PC BIOSes and vBIOSes know how to boot 
from a ISO from CD-ROM drive, hence it can extend the KVM-unit-tests 
a lot.

The patch set provides two approaches to create ISO. One is via 
"make iso". It wrap each ELF in foler x86 into a ISO without any 
parameters passed to the test cases.  The other is via script 
create_iso.sh. The script wraps the ELF according to the configure
file unittests.cfg which descripes various parameters for each testing.

Patch History:
V1: Initial version.
V2: Add the second parament to the script create_iso.sh, that could 
pass environment variables into test cases via the file.
V3: Add some failure handle.

 lib/grub/grub.cfg   |   7 +++
 x86/Makefile.common |  18 +++++++-
 x86/create_iso.sh   | 110 ++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 134 insertions(+), 1 deletion(-)
 create mode 100644 lib/grub/grub.cfg
 create mode 100755 x86/create_iso.sh

-- 
2.27.0

[PATCH v3 1/2] x86: Build ISO images from x86/*.elf

[Yi Sun]

Make use of tool grub-mkresure to wrap x86/*.elf in ISO images.
VMM could load test cases just like a CD-ROM, which could
extend usage of those cases.

Refine Makefile to clean *.iso when running 'make clean'.

Signed-off-by: Yi Sun <yi.sun@intel.com>

diff --git a/lib/grub/grub.cfg b/lib/grub/grub.cfg
new file mode 100644
index 0000000..b287cf4
--- /dev/null
+++ b/lib/grub/grub.cfg
@@ -0,0 +1,7 @@
+set timeout=0
+set default=0
+
+menuentry "my os" {
+    multiboot /boot/kernel.bin
+    boot
+}
diff --git a/x86/Makefile.common b/x86/Makefile.common
index 52bb7aa..62eea51 100644
--- a/x86/Makefile.common
+++ b/x86/Makefile.common
@@ -50,6 +50,22 @@ FLATLIBS = lib/libcflat.a
 	$(OBJCOPY) -O elf32-i386 $^ $@
 	@chmod a-x $@
 
+grub_cfg := lib/grub/grub.cfg
+
+elf_files := $(wildcard ./x86/*.elf)
+iso_files := $(patsubst %.elf,%.iso,$(elf_files))
+
+%.iso: %.elf
+	@echo "Creating ISO for case: $(notdir $<)"
+	@rm -rf build/isofiles
+	@mkdir -p build/isofiles/boot/grub
+	@cp $< build/isofiles/boot/kernel.bin
+	@cp $(grub_cfg) build/isofiles/boot/grub
+	@grub-mkrescue -o $@ build/isofiles 2> /dev/null
+
+iso: $(iso_files)
+	echo "All ISO created successfully!"
+
 tests-common = $(TEST_DIR)/vmexit.flat $(TEST_DIR)/tsc.flat \
                $(TEST_DIR)/smptest.flat  \
                $(TEST_DIR)/realmode.flat $(TEST_DIR)/msr.flat \
@@ -81,5 +97,5 @@ $(TEST_DIR)/hyperv_stimer.elf: $(TEST_DIR)/hyperv.o
 $(TEST_DIR)/hyperv_connections.elf: $(TEST_DIR)/hyperv.o
 
 arch_clean:
-	$(RM) $(TEST_DIR)/*.o $(TEST_DIR)/*.flat $(TEST_DIR)/*.elf \
+	$(RM) $(TEST_DIR)/*.o $(TEST_DIR)/*.flat $(TEST_DIR)/*.elf $(TEST_DIR)/*.iso \
 	$(TEST_DIR)/.*.d lib/x86/.*.d \
-- 
2.27.0

[PATCH v3 2/2] x86: Create ISO images according to unittests.cfg

[Yi Sun]

Create ISO image according to the configure file unittests.cfg,
where describes the parameters of each test case.

The grub.cfg in ios/boot/grub/ looks as following, taking case
'vmx_apicv_test' as an example:

set timeout=0
set default=0

menuentry "vmx.elf" {
    multiboot /boot/vmx.elf apic_reg_virt_test virt_x2apic_mode_test
    module   /boot/module
    boot
}

[1] All parameters are from '-append xxx' in the unittests.cfg.
[2] The file 'module' is a fixed file that can be passed by paramters
of script ./create_iso.sh.

Signed-off-by: Yi Sun <yi.sun@intel.com>

diff --git a/x86/create_iso.sh b/x86/create_iso.sh
new file mode 100755
index 0000000..4fe8e5b
--- /dev/null
+++ b/x86/create_iso.sh
@@ -0,0 +1,112 @@
+#!/bin/bash
+config_file=$1
+module_file=$2
+
+opts=
+extra_params=" "
+kernel=
+smp=
+testname=
+
+dir_x86=`dirname $0`
+[[ ${dir_x86:0:1} != "/" ]] && dir_x86=`pwd`/$dir_x86
+
+usage() {
+	cat << EOF
+Usage: create_iso.sh <configure> [module_file]
+    configure:
+        Usually just pass unittests.cfg to the create_iso.sh.
+
+    module_file:
+        This is optional. You can pass some environment to the kernels
+        The module.file looks as following:
+            NR_CPUS=56
+	    MEMSIZE=4096
+	    TEST_DEVICE=0
+	    BOOTLOADER=1
+EOF
+}
+
+grub_cfg() {
+	local kernel_elf=$1
+	local kernel_para=$2
+	local module_file=$3
+
+	if [[ "${module_file}" != "" ]]; then
+		module_cmd="module   /boot/${module_file}"
+	fi
+
+	cat << EOF
+set timeout=0
+set default=0
+
+menuentry "${kernel_elf}" {
+    multiboot /boot/${kernel_elf} ${kernel_para}
+    ${module_cmd}
+    boot
+}
+EOF
+}
+
+create_iso() {
+	local case_name=$1
+	local kernel_elf=$2
+	local kernel_params=$3
+	local module_file=$4
+
+	if [ -f $kernel_elf ]; then
+		rm -rf build/isofiles
+		mkdir -p build/isofiles/boot/grub
+
+		cp $kernel_elf build/isofiles/boot/
+		[[ -f $module_file ]] && cp $module_file build/isofiles/boot/
+
+		grub_cfg ${kernel_elf##*/} "${kernel_params}" ${module_file##*/} > build/isofiles/boot/grub/grub.cfg
+
+		rm -rf ${testname}.iso
+		grub-mkrescue -o ${dir_x86}/${case_name}.iso build/isofiles >& /dev/null
+		if [ $? == 0 ]; then
+			echo "Creating ISO for case: ${case_name}"
+		else
+			echo "[FAIL] grub-mkrescue: Please install grub-mkrescue and xorriso correctly."
+			exit 1
+		fi
+	fi
+}
+
+if [[ "$config_file" == "" || ! -f $config_file ]]; then
+	echo "[FAIL] A configure file is required, usually pass unittests.cfg as the first parameter"
+	usage
+	exit 1
+fi
+
+nline=`wc $config_file | cut -d' ' -f 2`
+
+while read -r line; do
+	if [[ "$line" =~ ^\[(.*)\]$ || $nline == 1 ]]; then
+		rematch=${BASH_REMATCH[1]}
+		if [[ "${testname}" != "" ]]; then
+			create_iso $testname ${dir_x86}/${kernel}.elf "${extra_params}" $module_file
+		fi
+		testname=$rematch
+		extra_params=" "
+		smp=1
+		kernel=""
+		opts=""
+		groups=""
+	elif [[ $line =~ ^file\ *=\ *(.*)\.flat$ ]]; then
+		kernel=${BASH_REMATCH[1]}
+	elif [[ $line =~ ^smp\ *=\ *(.*)$ ]]; then
+		smp=${BASH_REMATCH[1]}
+	elif [[ $line =~ ^extra_params\ *=\ *(.*)$ ]]; then
+		opts=${BASH_REMATCH[1]}
+		if [[ "$opts" =~ .*append\ (.*)$ ]]; then
+			extra_params=${BASH_REMATCH[1]}
+			extra_params=`echo $extra_params | sed 's/\"//g'`
+		fi
+	elif [[ $line =~ ^groups\ *=\ *(.*)$ ]]; then
+		groups=${BASH_REMATCH[1]}
+	fi
+	(( nline -= 1))
+
+done < $config_file
-- 
2.27.0

Re: [PATCH v3 0/2] Wrap EFL binaries into ISO images

[Nadav Amit]

> On Jun 13, 2021, at 5:47 AM, Yi Sun <yi.sun@intel.com> wrote:
> 
> This patch set make use of tool 'grub-mkrescue' to wrap ELF binaries 
> into bootable ISO images.
> 
> Cases in kvm-unit-tests can be run with QEMU. But the problem is that
> some newer VMMs such as Crosvm/Cloud-hyperviosr does NOT support 
> multiboot protocol with which QEMU loads and executes those testing 
> binaries correctly. This patch set can wrap each kvm-unit-tests EFL 
> binaries into a bootable ISO image aiming to adapt it to more usage 
> scenarios. As we know, all PC BIOSes and vBIOSes know how to boot 
> from a ISO from CD-ROM drive, hence it can extend the KVM-unit-tests 
> a lot.
> 
> The patch set provides two approaches to create ISO. One is via 
> "make iso". It wrap each ELF in foler x86 into a ISO without any 
> parameters passed to the test cases.  The other is via script 
> create_iso.sh. The script wraps the ELF according to the configure
> file unittests.cfg which descripes various parameters for each testing.
> 
> Patch History:
> V1: Initial version.
> V2: Add the second parament to the script create_iso.sh, that could 
> pass environment variables into test cases via the file.
> V3: Add some failure handle.

Thanks for doing that. I find your work very useful (at least for me).

For what it worth in kvm-unit-tests, I should have gave before:

Tested-by: Nadav Amit <nadav.amit@gmail.com>


Thanks,
Nadav