From: Robert Foley <robert.foley@linaro.org>
To: qemu-devel@nongnu.org
Cc: philmd@redhat.com, alex.bennee@linaro.org,
robert.foley@linaro.org, peter.puhov@linaro.org
Subject: [PATCH v7 05/12] tests/vm: Added configuration file support
Date: Tue, 19 May 2020 09:22:52 -0400 [thread overview]
Message-ID: <20200519132259.405-6-robert.foley@linaro.org> (raw)
In-Reply-To: <20200519132259.405-1-robert.foley@linaro.org>
Changes to tests/vm/basevm.py to allow accepting a configuration file
as a parameter. Allows for specifying VM options such as
cpu, machine, memory, and arbitrary qemu arguments for specifying options
such as NUMA configuration.
Also added an example conf_example_aarch64.yml and conf_example_x86.yml.
Signed-off-by: Robert Foley <robert.foley@linaro.org>
Reviewed-by: Peter Puhov <peter.puhov@linaro.org>
---
configure | 9 ++++++
tests/vm/Makefile.include | 6 ++++
tests/vm/basevm.py | 40 +++++++++++++++++++++++-
tests/vm/conf_example_aarch64.yml | 51 +++++++++++++++++++++++++++++++
tests/vm/conf_example_x86.yml | 50 ++++++++++++++++++++++++++++++
5 files changed, 155 insertions(+), 1 deletion(-)
create mode 100644 tests/vm/conf_example_aarch64.yml
create mode 100644 tests/vm/conf_example_x86.yml
diff --git a/configure b/configure
index a574250524..89d11aa5d4 100755
--- a/configure
+++ b/configure
@@ -950,6 +950,13 @@ do
fi
done
+# Check for existence of python3 yaml, needed to
+# import yaml config files into vm-build.
+python_yaml="no"
+if $(python3 -c "import yaml" 2> /dev/null); then
+ python_yaml="yes"
+fi
+
: ${smbd=${SMBD-/usr/sbin/smbd}}
# Default objcc to clang if available, otherwise use CC
@@ -6581,6 +6588,7 @@ if test "$docs" != "no"; then
echo "sphinx-build $sphinx_build"
fi
echo "genisoimage $genisoimage"
+echo "python_yaml $python_yaml"
echo "slirp support $slirp $(echo_version $slirp $slirp_version)"
if test "$slirp" != "no" ; then
echo "smbd $smbd"
@@ -7642,6 +7650,7 @@ echo "PYTHON=$python" >> $config_host_mak
echo "SPHINX_BUILD=$sphinx_build" >> $config_host_mak
echo "SPHINX_WERROR=$sphinx_werror" >> $config_host_mak
echo "GENISOIMAGE=$genisoimage" >> $config_host_mak
+echo "PYTHON_YAML=$python_yaml" >> $config_host_mak
echo "CC=$cc" >> $config_host_mak
if $iasl -h > /dev/null 2>&1; then
echo "IASL=$iasl" >> $config_host_mak
diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include
index 74ab522c55..d9b34eae63 100644
--- a/tests/vm/Makefile.include
+++ b/tests/vm/Makefile.include
@@ -43,6 +43,12 @@ endif
@echo " V=1 - Enable verbose ouput on host and guest commands"
@echo " QEMU=/path/to/qemu - Change path to QEMU binary"
@echo " QEMU_IMG=/path/to/qemu-img - Change path to qemu-img tool"
+ifeq ($(PYTHON_YAML),yes)
+ @echo " QEMU_CONFIG=/path/conf.yml - Change path to VM configuration .yml file."
+else
+ @echo " (install python3-yaml to enable support for yaml file to configure a VM.)"
+endif
+ @echo " See conf_example_*.yml for file format details."
vm-build-all: $(addprefix vm-build-, $(IMAGES))
diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
index 86ec0df24a..0bc1bad839 100644
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -478,9 +478,43 @@ class BaseVM(object):
cwd=cidir,
stdin=self._devnull, stdout=self._stdout,
stderr=self._stdout)
-
return os.path.join(cidir, "cloud-init.iso")
+def parse_config(config, args):
+ """ Parse yaml config and populate our config structure.
+ The yaml config allows the user to override the
+ defaults for VM parameters. In many cases these
+ defaults can be overridden without rebuilding the VM."""
+ if args.config:
+ config_file = args.config
+ elif 'QEMU_CONFIG' in os.environ:
+ config_file = os.environ['QEMU_CONFIG']
+ else:
+ return config
+ if not os.path.exists(config_file):
+ raise Exception("config file {} does not exist".format(config_file))
+ # We gracefully handle importing the yaml module
+ # since it might not be installed.
+ # If we are here it means the user supplied a .yml file,
+ # so if the yaml module is not installed we will exit with error.
+ try:
+ import yaml
+ except ImportError:
+ print("The python3-yaml package is needed "\
+ "to support config.yaml files")
+ # Instead of raising an exception we exit to avoid
+ # a raft of messy (expected) errors to stdout.
+ exit(1)
+ with open(config_file) as f:
+ yaml_dict = yaml.safe_load(f)
+
+ if 'qemu-conf' in yaml_dict:
+ config.update(yaml_dict['qemu-conf'])
+ else:
+ raise Exception("config file {} is not valid"\
+ " missing qemu-conf".format(config_file))
+ return config
+
def parse_args(vmcls):
def get_default_jobs():
@@ -517,6 +551,9 @@ def parse_args(vmcls):
help="run tests with a snapshot")
parser.add_option("--genisoimage", default="genisoimage",
help="iso imaging tool")
+ parser.add_option("--config", "-c", default=None,
+ help="Provide config yaml for configuration. "\
+ "See config_example.yaml for example.")
parser.disable_interspersed_args()
return parser.parse_args()
@@ -528,6 +565,7 @@ def main(vmcls, config=None):
if not argv and not args.build_qemu and not args.build_image:
print("Nothing to do?")
return 1
+ config = parse_config(config, args)
logging.basicConfig(level=(logging.DEBUG if args.debug
else logging.WARN))
vm = vmcls(args, config=config)
diff --git a/tests/vm/conf_example_aarch64.yml b/tests/vm/conf_example_aarch64.yml
new file mode 100644
index 0000000000..9d44ae356f
--- /dev/null
+++ b/tests/vm/conf_example_aarch64.yml
@@ -0,0 +1,51 @@
+#
+# Example yaml for use by any of the scripts in tests/vm.
+# Can be provided as an environment variable QEMU_CONFIG
+#
+qemu-conf:
+
+ # If any of the below are not provided, we will just use the qemu defaults.
+
+ # Login username and password(has to be sudo enabled)
+ guest_user: qemu
+ guest_pass: "qemupass"
+
+ # Password for root user can be different from guest.
+ root_pass: "qemupass"
+
+ # If one key is provided, both must be provided.
+ #ssh_key: /complete/path/of/your/keyfile/id_rsa
+ #ssh_pub_key: /complete/path/of/your/keyfile/id_rsa.pub
+
+ cpu: max
+ machine: virt,gic-version=max
+ memory: 16G
+
+ # The below is a example for how to configure NUMA topology with
+ # 4 NUMA nodes and 2 different NUMA distances.
+ qemu_args: "-smp cpus=16,sockets=2,cores=8
+ -numa node,cpus=0-3,nodeid=0 -numa node,cpus=4-7,nodeid=1
+ -numa node,cpus=8-11,nodeid=2 -numa node,cpus=12-15,nodeid=3
+ -numa dist,src=0,dst=1,val=15 -numa dist,src=2,dst=3,val=15
+ -numa dist,src=0,dst=2,val=20 -numa dist,src=0,dst=3,val=20
+ -numa dist,src=1,dst=2,val=20 -numa dist,src=1,dst=3,val=20"
+
+ # By default we do not set the DNS.
+ # You override the defaults by setting the below.
+ #dns: 1.234.567.89
+
+ # By default we will use a "block" device, but
+ # you can also boot from a "scsi" device.
+ # Just keep in mind your scripts might need to change
+ # As you will have /dev/sda instead of /dev/vda (for block device)
+ boot_dev_type: "block"
+
+ # By default the ssh port is not fixed.
+ # A fixed ssh port makes it easier for automated tests.
+ #ssh_port: 5555
+
+ # To install a different set of packages, provide a command to issue
+ #install_cmds: "apt-get update ; apt-get build-dep -y qemu"
+
+ # Or to skip the install entirely, just provide ""
+ #install_cmds: ""
diff --git a/tests/vm/conf_example_x86.yml b/tests/vm/conf_example_x86.yml
new file mode 100644
index 0000000000..78d3f5830f
--- /dev/null
+++ b/tests/vm/conf_example_x86.yml
@@ -0,0 +1,50 @@
+#
+# Example yaml for use by any of the x86 based scripts in tests/vm.
+# Can be provided as an environment variable QEMU_CONFIG
+#
+qemu-conf:
+
+ # If any of the below are not provided, we will just use the qemu defaults.
+
+ # Login username and password(has to be sudo enabled)
+ guest_user: "qemu"
+ guest_pass: "qemupass"
+
+ # Password for root user can be different from guest.
+ root_pass: "qemupass"
+
+ # Provide default ssh keys of current user.
+ # You need to edit the below for your user.
+ #ssh_key_file: /home/<user>/.ssh/id_rsa
+ #ssh_pub_key_file: /home/<user>/.ssh/id_rsa.pub
+
+ cpu: max
+ machine: pc
+ memory: 8G
+
+ # The below is a example for how to configure NUMA topology with
+ # 4 NUMA nodes and 2 different NUMA distances.
+ qemu_args: "-smp cpus=8,sockets=2,cores=4
+ -object memory-backend-ram,size=4G,policy=bind,host-nodes=0,id=ram-node0
+ -object memory-backend-ram,size=4G,policy=bind,host-nodes=0,id=ram-node1
+ -object memory-backend-ram,size=4G,policy=bind,host-nodes=1,id=ram-node2
+ -object memory-backend-ram,size=4G,policy=bind,host-nodes=1,id=ram-node3
+ -numa node,cpus=0-1,nodeid=0 -numa node,cpus=2-3,nodeid=1
+ -numa node,cpus=4-5,nodeid=2 -numa node,cpus=6-7,nodeid=3
+ -numa dist,src=0,dst=1,val=15 -numa dist,src=2,dst=3,val=15
+ -numa dist,src=0,dst=2,val=20 -numa dist,src=0,dst=3,val=20
+ -numa dist,src=1,dst=2,val=20 -numa dist,src=1,dst=3,val=20"
+
+ # By default we do not set the DNS.
+ # You override the defaults by setting the below.
+ #dns: "1.234.567.89"
+
+ # By default we will use a "block" device, but
+ # you can also boot from a "scsi" device.
+ # Just keep in mind your scripts might need to change
+ # As you will have /dev/sda instead of /dev/vda (for block device)
+ boot_dev_type: "block"
+
+ # By default the ssh port is not fixed.
+ # A fixed ssh port makes it easier for automated tests.
+ ssh_port: 5555
--
2.17.1
next prev parent reply other threads:[~2020-05-19 13:29 UTC|newest]
Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-05-19 13:22 [PATCH v7 00/12] tests/vm: Add support for aarch64 VMs Robert Foley
2020-05-19 13:22 ` [PATCH v7 01/12] configure: add alternate binary for genisoimage Robert Foley
2020-05-19 13:22 ` [PATCH v7 02/12] tests/vm: pass --genisoimage to basevm script Robert Foley
2020-05-19 13:22 ` [PATCH v7 03/12] tests/vm: pass args through to BaseVM's __init__ Robert Foley
2020-05-20 21:49 ` Alex Bennée
2020-05-22 12:58 ` Robert Foley
2020-05-19 13:22 ` [PATCH v7 04/12] tests/vm: Add configuration to basevm.py Robert Foley
2020-05-22 14:22 ` Alex Bennée
2020-05-19 13:22 ` Robert Foley [this message]
2020-05-22 14:26 ` [PATCH v7 05/12] tests/vm: Added configuration file support Alex Bennée
2020-05-19 13:22 ` [PATCH v7 06/12] tests/vm: Pass --debug through for vm-boot-ssh Robert Foley
2020-05-22 14:29 ` Alex Bennée
2020-05-19 13:22 ` [PATCH v7 07/12] tests/vm: Add ability to select QEMU from current build Robert Foley
2020-05-22 14:40 ` Alex Bennée
2020-05-22 18:53 ` Robert Foley
2020-05-19 13:22 ` [PATCH v7 08/12] tests/vm: allow wait_ssh() to specify command Robert Foley
2020-05-19 13:22 ` [PATCH v7 09/12] tests/vm: Added a new script for ubuntu.aarch64 Robert Foley
2020-05-22 15:34 ` Alex Bennée
2020-05-22 19:24 ` Robert Foley
2020-05-19 13:22 ` [PATCH v7 10/12] tests/vm: Added a new script for centos.aarch64 Robert Foley
2020-05-22 15:59 ` Alex Bennée
2020-05-22 19:44 ` Robert Foley
2020-05-19 13:22 ` [PATCH v7 11/12] tests/vm: change scripts to use self._config Robert Foley
2020-05-19 13:22 ` [PATCH v7 12/12] tests/vm: Add workaround to consume console Robert Foley
2020-05-22 16:31 ` Alex Bennée
2020-05-22 20:44 ` Robert Foley
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200519132259.405-6-robert.foley@linaro.org \
--to=robert.foley@linaro.org \
--cc=alex.bennee@linaro.org \
--cc=peter.puhov@linaro.org \
--cc=philmd@redhat.com \
--cc=qemu-devel@nongnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).