From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.6 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E7F2C35247 for ; Wed, 5 Feb 2020 21:39:26 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 64DDA20720 for ; Wed, 5 Feb 2020 21:39:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="jYLeZz1k" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 64DDA20720 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:57296 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1izSOL-0006r5-Kr for qemu-devel@archiver.kernel.org; Wed, 05 Feb 2020 16:39:25 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:47428) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1izSJ9-0005c7-Nf for qemu-devel@nongnu.org; Wed, 05 Feb 2020 16:34:05 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1izSJ7-0006Pa-Rc for qemu-devel@nongnu.org; Wed, 05 Feb 2020 16:34:03 -0500 Received: from mail-pg1-x544.google.com ([2607:f8b0:4864:20::544]:37972) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1izSJ7-0006N2-IQ for qemu-devel@nongnu.org; Wed, 05 Feb 2020 16:34:01 -0500 Received: by mail-pg1-x544.google.com with SMTP id a33so1595500pgm.5 for ; Wed, 05 Feb 2020 13:34:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=IQap2T8JqAz08Y5Tf6O+E0PgwZXgb93paPBw5e7BGIg=; b=jYLeZz1kOa+bKXYWeahYthoPBBEwPBx9dWOz8i1El5IF+7cZMVEiVDA1nrfBl73aR1 GZ3JKMDYMTHejAcNM/To88Bs74XQlJJfBAkA9fN0YWpUKghRhw52KW2cw4LQWdjx97q5 FX/cU58rObL45q8lVPf4qxbrN66M2s7mcESGpX6C+Aen6DTV0CabLVVSKl3zRyigYY8j Z9bd6rVrR3TTcGPIswq2pcip836vuKP9TXRDQlQMlx/6a4uOAyzWnsP8k/u/XRiDFjzz vZ//kB2Krgy8WMy8FxJ6pv1ni/PXkrimqmfm3mJx2ufnhZLWHXeNtGIuyMpCXe8VFyRk m3QQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=IQap2T8JqAz08Y5Tf6O+E0PgwZXgb93paPBw5e7BGIg=; b=OVbLfa8kChyhZvWYgNg+Vse3TUHNLxabDrnjuNcyq0hDBS9jqreIY7fkZbOIPOBr4G OjRdKfQY7CW299BmVTD6eRoouHX1rrW8xY1Jp6qj1yJqNNAHezzSrb8OVHIXpvmnxTaM zL3EJT/dQWF/mcsoOoQMecdT1isEzJxhjBskf3UTDy4B/3wKUjpKnC0FfyBgD6L5W7UI UBhtuqStUMSx2P9vBdc7yGCT2Ce02vKBYicE4m/nSxqE6feAlzRwyOsOFybzTWkQA/rc v6bo4JEbW7M0eJLUbPY+/S87J0RoztGOmd0C/V6BpGZMN4v6D1/1MOsMEiyfFaNP2V6+ 9XGg== X-Gm-Message-State: APjAAAWhDbDi8Z8UKlUiWu5s5uEX2Zx+yHRBZtaELNfQaA2unNXn+JXM Yvx4CP1AsPUY5NXEgEOFyLvRxApRVbg= X-Google-Smtp-Source: APXvYqyKBihD+AWn3Cj9ol35zNTjRFEKJBkGRhEzZhGCS04GBgulgIz4tEqVBdjQx5UOz4oHTzlSJw== X-Received: by 2002:a62:e30f:: with SMTP id g15mr3461pfh.124.1580938440027; Wed, 05 Feb 2020 13:34:00 -0800 (PST) Received: from Rfoley-MA01.usrd.futurewei.com ([12.111.81.71]) by smtp.gmail.com with ESMTPSA id p4sm724576pgh.14.2020.02.05.13.33.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Feb 2020 13:33:59 -0800 (PST) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v1 08/14] tests/vm: Added configuration file support Date: Wed, 5 Feb 2020 16:29:14 -0500 Message-Id: <20200205212920.467-9-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200205212920.467-1-robert.foley@linaro.org> References: <20200205212920.467-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::544 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fam@euphon.net, philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Changes to tests/vm/basevm.py to allow accepting a configuration file as a parameter. Allows for specifying VM options such as cpu, machine, memory, and arbitrary qemu arguments for specifying options such as NUMA configuration. Also added an example conf_example_aarch64.yml and conf_example_x86.yml. Signed-off-by: Robert Foley --- tests/vm/Makefile.include | 2 ++ tests/vm/basevm.py | 29 +++++++++++++++++- tests/vm/conf_example_aarch64.yml | 51 +++++++++++++++++++++++++++++++ tests/vm/conf_example_x86.yml | 50 ++++++++++++++++++++++++++++++ 4 files changed, 131 insertions(+), 1 deletion(-) create mode 100644 tests/vm/conf_example_aarch64.yml create mode 100644 tests/vm/conf_example_x86.yml diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index 778e506755..e9ed33226d 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -35,6 +35,8 @@ vm-help vm-test: @echo " V=1 - Enable verbose ouput on host and guest commands" @echo " QEMU=/path/to/qemu - Change path to QEMU binary" @echo " QEMU_IMG=/path/to/qemu-img - Change path to qemu-img tool" + @echo " QEMU_CONFIG=/path/conf.yml - Change path to VM configuration .yml file." + @echo " See conf_example_*.yml for file format details." vm-build-all: $(addprefix vm-build-, $(IMAGES)) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index 33004934af..f488d4103c 100755 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -32,6 +32,7 @@ import shutil import multiprocessing import traceback from socket_thread import SocketThread +import yaml SSH_KEY_FILE = os.path.join(os.path.dirname(__file__), "..", "keys", "id_rsa") @@ -500,9 +501,31 @@ class BaseVM(object): cwd=cidir, stdin=self._devnull, stdout=self._stdout, stderr=self._stdout) - return os.path.join(cidir, "cloud-init.iso") +def parse_config(config, args): + """ Parse yaml config and populate our config structure. + The yaml config allows the user to override the + defaults for VM parameters. In many cases these + defaults can be overridden without rebuilding the VM.""" + if args.config: + config_file = args.config + elif 'QEMU_CONFIG' in os.environ: + config_file = os.environ['QEMU_CONFIG'] + else: + return config + if not os.path.exists(config_file): + raise Exception("config file {} does not exist".format(config_file)) + with open(config_file) as f: + yaml_dict = yaml.safe_load(f) + + if 'qemu-conf' in yaml_dict: + config.update(yaml_dict['qemu-conf']) + else: + raise Exception("config file {} is not valid"\ + " missing qemu-conf".format(config_file)) + return config + def parse_args(vmcls): def get_default_jobs(): @@ -537,6 +560,9 @@ def parse_args(vmcls): help="Interactively run command") parser.add_option("--snapshot", "-s", action="store_true", help="run tests with a snapshot") + parser.add_option("--config", "-c", default=None, + help="Provide config yaml for configuration. "\ + "See config_example.yaml for example.") parser.disable_interspersed_args() return parser.parse_args() @@ -548,6 +574,7 @@ def main(vmcls, config=None): if not argv and not args.build_qemu and not args.build_image: print("Nothing to do?") return 1 + config = parse_config(config, args) logging.basicConfig(level=(logging.DEBUG if args.debug else logging.WARN)) vm = vmcls(debug=args.debug, vcpus=args.jobs, config=config) diff --git a/tests/vm/conf_example_aarch64.yml b/tests/vm/conf_example_aarch64.yml new file mode 100644 index 0000000000..9d44ae356f --- /dev/null +++ b/tests/vm/conf_example_aarch64.yml @@ -0,0 +1,51 @@ +# +# Example yaml for use by any of the scripts in tests/vm. +# Can be provided as an environment variable QEMU_CONFIG +# +qemu-conf: + + # If any of the below are not provided, we will just use the qemu defaults. + + # Login username and password(has to be sudo enabled) + guest_user: qemu + guest_pass: "qemupass" + + # Password for root user can be different from guest. + root_pass: "qemupass" + + # If one key is provided, both must be provided. + #ssh_key: /complete/path/of/your/keyfile/id_rsa + #ssh_pub_key: /complete/path/of/your/keyfile/id_rsa.pub + + cpu: max + machine: virt,gic-version=max + memory: 16G + + # The below is a example for how to configure NUMA topology with + # 4 NUMA nodes and 2 different NUMA distances. + qemu_args: "-smp cpus=16,sockets=2,cores=8 + -numa node,cpus=0-3,nodeid=0 -numa node,cpus=4-7,nodeid=1 + -numa node,cpus=8-11,nodeid=2 -numa node,cpus=12-15,nodeid=3 + -numa dist,src=0,dst=1,val=15 -numa dist,src=2,dst=3,val=15 + -numa dist,src=0,dst=2,val=20 -numa dist,src=0,dst=3,val=20 + -numa dist,src=1,dst=2,val=20 -numa dist,src=1,dst=3,val=20" + + # By default we do not set the DNS. + # You override the defaults by setting the below. + #dns: 1.234.567.89 + + # By default we will use a "block" device, but + # you can also boot from a "scsi" device. + # Just keep in mind your scripts might need to change + # As you will have /dev/sda instead of /dev/vda (for block device) + boot_dev_type: "block" + + # By default the ssh port is not fixed. + # A fixed ssh port makes it easier for automated tests. + #ssh_port: 5555 + + # To install a different set of packages, provide a command to issue + #install_cmds: "apt-get update ; apt-get build-dep -y qemu" + + # Or to skip the install entirely, just provide "" + #install_cmds: "" diff --git a/tests/vm/conf_example_x86.yml b/tests/vm/conf_example_x86.yml new file mode 100644 index 0000000000..78d3f5830f --- /dev/null +++ b/tests/vm/conf_example_x86.yml @@ -0,0 +1,50 @@ +# +# Example yaml for use by any of the x86 based scripts in tests/vm. +# Can be provided as an environment variable QEMU_CONFIG +# +qemu-conf: + + # If any of the below are not provided, we will just use the qemu defaults. + + # Login username and password(has to be sudo enabled) + guest_user: "qemu" + guest_pass: "qemupass" + + # Password for root user can be different from guest. + root_pass: "qemupass" + + # Provide default ssh keys of current user. + # You need to edit the below for your user. + #ssh_key_file: /home//.ssh/id_rsa + #ssh_pub_key_file: /home//.ssh/id_rsa.pub + + cpu: max + machine: pc + memory: 8G + + # The below is a example for how to configure NUMA topology with + # 4 NUMA nodes and 2 different NUMA distances. + qemu_args: "-smp cpus=8,sockets=2,cores=4 + -object memory-backend-ram,size=4G,policy=bind,host-nodes=0,id=ram-node0 + -object memory-backend-ram,size=4G,policy=bind,host-nodes=0,id=ram-node1 + -object memory-backend-ram,size=4G,policy=bind,host-nodes=1,id=ram-node2 + -object memory-backend-ram,size=4G,policy=bind,host-nodes=1,id=ram-node3 + -numa node,cpus=0-1,nodeid=0 -numa node,cpus=2-3,nodeid=1 + -numa node,cpus=4-5,nodeid=2 -numa node,cpus=6-7,nodeid=3 + -numa dist,src=0,dst=1,val=15 -numa dist,src=2,dst=3,val=15 + -numa dist,src=0,dst=2,val=20 -numa dist,src=0,dst=3,val=20 + -numa dist,src=1,dst=2,val=20 -numa dist,src=1,dst=3,val=20" + + # By default we do not set the DNS. + # You override the defaults by setting the below. + #dns: "1.234.567.89" + + # By default we will use a "block" device, but + # you can also boot from a "scsi" device. + # Just keep in mind your scripts might need to change + # As you will have /dev/sda instead of /dev/vda (for block device) + boot_dev_type: "block" + + # By default the ssh port is not fixed. + # A fixed ssh port makes it easier for automated tests. + ssh_port: 5555 -- 2.17.1