From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1CFF2C433EF for ; Wed, 22 Sep 2021 16:24:00 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 75665611F2 for ; Wed, 22 Sep 2021 16:23:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 75665611F2 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=greensocs.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:43346 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mT52M-0001me-Eg for qemu-devel@archiver.kernel.org; Wed, 22 Sep 2021 12:23:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:44700) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mT4uj-00038P-Ae; Wed, 22 Sep 2021 12:16:05 -0400 Received: from beetle.greensocs.com ([5.135.226.135]:38792) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mT4uO-00057v-DZ; Wed, 22 Sep 2021 12:16:05 -0400 Received: from crumble.bar.greensocs.com (unknown [172.17.10.6]) by beetle.greensocs.com (Postfix) with ESMTPS id D4A5120787; Wed, 22 Sep 2021 16:15:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=greensocs.com; s=mail; t=1632327333; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=p52VqvjsCznljFY2xO4Rt/Nf2Duz4eWTG3QEtLzwHk4=; b=ew6f8RpIgUvMpeGq1oW3jHrqPK8H2iO+ccW5V5LopATHnIAc/5JmP0kO1GUCNtrVUBg0cJ QT30XZEqF9IENJ34znQQjNMjtUpMYNa0HJ6F++N+WdqDnf2XtZ2AtpOoC8qXWKxmwJ7ZAK u3YYtGzUu+N4fWJfxLOXmE9BpPy7KnE= From: Damien Hedde To: qemu-devel@nongnu.org Subject: [RFC PATCH v2 00/16] Initial support for machine creation via QMP Date: Wed, 22 Sep 2021 18:13:49 +0200 Message-Id: <20210922161405.140018-1-damien.hedde@greensocs.com> X-Mailer: git-send-email 2.33.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam: Yes Received-SPF: pass client-ip=5.135.226.135; envelope-from=damien.hedde@greensocs.com; helo=beetle.greensocs.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , "Michael S. Tsirkin" , David Hildenbrand , Peter Xu , mirela.grujic@greensocs.com, Alistair Francis , Gerd Hoffmann , Ani Sinha , Eric Blake , Stefano Stabellini , xen-devel@lists.xenproject.org, Paul Durrant , Markus Armbruster , Anthony Perard , =?UTF-8?q?Marc-Andr=C3=A9=20Lureau?= , Eduardo Habkost , "Dr. David Alan Gilbert" , Eric Auger , Paolo Bonzini , qemu-riscv@nongnu.org, Damien Hedde , =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= , mark.burton@greensocs.com, edgari@xilinx.com, Igor Mammedov Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Hi, The goal of this work is to bring dynamic machine creation to QEMU: we want to setup a machine without compiling a specific machine C code. It would ease supporting highly configurable platforms (for example resulting from an automated design flow). The requirements for such configuration include begin able to specify the number of cores, available peripherals, emmory mapping, IRQ mapping, etc. This series focuses on the first step: populating a machine with devices during its creation. We propose patches to support this using QMP commands. This is a working set of patches and improves over the earlier rfc (posted in May): https://lists.gnu.org/archive/html/qemu-devel/2021-05/msg03706.html Although it is working and could be merged, it is tag as an RFC: we probably need to discuss the conditions for allowing a device to be created at an early stage. Patches 6, 10 and 13, 15 and 16 depend on such conditions and are subject to change. Other patches are unrelated to this point. We address several issues in this series. They are detailed below. ## 1. Stoping QEMU to populate the machine with devices QEMU goes through several steps (called _machine phases_) when creating the machine: 'no-machine', 'machine-created', 'accel-created', 'initialized', and finally 'ready'. At 'ready' phase, QEMU is ready to start (see Paolo's page https://wiki.qemu.org/User:Paolo_Bonzini/Machine_init_sequence for more details). Using the -preconfig CLI option, QEMU can be stopped today during the 'accel-created' phase. Then the 'x-exit-preconfig' QMP command triggers QEMU moving forwards to the completion of the machine creation ('ready' phase). The devices are created during the 'initialized' phase. In this phase the machine init() method has been executed and thus machine properties have been handled. Although the sysbus exists and the machine may have been populated by the init(), _machine_init_done_ notifiers have not been called yet. At this point we can add more devices to a machine. We propose to add 2 QMP commands: + The 'query-machine-phase' command would return the current machine phase. + The 'x-machine-init' command would advance the machine phase to 'initialized'. 'x-exit-preconfig' could then still be used to advance to the last phase. ## 2. Adding devices Right now, the user can create devices in 2 ways: using '-device' CLI option or 'device_add' QMP command. Both are executed after the machine is ready: such devices are hot-plugged. We propose to allow 'device_add' QMP command to be used during the 'initialized' phase. In this series, we keep the constraint that the device must be 'user-creatable' (this is a device class flag). We do not see any reason why a device the user can hot-plug could not be created at an earlier stage. This part is still RFC because, as Peter mentioned it (in this thread https://lists.gnu.org/archive/html/qemu-devel/2021-08/msg01933.html), we may want additional or distinct conditions for: + device we can hot-plug + device we can add in '-preconfig' (cold-plug) We are open to suggestions. We could for example add a 'preconfig-creatable' or 'init-creatable' flag to device class, which can identify a set of devices we can create this way. The main addition is how we handle the case of sysbus devices. Sysbus devices are particular because unlike, for example, pci devices, you have to manually handle the memory mapping and interrupts wiring. So right now, a sysbus device is dynamically creatable (using -device CLI option or device_add QMP command) only if: + it is 'user_creatable' (like any other device), + and it is in the current machine sysbus device allow list. In this series, we propose to relax the second constraint during the earlier phases of machine creation so that when using -preconfig we can create any 'user-creatable' sysbus device. When the machine progresses to the 'ready' phase, sysbus devices creation will come back to the legacy behavior: it will be possible only based on the per-machine authorization basis. For sysbus devices, wiring interrupts is not a problem as we can use the 'qom-set' QMP command, but memory mapping is. ## 3. Memory mapping There is no point allowing the creation sysbus devices if we cannot map them onto the memory bus (the 'sysbus'). As far as we know, right now, there is no way to add memory mapping for sysbus device using QMP commands. We propose a 'x-sysbus-mmio-map' command to do this. This command would only be allowed during the 'initialized' phase when using -preconfig. ## 4. Working example The last patches of the series add and modify devices in order to build a working machine starting from the 'none' machine. We add a new sysbus device modeling a simple memory (ram or rom). We also set 'user-creatable' flag of some sysbus devices. These are trivial patches, but they depends on the conditions we choose to allow creating devices with -preconfig. Therefore, there is really no need to review them until we settled on the device conditions first. With these devices (memory, ibex_uart, ibex_plic) we can dynamically configure a part (we did not add the timer, but we could) the opentitan machine very easily and run firmwares which demonstrates interrupts and memory-mapping are working. We use the existing qmp-shell script to issue machine devices from a qmp commands script file which contains qmp commands listed in a file. The following qmp commands add some memories, an interrupt controller and an uart with an interrupt. cat > opentitan.qmp < X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE7FFC433F5 for ; Wed, 22 Sep 2021 17:11:06 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9872D60F23 for ; Wed, 22 Sep 2021 17:11:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 9872D60F23 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=greensocs.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.192937.343699 (Exim 4.92) (envelope-from ) id 1mT5lm-0001BA-1E; Wed, 22 Sep 2021 17:10:54 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 192937.343699; Wed, 22 Sep 2021 17:10:54 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mT5ll-0001B3-UF; Wed, 22 Sep 2021 17:10:53 +0000 Received: by outflank-mailman (input) for mailman id 192937; Wed, 22 Sep 2021 16:31:39 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mT59n-0004Al-3E for xen-devel@lists.xenproject.org; Wed, 22 Sep 2021 16:31:39 +0000 Received: from beetle.greensocs.com (unknown [5.135.226.135]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 8e312bdb-1bc2-11ec-b9b9-12813bfff9fa; Wed, 22 Sep 2021 16:31:37 +0000 (UTC) Received: from crumble.bar.greensocs.com (unknown [172.17.10.6]) by beetle.greensocs.com (Postfix) with ESMTPS id D4A5120787; Wed, 22 Sep 2021 16:15:32 +0000 (UTC) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 8e312bdb-1bc2-11ec-b9b9-12813bfff9fa DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=greensocs.com; s=mail; t=1632327333; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=p52VqvjsCznljFY2xO4Rt/Nf2Duz4eWTG3QEtLzwHk4=; b=ew6f8RpIgUvMpeGq1oW3jHrqPK8H2iO+ccW5V5LopATHnIAc/5JmP0kO1GUCNtrVUBg0cJ QT30XZEqF9IENJ34znQQjNMjtUpMYNa0HJ6F++N+WdqDnf2XtZ2AtpOoC8qXWKxmwJ7ZAK u3YYtGzUu+N4fWJfxLOXmE9BpPy7KnE= From: Damien Hedde To: qemu-devel@nongnu.org Cc: Damien Hedde , Alistair Francis , =?UTF-8?q?Marc-Andr=C3=A9=20Lureau?= , Paolo Bonzini , Eduardo Habkost , Marcel Apfelbaum , =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= , "Michael S. Tsirkin" , Igor Mammedov , Ani Sinha , Gerd Hoffmann , Eric Auger , Stefano Stabellini , Anthony Perard , Paul Durrant , Peter Xu , David Hildenbrand , "Dr. David Alan Gilbert" , Markus Armbruster , Eric Blake , qemu-riscv@nongnu.org, xen-devel@lists.xenproject.org, mark.burton@greensocs.com, mirela.grujic@greensocs.com, edgari@xilinx.com, Peter Maydell Subject: [RFC PATCH v2 00/16] Initial support for machine creation via QMP Date: Wed, 22 Sep 2021 18:13:49 +0200 Message-Id: <20210922161405.140018-1-damien.hedde@greensocs.com> X-Mailer: git-send-email 2.33.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam: Yes Hi, The goal of this work is to bring dynamic machine creation to QEMU: we want to setup a machine without compiling a specific machine C code. It would ease supporting highly configurable platforms (for example resulting from an automated design flow). The requirements for such configuration include begin able to specify the number of cores, available peripherals, emmory mapping, IRQ mapping, etc. This series focuses on the first step: populating a machine with devices during its creation. We propose patches to support this using QMP commands. This is a working set of patches and improves over the earlier rfc (posted in May): https://lists.gnu.org/archive/html/qemu-devel/2021-05/msg03706.html Although it is working and could be merged, it is tag as an RFC: we probably need to discuss the conditions for allowing a device to be created at an early stage. Patches 6, 10 and 13, 15 and 16 depend on such conditions and are subject to change. Other patches are unrelated to this point. We address several issues in this series. They are detailed below. ## 1. Stoping QEMU to populate the machine with devices QEMU goes through several steps (called _machine phases_) when creating the machine: 'no-machine', 'machine-created', 'accel-created', 'initialized', and finally 'ready'. At 'ready' phase, QEMU is ready to start (see Paolo's page https://wiki.qemu.org/User:Paolo_Bonzini/Machine_init_sequence for more details). Using the -preconfig CLI option, QEMU can be stopped today during the 'accel-created' phase. Then the 'x-exit-preconfig' QMP command triggers QEMU moving forwards to the completion of the machine creation ('ready' phase). The devices are created during the 'initialized' phase. In this phase the machine init() method has been executed and thus machine properties have been handled. Although the sysbus exists and the machine may have been populated by the init(), _machine_init_done_ notifiers have not been called yet. At this point we can add more devices to a machine. We propose to add 2 QMP commands: + The 'query-machine-phase' command would return the current machine phase. + The 'x-machine-init' command would advance the machine phase to 'initialized'. 'x-exit-preconfig' could then still be used to advance to the last phase. ## 2. Adding devices Right now, the user can create devices in 2 ways: using '-device' CLI option or 'device_add' QMP command. Both are executed after the machine is ready: such devices are hot-plugged. We propose to allow 'device_add' QMP command to be used during the 'initialized' phase. In this series, we keep the constraint that the device must be 'user-creatable' (this is a device class flag). We do not see any reason why a device the user can hot-plug could not be created at an earlier stage. This part is still RFC because, as Peter mentioned it (in this thread https://lists.gnu.org/archive/html/qemu-devel/2021-08/msg01933.html), we may want additional or distinct conditions for: + device we can hot-plug + device we can add in '-preconfig' (cold-plug) We are open to suggestions. We could for example add a 'preconfig-creatable' or 'init-creatable' flag to device class, which can identify a set of devices we can create this way. The main addition is how we handle the case of sysbus devices. Sysbus devices are particular because unlike, for example, pci devices, you have to manually handle the memory mapping and interrupts wiring. So right now, a sysbus device is dynamically creatable (using -device CLI option or device_add QMP command) only if: + it is 'user_creatable' (like any other device), + and it is in the current machine sysbus device allow list. In this series, we propose to relax the second constraint during the earlier phases of machine creation so that when using -preconfig we can create any 'user-creatable' sysbus device. When the machine progresses to the 'ready' phase, sysbus devices creation will come back to the legacy behavior: it will be possible only based on the per-machine authorization basis. For sysbus devices, wiring interrupts is not a problem as we can use the 'qom-set' QMP command, but memory mapping is. ## 3. Memory mapping There is no point allowing the creation sysbus devices if we cannot map them onto the memory bus (the 'sysbus'). As far as we know, right now, there is no way to add memory mapping for sysbus device using QMP commands. We propose a 'x-sysbus-mmio-map' command to do this. This command would only be allowed during the 'initialized' phase when using -preconfig. ## 4. Working example The last patches of the series add and modify devices in order to build a working machine starting from the 'none' machine. We add a new sysbus device modeling a simple memory (ram or rom). We also set 'user-creatable' flag of some sysbus devices. These are trivial patches, but they depends on the conditions we choose to allow creating devices with -preconfig. Therefore, there is really no need to review them until we settled on the device conditions first. With these devices (memory, ibex_uart, ibex_plic) we can dynamically configure a part (we did not add the timer, but we could) the opentitan machine very easily and run firmwares which demonstrates interrupts and memory-mapping are working. We use the existing qmp-shell script to issue machine devices from a qmp commands script file which contains qmp commands listed in a file. The following qmp commands add some memories, an interrupt controller and an uart with an interrupt. cat > opentitan.qmp <