From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 919B8C63777 for ; Mon, 23 Nov 2020 06:55:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 470A22078E for ; Mon, 23 Nov 2020 06:55:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727849AbgKWGyy (ORCPT ); Mon, 23 Nov 2020 01:54:54 -0500 Received: from szxga06-in.huawei.com ([45.249.212.32]:7964 "EHLO szxga06-in.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725275AbgKWGyy (ORCPT ); Mon, 23 Nov 2020 01:54:54 -0500 Received: from DGGEMS405-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga06-in.huawei.com (SkyGuard) with ESMTP id 4CfdDn12VvzhgH2; Mon, 23 Nov 2020 14:54:33 +0800 (CST) Received: from DESKTOP-7FEPK9S.china.huawei.com (10.174.187.74) by DGGEMS405-HUB.china.huawei.com (10.3.19.205) with Microsoft SMTP Server id 14.3.487.0; Mon, 23 Nov 2020 14:54:38 +0800 From: Shenming Lu To: Marc Zyngier , James Morse , "Julien Thierry" , Suzuki K Poulose , Eric Auger , , , , , Christoffer Dall CC: Alex Williamson , Kirti Wankhede , Cornelia Huck , Neo Jia , , , Subject: [RFC PATCH v1 0/4] KVM: arm64: Add VLPI migration support on GICv4.1 Date: Mon, 23 Nov 2020 14:54:06 +0800 Message-ID: <20201123065410.1915-1-lushenming@huawei.com> X-Mailer: git-send-email 2.27.0.windows.1 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-Originating-IP: [10.174.187.74] X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In GICv4.1, migration has been supported except for (directly-injected) VLPI. And GICv4.1 spec explicitly gives a way to get the VLPI's pending state (which was crucially missing in GICv4.0). So we make VLPI migration capable on GICv4.1 in this patch set. In order to support VLPI migration, we need to save and restore all required configuration information and pending states of VLPIs. But in fact, the configuration information of VLPIs has already been saved (or will be reallocated on the dst host...) in vgic(kvm) migration. So we only have to migrate the pending states of VLPIs specially. Below is the related workflow in migration. On the save path: In migration completion: pause all vCPUs | call each VM state change handler: pause other devices (just keep from sending interrupts, and such as VFIO migration protocol has already realized it [1]) | flush ITS tables into guest RAM | flush RDIST pending tables (also flush VLPI state here) | ... On the resume path: load each device's state: restore ITS tables (include pending tables) from guest RAM | for other (PCI) devices (paused), if configured to have VLPIs, establish the forwarding paths of their VLPIs (and transfer the pending states from kvm's vgic to VPT here) Yet TODO: - For some reason, such as for VFIO PCI devices, there may be repeated resettings of HW VLPI configuration in load_state, resulting in the loss of pending state. A very intuitive solution is to retrieve the pending state in unset_forwarding (and this should be so regardless of migration). But at normal run time, this function may be called when all devices are running, in which case the unmapping of VPE is not allowed. It seems to be an almost insoluble bug... There are other possible solutions as follows: 1) avoid unset_forwarding being called from QEMU in resuming (simply allocate all needed vectors first), which is more reasonable and efficient. 2) add a new dedicated interface to transfer these pending states to HW in GIC VM state change handler corresponding to save_pending_tables. ... Any comments and suggestions are very welcome. Besides, we have tested this series in VFIO migration, and nothing else goes wrong (with two issues committed [2][3]). Links: [1] vfio: UAPI for migration interface for device state: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/ commit/?id=a8a24f3f6e38103b77cf399c38eb54e1219d00d6 [2] vfio: Move the saving of the config space to the right place in VFIO migration: https://patchwork.ozlabs.org/patch/1400246/ [3] vfio: Set the priority of VFIO VM state change handler explicitly: https://patchwork.ozlabs.org/patch/1401280/ Shenming Lu (2): KVM: arm64: GICv4.1: Try to save hw pending state in save_pending_tables KVM: arm64: GICv4.1: Give a chance to save VLPI's pending state Zenghui Yu (2): irqchip/gic-v4.1: Plumb get_irqchip_state VLPI callback KVM: arm64: GICv4.1: Restore VLPI's pending state to physical side .../virt/kvm/devices/arm-vgic-its.rst | 2 +- arch/arm64/kvm/vgic/vgic-its.c | 6 +- arch/arm64/kvm/vgic/vgic-v3.c | 62 +++++++++++++++++-- arch/arm64/kvm/vgic/vgic-v4.c | 12 ++++ drivers/irqchip/irq-gic-v3-its.c | 38 ++++++++++++ 5 files changed, 110 insertions(+), 10 deletions(-) -- 2.23.0 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EEA3DC63777 for ; Mon, 23 Nov 2020 08:35:56 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id 1AE7620888 for ; Mon, 23 Nov 2020 08:35:55 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1AE7620888 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvmarm-bounces@lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 8069C4B986; Mon, 23 Nov 2020 03:35:55 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bH8hQaGkJ4JH; Mon, 23 Nov 2020 03:35:52 -0500 (EST) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 062CA4BB3B; Mon, 23 Nov 2020 03:35:51 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 173314BA5E for ; Mon, 23 Nov 2020 01:54:58 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SwT1Dn4JpY1k for ; Mon, 23 Nov 2020 01:54:56 -0500 (EST) Received: from szxga06-in.huawei.com (szxga06-in.huawei.com [45.249.212.32]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 6C5204BA50 for ; Mon, 23 Nov 2020 01:54:55 -0500 (EST) Received: from DGGEMS405-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga06-in.huawei.com (SkyGuard) with ESMTP id 4CfdDn12VvzhgH2; Mon, 23 Nov 2020 14:54:33 +0800 (CST) Received: from DESKTOP-7FEPK9S.china.huawei.com (10.174.187.74) by DGGEMS405-HUB.china.huawei.com (10.3.19.205) with Microsoft SMTP Server id 14.3.487.0; Mon, 23 Nov 2020 14:54:38 +0800 From: Shenming Lu To: Marc Zyngier , James Morse , "Julien Thierry" , Suzuki K Poulose , Eric Auger , , , , , Christoffer Dall Subject: [RFC PATCH v1 0/4] KVM: arm64: Add VLPI migration support on GICv4.1 Date: Mon, 23 Nov 2020 14:54:06 +0800 Message-ID: <20201123065410.1915-1-lushenming@huawei.com> X-Mailer: git-send-email 2.27.0.windows.1 MIME-Version: 1.0 X-Originating-IP: [10.174.187.74] X-CFilter-Loop: Reflected X-Mailman-Approved-At: Mon, 23 Nov 2020 03:35:49 -0500 Cc: Neo Jia , Cornelia Huck , Kirti Wankhede , lushenming@huawei.com, Alex Williamson X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu In GICv4.1, migration has been supported except for (directly-injected) VLPI. And GICv4.1 spec explicitly gives a way to get the VLPI's pending state (which was crucially missing in GICv4.0). So we make VLPI migration capable on GICv4.1 in this patch set. In order to support VLPI migration, we need to save and restore all required configuration information and pending states of VLPIs. But in fact, the configuration information of VLPIs has already been saved (or will be reallocated on the dst host...) in vgic(kvm) migration. So we only have to migrate the pending states of VLPIs specially. Below is the related workflow in migration. On the save path: In migration completion: pause all vCPUs | call each VM state change handler: pause other devices (just keep from sending interrupts, and such as VFIO migration protocol has already realized it [1]) | flush ITS tables into guest RAM | flush RDIST pending tables (also flush VLPI state here) | ... On the resume path: load each device's state: restore ITS tables (include pending tables) from guest RAM | for other (PCI) devices (paused), if configured to have VLPIs, establish the forwarding paths of their VLPIs (and transfer the pending states from kvm's vgic to VPT here) Yet TODO: - For some reason, such as for VFIO PCI devices, there may be repeated resettings of HW VLPI configuration in load_state, resulting in the loss of pending state. A very intuitive solution is to retrieve the pending state in unset_forwarding (and this should be so regardless of migration). But at normal run time, this function may be called when all devices are running, in which case the unmapping of VPE is not allowed. It seems to be an almost insoluble bug... There are other possible solutions as follows: 1) avoid unset_forwarding being called from QEMU in resuming (simply allocate all needed vectors first), which is more reasonable and efficient. 2) add a new dedicated interface to transfer these pending states to HW in GIC VM state change handler corresponding to save_pending_tables. ... Any comments and suggestions are very welcome. Besides, we have tested this series in VFIO migration, and nothing else goes wrong (with two issues committed [2][3]). Links: [1] vfio: UAPI for migration interface for device state: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/ commit/?id=a8a24f3f6e38103b77cf399c38eb54e1219d00d6 [2] vfio: Move the saving of the config space to the right place in VFIO migration: https://patchwork.ozlabs.org/patch/1400246/ [3] vfio: Set the priority of VFIO VM state change handler explicitly: https://patchwork.ozlabs.org/patch/1401280/ Shenming Lu (2): KVM: arm64: GICv4.1: Try to save hw pending state in save_pending_tables KVM: arm64: GICv4.1: Give a chance to save VLPI's pending state Zenghui Yu (2): irqchip/gic-v4.1: Plumb get_irqchip_state VLPI callback KVM: arm64: GICv4.1: Restore VLPI's pending state to physical side .../virt/kvm/devices/arm-vgic-its.rst | 2 +- arch/arm64/kvm/vgic/vgic-its.c | 6 +- arch/arm64/kvm/vgic/vgic-v3.c | 62 +++++++++++++++++-- arch/arm64/kvm/vgic/vgic-v4.c | 12 ++++ drivers/irqchip/irq-gic-v3-its.c | 38 ++++++++++++ 5 files changed, 110 insertions(+), 10 deletions(-) -- 2.23.0 _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A350C2D0E4 for ; Mon, 23 Nov 2020 06:55:50 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EC6D52078E for ; Mon, 23 Nov 2020 06:55:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="sP+TulNR" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EC6D52078E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:Message-ID:Date:Subject:To:From: Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender :Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Owner; bh=nzgU2UKS4JvwycLPdYHcoKVRRbJhOb9u8dZVn7b9D/I=; b=sP+TulNRRm+jd88CyFAM0MdobM ppTaAJGEtt36plJ5dyrBoqe4W6NqmGCWhkKpj+zXbVpL2PY6kt6m1rjqXDikYmvKtj6zK1yDd/Djd OwebIfkBwhJCewuSE+5Q45ZnQC+qgODL/w7Lhvh9iGpdSwKDMdlby+00ypRou+lLAKcXNERwQgDYr DiB/V7RBL6VCN8M9bCXMqL9e79x6UHR9p1ZrigKlH7PE/LhuNAhPhGyGtius7oxOqSxLtt+2lAJzC xE+Xtr9VRtVARNG/LczPPGFHTGo/iK975QVLBiCzGni0kmP93m1TXsH/+gK5T9woZzukaByYW2Kgd f8S7L8Xg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kh5kx-0007zd-LW; Mon, 23 Nov 2020 06:55:23 +0000 Received: from szxga06-in.huawei.com ([45.249.212.32]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kh5ki-0007v9-Rf for linux-arm-kernel@lists.infradead.org; Mon, 23 Nov 2020 06:55:10 +0000 Received: from DGGEMS405-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga06-in.huawei.com (SkyGuard) with ESMTP id 4CfdDn12VvzhgH2; Mon, 23 Nov 2020 14:54:33 +0800 (CST) Received: from DESKTOP-7FEPK9S.china.huawei.com (10.174.187.74) by DGGEMS405-HUB.china.huawei.com (10.3.19.205) with Microsoft SMTP Server id 14.3.487.0; Mon, 23 Nov 2020 14:54:38 +0800 From: Shenming Lu To: Marc Zyngier , James Morse , "Julien Thierry" , Suzuki K Poulose , Eric Auger , , , , , Christoffer Dall Subject: [RFC PATCH v1 0/4] KVM: arm64: Add VLPI migration support on GICv4.1 Date: Mon, 23 Nov 2020 14:54:06 +0800 Message-ID: <20201123065410.1915-1-lushenming@huawei.com> X-Mailer: git-send-email 2.27.0.windows.1 MIME-Version: 1.0 X-Originating-IP: [10.174.187.74] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201123_015509_398349_1B46A727 X-CRM114-Status: GOOD ( 12.48 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Neo Jia , Cornelia Huck , Kirti Wankhede , lushenming@huawei.com, Alex Williamson , yuzenghui@huawei.com, wanghaibin.wang@huawei.com Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In GICv4.1, migration has been supported except for (directly-injected) VLPI. And GICv4.1 spec explicitly gives a way to get the VLPI's pending state (which was crucially missing in GICv4.0). So we make VLPI migration capable on GICv4.1 in this patch set. In order to support VLPI migration, we need to save and restore all required configuration information and pending states of VLPIs. But in fact, the configuration information of VLPIs has already been saved (or will be reallocated on the dst host...) in vgic(kvm) migration. So we only have to migrate the pending states of VLPIs specially. Below is the related workflow in migration. On the save path: In migration completion: pause all vCPUs | call each VM state change handler: pause other devices (just keep from sending interrupts, and such as VFIO migration protocol has already realized it [1]) | flush ITS tables into guest RAM | flush RDIST pending tables (also flush VLPI state here) | ... On the resume path: load each device's state: restore ITS tables (include pending tables) from guest RAM | for other (PCI) devices (paused), if configured to have VLPIs, establish the forwarding paths of their VLPIs (and transfer the pending states from kvm's vgic to VPT here) Yet TODO: - For some reason, such as for VFIO PCI devices, there may be repeated resettings of HW VLPI configuration in load_state, resulting in the loss of pending state. A very intuitive solution is to retrieve the pending state in unset_forwarding (and this should be so regardless of migration). But at normal run time, this function may be called when all devices are running, in which case the unmapping of VPE is not allowed. It seems to be an almost insoluble bug... There are other possible solutions as follows: 1) avoid unset_forwarding being called from QEMU in resuming (simply allocate all needed vectors first), which is more reasonable and efficient. 2) add a new dedicated interface to transfer these pending states to HW in GIC VM state change handler corresponding to save_pending_tables. ... Any comments and suggestions are very welcome. Besides, we have tested this series in VFIO migration, and nothing else goes wrong (with two issues committed [2][3]). Links: [1] vfio: UAPI for migration interface for device state: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/ commit/?id=a8a24f3f6e38103b77cf399c38eb54e1219d00d6 [2] vfio: Move the saving of the config space to the right place in VFIO migration: https://patchwork.ozlabs.org/patch/1400246/ [3] vfio: Set the priority of VFIO VM state change handler explicitly: https://patchwork.ozlabs.org/patch/1401280/ Shenming Lu (2): KVM: arm64: GICv4.1: Try to save hw pending state in save_pending_tables KVM: arm64: GICv4.1: Give a chance to save VLPI's pending state Zenghui Yu (2): irqchip/gic-v4.1: Plumb get_irqchip_state VLPI callback KVM: arm64: GICv4.1: Restore VLPI's pending state to physical side .../virt/kvm/devices/arm-vgic-its.rst | 2 +- arch/arm64/kvm/vgic/vgic-its.c | 6 +- arch/arm64/kvm/vgic/vgic-v3.c | 62 +++++++++++++++++-- arch/arm64/kvm/vgic/vgic-v4.c | 12 ++++ drivers/irqchip/irq-gic-v3-its.c | 38 ++++++++++++ 5 files changed, 110 insertions(+), 10 deletions(-) -- 2.23.0 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel