From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3F81C388F7 for ; Thu, 22 Oct 2020 14:07:54 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2DA872225F for ; Thu, 22 Oct 2020 14:07:54 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Xkh8LyRU" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2DA872225F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:49252 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kVbFx-0004uc-56 for qemu-devel@archiver.kernel.org; Thu, 22 Oct 2020 10:07:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:43668) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kVbFB-000449-G4 for qemu-devel@nongnu.org; Thu, 22 Oct 2020 10:07:05 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:41895) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1kVbF8-0000d3-GL for qemu-devel@nongnu.org; Thu, 22 Oct 2020 10:07:05 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1603375620; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+TdepzQIfxqBzVYXiRDzV4IiG9bQHD4xd6V99tqPBds=; b=Xkh8LyRUV2MZ1MB7tqeHtPIAN0c4RNOwLVRZsajhKcmt2OhgDU0YzbiIN3PFj5iCD4AxHf Qo8MBeVPuZ7Ccqm6vWJuJJwzfUec6ze5Y6WGPMMvU1N5d8eY38tQcBbiHN0WJVaPAbxour FrvUDbiQcKSU8rYpCbCG0X3IfQunRKw= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-207-qzqpbZ5TP8WVuQZ2yWl_gA-1; Thu, 22 Oct 2020 10:06:57 -0400 X-MC-Unique: qzqpbZ5TP8WVuQZ2yWl_gA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 23B8F1007464; Thu, 22 Oct 2020 14:06:54 +0000 (UTC) Received: from w520.home (ovpn-112-213.phx2.redhat.com [10.3.112.213]) by smtp.corp.redhat.com (Postfix) with ESMTP id 72CFA5C1C7; Thu, 22 Oct 2020 14:06:49 +0000 (UTC) Date: Thu, 22 Oct 2020 08:06:48 -0600 From: Alex Williamson To: Kirti Wankhede Subject: Re: [PATCH v27 03/17] vfio: Add save and load functions for VFIO PCI devices Message-ID: <20201022080648.1af7b85c@w520.home> In-Reply-To: <1603365127-14202-4-git-send-email-kwankhede@nvidia.com> References: <1603365127-14202-1-git-send-email-kwankhede@nvidia.com> <1603365127-14202-4-git-send-email-kwankhede@nvidia.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=alex.williamson@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Received-SPF: pass client-ip=216.205.24.124; envelope-from=alex.williamson@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/22 08:33:10 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cohuck@redhat.com, cjia@nvidia.com, zhi.wang.linux@gmail.com, aik@ozlabs.ru, Zhengxiao.zx@Alibaba-inc.com, shuangtai.tst@alibaba-inc.com, qemu-devel@nongnu.org, peterx@redhat.com, eauger@redhat.com, yi.l.liu@intel.com, quintela@redhat.com, ziye.yang@intel.com, armbru@redhat.com, mlevitsk@redhat.com, pasic@linux.ibm.com, felipe@nutanix.com, zhi.a.wang@intel.com, mcrossley@nvidia.com, kevin.tian@intel.com, yan.y.zhao@intel.com, dgilbert@redhat.com, changpeng.liu@intel.com, eskultet@redhat.com, Ken.Xue@amd.com, jonathan.davies@nutanix.com, pbonzini@redhat.com, dnigam@nvidia.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" On Thu, 22 Oct 2020 16:41:53 +0530 Kirti Wankhede wrote: > Added functions to save and restore PCI device specific data, > specifically config space of PCI device. > > Signed-off-by: Kirti Wankhede > Reviewed-by: Neo Jia > --- > hw/vfio/pci.c | 48 +++++++++++++++++++++++++++++++++++++++++++ > include/hw/vfio/vfio-common.h | 2 ++ > 2 files changed, 50 insertions(+) > > diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c > index bffd5bfe3b78..1036a5332772 100644 > --- a/hw/vfio/pci.c > +++ b/hw/vfio/pci.c > @@ -41,6 +41,7 @@ > #include "trace.h" > #include "qapi/error.h" > #include "migration/blocker.h" > +#include "migration/qemu-file.h" > > #define TYPE_VFIO_PCI_NOHOTPLUG "vfio-pci-nohotplug" > > @@ -2401,11 +2402,58 @@ static Object *vfio_pci_get_object(VFIODevice *vbasedev) > return OBJECT(vdev); > } > > +static bool vfio_msix_enabled(void *opaque, int version_id) > +{ > + PCIDevice *pdev = opaque; > + > + return msix_enabled(pdev); Why msix_enabled() rather than msix_present()? It seems that even if MSI-X is not enabled at the point in time where this is called, there's still emulated state in the vector table. For example if the guest has written the vectors but has not yet enabled the capability at the point where we start a migration, this test might cause the guest on the target to enable MSI-X with uninitialized data in the vector table. > +} > + > +const VMStateDescription vmstate_vfio_pci_config = { > + .name = "VFIOPCIDevice", > + .version_id = 1, > + .minimum_version_id = 1, > + .fields = (VMStateField[]) { > + VMSTATE_PCI_DEVICE(pdev, VFIOPCIDevice), > + VMSTATE_MSIX_TEST(pdev, VFIOPCIDevice, vfio_msix_enabled), MSI (not-X) state is entirely in config space, so doesn't need a separate field, correct? Otherwise this looks quite a bit cleaner than previous version, I hope VMState experts can confirm this is sufficiently extensible within the migration framework. Thanks, Alex > + VMSTATE_END_OF_LIST() > + } > +}; > + > +static void vfio_pci_save_config(VFIODevice *vbasedev, QEMUFile *f) > +{ > + VFIOPCIDevice *vdev = container_of(vbasedev, VFIOPCIDevice, vbasedev); > + > + vmstate_save_state(f, &vmstate_vfio_pci_config, vdev, NULL); > +} > + > +static int vfio_pci_load_config(VFIODevice *vbasedev, QEMUFile *f) > +{ > + VFIOPCIDevice *vdev = container_of(vbasedev, VFIOPCIDevice, vbasedev); > + PCIDevice *pdev = &vdev->pdev; > + int ret; > + > + ret = vmstate_load_state(f, &vmstate_vfio_pci_config, vdev, 1); > + if (ret) { > + return ret; > + } > + > + if (msi_enabled(pdev)) { > + vfio_msi_enable(vdev); > + } else if (msix_enabled(pdev)) { > + vfio_msix_enable(vdev); > + } > + > + return ret; > +} > + > static VFIODeviceOps vfio_pci_ops = { > .vfio_compute_needs_reset = vfio_pci_compute_needs_reset, > .vfio_hot_reset_multi = vfio_pci_hot_reset_multi, > .vfio_eoi = vfio_intx_eoi, > .vfio_get_object = vfio_pci_get_object, > + .vfio_save_config = vfio_pci_save_config, > + .vfio_load_config = vfio_pci_load_config, > }; > > int vfio_populate_vga(VFIOPCIDevice *vdev, Error **errp) > diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h > index fe99c36a693a..ba6169cd926e 100644 > --- a/include/hw/vfio/vfio-common.h > +++ b/include/hw/vfio/vfio-common.h > @@ -120,6 +120,8 @@ struct VFIODeviceOps { > int (*vfio_hot_reset_multi)(VFIODevice *vdev); > void (*vfio_eoi)(VFIODevice *vdev); > Object *(*vfio_get_object)(VFIODevice *vdev); > + void (*vfio_save_config)(VFIODevice *vdev, QEMUFile *f); > + int (*vfio_load_config)(VFIODevice *vdev, QEMUFile *f); > }; > > typedef struct VFIOGroup {