From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1759435AbcLANgf (ORCPT ); Thu, 1 Dec 2016 08:36:35 -0500 Received: from cn.fujitsu.com ([59.151.112.132]:27551 "EHLO heian.cn.fujitsu.com" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1758059AbcLANgW (ORCPT ); Thu, 1 Dec 2016 08:36:22 -0500 X-IronPort-AV: E=Sophos;i="5.22,518,1449504000"; d="scan'208";a="13511495" Subject: Re: [PATCH] vfio/pci: Support error recovery To: Alex Williamson References: <1480246457-10368-1-git-send-email-caoj.fnst@cn.fujitsu.com> <20161130210413.5161aab1@t450s.home> CC: , , , From: Cao jin Message-ID: <58402830.3060606@cn.fujitsu.com> Date: Thu, 1 Dec 2016 21:40:00 +0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <20161130210413.5161aab1@t450s.home> Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: 7bit X-Originating-IP: [10.167.226.69] X-yoursite-MailScanner-ID: 33AA741B5F58.A86B4 X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: caoj.fnst@cn.fujitsu.com Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 12/01/2016 12:04 PM, Alex Williamson wrote: > On Sun, 27 Nov 2016 19:34:17 +0800 > Cao jin wrote: > >> It is user space driver's or device-specific driver's(in guest) responsbility >> to do a serious recovery when error happened. Link-reset is one part of >> recovery, when pci device is assigned to VM via vfio, link-reset will do >> twice in host & guest separately, which will cause many trouble for a >> successful recovery, so, disable the vfio-pci's link-reset in aer driver >> in host, this is a keypoint for guest to do error recovery successfully. >> >> CC: alex.williamson@redhat.com >> CC: mst@redhat.com >> Signed-off-by: Cao jin >> --- >> This is actually a RFC version(has debug lines left), and has minor changes in >> aer driver, so I think maybe it is better not to CC pci guys in this round. >> Later will do. >> >> drivers/pci/pcie/aer/aerdrv_core.c | 12 ++++++- >> drivers/vfio/pci/vfio_pci.c | 63 +++++++++++++++++++++++++++++++++++-- >> drivers/vfio/pci/vfio_pci_private.h | 2 ++ >> 3 files changed, 74 insertions(+), 3 deletions(-) >> >> diff --git a/drivers/pci/pcie/aer/aerdrv_core.c b/drivers/pci/pcie/aer/aerdrv_core.c >> index 521e39c..289fb8e 100644 >> --- a/drivers/pci/pcie/aer/aerdrv_core.c >> +++ b/drivers/pci/pcie/aer/aerdrv_core.c >> @@ -496,7 +496,17 @@ static void do_recovery(struct pci_dev *dev, int severity) >> "error_detected", >> report_error_detected); >> >> - if (severity == AER_FATAL) { >> + /* vfio-pci as a general meta driver, it actually couldn't do any real >> + * recovery for device. It is user space driver, or device-specific >> + * driver in guest who should take care of the serious error recovery, >> + * link reset actually is one part of whole recovery. Doing reset_link >> + * in aer driver of host kernel for vfio-pci devices will cause many >> + * trouble for user space driver or guest's device-specific driver, >> + * for example: the serious recovery often need to read register in >> + * config space, but if register reading happens during link-resetting, >> + * it is quite possible to return invalid value like all F's, which >> + * will result in unpredictable error. */ >> + if (severity == AER_FATAL && strcmp(dev->driver->name, "vfio-pci")) { >> result = reset_link(dev); >> if (result != PCI_ERS_RESULT_RECOVERED) >> goto failed; > > This is not acceptable. If we want to make a path through AER recovery > that does not reset the link, there should be a way for the driver to > request it. Testing the driver name is a horrible hack. The other > question is what guarantees does vfio make that the device does get > reset? I am not sure how vfio guarantee that...When device is assigned to VM, we have that guarantees(aer driver in guest driver will do that), so I think it is a well-behaved user space driver's responsibility to do link reset? And I think if there is a user space driver, it is surely its responsibility to consider how to do serious error recovery, like I said before, vfio, as a general meta driver, it surely don't know how each device does its specific recovery, except bus/slot reset > If an AER fault occurs and the user doesn't do a reset, what > happens when that device is released and a host driver tries to make > use of it? The user makes no commitment to do a reset and there are > only limited configurations where we even allow the user to perform a > reset. > Limited? Do you mean the things __pci_dev_reset() can do? ... > >> + aer_cap_offset = pci_find_ext_capability(vdev->pdev, PCI_EXT_CAP_ID_ERR); >> + ret = pci_read_config_dword(vdev->pdev, aer_cap_offset + >> + PCI_ERR_UNCOR_STATUS, &uncor_status); >> + if (ret) >> + return PCI_ERS_RESULT_DISCONNECT; >> + >> + pr_err("device %d got AER detect notification. uncorrectable error status = 0x%x\n", pdev->devfn, uncor_status);//to be removed >> mutex_lock(&vdev->igate); >> + >> + vdev->aer_recovering = true; >> + reinit_completion(&vdev->aer_error_completion); >> + >> + /* suspend config space access from user space, >> + * when vfio-pci's error recovery process is on */ >> + pci_cfg_access_trylock(vdev->pdev); >> >> - if (vdev->err_trigger) >> - eventfd_signal(vdev->err_trigger, 1); >> + if (vdev->err_trigger && uncor_status) { >> + pr_err("device %d signal uncor status to user space", pdev->devfn);//may be removed >> + /* signal uncorrectable error status to user space */ >> + eventfd_signal(vdev->err_trigger, uncor_status); >> + } > > } else... what? By bypassing the AER link reset we've assumed > responsibility for resetting the device. Even if we signal the user, > what guarantee do we have that the device is recovered? > else...consider it as a fake error notification and ignore? I am not sure I understand your thoughts totally, but it seems my previous comments apply, that is: it is well-behaved user space driver's responsibility to do a serious recovery. In my understanding, user space driver has 2 category: one is VM(has guest OS running inside), the other is ordinary user space program. When device is assigned to a VM, (qemu + guest OS) will do fully steps to do recovery(roughly is what struct pci_error_handlers has). So, equally, if it is a ordinary user space program acting as the driver, the responsibility belongs to it. Sincerely, Cao jin >> >> mutex_unlock(&vdev->igate); >> >> @@ -1199,8 +1232,34 @@ static pci_ers_result_t vfio_pci_aer_err_detected(struct pci_dev *pdev, >> return PCI_ERS_RESULT_CAN_RECOVER; >> } >> >> +static void vfio_pci_aer_resume(struct pci_dev *pdev) >> +{ >> + struct vfio_pci_device *vdev; >> + struct vfio_device *device; >> + >> + device = vfio_device_get_from_dev(&pdev->dev); >> + if (device == NULL) >> + return; >> + >> + vdev = vfio_device_data(device); >> + if (vdev == NULL) { >> + vfio_device_put(device); >> + return; >> + } >> + >> + /* vfio-pci's error recovery is done, time to >> + * resume pci config space's accesses */ >> + pci_cfg_access_unlock(vdev->pdev); >> + >> + vdev->aer_recovering = false; >> + complete_all(&vdev->aer_error_completion); >> + >> + vfio_device_put(device); >> +} >> + >> static const struct pci_error_handlers vfio_err_handlers = { >> .error_detected = vfio_pci_aer_err_detected, >> + .resume = vfio_pci_aer_resume, >> }; >> >> static struct pci_driver vfio_pci_driver = { >> diff --git a/drivers/vfio/pci/vfio_pci_private.h b/drivers/vfio/pci/vfio_pci_private.h >> index 8a7d546..ebf1041 100644 >> --- a/drivers/vfio/pci/vfio_pci_private.h >> +++ b/drivers/vfio/pci/vfio_pci_private.h >> @@ -83,6 +83,8 @@ struct vfio_pci_device { >> bool bardirty; >> bool has_vga; >> bool needs_reset; >> + bool aer_recovering; >> + struct completion aer_error_completion; >> struct pci_saved_state *pci_saved_state; >> int refcnt; >> struct eventfd_ctx *err_trigger; > > > > . >