From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Google-Smtp-Source: AB8JxZrSViLSfqrlNyYO6SVuKNd7l6xNPj7+G/9EdJM5PGe+WacCfveP46lflvRU6eFyhy3ne0DY ARC-Seal: i=1; a=rsa-sha256; t=1525935678; cv=none; d=google.com; s=arc-20160816; b=FeRSkIxaWQvChDDklpNim7X0Lndzez/B5tqjrGqC1Go5MmkRQjvm8ImeDphn6qi1Wn mTCCHwmhbKIHKcWhgW2BL06VHat8ZRvwHPr/Av6yqzCw1QugE7/9/cHQ+qHgBrjGbkce 7hNXChc5ecD7A5aQKXAQ+qpWuhWpfQPPMZfxaVRpcWpBlLXUWjEndtvz57XE1rfBG0aR RlSXb4IOqco7LfXtrdQVlAS6+4enZr45pJUBCW3pSCogS521ptvL+DmmUWUr+VT9Fk0C jTB7ayOXEraTh52sso73CTE/B1KNXIf8wRkPQ0WhT51aprF9TdK0mNmdb5wAKJhDjZGy SBlA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=user-agent:message-id:references:in-reply-to:subject:cc:to:from :date:content-transfer-encoding:mime-version:dkim-signature :dkim-signature:arc-authentication-results; bh=u2NilxRpuVkM9oz6b8+YVgNmWJo+SOps/fsB2jiEzYY=; b=t1YLbW3AXq1iCWUOXTCC0M5gEowaH3APWg7ong6qU6q9m9w2zMHVCoDwiYi/aQYySt FeWMHW3je4P/oAZPz4NRIKvgZpzmpeyZR9xPr83xPcNUXoEDhZhshfVs/R+cFdggdfX+ lgaAHCvbIIECENoPXtcy3/z8q+BRSOvw+v9+WA/siMlHX8aRiX3Ft1Sr+30zmxZJ/m+k lOxaXEqBiGBW2G6012uTmFTPGy5n3UbLUjRtkgsNEmAFmOF1clLgaOWpx0ch6EOwLLB+ K3c0JRYnR5esJBovdMYnRq+Z1bbHA/+YcBB25kHziKS58EGfON3iEIf0v83Z/l7a37AP 5spg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@codeaurora.org header.s=default header.b=Ra+l71gM; dkim=pass header.i=@codeaurora.org header.s=default header.b=hNKtxQdm; spf=pass (google.com: domain of poza@codeaurora.org designates 198.145.29.96 as permitted sender) smtp.mailfrom=poza@codeaurora.org Authentication-Results: mx.google.com; dkim=pass header.i=@codeaurora.org header.s=default header.b=Ra+l71gM; dkim=pass header.i=@codeaurora.org header.s=default header.b=hNKtxQdm; spf=pass (google.com: domain of poza@codeaurora.org designates 198.145.29.96 as permitted sender) smtp.mailfrom=poza@codeaurora.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Thu, 10 May 2018 12:31:16 +0530 From: poza@codeaurora.org To: Bjorn Helgaas Cc: Bjorn Helgaas , Philippe Ombredanne , Thomas Gleixner , Greg Kroah-Hartman , Kate Stewart , linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, Dongdong Liu , Keith Busch , Wei Zhang , Sinan Kaya , Timur Tabi , linux-pci-owner@vger.kernel.org Subject: Re: [PATCH v15 3/9] PCI/AER: Handle ERR_FATAL with removal and re-enumeration of devices In-Reply-To: <20180509232111.GA43216@bhelgaas-glaptop.roam.corp.google.com> References: <1525323838-1735-1-git-send-email-poza@codeaurora.org> <1525323838-1735-4-git-send-email-poza@codeaurora.org> <20180508235330.GN161390@bhelgaas-glaptop.roam.corp.google.com> <20180509130730.GA236548@bhelgaas-glaptop.roam.corp.google.com> <20180509232111.GA43216@bhelgaas-glaptop.roam.corp.google.com> Message-ID: User-Agent: Roundcube Webmail/1.2.5 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: =?utf-8?q?1599417977501212020?= X-GMAIL-MSGID: =?utf-8?q?1600059530471727761?= X-Mailing-List: linux-kernel@vger.kernel.org List-ID: On 2018-05-10 04:51, Bjorn Helgaas wrote: > On Wed, May 09, 2018 at 06:44:53PM +0530, poza@codeaurora.org wrote: >> On 2018-05-09 18:37, Bjorn Helgaas wrote: >> > On Tue, May 08, 2018 at 06:53:30PM -0500, Bjorn Helgaas wrote: >> > > On Thu, May 03, 2018 at 01:03:52AM -0400, Oza Pawandeep wrote: >> > > > This patch alters the behavior of handling of ERR_FATAL, where removal >> > > > of devices is initiated, followed by reset link, followed by >> > > > re-enumeration. >> > > > >> > > > So the errors are handled in a different way as follows: >> > > > ERR_NONFATAL => call driver recovery entry points >> > > > ERR_FATAL => remove and re-enumerate >> > > > >> > > > please refer to Documentation/PCI/pci-error-recovery.txt for more details. >> > > > >> > > > Signed-off-by: Oza Pawandeep >> > > > >> > > > diff --git a/drivers/pci/pcie/aer/aerdrv.c b/drivers/pci/pcie/aer/aerdrv.c >> > > > index 779b387..206f590 100644 >> > > > --- a/drivers/pci/pcie/aer/aerdrv.c >> > > > +++ b/drivers/pci/pcie/aer/aerdrv.c >> > > > @@ -330,6 +330,13 @@ static pci_ers_result_t aer_root_reset(struct pci_dev *dev) >> > > > reg32 |= ROOT_PORT_INTR_ON_MESG_MASK; >> > > > pci_write_config_dword(dev, pos + PCI_ERR_ROOT_COMMAND, reg32); >> > > > >> > > > + /* >> > > > + * This function is called only on ERR_FATAL now, and since >> > > > + * the pci_report_resume is called only in ERR_NONFATAL case, >> > > > + * the clearing part has to be taken care here. >> > > > + */ >> > > > + aer_error_resume(dev); >> > > >> > > I don't understand this part. Previously the ERR_FATAL path looked >> > > like >> > > this: >> > > >> > > do_recovery >> > > reset_link >> > > driver->reset_link >> > > aer_root_reset >> > > pci_reset_bridge_secondary_bus # <-- reset >> > > broadcast_error_message(..., report_resume) >> > > pci_walk_bus(..., report_resume, ...) >> > > report_resume >> > > if (cb == report_resume) >> > > pci_cleanup_aer_uncorrect_error_status >> > > pci_write_config_dword(PCI_ERR_UNCOR_STATUS) # <-- clear >> > > status >> > > >> > > After this patch, it will look like this: >> > > >> > > do_recovery >> > > do_fatal_recovery >> > > pci_cleanup_aer_uncorrect_error_status >> > > pci_write_config_dword(PCI_ERR_UNCOR_STATUS) # <-- clear >> > > status >> > > reset_link >> > > driver->reset_link >> > > aer_root_reset >> > > pci_reset_bridge_secondary_bus # <-- reset >> > > aer_error_resume >> > > pcie_capability_write_word(PCI_EXP_DEVSTA) # >> > > <-- clear more >> > > pci_write_config_dword(PCI_ERR_UNCOR_STATUS) # >> > > <-- clear status >> > > >> > > So if I'm understanding correctly, the new path clears the status too >> > > early, then clears it again (plus clearing DEVSTA, which we didn't do >> > > before) later. >> > > >> > > I would think we would want to leave aer_root_reset() alone, and >> > > just move >> > > the pci_cleanup_aer_uncorrect_error_status() in do_fatal_recovery() >> > > down so >> > > it happens after we call reset_link(). That way the reset/clear >> > > sequence >> > > would be the same as it was before. >> > >> > I've been fiddling with this a bit myself and will post the results to >> > see >> > what you think. >> >> >> ok so you are suggesting to move >> pci_cleanup_aer_uncorrect_error_status down >> which I can do. >> >> And not to call aer_error_resume, because you think its clearing the >> status >> again. >> >> following code: calls aer_error_resume. >> pci_broadcast_error_message() >> /* >> * If the error is reported by an end point, we think >> this >> * error is related to the upstream link of the end >> point. >> */ >> if (state == pci_channel_io_normal) >> /* >> * the error is non fatal so the bus is ok, >> just >> invoke >> * the callback for the function that logged >> the >> error. >> */ >> cb(dev, &result_data); >> else >> pci_walk_bus(dev->bus, cb, &result_data); > > Holy crap, I thought this could not possibly get any more complicated, > but you're right; we do actually call aer_error_resume() today via an > extremely convoluted path: > > do_recovery(pci_dev) > broadcast_error_message(..., error_detected, ...) > if (AER_FATAL) > reset_link(pci_dev) > udev = BRIDGE ? pci_dev : pci_dev->bus->self > driver->reset_link(udev) > aer_root_reset(udev) > if (CAN_RECOVER) > broadcast_error_message(..., mmio_enabled, ...) > if (NEED_RESET) > broadcast_error_message(..., slot_reset, ...) > broadcast_error_message(dev, ..., report_resume, ...) > if (BRIDGE) > report_resume > driver->resume > pcie_portdrv_err_resume > device_for_each_child(..., resume_iter) > resume_iter > driver->error_resume > aer_error_resume > pci_cleanup_aer_uncorrect_error_status(pci_dev) # only if > BRIDGE > pci_write_config_dword(PCI_ERR_UNCOR_STATUS) > > aerdriver is the only port service driver that implements > .error_resume(), and aerdriver only binds to root ports. I can't > really believe all these device_for_each_child()/resume_iter() > gyrations are necessary when this is AER code calling AER code. > > Bjorn here is the code of do_fatal_recovery, where I have moved the things down and doing only if it is bridge. let me know how this looks to you, so then I can post v16. static pci_ers_result_t do_fatal_recovery(struct pci_dev *dev, int severity) { struct pci_dev *udev; struct pci_bus *parent; struct pci_dev *pdev, *temp; struct aer_broadcast_data result_data; pci_ers_result_t result = PCI_ERS_RESULT_RECOVERED; if (dev->hdr_type == PCI_HEADER_TYPE_BRIDGE) udev = dev; else udev = dev->bus->self; parent = udev->subordinate; pci_lock_rescan_remove(); list_for_each_entry_safe_reverse(pdev, temp, &parent->devices, bus_list) { pci_dev_get(pdev); pci_dev_set_disconnected(pdev, NULL); if (pci_has_subordinate(pdev)) pci_walk_bus(pdev->subordinate, pci_dev_set_disconnected, NULL); pci_stop_and_remove_bus_device(pdev); pci_dev_put(pdev); } result = reset_link(udev, severity); if (severity == AER_FATAL && dev->hdr_type == PCI_HEADER_TYPE_BRIDGE) { pci_walk_bus(dev->subordinate, report_resume, &result_data); pci_cleanup_aer_uncorrect_error_status(dev); dev->error_state = pci_channel_io_normal; } if (result == PCI_ERS_RESULT_RECOVERED) if (pcie_wait_for_link(udev, true)) pci_rescan_bus(udev->bus); pci_unlock_rescan_remove(); return result; } Regards, Oza.