Re: [PATCH] PCI/ERR: Resolve regression in pcie_do_recovery

[Jay Vosburgh <jay.vosburgh@canonical.com>]

"Kuppuswamy, Sathyanarayanan" wrote:

>Hi Jay,
>
>On 4/29/20 6:15 PM, Kuppuswamy, Sathyanarayanan wrote:
>>
>>
>> On 4/29/20 5:42 PM, Jay Vosburgh wrote:
>>>     Commit 6d2c89441571 ("PCI/ERR: Update error status after
>>> reset_link()"), introduced a regression, as pcie_do_recovery will
>>> discard the status result from report_frozen_detected.  This can cause a
>>> failure to recover if _NEED_RESET is returned by report_frozen_detected
>>> and report_slot_reset is not invoked.
>>>
>>>     Such an event can be induced for testing purposes by reducing
>>> the Max_Payload_Size of a PCIe bridge to less than that of a device
>>> downstream from the bridge, and then initating I/O through the device,
>>> resulting in oversize transactions.  In the presence of DPC, this
>>> results in a containment event and attempted reset and recovery via
>>> pcie_do_recovery.  After 6d2c89441571 report_slot_reset is not invoked,
>>> and the device does not recover.
>>
>> I think this issue is related to the issue discussed in following
>> thread (DPC non-hotplug support).
>>
>> https://lkml.org/lkml/2020/3/28/328
>>
>> If my assumption is correct, you are dealing with devices which are
>> not hotplug capable. If the devices are hotplug capable then you don't
>> need to proceed to report_slot_reset(), since hotplug handler will
>> remove/re-enumerate the devices correctly.

	Correct, this particular device (a network card) is in a
non-hotplug slot.

>Can you check whether following fix works for you?

	Yes, it does.

	I fixed up the whitespace and made a minor change to add braces
in what look like the correct places around the "if (reset_link)" block;
the patch I tested with is below.  I'll also install this on another
machine with hotplug capable slots to test there as well.

diff --git a/drivers/pci/pcie/err.c b/drivers/pci/pcie/err.c
index 14bb8f54723e..db80e1ecb2dc 100644
--- a/drivers/pci/pcie/err.c
+++ b/drivers/pci/pcie/err.c
@@ -165,13 +165,24 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,
 	pci_dbg(dev, "broadcast error_detected message\n");
 	if (state == pci_channel_io_frozen) {
 		pci_walk_bus(bus, report_frozen_detected, &status);
-		status = reset_link(dev);
-		if (status != PCI_ERS_RESULT_RECOVERED) {
+		status = PCI_ERS_RESULT_NEED_RESET;
+	} else {
+		pci_walk_bus(bus, report_normal_detected, &status);
+	}
+
+	if (status == PCI_ERS_RESULT_NEED_RESET) {
+		if (reset_link) {
+			if (reset_link(dev) != PCI_ERS_RESULT_RECOVERED)
+				status = PCI_ERS_RESULT_DISCONNECT;
+		} else {
+			if (pci_bus_error_reset(dev))
+				status = PCI_ERS_RESULT_DISCONNECT;
+		}
+
+		if (status == PCI_ERS_RESULT_DISCONNECT) {
 			pci_warn(dev, "link reset failed\n");
 			goto failed;
 		}
-	} else {
-		pci_walk_bus(bus, report_normal_detected, &status);
 	}
 
 	if (status == PCI_ERS_RESULT_CAN_RECOVER) {


	-J

>This includes support for bus_reset in recovery function itself.
>
>index 14bb8f54723e..c9eaab68ab7a 100644
>--- a/drivers/pci/pcie/err.c
>+++ b/drivers/pci/pcie/err.c
>@@ -165,13 +165,23 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,
>        pci_dbg(dev, "broadcast error_detected message\n");
>        if (state == pci_channel_io_frozen) {
>        if (state == pci_channel_io_frozen) {
>                pci_walk_bus(bus, report_frozen_detected, &status);
>-               status = reset_link(dev);
>-               if (status != PCI_ERS_RESULT_RECOVERED) {
>+               status = PCI_ERS_RESULT_NEED_RESET;
>+       } else {
>+               pci_walk_bus(bus, report_normal_detected, &status);
>+       }
>+
>+       if (status == PCI_ERS_RESULT_NEED_RESET) {
>+               if (reset_link)
>+                       if (reset_link(dev) != PCI_ERS_RESULT_RECOVERED)
>+                               status = PCI_ERS_RESULT_DISCONNECT;
>+               else
>+                       if (pci_bus_error_reset(dev))
>+                               status = PCI_ERS_RESULT_DISCONNECT;
>+
>+               if (status == PCI_ERS_RESULT_DISCONNECT) {
>                        pci_warn(dev, "link reset failed\n");
>                        goto failed;
>                }
>-       } else {
>-               pci_walk_bus(bus, report_normal_detected, &status);
>        }
>
>        if (status == PCI_ERS_RESULT_CAN_RECOVER) {
>
>
>>
>>>
>>>     Inspection shows a similar path is plausible for a return of
>>> _CAN_RECOVER and the invocation of report_mmio_enabled.
>>>
>>>     Resolve this by preserving the result of report_frozen_detected if
>>> reset_link does not return _DISCONNECT.
>>>
>>> Fixes: 6d2c89441571 ("PCI/ERR: Update error status after reset_link()")
>>> Signed-off-by: Jay Vosburgh <jay.vosburgh@canonical.com>
>>>
>>> ---
>>>   drivers/pci/pcie/err.c | 11 +++++++++--
>>>   1 file changed, 9 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/drivers/pci/pcie/err.c b/drivers/pci/pcie/err.c
>>> index 14bb8f54723e..e4274562f3a0 100644
>>> --- a/drivers/pci/pcie/err.c
>>> +++ b/drivers/pci/pcie/err.c
>>> @@ -164,10 +164,17 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev
>>> *dev,
>>>       pci_dbg(dev, "broadcast error_detected message\n");
>>>       if (state == pci_channel_io_frozen) {
>>> +        pci_ers_result_t status2;
>>> +
>>>           pci_walk_bus(bus, report_frozen_detected, &status);
>>> -        status = reset_link(dev);
>>> -        if (status != PCI_ERS_RESULT_RECOVERED) {
>>> +        /* preserve status from report_frozen_detected to
>>> +         * insure report_mmio_enabled or report_slot_reset are
>>> +         * invoked even if reset_link returns _RECOVERED.
>>> +         */
>>> +        status2 = reset_link(dev);
>>> +        if (status2 != PCI_ERS_RESULT_RECOVERED) {
>>>               pci_warn(dev, "link reset failed\n");
>>> +            status = status2;
>>>               goto failed;
>>>           }
>>>       } else {
>>>

---
	-Jay Vosburgh, jay.vosburgh@canonical.com