* Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed [not found] <20210702045120.22855-1-aaron.ma@canonical.com> @ 2021-07-04 14:28 ` Pali Rohár 2021-07-05 7:23 ` Aaron Ma 2021-07-05 23:02 ` Krzysztof Wilczyński 2021-07-06 20:12 ` Bjorn Helgaas 1 sibling, 2 replies; 14+ messages in thread From: Pali Rohár @ 2021-07-04 14:28 UTC (permalink / raw) To: Aaron Ma, Bjorn Helgaas, Krzysztof Wilczyński Cc: jesse.brandeburg, anthony.l.nguyen, davem, kuba, intel-wired-lan, netdev, linux-kernel, linux-pci, Marek Behún + Bjorn, Krzysztof and linux-pci On Friday 02 July 2021 12:51:19 Aaron Ma wrote: > Check PCI state when rd/wr iomem. > Implement wr32 function as rd32 too. > > When unplug TBT dock with i225, rd/wr PCI iomem will cause error log: > Trace: > BUG: unable to handle page fault for address: 000000000000b604 > Oops: 0000 [#1] SMP NOPTI > RIP: 0010:igc_rd32+0x1c/0x90 [igc] > Call Trace: > igc_ptp_suspend+0x6c/0xa0 [igc] > igc_ptp_stop+0x12/0x50 [igc] > igc_remove+0x7f/0x1c0 [igc] > pci_device_remove+0x3e/0xb0 > __device_release_driver+0x181/0x240 > > Signed-off-by: Aaron Ma <aaron.ma@canonical.com> > --- > drivers/net/ethernet/intel/igc/igc_main.c | 16 ++++++++++++++++ > drivers/net/ethernet/intel/igc/igc_regs.h | 7 ++----- > 2 files changed, 18 insertions(+), 5 deletions(-) > > diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c > index f1adf154ec4a..606b72cb6193 100644 > --- a/drivers/net/ethernet/intel/igc/igc_main.c > +++ b/drivers/net/ethernet/intel/igc/igc_main.c > @@ -5292,6 +5292,10 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > u32 value = 0; > > + if (igc->pdev && > + igc->pdev->error_state == pci_channel_io_perm_failure) Hello! This code pattern and commit message looks like that we could use pci_dev_is_disconnected() helper function for checking if device is still connected or was disconnected. Apparently pci_dev_is_disconnected() is defined only in private header file drivers/pci/pci.h and not in public include/linux/pci.h. Aaron: can you check if pci_dev_is_disconnected() is really something which should be used and it helps you? Bjorn, Krzysztof: what do you think about lifting helper function pci_dev_is_disconnected() to be available to all drivers and not only in PCI subsystem? I think that such helper function makes driver code more readable and can be useful also for other drivers which are checking if return value is all F's. > + return 0; Aaron: should not you return all F's on error? Because few lines below in this function is returned value with all F's when PCIe link lost. > + > value = readl(&hw_addr[reg]); Anyway, this code looks to be racy. When pci_channel_io_perm_failure is set (e.g. by hotplug interrupt) after checking for pdev->error_state and prior executing above readl() then mentioned fatal error still occurs. > > /* reads should not return all F's */ > @@ -5308,6 +5312,18 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > return value; > } > > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val) > +{ > + struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw); > + u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > + > + if (igc->pdev && > + igc->pdev->error_state == pci_channel_io_perm_failure) > + return; > + > + writel((val), &hw_addr[(reg)]); > +} > + > int igc_set_spd_dplx(struct igc_adapter *adapter, u32 spd, u8 dplx) > { > struct igc_mac_info *mac = &adapter->hw.mac; > diff --git a/drivers/net/ethernet/intel/igc/igc_regs.h b/drivers/net/ethernet/intel/igc/igc_regs.h > index cc174853554b..eb4be87d0e8b 100644 > --- a/drivers/net/ethernet/intel/igc/igc_regs.h > +++ b/drivers/net/ethernet/intel/igc/igc_regs.h > @@ -260,13 +260,10 @@ struct igc_hw; > u32 igc_rd32(struct igc_hw *hw, u32 reg); > > /* write operations, indexed using DWORDS */ > -#define wr32(reg, val) \ > -do { \ > - u8 __iomem *hw_addr = READ_ONCE((hw)->hw_addr); \ > - writel((val), &hw_addr[(reg)]); \ > -} while (0) > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val); > > #define rd32(reg) (igc_rd32(hw, reg)) > +#define wr32(reg, val) (igc_wr32(hw, reg, val)) > > #define wrfl() ((void)rd32(IGC_STATUS)) > > -- > 2.30.2 > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed 2021-07-04 14:28 ` [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed Pali Rohár @ 2021-07-05 7:23 ` Aaron Ma 2021-07-05 23:02 ` Krzysztof Wilczyński 1 sibling, 0 replies; 14+ messages in thread From: Aaron Ma @ 2021-07-05 7:23 UTC (permalink / raw) To: Pali Rohár, Bjorn Helgaas, Krzysztof Wilczyński Cc: jesse.brandeburg, anthony.l.nguyen, davem, kuba, intel-wired-lan, netdev, linux-kernel, linux-pci, Marek Behún On 7/4/21 10:28 PM, Pali Rohár wrote: > + Bjorn, Krzysztof and linux-pci > > On Friday 02 July 2021 12:51:19 Aaron Ma wrote: >> Check PCI state when rd/wr iomem. >> Implement wr32 function as rd32 too. >> >> When unplug TBT dock with i225, rd/wr PCI iomem will cause error log: >> Trace: >> BUG: unable to handle page fault for address: 000000000000b604 >> Oops: 0000 [#1] SMP NOPTI >> RIP: 0010:igc_rd32+0x1c/0x90 [igc] >> Call Trace: >> igc_ptp_suspend+0x6c/0xa0 [igc] >> igc_ptp_stop+0x12/0x50 [igc] >> igc_remove+0x7f/0x1c0 [igc] >> pci_device_remove+0x3e/0xb0 >> __device_release_driver+0x181/0x240 >> >> Signed-off-by: Aaron Ma <aaron.ma@canonical.com> >> --- >> drivers/net/ethernet/intel/igc/igc_main.c | 16 ++++++++++++++++ >> drivers/net/ethernet/intel/igc/igc_regs.h | 7 ++----- >> 2 files changed, 18 insertions(+), 5 deletions(-) >> >> diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c >> index f1adf154ec4a..606b72cb6193 100644 >> --- a/drivers/net/ethernet/intel/igc/igc_main.c >> +++ b/drivers/net/ethernet/intel/igc/igc_main.c >> @@ -5292,6 +5292,10 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) >> u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); >> u32 value = 0; >> >> + if (igc->pdev && >> + igc->pdev->error_state == pci_channel_io_perm_failure) > > Hello! This code pattern and commit message looks like that we could use > pci_dev_is_disconnected() helper function for checking if device is > still connected or was disconnected. > > Apparently pci_dev_is_disconnected() is defined only in private header > file drivers/pci/pci.h and not in public include/linux/pci.h. > > Aaron: can you check if pci_dev_is_disconnected() is really something > which should be used and it helps you? > Hi Pali, How about using pci_channel_offline instead? It's ready and also safe for frozen state, and verified on hw. > Bjorn, Krzysztof: what do you think about lifting helper function > pci_dev_is_disconnected() to be available to all drivers and not only in > PCI subsystem? > > I think that such helper function makes driver code more readable and > can be useful also for other drivers which are checking if return value > is all F's. > >> + return 0; > > Aaron: should not you return all F's on error? Because few lines below > in this function is returned value with all F's when PCIe link lost. > If you agree with the above change, I can fix it to "return -1" in v2. Thanks for your comments, Aaron >> + >> value = readl(&hw_addr[reg]); > > Anyway, this code looks to be racy. When pci_channel_io_perm_failure is > set (e.g. by hotplug interrupt) after checking for pdev->error_state and > prior executing above readl() then mentioned fatal error still occurs. > >> >> /* reads should not return all F's */ >> @@ -5308,6 +5312,18 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) >> return value; >> } >> >> +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val) >> +{ >> + struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw); >> + u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); >> + >> + if (igc->pdev && >> + igc->pdev->error_state == pci_channel_io_perm_failure) >> + return; >> + >> + writel((val), &hw_addr[(reg)]); >> +} >> + >> int igc_set_spd_dplx(struct igc_adapter *adapter, u32 spd, u8 dplx) >> { >> struct igc_mac_info *mac = &adapter->hw.mac; >> diff --git a/drivers/net/ethernet/intel/igc/igc_regs.h b/drivers/net/ethernet/intel/igc/igc_regs.h >> index cc174853554b..eb4be87d0e8b 100644 >> --- a/drivers/net/ethernet/intel/igc/igc_regs.h >> +++ b/drivers/net/ethernet/intel/igc/igc_regs.h >> @@ -260,13 +260,10 @@ struct igc_hw; >> u32 igc_rd32(struct igc_hw *hw, u32 reg); >> >> /* write operations, indexed using DWORDS */ >> -#define wr32(reg, val) \ >> -do { \ >> - u8 __iomem *hw_addr = READ_ONCE((hw)->hw_addr); \ >> - writel((val), &hw_addr[(reg)]); \ >> -} while (0) >> +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val); >> >> #define rd32(reg) (igc_rd32(hw, reg)) >> +#define wr32(reg, val) (igc_wr32(hw, reg, val)) >> >> #define wrfl() ((void)rd32(IGC_STATUS)) >> >> -- >> 2.30.2 >> ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed 2021-07-04 14:28 ` [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed Pali Rohár 2021-07-05 7:23 ` Aaron Ma @ 2021-07-05 23:02 ` Krzysztof Wilczyński 2021-07-06 14:23 ` Pali Rohár 1 sibling, 1 reply; 14+ messages in thread From: Krzysztof Wilczyński @ 2021-07-05 23:02 UTC (permalink / raw) To: Pali Rohár Cc: Aaron Ma, Bjorn Helgaas, jesse.brandeburg, anthony.l.nguyen, davem, kuba, intel-wired-lan, netdev, linux-kernel, linux-pci, Marek Behún Hi Pali, [...] > Aaron: can you check if pci_dev_is_disconnected() is really something > which should be used and it helps you? While having a closer look, I've noticed that quite a few of the network drivers handle this somewhat, as I see that a lot of them have some sort of I/O error handles set where a check for "pci_channel_io_perm_failure" seem to be having place. This is also true for this driver looking at the igc_io_error_detected(). Is this not working for the igc driver? Or is this for something completely different? Having said all that, I am not an expert in network drivers, so pardon me if I am asking about something completely different, and I apologise if that is the case. > Bjorn, Krzysztof: what do you think about lifting helper function > pci_dev_is_disconnected() to be available to all drivers and not only in > PCI subsystem? No objections from me, if we believe it's useful and that it might encourage people to use a common API. Currently, I can see at least five potential users of this helper. Krzysztof ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed 2021-07-05 23:02 ` Krzysztof Wilczyński @ 2021-07-06 14:23 ` Pali Rohár 0 siblings, 0 replies; 14+ messages in thread From: Pali Rohár @ 2021-07-06 14:23 UTC (permalink / raw) To: Krzysztof Wilczyński Cc: Aaron Ma, Bjorn Helgaas, jesse.brandeburg, anthony.l.nguyen, davem, kuba, intel-wired-lan, netdev, linux-kernel, linux-pci, Marek Behún On Tuesday 06 July 2021 01:02:12 Krzysztof Wilczyński wrote: > Hi Pali, > > [...] > > Aaron: can you check if pci_dev_is_disconnected() is really something > > which should be used and it helps you? > > While having a closer look, I've noticed that quite a few of the network > drivers handle this somewhat, as I see that a lot of them have some sort > of I/O error handles set where a check for "pci_channel_io_perm_failure" > seem to be having place. This is also true for this driver looking at > the igc_io_error_detected(). > > Is this not working for the igc driver? Or is this for something > completely different? I guess that this callback is called when Bridge receive some kind of fatal error. Non-AER-aware bridges probably do not have to inform system that error happened and kernel would not call this callback. So I guess it depends on to which "machine" you need this network adapter. So in my opinion this callback is there for PCI subsystem to inform driver that error happened and let driver to do any hw specific recovery if it is possible. But I think problem described here can be slightly different. It is needed to check if device is still alive or was disconnected. > Having said all that, I am not an expert in network drivers, so pardon > me if I am asking about something completely different, and I apologise > if that is the case. > > > Bjorn, Krzysztof: what do you think about lifting helper function > > pci_dev_is_disconnected() to be available to all drivers and not only in > > PCI subsystem? > > No objections from me, if we believe it's useful and that it might > encourage people to use a common API. Currently, I can see at least > five potential users of this helper. > > Krzysztof ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed [not found] <20210702045120.22855-1-aaron.ma@canonical.com> 2021-07-04 14:28 ` [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed Pali Rohár @ 2021-07-06 20:12 ` Bjorn Helgaas 2021-07-07 21:53 ` Pali Rohár 1 sibling, 1 reply; 14+ messages in thread From: Bjorn Helgaas @ 2021-07-06 20:12 UTC (permalink / raw) To: Aaron Ma Cc: jesse.brandeburg, anthony.l.nguyen, davem, kuba, intel-wired-lan, netdev, linux-kernel, Pali Rohár, Krzysztof Wilczyński, linux-pci On Fri, Jul 02, 2021 at 12:51:19PM +0800, Aaron Ma wrote: > Check PCI state when rd/wr iomem. > Implement wr32 function as rd32 too. > > When unplug TBT dock with i225, rd/wr PCI iomem will cause error log: > Trace: > BUG: unable to handle page fault for address: 000000000000b604 > Oops: 0000 [#1] SMP NOPTI > RIP: 0010:igc_rd32+0x1c/0x90 [igc] > Call Trace: > igc_ptp_suspend+0x6c/0xa0 [igc] > igc_ptp_stop+0x12/0x50 [igc] > igc_remove+0x7f/0x1c0 [igc] > pci_device_remove+0x3e/0xb0 > __device_release_driver+0x181/0x240 > > Signed-off-by: Aaron Ma <aaron.ma@canonical.com> > --- > drivers/net/ethernet/intel/igc/igc_main.c | 16 ++++++++++++++++ > drivers/net/ethernet/intel/igc/igc_regs.h | 7 ++----- > 2 files changed, 18 insertions(+), 5 deletions(-) > > diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c > index f1adf154ec4a..606b72cb6193 100644 > --- a/drivers/net/ethernet/intel/igc/igc_main.c > +++ b/drivers/net/ethernet/intel/igc/igc_main.c > @@ -5292,6 +5292,10 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > u32 value = 0; > > + if (igc->pdev && > + igc->pdev->error_state == pci_channel_io_perm_failure) > + return 0; I don't think this solves the problem. - Driver calls igc_rd32(). - "if (pci_channel_io_perm_failure)" evaluates to false (error_state does not indicate an error). - Device is unplugged. - igc_rd32() calls readl(), which performs MMIO read, which fails because the device is no longer present. readl() returns ~0 on most platforms. - Same page fault occurs. The only way is to check *after* the MMIO read to see whether an error occurred. On most platforms that means checking for ~0 data. If you see that, a PCI error *may* have occurred. If you know that ~0 can never be valid, e.g., if you're reading a register where ~0 is not a valid value, you know for sure that an error has occurred. If ~0 might be a valid value, e.g., if you're reading a buffer that contains arbitrary data, you have to look harder. You might read a register than cannot contain ~0, and see if you get the data you expect. Or you might read the Vendor ID or something from config space. > value = readl(&hw_addr[reg]); > > /* reads should not return all F's */ > @@ -5308,6 +5312,18 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > return value; > } > > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val) > +{ > + struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw); > + u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > + > + if (igc->pdev && > + igc->pdev->error_state == pci_channel_io_perm_failure) > + return; > + > + writel((val), &hw_addr[(reg)]); > +} > + > int igc_set_spd_dplx(struct igc_adapter *adapter, u32 spd, u8 dplx) > { > struct igc_mac_info *mac = &adapter->hw.mac; > diff --git a/drivers/net/ethernet/intel/igc/igc_regs.h b/drivers/net/ethernet/intel/igc/igc_regs.h > index cc174853554b..eb4be87d0e8b 100644 > --- a/drivers/net/ethernet/intel/igc/igc_regs.h > +++ b/drivers/net/ethernet/intel/igc/igc_regs.h > @@ -260,13 +260,10 @@ struct igc_hw; > u32 igc_rd32(struct igc_hw *hw, u32 reg); > > /* write operations, indexed using DWORDS */ > -#define wr32(reg, val) \ > -do { \ > - u8 __iomem *hw_addr = READ_ONCE((hw)->hw_addr); \ > - writel((val), &hw_addr[(reg)]); \ > -} while (0) > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val); > > #define rd32(reg) (igc_rd32(hw, reg)) > +#define wr32(reg, val) (igc_wr32(hw, reg, val)) > > #define wrfl() ((void)rd32(IGC_STATUS)) > > -- > 2.30.2 > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed 2021-07-06 20:12 ` Bjorn Helgaas @ 2021-07-07 21:53 ` Pali Rohár 2021-07-07 22:10 ` Bjorn Helgaas 0 siblings, 1 reply; 14+ messages in thread From: Pali Rohár @ 2021-07-07 21:53 UTC (permalink / raw) To: Bjorn Helgaas Cc: Aaron Ma, jesse.brandeburg, anthony.l.nguyen, davem, kuba, intel-wired-lan, netdev, linux-kernel, Krzysztof Wilczyński, linux-pci On Tuesday 06 July 2021 15:12:41 Bjorn Helgaas wrote: > On Fri, Jul 02, 2021 at 12:51:19PM +0800, Aaron Ma wrote: > > Check PCI state when rd/wr iomem. > > Implement wr32 function as rd32 too. > > > > When unplug TBT dock with i225, rd/wr PCI iomem will cause error log: > > Trace: > > BUG: unable to handle page fault for address: 000000000000b604 > > Oops: 0000 [#1] SMP NOPTI > > RIP: 0010:igc_rd32+0x1c/0x90 [igc] > > Call Trace: > > igc_ptp_suspend+0x6c/0xa0 [igc] > > igc_ptp_stop+0x12/0x50 [igc] > > igc_remove+0x7f/0x1c0 [igc] > > pci_device_remove+0x3e/0xb0 > > __device_release_driver+0x181/0x240 > > > > Signed-off-by: Aaron Ma <aaron.ma@canonical.com> > > --- > > drivers/net/ethernet/intel/igc/igc_main.c | 16 ++++++++++++++++ > > drivers/net/ethernet/intel/igc/igc_regs.h | 7 ++----- > > 2 files changed, 18 insertions(+), 5 deletions(-) > > > > diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c > > index f1adf154ec4a..606b72cb6193 100644 > > --- a/drivers/net/ethernet/intel/igc/igc_main.c > > +++ b/drivers/net/ethernet/intel/igc/igc_main.c > > @@ -5292,6 +5292,10 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > > u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > > u32 value = 0; > > > > + if (igc->pdev && > > + igc->pdev->error_state == pci_channel_io_perm_failure) > > + return 0; > > I don't think this solves the problem. > > - Driver calls igc_rd32(). > > - "if (pci_channel_io_perm_failure)" evaluates to false (error_state > does not indicate an error). > > - Device is unplugged. > > - igc_rd32() calls readl(), which performs MMIO read, which fails > because the device is no longer present. readl() returns ~0 on > most platforms. > > - Same page fault occurs. Hi Bjorn! I think that backtrace show that this error happens when PCIe hotplug get interrupt that device was unplugged and PCIe hotplug code calls remove/unbind procedure to stop unplugged driver. And in this case really does not make sense to try issuing MMIO read, device is already unplugged. I looked that PCIe hotplug driver calls pci_dev_set_disconnected() when this unplug interrupt happens and pci_dev_set_disconnected() just sets pci_channel_io_perm_failure flag. drivers/pci/pci.h provides function pci_dev_is_disconnected() which checks if that flag pci_channel_io_perm_failure is set. So I think that pci_dev_is_disconnected() is useful and could be exported also to drivers (like this one) so they can check if pci_dev_set_disconnected() was called in past and PCI driver is now in unbind/cleanup/remove state because PCIe device is already disconnected and not accessible anymore. But maybe this check should be on other place in driver unbound procedure and not in general MMIO read function? > The only way is to check *after* the MMIO read to see whether an error > occurred. On most platforms that means checking for ~0 data. If you > see that, a PCI error *may* have occurred. > > If you know that ~0 can never be valid, e.g., if you're reading a > register where ~0 is not a valid value, you know for sure that an > error has occurred. > > If ~0 might be a valid value, e.g., if you're reading a buffer that > contains arbitrary data, you have to look harder. You might read a > register than cannot contain ~0, and see if you get the data you > expect. Or you might read the Vendor ID or something from config > space. > > > value = readl(&hw_addr[reg]); > > > > /* reads should not return all F's */ > > @@ -5308,6 +5312,18 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > > return value; > > } > > > > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val) > > +{ > > + struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw); > > + u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > > + > > + if (igc->pdev && > > + igc->pdev->error_state == pci_channel_io_perm_failure) > > + return; > > + > > + writel((val), &hw_addr[(reg)]); > > +} > > + > > int igc_set_spd_dplx(struct igc_adapter *adapter, u32 spd, u8 dplx) > > { > > struct igc_mac_info *mac = &adapter->hw.mac; > > diff --git a/drivers/net/ethernet/intel/igc/igc_regs.h b/drivers/net/ethernet/intel/igc/igc_regs.h > > index cc174853554b..eb4be87d0e8b 100644 > > --- a/drivers/net/ethernet/intel/igc/igc_regs.h > > +++ b/drivers/net/ethernet/intel/igc/igc_regs.h > > @@ -260,13 +260,10 @@ struct igc_hw; > > u32 igc_rd32(struct igc_hw *hw, u32 reg); > > > > /* write operations, indexed using DWORDS */ > > -#define wr32(reg, val) \ > > -do { \ > > - u8 __iomem *hw_addr = READ_ONCE((hw)->hw_addr); \ > > - writel((val), &hw_addr[(reg)]); \ > > -} while (0) > > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val); > > > > #define rd32(reg) (igc_rd32(hw, reg)) > > +#define wr32(reg, val) (igc_wr32(hw, reg, val)) > > > > #define wrfl() ((void)rd32(IGC_STATUS)) > > > > -- > > 2.30.2 > > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed 2021-07-07 21:53 ` Pali Rohár @ 2021-07-07 22:10 ` Bjorn Helgaas 2021-07-08 2:04 ` Oliver O'Halloran 0 siblings, 1 reply; 14+ messages in thread From: Bjorn Helgaas @ 2021-07-07 22:10 UTC (permalink / raw) To: Pali Rohár Cc: Aaron Ma, jesse.brandeburg, anthony.l.nguyen, davem, kuba, intel-wired-lan, netdev, linux-kernel, Krzysztof Wilczyński, linux-pci On Wed, Jul 07, 2021 at 11:53:37PM +0200, Pali Rohár wrote: > On Tuesday 06 July 2021 15:12:41 Bjorn Helgaas wrote: > > On Fri, Jul 02, 2021 at 12:51:19PM +0800, Aaron Ma wrote: > > > Check PCI state when rd/wr iomem. > > > Implement wr32 function as rd32 too. > > > > > > When unplug TBT dock with i225, rd/wr PCI iomem will cause error log: > > > Trace: > > > BUG: unable to handle page fault for address: 000000000000b604 > > > Oops: 0000 [#1] SMP NOPTI > > > RIP: 0010:igc_rd32+0x1c/0x90 [igc] > > > Call Trace: > > > igc_ptp_suspend+0x6c/0xa0 [igc] > > > igc_ptp_stop+0x12/0x50 [igc] > > > igc_remove+0x7f/0x1c0 [igc] > > > pci_device_remove+0x3e/0xb0 > > > __device_release_driver+0x181/0x240 > > > > > > Signed-off-by: Aaron Ma <aaron.ma@canonical.com> > > > --- > > > drivers/net/ethernet/intel/igc/igc_main.c | 16 ++++++++++++++++ > > > drivers/net/ethernet/intel/igc/igc_regs.h | 7 ++----- > > > 2 files changed, 18 insertions(+), 5 deletions(-) > > > > > > diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c > > > index f1adf154ec4a..606b72cb6193 100644 > > > --- a/drivers/net/ethernet/intel/igc/igc_main.c > > > +++ b/drivers/net/ethernet/intel/igc/igc_main.c > > > @@ -5292,6 +5292,10 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > > > u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > > > u32 value = 0; > > > > > > + if (igc->pdev && > > > + igc->pdev->error_state == pci_channel_io_perm_failure) > > > + return 0; > > > > I don't think this solves the problem. > > > > - Driver calls igc_rd32(). > > > > - "if (pci_channel_io_perm_failure)" evaluates to false (error_state > > does not indicate an error). > > > > - Device is unplugged. > > > > - igc_rd32() calls readl(), which performs MMIO read, which fails > > because the device is no longer present. readl() returns ~0 on > > most platforms. > > > > - Same page fault occurs. > > Hi Bjorn! I think that backtrace show that this error happens when PCIe > hotplug get interrupt that device was unplugged and PCIe hotplug code > calls remove/unbind procedure to stop unplugged driver. > > And in this case really does not make sense to try issuing MMIO read, > device is already unplugged. > > I looked that PCIe hotplug driver calls pci_dev_set_disconnected() when > this unplug interrupt happens and pci_dev_set_disconnected() just sets > pci_channel_io_perm_failure flag. > > drivers/pci/pci.h provides function pci_dev_is_disconnected() which > checks if that flag pci_channel_io_perm_failure is set. > > So I think that pci_dev_is_disconnected() is useful and could be > exported also to drivers (like this one) so they can check if > pci_dev_set_disconnected() was called in past and PCI driver is now in > unbind/cleanup/remove state because PCIe device is already disconnected > and not accessible anymore. > > But maybe this check should be on other place in driver unbound > procedure and not in general MMIO read function? If we add the check as proposed in this patch, I think people will read it and think this is the correct way to avoid MMIO errors. It does happen to avoid some MMIO errors, but it cannot avoid them all, so it's not a complete solution and it gives a false sense of security. A complete solution requires a test *after* the MMIO read. If you have the test after the read, you don't really need one before. Sure, testing before means you can avoid one MMIO read failure in some cases. But avoiding that failure costs quite a lot in code clutter. > > The only way is to check *after* the MMIO read to see whether an error > > occurred. On most platforms that means checking for ~0 data. If you > > see that, a PCI error *may* have occurred. > > > > If you know that ~0 can never be valid, e.g., if you're reading a > > register where ~0 is not a valid value, you know for sure that an > > error has occurred. > > > > If ~0 might be a valid value, e.g., if you're reading a buffer that > > contains arbitrary data, you have to look harder. You might read a > > register than cannot contain ~0, and see if you get the data you > > expect. Or you might read the Vendor ID or something from config > > space. > > > > > value = readl(&hw_addr[reg]); > > > > > > /* reads should not return all F's */ > > > @@ -5308,6 +5312,18 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > > > return value; > > > } > > > > > > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val) > > > +{ > > > + struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw); > > > + u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > > > + > > > + if (igc->pdev && > > > + igc->pdev->error_state == pci_channel_io_perm_failure) > > > + return; > > > + > > > + writel((val), &hw_addr[(reg)]); > > > +} > > > + > > > int igc_set_spd_dplx(struct igc_adapter *adapter, u32 spd, u8 dplx) > > > { > > > struct igc_mac_info *mac = &adapter->hw.mac; > > > diff --git a/drivers/net/ethernet/intel/igc/igc_regs.h b/drivers/net/ethernet/intel/igc/igc_regs.h > > > index cc174853554b..eb4be87d0e8b 100644 > > > --- a/drivers/net/ethernet/intel/igc/igc_regs.h > > > +++ b/drivers/net/ethernet/intel/igc/igc_regs.h > > > @@ -260,13 +260,10 @@ struct igc_hw; > > > u32 igc_rd32(struct igc_hw *hw, u32 reg); > > > > > > /* write operations, indexed using DWORDS */ > > > -#define wr32(reg, val) \ > > > -do { \ > > > - u8 __iomem *hw_addr = READ_ONCE((hw)->hw_addr); \ > > > - writel((val), &hw_addr[(reg)]); \ > > > -} while (0) > > > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val); > > > > > > #define rd32(reg) (igc_rd32(hw, reg)) > > > +#define wr32(reg, val) (igc_wr32(hw, reg, val)) > > > > > > #define wrfl() ((void)rd32(IGC_STATUS)) > > > > > > -- > > > 2.30.2 > > > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed 2021-07-07 22:10 ` Bjorn Helgaas @ 2021-07-08 2:04 ` Oliver O'Halloran 2021-07-08 15:45 ` Bjorn Helgaas 0 siblings, 1 reply; 14+ messages in thread From: Oliver O'Halloran @ 2021-07-08 2:04 UTC (permalink / raw) To: Bjorn Helgaas Cc: Pali Rohár, Aaron Ma, jesse.brandeburg, anthony.l.nguyen, David S. Miller, Jakub Kicinski, intel-wired-lan, netdev, Linux Kernel Mailing List, Krzysztof Wilczyński, linux-pci On Thu, Jul 8, 2021 at 8:40 AM Bjorn Helgaas <helgaas@kernel.org> wrote: > > If we add the check as proposed in this patch, I think people will > read it and think this is the correct way to avoid MMIO errors. It > does happen to avoid some MMIO errors, but it cannot avoid them all, > so it's not a complete solution and it gives a false sense of > security. I think it's helpful to classify MMIO errors as either benign or poisonous with the poison MMIOs causing some kind of crash. Most of the discussions about pci_dev_is_disconnected(), including this one, seem to stem from people trying to use it to avoid the poison case. I agree that using pci_dev_is_disconnected() that way is hacky and doesn't really fix the problem, but considering poison MMIOs usually stem from broken hardware or firmware maybe we should allow it anyway. We can't do anything better and it's an improvement compared to crashing. > A complete solution requires a test *after* the MMIO read. If you > have the test after the read, you don't really need one before. Sure, > testing before means you can avoid one MMIO read failure in some > cases. But avoiding that failure costs quite a lot in code clutter. It's not that much clutter if the checks are buried in the MMIO helpers which most drivers define. Speaking of which: > u32 igc_rd32(struct igc_hw *hw, u32 reg) > { > struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw); > u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > u32 value = 0; > > value = readl(&hw_addr[reg]); > > /* reads should not return all F's */ > if (!(~value) && (!reg || !(~readl(hw_addr)))) { > struct net_device *netdev = igc->netdev; > > hw->hw_addr = NULL; > netif_device_detach(netdev); > netdev_err(netdev, "PCIe link lost, device now detached\n"); > WARN(pci_device_is_present(igc->pdev), > "igc: Failed to read reg 0x%x!\n", reg); > } > > return value; > } I think I found where that page fault is coming from. I wonder if we should provide drivers some way of invoking the error recovery mechanisms manually or even just flagging itself as broken. Right now even if the driver bothers with synchronous error detection the driver can't really do anything other than parking itself and hoping AER/EEH recovery kicks in. Oliver ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed 2021-07-08 2:04 ` Oliver O'Halloran @ 2021-07-08 15:45 ` Bjorn Helgaas 2021-07-18 16:31 ` Oliver O'Halloran 0 siblings, 1 reply; 14+ messages in thread From: Bjorn Helgaas @ 2021-07-08 15:45 UTC (permalink / raw) To: Oliver O'Halloran Cc: Pali Rohár, Aaron Ma, jesse.brandeburg, anthony.l.nguyen, David S. Miller, Jakub Kicinski, intel-wired-lan, netdev, Linux Kernel Mailing List, Krzysztof Wilczyński, linux-pci On Thu, Jul 08, 2021 at 12:04:02PM +1000, Oliver O'Halloran wrote: > On Thu, Jul 8, 2021 at 8:40 AM Bjorn Helgaas <helgaas@kernel.org> wrote: > > > > If we add the check as proposed in this patch, I think people will > > read it and think this is the correct way to avoid MMIO errors. It > > does happen to avoid some MMIO errors, but it cannot avoid them all, > > so it's not a complete solution and it gives a false sense of > > security. > > I think it's helpful to classify MMIO errors as either benign or > poisonous with the poison MMIOs causing some kind of crash. Most of > the discussions about pci_dev_is_disconnected(), including this one, > seem to stem from people trying to use it to avoid the poison case. I > agree that using pci_dev_is_disconnected() that way is hacky and > doesn't really fix the problem, but considering poison MMIOs usually > stem from broken hardware or firmware maybe we should allow it > anyway. We can't do anything better and it's an improvement compared > to crashing. Apologies for rehashing what's probably obvious to everybody but me. I'm trying to get a better handle on benign vs poisonous errors. MMIO means CPU reads or writes to the device. In PCI, writes are posted and don't receive a response, so a driver will never see writel() return an error (although an error may be reported asynchronously via AER or similar). So I think we're mostly talking about CPU reads here. We expect a PCI response containing the data. Sometimes there's no response or an error response. The behavior of the host bridge in these error cases is not defined by PCI, so what the CPU sees is not consistent across platforms. In some cases, the bridge handles this as a catastrophic error that forces a system restart. But in most cases, at least on x86, the bridge logs an error and fabricates ~0 data so the CPU read can complete. Then it's up to software to recognize that an error occurred and decide what to do about it. Is this a benign or a poisonous error? I'd say this is a benign error. It certainly can't be ignored, but as long as the driver recognizes the error, it should be able to deal with it without crashing the whole system and forcing a restart. Bjorn ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed 2021-07-08 15:45 ` Bjorn Helgaas @ 2021-07-18 16:31 ` Oliver O'Halloran 2021-07-18 22:50 ` Pali Rohár 0 siblings, 1 reply; 14+ messages in thread From: Oliver O'Halloran @ 2021-07-18 16:31 UTC (permalink / raw) To: Bjorn Helgaas Cc: Pali Rohár, Aaron Ma, jesse.brandeburg, anthony.l.nguyen, David S. Miller, Jakub Kicinski, intel-wired-lan, netdev, Linux Kernel Mailing List, Krzysztof Wilczyński, linux-pci On Fri, Jul 9, 2021 at 1:45 AM Bjorn Helgaas <helgaas@kernel.org> wrote: > > *snip* > > Apologies for rehashing what's probably obvious to everybody but me. > I'm trying to get a better handle on benign vs poisonous errors. > > MMIO means CPU reads or writes to the device. In PCI, writes are > posted and don't receive a response, so a driver will never see > writel() return an error (although an error may be reported > asynchronously via AER or similar). > > So I think we're mostly talking about CPU reads here. We expect a PCI > response containing the data. Sometimes there's no response or an > error response. The behavior of the host bridge in these error cases > is not defined by PCI, so what the CPU sees is not consistent across > platforms. In some cases, the bridge handles this as a catastrophic > error that forces a system restart. > > But in most cases, at least on x86, the bridge logs an error and > fabricates ~0 data so the CPU read can complete. Then it's up to > software to recognize that an error occurred and decide what to do > about it. Is this a benign or a poisonous error? > > I'd say this is a benign error. It certainly can't be ignored, but as > long as the driver recognizes the error, it should be able to deal > with it without crashing the whole system and forcing a restart. I was thinking more in terms of what the driver author sees rather than what's happening on the CPU side. The crash seen in the OP appears to be because the code is "doing an MMIO." However, the reasons for the crash have nothing to do with the actual mechanics of the operation (which should be benign). The point I was making is that the pattern of: if (is_disconnected()) return failure; return do_mmio_read(addr); does have some utility as a last-ditch attempt to prevent crashes in the face of obnoxious bridges or bad hardware. Granted, that should be a platform concern rather than something that should ever appear in driver code, but considering drivers open-code readl()/writel() calls there's not really any place to put that sort of workaround. That all said, the case in the OP is due to an entirely avoidable driver bug and that sort of hack is absolutely the wrong thing to do. Oliver ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed 2021-07-18 16:31 ` Oliver O'Halloran @ 2021-07-18 22:50 ` Pali Rohár 2021-07-19 2:49 ` Oliver O'Halloran 0 siblings, 1 reply; 14+ messages in thread From: Pali Rohár @ 2021-07-18 22:50 UTC (permalink / raw) To: Oliver O'Halloran Cc: Bjorn Helgaas, Aaron Ma, jesse.brandeburg, anthony.l.nguyen, David S. Miller, Jakub Kicinski, intel-wired-lan, netdev, Linux Kernel Mailing List, Krzysztof Wilczyński, linux-pci On Monday 19 July 2021 02:31:10 Oliver O'Halloran wrote: > On Fri, Jul 9, 2021 at 1:45 AM Bjorn Helgaas <helgaas@kernel.org> wrote: > > > > *snip* > > > > Apologies for rehashing what's probably obvious to everybody but me. > > I'm trying to get a better handle on benign vs poisonous errors. > > > > MMIO means CPU reads or writes to the device. In PCI, writes are > > posted and don't receive a response, so a driver will never see > > writel() return an error (although an error may be reported > > asynchronously via AER or similar). > > > > So I think we're mostly talking about CPU reads here. We expect a PCI > > response containing the data. Sometimes there's no response or an > > error response. The behavior of the host bridge in these error cases > > is not defined by PCI, so what the CPU sees is not consistent across > > platforms. In some cases, the bridge handles this as a catastrophic > > error that forces a system restart. > > > > But in most cases, at least on x86, the bridge logs an error and > > fabricates ~0 data so the CPU read can complete. Then it's up to > > software to recognize that an error occurred and decide what to do > > about it. Is this a benign or a poisonous error? > > > > I'd say this is a benign error. It certainly can't be ignored, but as > > long as the driver recognizes the error, it should be able to deal > > with it without crashing the whole system and forcing a restart. > > I was thinking more in terms of what the driver author sees rather > than what's happening on the CPU side. The crash seen in the OP > appears to be because the code is "doing an MMIO." However, the > reasons for the crash have nothing to do with the actual mechanics of > the operation (which should be benign). The point I was making is that > the pattern of: > > if (is_disconnected()) > return failure; > return do_mmio_read(addr); > > does have some utility as a last-ditch attempt to prevent crashes in > the face of obnoxious bridges or bad hardware. Granted, that should be > a platform concern rather than something that should ever appear in > driver code, but considering drivers open-code readl()/writel() calls > there's not really any place to put that sort of workaround. > > That all said, the case in the OP is due to an entirely avoidable > driver bug and that sort of hack is absolutely the wrong thing to do. > > Oliver And do we have some solution for this kind of issue? There are more PCIe controllers / platforms which do not like MMIO read/write operation when card / link is not connected. If we do not provide a way how to solve these problems then we can expect that people would just hack ethernet / wifi / ... device drivers which are currently crashing by patches like in this thread. Maybe PCI subsystem could provide wrapper function which implements above pattern and which can be used by device drivers? ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed 2021-07-18 22:50 ` Pali Rohár @ 2021-07-19 2:49 ` Oliver O'Halloran 2021-07-19 8:13 ` Pali Rohár 2021-07-20 0:17 ` Bjorn Helgaas 0 siblings, 2 replies; 14+ messages in thread From: Oliver O'Halloran @ 2021-07-19 2:49 UTC (permalink / raw) To: Pali Rohár Cc: Bjorn Helgaas, Aaron Ma, jesse.brandeburg, anthony.l.nguyen, David S. Miller, Jakub Kicinski, intel-wired-lan, netdev, Linux Kernel Mailing List, Krzysztof Wilczyński, linux-pci On Mon, Jul 19, 2021 at 8:51 AM Pali Rohár <pali@kernel.org> wrote: > > And do we have some solution for this kind of issue? There are more PCIe > controllers / platforms which do not like MMIO read/write operation when > card / link is not connected. Do you have some actual examples? The few times I've seen those crashes were due to broken firmware-first error handling. The AER notifications would be escalated into some kind of ACPI error which the kernel didn't have a good way of dealing with so it panicked instead. Assuming it is a real problem then as Bjorn pointed out this sort of hack doesn't really fix the issue because hotplug and AER notifications are fundamentally asynchronous. If the driver is actively using the device when the error / removal happens then the pci_dev_is_disconnected() check will pass and the MMIO will go through. If the MMIO is poisonous because of dumb hardware then this sort of hack will only paper over the issue. > If we do not provide a way how to solve these problems then we can > expect that people would just hack ethernet / wifi / ... device drivers > which are currently crashing by patches like in this thread. > > Maybe PCI subsystem could provide wrapper function which implements > above pattern and which can be used by device drivers? We could do that and I think there was a proposal to add some pci_readl(pdev, <addr>) style wrappers at one point. On powerpc there's hooks in the arch provided MMIO functions to detect error responses and kick off the error handling machinery when a problem is detected. Those hooks are mainly there to help the platform detect errors though and they don't make life much easier for drivers. Due to locking concerns the driver's .error_detected() callback cannot be called in the MMIO hook so even when the platform detects errors synchronously the driver notifications must happen asynchronously. In the meanwhile the driver still needs to handle the 0xFFs response safely and there's not much we can do from the platform side to help there. Oliver ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed 2021-07-19 2:49 ` Oliver O'Halloran @ 2021-07-19 8:13 ` Pali Rohár 2021-07-20 0:17 ` Bjorn Helgaas 1 sibling, 0 replies; 14+ messages in thread From: Pali Rohár @ 2021-07-19 8:13 UTC (permalink / raw) To: Oliver O'Halloran Cc: Bjorn Helgaas, Aaron Ma, jesse.brandeburg, anthony.l.nguyen, David S. Miller, Jakub Kicinski, intel-wired-lan, netdev, Linux Kernel Mailing List, Krzysztof Wilczyński, linux-pci On Monday 19 July 2021 12:49:18 Oliver O'Halloran wrote: > On Mon, Jul 19, 2021 at 8:51 AM Pali Rohár <pali@kernel.org> wrote: > > > > And do we have some solution for this kind of issue? There are more PCIe > > controllers / platforms which do not like MMIO read/write operation when > > card / link is not connected. > > Do you have some actual examples? The few times I've seen those > crashes were due to broken firmware-first error handling. The AER > notifications would be escalated into some kind of ACPI error which > the kernel didn't have a good way of dealing with so it panicked > instead. I have experience and examples with pci aardvark controller. When card is disconnected it sends synchronous abort to CPU when doing MMIO read operation. One example is in this linux-usb thread: https://lore.kernel.org/linux-usb/20210505120117.4wpmo6fhvzznf3wv@pali/t/#u I can trigger this issue at least for xhci, nvme and ath drivers. > Assuming it is a real problem then as Bjorn pointed out this sort of > hack doesn't really fix the issue because hotplug and AER > notifications are fundamentally asynchronous. In case of pci aardvark it is not AER notification. And for MMIO read it is synchronous abort. Anyway, hotplug events are really asynchronous, but there is main issue that this hotplug disconnect event instruct device driver to "unbind" and e.g. these ethernet or usb controllers try to do MMIO operations in their cleanup / remove / unbind phase, even when card is already "disconnected" in PCI subsystem. > If the driver is > actively using the device when the error / removal happens then the > pci_dev_is_disconnected() check will pass and the MMIO will go > through. If the MMIO is poisonous because of dumb hardware then this > sort of hack will only paper over the issue. > > > If we do not provide a way how to solve these problems then we can > > expect that people would just hack ethernet / wifi / ... device drivers > > which are currently crashing by patches like in this thread. > > > > Maybe PCI subsystem could provide wrapper function which implements > > above pattern and which can be used by device drivers? > > We could do that and I think there was a proposal to add some > pci_readl(pdev, <addr>) style wrappers at one point. On powerpc > there's hooks in the arch provided MMIO functions to detect error > responses and kick off the error handling machinery when a problem is > detected. Those hooks are mainly there to help the platform detect > errors though and they don't make life much easier for drivers. Due to > locking concerns the driver's .error_detected() callback cannot be > called in the MMIO hook so even when the platform detects errors > synchronously the driver notifications must happen asynchronously. In > the meanwhile the driver still needs to handle the 0xFFs response > safely and there's not much we can do from the platform side to help > there. > > Oliver ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed 2021-07-19 2:49 ` Oliver O'Halloran 2021-07-19 8:13 ` Pali Rohár @ 2021-07-20 0:17 ` Bjorn Helgaas 1 sibling, 0 replies; 14+ messages in thread From: Bjorn Helgaas @ 2021-07-20 0:17 UTC (permalink / raw) To: Oliver O'Halloran Cc: Pali Rohár, Aaron Ma, jesse.brandeburg, anthony.l.nguyen, David S. Miller, Jakub Kicinski, intel-wired-lan, netdev, Linux Kernel Mailing List, Krzysztof Wilczyński, linux-pci On Mon, Jul 19, 2021 at 12:49:18PM +1000, Oliver O'Halloran wrote: > On Mon, Jul 19, 2021 at 8:51 AM Pali Rohár <pali@kernel.org> wrote: > > > > And do we have some solution for this kind of issue? There are more PCIe > > controllers / platforms which do not like MMIO read/write operation when > > card / link is not connected. > > Do you have some actual examples? The few times I've seen those > crashes were due to broken firmware-first error handling. The AER > notifications would be escalated into some kind of ACPI error which > the kernel didn't have a good way of dealing with so it panicked > instead. > > Assuming it is a real problem then as Bjorn pointed out this sort of > hack doesn't really fix the issue because hotplug and AER > notifications are fundamentally asynchronous. If the driver is > actively using the device when the error / removal happens then the > pci_dev_is_disconnected() check will pass and the MMIO will go > through. If the MMIO is poisonous because of dumb hardware then this > sort of hack will only paper over the issue. > > > If we do not provide a way how to solve these problems then we can > > expect that people would just hack ethernet / wifi / ... device drivers > > which are currently crashing by patches like in this thread. > > > > Maybe PCI subsystem could provide wrapper function which implements > > above pattern and which can be used by device drivers? > > We could do that and I think there was a proposal to add some > pci_readl(pdev, <addr>) style wrappers at one point. Obviously this wouldn't help user-space mmaps, but in the kernel, Documentation/driver-api/device-io.rst [1] does say that drivers are supposed to use readl() et al even though on most arches it "works" to just dereference the result of ioremap(), so maybe we could make a useful wrapper. Seems like we should do *something*, even if it's just a generic #define and some examples. I took a stab at this [2] a couple years ago, but it was only for the PCI core, and it didn't go anywhere. [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/driver-api/device-io.rst?id=v5.13#n160 [2] https://lore.kernel.org/linux-pci/20190822200551.129039-1-helgaas@kernel.org/ > On powerpc > there's hooks in the arch provided MMIO functions to detect error > responses and kick off the error handling machinery when a problem is > detected. Those hooks are mainly there to help the platform detect > errors though and they don't make life much easier for drivers. Due to > locking concerns the driver's .error_detected() callback cannot be > called in the MMIO hook so even when the platform detects errors > synchronously the driver notifications must happen asynchronously. In > the meanwhile the driver still needs to handle the 0xFFs response > safely and there's not much we can do from the platform side to help > there. > > Oliver ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2021-07-20 0:58 UTC | newest] Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <20210702045120.22855-1-aaron.ma@canonical.com> 2021-07-04 14:28 ` [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed Pali Rohár 2021-07-05 7:23 ` Aaron Ma 2021-07-05 23:02 ` Krzysztof Wilczyński 2021-07-06 14:23 ` Pali Rohár 2021-07-06 20:12 ` Bjorn Helgaas 2021-07-07 21:53 ` Pali Rohár 2021-07-07 22:10 ` Bjorn Helgaas 2021-07-08 2:04 ` Oliver O'Halloran 2021-07-08 15:45 ` Bjorn Helgaas 2021-07-18 16:31 ` Oliver O'Halloran 2021-07-18 22:50 ` Pali Rohár 2021-07-19 2:49 ` Oliver O'Halloran 2021-07-19 8:13 ` Pali Rohár 2021-07-20 0:17 ` Bjorn Helgaas
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).