Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed

Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed

[Pali Rohár]

+ Bjorn, Krzysztof and linux-pci

On Friday 02 July 2021 12:51:19 Aaron Ma wrote:
> Check PCI state when rd/wr iomem.
> Implement wr32 function as rd32 too.
> 
> When unplug TBT dock with i225, rd/wr PCI iomem will cause error log:
> Trace:
> BUG: unable to handle page fault for address: 000000000000b604
> Oops: 0000 [#1] SMP NOPTI
> RIP: 0010:igc_rd32+0x1c/0x90 [igc]
> Call Trace:
> igc_ptp_suspend+0x6c/0xa0 [igc]
> igc_ptp_stop+0x12/0x50 [igc]
> igc_remove+0x7f/0x1c0 [igc]
> pci_device_remove+0x3e/0xb0
> __device_release_driver+0x181/0x240
> 
> Signed-off-by: Aaron Ma <aaron.ma@canonical.com>
> ---
>  drivers/net/ethernet/intel/igc/igc_main.c | 16 ++++++++++++++++
>  drivers/net/ethernet/intel/igc/igc_regs.h |  7 ++-----
>  2 files changed, 18 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c
> index f1adf154ec4a..606b72cb6193 100644
> --- a/drivers/net/ethernet/intel/igc/igc_main.c
> +++ b/drivers/net/ethernet/intel/igc/igc_main.c
> @@ -5292,6 +5292,10 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg)
>  	u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr);
>  	u32 value = 0;
>  
> +	if (igc->pdev &&
> +		igc->pdev->error_state == pci_channel_io_perm_failure)

Hello! This code pattern and commit message looks like that we could use
pci_dev_is_disconnected() helper function for checking if device is
still connected or was disconnected.

Apparently pci_dev_is_disconnected() is defined only in private header
file drivers/pci/pci.h and not in public include/linux/pci.h.

Aaron: can you check if pci_dev_is_disconnected() is really something
which should be used and it helps you?

Bjorn, Krzysztof: what do you think about lifting helper function
pci_dev_is_disconnected() to be available to all drivers and not only in
PCI subsystem?

I think that such helper function makes driver code more readable and
can be useful also for other drivers which are checking if return value
is all F's.

> +		return 0;

Aaron: should not you return all F's on error? Because few lines below
in this function is returned value with all F's when PCIe link lost.

> +
>  	value = readl(&hw_addr[reg]);

Anyway, this code looks to be racy. When pci_channel_io_perm_failure is
set (e.g. by hotplug interrupt) after checking for pdev->error_state and
prior executing above readl() then mentioned fatal error still occurs.

>  
>  	/* reads should not return all F's */
> @@ -5308,6 +5312,18 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg)
>  	return value;
>  }
>  
> +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val)
> +{
> +	struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw);
> +	u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr);
> +
> +	if (igc->pdev &&
> +		igc->pdev->error_state == pci_channel_io_perm_failure)
> +		return;
> +
> +	writel((val), &hw_addr[(reg)]);
> +}
> +
>  int igc_set_spd_dplx(struct igc_adapter *adapter, u32 spd, u8 dplx)
>  {
>  	struct igc_mac_info *mac = &adapter->hw.mac;
> diff --git a/drivers/net/ethernet/intel/igc/igc_regs.h b/drivers/net/ethernet/intel/igc/igc_regs.h
> index cc174853554b..eb4be87d0e8b 100644
> --- a/drivers/net/ethernet/intel/igc/igc_regs.h
> +++ b/drivers/net/ethernet/intel/igc/igc_regs.h
> @@ -260,13 +260,10 @@ struct igc_hw;
>  u32 igc_rd32(struct igc_hw *hw, u32 reg);
>  
>  /* write operations, indexed using DWORDS */
> -#define wr32(reg, val) \
> -do { \
> -	u8 __iomem *hw_addr = READ_ONCE((hw)->hw_addr); \
> -	writel((val), &hw_addr[(reg)]); \
> -} while (0)
> +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val);
>  
>  #define rd32(reg) (igc_rd32(hw, reg))
> +#define wr32(reg, val) (igc_wr32(hw, reg, val))
>  
>  #define wrfl() ((void)rd32(IGC_STATUS))
>  
> -- 
> 2.30.2
> 

Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed

[Aaron Ma]

On 7/4/21 10:28 PM, Pali Rohár wrote:
> + Bjorn, Krzysztof and linux-pci
> 
> On Friday 02 July 2021 12:51:19 Aaron Ma wrote:
>> Check PCI state when rd/wr iomem.
>> Implement wr32 function as rd32 too.
>>
>> When unplug TBT dock with i225, rd/wr PCI iomem will cause error log:
>> Trace:
>> BUG: unable to handle page fault for address: 000000000000b604
>> Oops: 0000 [#1] SMP NOPTI
>> RIP: 0010:igc_rd32+0x1c/0x90 [igc]
>> Call Trace:
>> igc_ptp_suspend+0x6c/0xa0 [igc]
>> igc_ptp_stop+0x12/0x50 [igc]
>> igc_remove+0x7f/0x1c0 [igc]
>> pci_device_remove+0x3e/0xb0
>> __device_release_driver+0x181/0x240
>>
>> Signed-off-by: Aaron Ma <aaron.ma@canonical.com>
>> ---
>>   drivers/net/ethernet/intel/igc/igc_main.c | 16 ++++++++++++++++
>>   drivers/net/ethernet/intel/igc/igc_regs.h |  7 ++-----
>>   2 files changed, 18 insertions(+), 5 deletions(-)
>>
>> diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c
>> index f1adf154ec4a..606b72cb6193 100644
>> --- a/drivers/net/ethernet/intel/igc/igc_main.c
>> +++ b/drivers/net/ethernet/intel/igc/igc_main.c
>> @@ -5292,6 +5292,10 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg)
>>   	u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr);
>>   	u32 value = 0;
>>   
>> +	if (igc->pdev &&
>> +		igc->pdev->error_state == pci_channel_io_perm_failure)
> 
> Hello! This code pattern and commit message looks like that we could use
> pci_dev_is_disconnected() helper function for checking if device is
> still connected or was disconnected.
> 
> Apparently pci_dev_is_disconnected() is defined only in private header
> file drivers/pci/pci.h and not in public include/linux/pci.h.
> 
> Aaron: can you check if pci_dev_is_disconnected() is really something
> which should be used and it helps you?
> 

Hi Pali,

How about using pci_channel_offline instead?
It's ready and also safe for frozen state, and verified on hw.

> Bjorn, Krzysztof: what do you think about lifting helper function
> pci_dev_is_disconnected() to be available to all drivers and not only in
> PCI subsystem?
> 
> I think that such helper function makes driver code more readable and
> can be useful also for other drivers which are checking if return value
> is all F's.
> 
>> +		return 0;
> 
> Aaron: should not you return all F's on error? Because few lines below
> in this function is returned value with all F's when PCIe link lost.
> 

If you agree with the above change, I can fix it to "return -1" in v2.

Thanks for your comments,
Aaron


>> +
>>   	value = readl(&hw_addr[reg]);
> 
> Anyway, this code looks to be racy. When pci_channel_io_perm_failure is
> set (e.g. by hotplug interrupt) after checking for pdev->error_state and
> prior executing above readl() then mentioned fatal error still occurs.
> 
>>   
>>   	/* reads should not return all F's */
>> @@ -5308,6 +5312,18 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg)
>>   	return value;
>>   }
>>   
>> +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val)
>> +{
>> +	struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw);
>> +	u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr);
>> +
>> +	if (igc->pdev &&
>> +		igc->pdev->error_state == pci_channel_io_perm_failure)
>> +		return;
>> +
>> +	writel((val), &hw_addr[(reg)]);
>> +}
>> +
>>   int igc_set_spd_dplx(struct igc_adapter *adapter, u32 spd, u8 dplx)
>>   {
>>   	struct igc_mac_info *mac = &adapter->hw.mac;
>> diff --git a/drivers/net/ethernet/intel/igc/igc_regs.h b/drivers/net/ethernet/intel/igc/igc_regs.h
>> index cc174853554b..eb4be87d0e8b 100644
>> --- a/drivers/net/ethernet/intel/igc/igc_regs.h
>> +++ b/drivers/net/ethernet/intel/igc/igc_regs.h
>> @@ -260,13 +260,10 @@ struct igc_hw;
>>   u32 igc_rd32(struct igc_hw *hw, u32 reg);
>>   
>>   /* write operations, indexed using DWORDS */
>> -#define wr32(reg, val) \
>> -do { \
>> -	u8 __iomem *hw_addr = READ_ONCE((hw)->hw_addr); \
>> -	writel((val), &hw_addr[(reg)]); \
>> -} while (0)
>> +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val);
>>   
>>   #define rd32(reg) (igc_rd32(hw, reg))
>> +#define wr32(reg, val) (igc_wr32(hw, reg, val))
>>   
>>   #define wrfl() ((void)rd32(IGC_STATUS))
>>   
>> -- 
>> 2.30.2
>>

Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed

[Krzysztof Wilczyński]

Hi Pali,

[...]
> Aaron: can you check if pci_dev_is_disconnected() is really something
> which should be used and it helps you?

While having a closer look, I've noticed that quite a few of the network
drivers handle this somewhat, as I see that a lot of them have some sort
of I/O error handles set where a check for "pci_channel_io_perm_failure"
seem to be having place.  This is also true for this driver looking at
the igc_io_error_detected().

Is this not working for the igc driver?  Or is this for something
completely different?

Having said all that, I am not an expert in network drivers, so pardon
me if I am asking about something completely different, and I apologise
if that is the case.

> Bjorn, Krzysztof: what do you think about lifting helper function
> pci_dev_is_disconnected() to be available to all drivers and not only in
> PCI subsystem?

No objections from me, if we believe it's useful and that it might
encourage people to use a common API.  Currently, I can see at least
five potential users of this helper.

	Krzysztof

Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed

[Pali Rohár]

On Tuesday 06 July 2021 01:02:12 Krzysztof Wilczyński wrote:
> Hi Pali,
> 
> [...]
> > Aaron: can you check if pci_dev_is_disconnected() is really something
> > which should be used and it helps you?
> 
> While having a closer look, I've noticed that quite a few of the network
> drivers handle this somewhat, as I see that a lot of them have some sort
> of I/O error handles set where a check for "pci_channel_io_perm_failure"
> seem to be having place.  This is also true for this driver looking at
> the igc_io_error_detected().
> 
> Is this not working for the igc driver?  Or is this for something
> completely different?

I guess that this callback is called when Bridge receive some kind of
fatal error. Non-AER-aware bridges probably do not have to inform system
that error happened and kernel would not call this callback. So I guess
it depends on to which "machine" you need this network adapter.

So in my opinion this callback is there for PCI subsystem to inform
driver that error happened and let driver to do any hw specific recovery
if it is possible.

But I think problem described here can be slightly different. It is
needed to check if device is still alive or was disconnected.

> Having said all that, I am not an expert in network drivers, so pardon
> me if I am asking about something completely different, and I apologise
> if that is the case.
> 
> > Bjorn, Krzysztof: what do you think about lifting helper function
> > pci_dev_is_disconnected() to be available to all drivers and not only in
> > PCI subsystem?
> 
> No objections from me, if we believe it's useful and that it might
> encourage people to use a common API.  Currently, I can see at least
> five potential users of this helper.
> 
> 	Krzysztof

Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed

[Bjorn Helgaas]

On Fri, Jul 02, 2021 at 12:51:19PM +0800, Aaron Ma wrote:
> Check PCI state when rd/wr iomem.
> Implement wr32 function as rd32 too.
> 
> When unplug TBT dock with i225, rd/wr PCI iomem will cause error log:
> Trace:
> BUG: unable to handle page fault for address: 000000000000b604
> Oops: 0000 [#1] SMP NOPTI
> RIP: 0010:igc_rd32+0x1c/0x90 [igc]
> Call Trace:
> igc_ptp_suspend+0x6c/0xa0 [igc]
> igc_ptp_stop+0x12/0x50 [igc]
> igc_remove+0x7f/0x1c0 [igc]
> pci_device_remove+0x3e/0xb0
> __device_release_driver+0x181/0x240
> 
> Signed-off-by: Aaron Ma <aaron.ma@canonical.com>
> ---
>  drivers/net/ethernet/intel/igc/igc_main.c | 16 ++++++++++++++++
>  drivers/net/ethernet/intel/igc/igc_regs.h |  7 ++-----
>  2 files changed, 18 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c
> index f1adf154ec4a..606b72cb6193 100644
> --- a/drivers/net/ethernet/intel/igc/igc_main.c
> +++ b/drivers/net/ethernet/intel/igc/igc_main.c
> @@ -5292,6 +5292,10 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg)
>  	u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr);
>  	u32 value = 0;
>  
> +	if (igc->pdev &&
> +		igc->pdev->error_state == pci_channel_io_perm_failure)
> +		return 0;

I don't think this solves the problem.

  - Driver calls igc_rd32().

  - "if (pci_channel_io_perm_failure)" evaluates to false (error_state
    does not indicate an error).

  - Device is unplugged.

  - igc_rd32() calls readl(), which performs MMIO read, which fails
    because the device is no longer present.  readl() returns ~0 on
    most platforms.

  - Same page fault occurs.

The only way is to check *after* the MMIO read to see whether an error
occurred.  On most platforms that means checking for ~0 data.  If you
see that, a PCI error *may* have occurred.

If you know that ~0 can never be valid, e.g., if you're reading a
register where ~0 is not a valid value, you know for sure that an
error has occurred.

If ~0 might be a valid value, e.g., if you're reading a buffer that
contains arbitrary data, you have to look harder.   You might read a
register than cannot contain ~0, and see if you get the data you
expect.  Or you might read the Vendor ID or something from config
space.

>  	value = readl(&hw_addr[reg]);
>  
>  	/* reads should not return all F's */
> @@ -5308,6 +5312,18 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg)
>  	return value;
>  }
>  
> +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val)
> +{
> +	struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw);
> +	u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr);
> +
> +	if (igc->pdev &&
> +		igc->pdev->error_state == pci_channel_io_perm_failure)
> +		return;
> +
> +	writel((val), &hw_addr[(reg)]);
> +}
> +
>  int igc_set_spd_dplx(struct igc_adapter *adapter, u32 spd, u8 dplx)
>  {
>  	struct igc_mac_info *mac = &adapter->hw.mac;
> diff --git a/drivers/net/ethernet/intel/igc/igc_regs.h b/drivers/net/ethernet/intel/igc/igc_regs.h
> index cc174853554b..eb4be87d0e8b 100644
> --- a/drivers/net/ethernet/intel/igc/igc_regs.h
> +++ b/drivers/net/ethernet/intel/igc/igc_regs.h
> @@ -260,13 +260,10 @@ struct igc_hw;
>  u32 igc_rd32(struct igc_hw *hw, u32 reg);
>  
>  /* write operations, indexed using DWORDS */
> -#define wr32(reg, val) \
> -do { \
> -	u8 __iomem *hw_addr = READ_ONCE((hw)->hw_addr); \
> -	writel((val), &hw_addr[(reg)]); \
> -} while (0)
> +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val);
>  
>  #define rd32(reg) (igc_rd32(hw, reg))
> +#define wr32(reg, val) (igc_wr32(hw, reg, val))
>  
>  #define wrfl() ((void)rd32(IGC_STATUS))
>  
> -- 
> 2.30.2
> 

Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed

[Pali Rohár]

On Tuesday 06 July 2021 15:12:41 Bjorn Helgaas wrote:
> On Fri, Jul 02, 2021 at 12:51:19PM +0800, Aaron Ma wrote:
> > Check PCI state when rd/wr iomem.
> > Implement wr32 function as rd32 too.
> > 
> > When unplug TBT dock with i225, rd/wr PCI iomem will cause error log:
> > Trace:
> > BUG: unable to handle page fault for address: 000000000000b604
> > Oops: 0000 [#1] SMP NOPTI
> > RIP: 0010:igc_rd32+0x1c/0x90 [igc]
> > Call Trace:
> > igc_ptp_suspend+0x6c/0xa0 [igc]
> > igc_ptp_stop+0x12/0x50 [igc]
> > igc_remove+0x7f/0x1c0 [igc]
> > pci_device_remove+0x3e/0xb0
> > __device_release_driver+0x181/0x240
> > 
> > Signed-off-by: Aaron Ma <aaron.ma@canonical.com>
> > ---
> >  drivers/net/ethernet/intel/igc/igc_main.c | 16 ++++++++++++++++
> >  drivers/net/ethernet/intel/igc/igc_regs.h |  7 ++-----
> >  2 files changed, 18 insertions(+), 5 deletions(-)
> > 
> > diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c
> > index f1adf154ec4a..606b72cb6193 100644
> > --- a/drivers/net/ethernet/intel/igc/igc_main.c
> > +++ b/drivers/net/ethernet/intel/igc/igc_main.c
> > @@ -5292,6 +5292,10 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg)
> >  	u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr);
> >  	u32 value = 0;
> >  
> > +	if (igc->pdev &&
> > +		igc->pdev->error_state == pci_channel_io_perm_failure)
> > +		return 0;
> 
> I don't think this solves the problem.
> 
>   - Driver calls igc_rd32().
> 
>   - "if (pci_channel_io_perm_failure)" evaluates to false (error_state
>     does not indicate an error).
> 
>   - Device is unplugged.
> 
>   - igc_rd32() calls readl(), which performs MMIO read, which fails
>     because the device is no longer present.  readl() returns ~0 on
>     most platforms.
> 
>   - Same page fault occurs.

Hi Bjorn! I think that backtrace show that this error happens when PCIe
hotplug get interrupt that device was unplugged and PCIe hotplug code
calls remove/unbind procedure to stop unplugged driver.

And in this case really does not make sense to try issuing MMIO read,
device is already unplugged.

I looked that PCIe hotplug driver calls pci_dev_set_disconnected() when
this unplug interrupt happens and pci_dev_set_disconnected() just sets
pci_channel_io_perm_failure flag.

drivers/pci/pci.h provides function pci_dev_is_disconnected() which
checks if that flag pci_channel_io_perm_failure is set.

So I think that pci_dev_is_disconnected() is useful and could be
exported also to drivers (like this one) so they can check if
pci_dev_set_disconnected() was called in past and PCI driver is now in
unbind/cleanup/remove state because PCIe device is already disconnected
and not accessible anymore.

But maybe this check should be on other place in driver unbound
procedure and not in general MMIO read function?

> The only way is to check *after* the MMIO read to see whether an error
> occurred.  On most platforms that means checking for ~0 data.  If you
> see that, a PCI error *may* have occurred.
> 
> If you know that ~0 can never be valid, e.g., if you're reading a
> register where ~0 is not a valid value, you know for sure that an
> error has occurred.
> 
> If ~0 might be a valid value, e.g., if you're reading a buffer that
> contains arbitrary data, you have to look harder.   You might read a
> register than cannot contain ~0, and see if you get the data you
> expect.  Or you might read the Vendor ID or something from config
> space.
> 
> >  	value = readl(&hw_addr[reg]);
> >  
> >  	/* reads should not return all F's */
> > @@ -5308,6 +5312,18 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg)
> >  	return value;
> >  }
> >  
> > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val)
> > +{
> > +	struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw);
> > +	u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr);
> > +
> > +	if (igc->pdev &&
> > +		igc->pdev->error_state == pci_channel_io_perm_failure)
> > +		return;
> > +
> > +	writel((val), &hw_addr[(reg)]);
> > +}
> > +
> >  int igc_set_spd_dplx(struct igc_adapter *adapter, u32 spd, u8 dplx)
> >  {
> >  	struct igc_mac_info *mac = &adapter->hw.mac;
> > diff --git a/drivers/net/ethernet/intel/igc/igc_regs.h b/drivers/net/ethernet/intel/igc/igc_regs.h
> > index cc174853554b..eb4be87d0e8b 100644
> > --- a/drivers/net/ethernet/intel/igc/igc_regs.h
> > +++ b/drivers/net/ethernet/intel/igc/igc_regs.h
> > @@ -260,13 +260,10 @@ struct igc_hw;
> >  u32 igc_rd32(struct igc_hw *hw, u32 reg);
> >  
> >  /* write operations, indexed using DWORDS */
> > -#define wr32(reg, val) \
> > -do { \
> > -	u8 __iomem *hw_addr = READ_ONCE((hw)->hw_addr); \
> > -	writel((val), &hw_addr[(reg)]); \
> > -} while (0)
> > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val);
> >  
> >  #define rd32(reg) (igc_rd32(hw, reg))
> > +#define wr32(reg, val) (igc_wr32(hw, reg, val))
> >  
> >  #define wrfl() ((void)rd32(IGC_STATUS))
> >  
> > -- 
> > 2.30.2
> > 

Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed

[Bjorn Helgaas]

On Wed, Jul 07, 2021 at 11:53:37PM +0200, Pali Rohár wrote:
> On Tuesday 06 July 2021 15:12:41 Bjorn Helgaas wrote:
> > On Fri, Jul 02, 2021 at 12:51:19PM +0800, Aaron Ma wrote:
> > > Check PCI state when rd/wr iomem.
> > > Implement wr32 function as rd32 too.
> > > 
> > > When unplug TBT dock with i225, rd/wr PCI iomem will cause error log:
> > > Trace:
> > > BUG: unable to handle page fault for address: 000000000000b604
> > > Oops: 0000 [#1] SMP NOPTI
> > > RIP: 0010:igc_rd32+0x1c/0x90 [igc]
> > > Call Trace:
> > > igc_ptp_suspend+0x6c/0xa0 [igc]
> > > igc_ptp_stop+0x12/0x50 [igc]
> > > igc_remove+0x7f/0x1c0 [igc]
> > > pci_device_remove+0x3e/0xb0
> > > __device_release_driver+0x181/0x240
> > > 
> > > Signed-off-by: Aaron Ma <aaron.ma@canonical.com>
> > > ---
> > >  drivers/net/ethernet/intel/igc/igc_main.c | 16 ++++++++++++++++
> > >  drivers/net/ethernet/intel/igc/igc_regs.h |  7 ++-----
> > >  2 files changed, 18 insertions(+), 5 deletions(-)
> > > 
> > > diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c
> > > index f1adf154ec4a..606b72cb6193 100644
> > > --- a/drivers/net/ethernet/intel/igc/igc_main.c
> > > +++ b/drivers/net/ethernet/intel/igc/igc_main.c
> > > @@ -5292,6 +5292,10 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg)
> > >  	u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr);
> > >  	u32 value = 0;
> > >  
> > > +	if (igc->pdev &&
> > > +		igc->pdev->error_state == pci_channel_io_perm_failure)
> > > +		return 0;
> > 
> > I don't think this solves the problem.
> > 
> >   - Driver calls igc_rd32().
> > 
> >   - "if (pci_channel_io_perm_failure)" evaluates to false (error_state
> >     does not indicate an error).
> > 
> >   - Device is unplugged.
> > 
> >   - igc_rd32() calls readl(), which performs MMIO read, which fails
> >     because the device is no longer present.  readl() returns ~0 on
> >     most platforms.
> > 
> >   - Same page fault occurs.
> 
> Hi Bjorn! I think that backtrace show that this error happens when PCIe
> hotplug get interrupt that device was unplugged and PCIe hotplug code
> calls remove/unbind procedure to stop unplugged driver.
> 
> And in this case really does not make sense to try issuing MMIO read,
> device is already unplugged.
> 
> I looked that PCIe hotplug driver calls pci_dev_set_disconnected() when
> this unplug interrupt happens and pci_dev_set_disconnected() just sets
> pci_channel_io_perm_failure flag.
> 
> drivers/pci/pci.h provides function pci_dev_is_disconnected() which
> checks if that flag pci_channel_io_perm_failure is set.
> 
> So I think that pci_dev_is_disconnected() is useful and could be
> exported also to drivers (like this one) so they can check if
> pci_dev_set_disconnected() was called in past and PCI driver is now in
> unbind/cleanup/remove state because PCIe device is already disconnected
> and not accessible anymore.
> 
> But maybe this check should be on other place in driver unbound
> procedure and not in general MMIO read function?

If we add the check as proposed in this patch, I think people will
read it and think this is the correct way to avoid MMIO errors.  It
does happen to avoid some MMIO errors, but it cannot avoid them all,
so it's not a complete solution and it gives a false sense of
security.

A complete solution requires a test *after* the MMIO read.  If you
have the test after the read, you don't really need one before.  Sure,
testing before means you can avoid one MMIO read failure in some
cases.  But avoiding that failure costs quite a lot in code clutter.

> > The only way is to check *after* the MMIO read to see whether an error
> > occurred.  On most platforms that means checking for ~0 data.  If you
> > see that, a PCI error *may* have occurred.
> > 
> > If you know that ~0 can never be valid, e.g., if you're reading a
> > register where ~0 is not a valid value, you know for sure that an
> > error has occurred.
> > 
> > If ~0 might be a valid value, e.g., if you're reading a buffer that
> > contains arbitrary data, you have to look harder.   You might read a
> > register than cannot contain ~0, and see if you get the data you
> > expect.  Or you might read the Vendor ID or something from config
> > space.
> > 
> > >  	value = readl(&hw_addr[reg]);
> > >  
> > >  	/* reads should not return all F's */
> > > @@ -5308,6 +5312,18 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg)
> > >  	return value;
> > >  }
> > >  
> > > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val)
> > > +{
> > > +	struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw);
> > > +	u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr);
> > > +
> > > +	if (igc->pdev &&
> > > +		igc->pdev->error_state == pci_channel_io_perm_failure)
> > > +		return;
> > > +
> > > +	writel((val), &hw_addr[(reg)]);
> > > +}
> > > +
> > >  int igc_set_spd_dplx(struct igc_adapter *adapter, u32 spd, u8 dplx)
> > >  {
> > >  	struct igc_mac_info *mac = &adapter->hw.mac;
> > > diff --git a/drivers/net/ethernet/intel/igc/igc_regs.h b/drivers/net/ethernet/intel/igc/igc_regs.h
> > > index cc174853554b..eb4be87d0e8b 100644
> > > --- a/drivers/net/ethernet/intel/igc/igc_regs.h
> > > +++ b/drivers/net/ethernet/intel/igc/igc_regs.h
> > > @@ -260,13 +260,10 @@ struct igc_hw;
> > >  u32 igc_rd32(struct igc_hw *hw, u32 reg);
> > >  
> > >  /* write operations, indexed using DWORDS */
> > > -#define wr32(reg, val) \
> > > -do { \
> > > -	u8 __iomem *hw_addr = READ_ONCE((hw)->hw_addr); \
> > > -	writel((val), &hw_addr[(reg)]); \
> > > -} while (0)
> > > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val);
> > >  
> > >  #define rd32(reg) (igc_rd32(hw, reg))
> > > +#define wr32(reg, val) (igc_wr32(hw, reg, val))
> > >  
> > >  #define wrfl() ((void)rd32(IGC_STATUS))
> > >  
> > > -- 
> > > 2.30.2
> > > 

Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed

[Oliver O'Halloran]

On Thu, Jul 8, 2021 at 8:40 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> If we add the check as proposed in this patch, I think people will
> read it and think this is the correct way to avoid MMIO errors.  It
> does happen to avoid some MMIO errors, but it cannot avoid them all,
> so it's not a complete solution and it gives a false sense of
> security.

I think it's helpful to classify MMIO errors as either benign or
poisonous with the poison MMIOs causing some kind of crash. Most of
the discussions about pci_dev_is_disconnected(), including this one,
seem to stem from people trying to use it to avoid the poison case. I
agree that using pci_dev_is_disconnected() that way is hacky and
doesn't really fix the problem, but considering poison MMIOs usually
stem from broken hardware or firmware  maybe we should allow it
anyway. We can't do anything better and it's an improvement compared
to crashing.

> A complete solution requires a test *after* the MMIO read.  If you
> have the test after the read, you don't really need one before.  Sure,
> testing before means you can avoid one MMIO read failure in some
> cases.  But avoiding that failure costs quite a lot in code clutter.

It's not that much clutter if the checks are buried in the MMIO
helpers which most drivers define. Speaking of which:

> u32 igc_rd32(struct igc_hw *hw, u32 reg)
> {
>   struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw);
>   u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr);
>   u32 value = 0;
>
>   value = readl(&hw_addr[reg]);
>
>   /* reads should not return all F's */
>   if (!(~value) && (!reg || !(~readl(hw_addr)))) {
>     struct net_device *netdev = igc->netdev;
>
>     hw->hw_addr = NULL;
>     netif_device_detach(netdev);
>     netdev_err(netdev, "PCIe link lost, device now detached\n");
>     WARN(pci_device_is_present(igc->pdev),
>          "igc: Failed to read reg 0x%x!\n", reg);
>   }
>
>   return value;
> }

I think I found where that page fault is coming from.

I wonder if we should provide drivers some way of invoking the error
recovery mechanisms manually or even just flagging itself as broken.
Right now even if the driver bothers with synchronous error detection
the driver can't really do anything other than parking itself and
hoping AER/EEH recovery kicks in.

Oliver

Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed

[Bjorn Helgaas]

On Thu, Jul 08, 2021 at 12:04:02PM +1000, Oliver O'Halloran wrote:
> On Thu, Jul 8, 2021 at 8:40 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> >
> > If we add the check as proposed in this patch, I think people will
> > read it and think this is the correct way to avoid MMIO errors.  It
> > does happen to avoid some MMIO errors, but it cannot avoid them all,
> > so it's not a complete solution and it gives a false sense of
> > security.
> 
> I think it's helpful to classify MMIO errors as either benign or
> poisonous with the poison MMIOs causing some kind of crash. Most of
> the discussions about pci_dev_is_disconnected(), including this one,
> seem to stem from people trying to use it to avoid the poison case. I
> agree that using pci_dev_is_disconnected() that way is hacky and
> doesn't really fix the problem, but considering poison MMIOs usually
> stem from broken hardware or firmware maybe we should allow it
> anyway. We can't do anything better and it's an improvement compared
> to crashing.

Apologies for rehashing what's probably obvious to everybody but me.
I'm trying to get a better handle on benign vs poisonous errors.

MMIO means CPU reads or writes to the device.  In PCI, writes are
posted and don't receive a response, so a driver will never see
writel() return an error (although an error may be reported
asynchronously via AER or similar).

So I think we're mostly talking about CPU reads here.  We expect a PCI
response containing the data.  Sometimes there's no response or an
error response.  The behavior of the host bridge in these error cases
is not defined by PCI, so what the CPU sees is not consistent across
platforms.  In some cases, the bridge handles this as a catastrophic
error that forces a system restart.

But in most cases, at least on x86, the bridge logs an error and
fabricates ~0 data so the CPU read can complete.  Then it's up to
software to recognize that an error occurred and decide what to do
about it.  Is this a benign or a poisonous error?

I'd say this is a benign error.  It certainly can't be ignored, but as
long as the driver recognizes the error, it should be able to deal
with it without crashing the whole system and forcing a restart.

Bjorn

Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed

[Oliver O'Halloran]

On Fri, Jul 9, 2021 at 1:45 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> *snip*
>
> Apologies for rehashing what's probably obvious to everybody but me.
> I'm trying to get a better handle on benign vs poisonous errors.
>
> MMIO means CPU reads or writes to the device.  In PCI, writes are
> posted and don't receive a response, so a driver will never see
> writel() return an error (although an error may be reported
> asynchronously via AER or similar).
>
> So I think we're mostly talking about CPU reads here.  We expect a PCI
> response containing the data.  Sometimes there's no response or an
> error response.  The behavior of the host bridge in these error cases
> is not defined by PCI, so what the CPU sees is not consistent across
> platforms.  In some cases, the bridge handles this as a catastrophic
> error that forces a system restart.
>
> But in most cases, at least on x86, the bridge logs an error and
> fabricates ~0 data so the CPU read can complete.  Then it's up to
> software to recognize that an error occurred and decide what to do
> about it.  Is this a benign or a poisonous error?
>
> I'd say this is a benign error. It certainly can't be ignored, but as
> long as the driver recognizes the error, it should be able to deal
> with it without crashing the whole system and forcing a restart.

I was thinking more in terms of what the driver author sees rather
than what's happening on the CPU side. The crash seen in the OP
appears to be because the code is "doing an MMIO." However, the
reasons for the crash have nothing to do with the actual mechanics of
the operation (which should be benign). The point I was making is that
the pattern of:

if (is_disconnected())
    return failure;
return do_mmio_read(addr);

does have some utility as a last-ditch attempt to prevent crashes in
the face of obnoxious bridges or bad hardware. Granted, that should be
a platform concern rather than something that should ever appear in
driver code, but considering drivers open-code readl()/writel() calls
there's not really any place to put that sort of workaround.

That all said, the case in the OP is due to an entirely avoidable
driver bug and that sort of hack is absolutely the wrong thing to do.

Oliver

Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed

[Pali Rohár]

On Monday 19 July 2021 02:31:10 Oliver O'Halloran wrote:
> On Fri, Jul 9, 2021 at 1:45 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> >
> > *snip*
> >
> > Apologies for rehashing what's probably obvious to everybody but me.
> > I'm trying to get a better handle on benign vs poisonous errors.
> >
> > MMIO means CPU reads or writes to the device.  In PCI, writes are
> > posted and don't receive a response, so a driver will never see
> > writel() return an error (although an error may be reported
> > asynchronously via AER or similar).
> >
> > So I think we're mostly talking about CPU reads here.  We expect a PCI
> > response containing the data.  Sometimes there's no response or an
> > error response.  The behavior of the host bridge in these error cases
> > is not defined by PCI, so what the CPU sees is not consistent across
> > platforms.  In some cases, the bridge handles this as a catastrophic
> > error that forces a system restart.
> >
> > But in most cases, at least on x86, the bridge logs an error and
> > fabricates ~0 data so the CPU read can complete.  Then it's up to
> > software to recognize that an error occurred and decide what to do
> > about it.  Is this a benign or a poisonous error?
> >
> > I'd say this is a benign error. It certainly can't be ignored, but as
> > long as the driver recognizes the error, it should be able to deal
> > with it without crashing the whole system and forcing a restart.
> 
> I was thinking more in terms of what the driver author sees rather
> than what's happening on the CPU side. The crash seen in the OP
> appears to be because the code is "doing an MMIO." However, the
> reasons for the crash have nothing to do with the actual mechanics of
> the operation (which should be benign). The point I was making is that
> the pattern of:
> 
> if (is_disconnected())
>     return failure;
> return do_mmio_read(addr);
> 
> does have some utility as a last-ditch attempt to prevent crashes in
> the face of obnoxious bridges or bad hardware. Granted, that should be
> a platform concern rather than something that should ever appear in
> driver code, but considering drivers open-code readl()/writel() calls
> there's not really any place to put that sort of workaround.
> 
> That all said, the case in the OP is due to an entirely avoidable
> driver bug and that sort of hack is absolutely the wrong thing to do.
> 
> Oliver

And do we have some solution for this kind of issue? There are more PCIe
controllers / platforms which do not like MMIO read/write operation when
card / link is not connected.

If we do not provide a way how to solve these problems then we can
expect that people would just hack ethernet / wifi / ... device drivers
which are currently crashing by patches like in this thread.

Maybe PCI subsystem could provide wrapper function which implements
above pattern and which can be used by device drivers?

Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed

[Oliver O'Halloran]

On Mon, Jul 19, 2021 at 8:51 AM Pali Rohár <pali@kernel.org> wrote:
>
> And do we have some solution for this kind of issue? There are more PCIe
> controllers / platforms which do not like MMIO read/write operation when
> card / link is not connected.

Do you have some actual examples? The few times I've seen those
crashes were due to broken firmware-first error handling. The AER
notifications would be escalated into some kind of ACPI error which
the kernel didn't have a good way of dealing with so it panicked
instead.

Assuming it is a real problem then as Bjorn pointed out this sort of
hack doesn't really fix the issue because hotplug and AER
notifications are fundamentally asynchronous. If the driver is
actively using the device when the error / removal happens then the
pci_dev_is_disconnected() check will pass and the MMIO will go
through. If the MMIO is poisonous because of dumb hardware then this
sort of hack will only paper over the issue.

> If we do not provide a way how to solve these problems then we can
> expect that people would just hack ethernet / wifi / ... device drivers
> which are currently crashing by patches like in this thread.
>
> Maybe PCI subsystem could provide wrapper function which implements
> above pattern and which can be used by device drivers?

We could do that and I think there was a proposal to add some
pci_readl(pdev, <addr>) style wrappers at one point. On powerpc
there's hooks in the arch provided MMIO functions to detect error
responses and kick off the error handling machinery when a problem is
detected. Those hooks are mainly there to help the platform detect
errors though and they don't make life much easier for drivers. Due to
locking concerns the driver's .error_detected() callback cannot be
called in the MMIO hook so even when the platform detects errors
synchronously the driver notifications must happen asynchronously. In
the meanwhile the driver still needs to handle the 0xFFs response
safely and there's not much we can do from the platform side to help
there.

Oliver

Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed

[Pali Rohár]

On Monday 19 July 2021 12:49:18 Oliver O'Halloran wrote:
> On Mon, Jul 19, 2021 at 8:51 AM Pali Rohár <pali@kernel.org> wrote:
> >
> > And do we have some solution for this kind of issue? There are more PCIe
> > controllers / platforms which do not like MMIO read/write operation when
> > card / link is not connected.
> 
> Do you have some actual examples? The few times I've seen those
> crashes were due to broken firmware-first error handling. The AER
> notifications would be escalated into some kind of ACPI error which
> the kernel didn't have a good way of dealing with so it panicked
> instead.

I have experience and examples with pci aardvark controller. When card
is disconnected it sends synchronous abort to CPU when doing MMIO read
operation. One example is in this linux-usb thread:

https://lore.kernel.org/linux-usb/20210505120117.4wpmo6fhvzznf3wv@pali/t/#u

I can trigger this issue at least for xhci, nvme and ath drivers.

> Assuming it is a real problem then as Bjorn pointed out this sort of
> hack doesn't really fix the issue because hotplug and AER
> notifications are fundamentally asynchronous.

In case of pci aardvark it is not AER notification. And for MMIO read it
is synchronous abort.

Anyway, hotplug events are really asynchronous, but there is main issue
that this hotplug disconnect event instruct device driver to "unbind"
and e.g. these ethernet or usb controllers try to do MMIO operations in
their cleanup / remove / unbind phase, even when card is already
"disconnected" in PCI subsystem.

> If the driver is
> actively using the device when the error / removal happens then the
> pci_dev_is_disconnected() check will pass and the MMIO will go
> through. If the MMIO is poisonous because of dumb hardware then this
> sort of hack will only paper over the issue.
> 
> > If we do not provide a way how to solve these problems then we can
> > expect that people would just hack ethernet / wifi / ... device drivers
> > which are currently crashing by patches like in this thread.
> >
> > Maybe PCI subsystem could provide wrapper function which implements
> > above pattern and which can be used by device drivers?
> 
> We could do that and I think there was a proposal to add some
> pci_readl(pdev, <addr>) style wrappers at one point. On powerpc
> there's hooks in the arch provided MMIO functions to detect error
> responses and kick off the error handling machinery when a problem is
> detected. Those hooks are mainly there to help the platform detect
> errors though and they don't make life much easier for drivers. Due to
> locking concerns the driver's .error_detected() callback cannot be
> called in the MMIO hook so even when the platform detects errors
> synchronously the driver notifications must happen asynchronously. In
> the meanwhile the driver still needs to handle the 0xFFs response
> safely and there's not much we can do from the platform side to help
> there.
> 
> Oliver

Re: [PATCH 1/2] igc: don't rd/wr iomem when PCI is removed

[Bjorn Helgaas]

On Mon, Jul 19, 2021 at 12:49:18PM +1000, Oliver O'Halloran wrote:
> On Mon, Jul 19, 2021 at 8:51 AM Pali Rohár <pali@kernel.org> wrote:
> >
> > And do we have some solution for this kind of issue? There are more PCIe
> > controllers / platforms which do not like MMIO read/write operation when
> > card / link is not connected.
> 
> Do you have some actual examples? The few times I've seen those
> crashes were due to broken firmware-first error handling. The AER
> notifications would be escalated into some kind of ACPI error which
> the kernel didn't have a good way of dealing with so it panicked
> instead.
> 
> Assuming it is a real problem then as Bjorn pointed out this sort of
> hack doesn't really fix the issue because hotplug and AER
> notifications are fundamentally asynchronous. If the driver is
> actively using the device when the error / removal happens then the
> pci_dev_is_disconnected() check will pass and the MMIO will go
> through. If the MMIO is poisonous because of dumb hardware then this
> sort of hack will only paper over the issue.
> 
> > If we do not provide a way how to solve these problems then we can
> > expect that people would just hack ethernet / wifi / ... device drivers
> > which are currently crashing by patches like in this thread.
> >
> > Maybe PCI subsystem could provide wrapper function which implements
> > above pattern and which can be used by device drivers?
> 
> We could do that and I think there was a proposal to add some
> pci_readl(pdev, <addr>) style wrappers at one point.

Obviously this wouldn't help user-space mmaps, but in the kernel,
Documentation/driver-api/device-io.rst [1] does say that drivers are
supposed to use readl() et al even though on most arches it "works"
to just dereference the result of ioremap(), so maybe we could make
a useful wrapper.

Seems like we should do *something*, even if it's just a generic
#define and some examples.  I took a stab at this [2] a couple years
ago, but it was only for the PCI core, and it didn't go anywhere.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/driver-api/device-io.rst?id=v5.13#n160
[2] https://lore.kernel.org/linux-pci/20190822200551.129039-1-helgaas@kernel.org/

> On powerpc
> there's hooks in the arch provided MMIO functions to detect error
> responses and kick off the error handling machinery when a problem is
> detected. Those hooks are mainly there to help the platform detect
> errors though and they don't make life much easier for drivers. Due to
> locking concerns the driver's .error_detected() callback cannot be
> called in the MMIO hook so even when the platform detects errors
> synchronously the driver notifications must happen asynchronously. In
> the meanwhile the driver still needs to handle the 0xFFs response
> safely and there's not much we can do from the platform side to help
> there.
> 
> Oliver