[PATCH V2 0/2] PCI/portdrv: Fix switch devctrl error report enable

[PATCH V2 0/2] PCI/portdrv: Fix switch devctrl error report enable

[Dongdong Liu]

PATCH 1/2 is to fix switch upstream/downstream port error report.
PATCH 2/2 is to make AER device configuration more reasonable.
Delete the upstream/downstream port AER configuraion in PATCH 2/2,
but if firmware does not enable upstream/downstream port error reporting, 
it seems current no place to enable them, so PATCH 2/2 need more
suggestion. Please see the commit log for more information. 

Changes from v1:
- Rebase on v4.15-rc2.
- Modify the change log.
- Add PATCH 2/2.

Dongdong Liu (2):
  PCI/portdrv: Fix switch devctrl error report enable
  PCI/AER: Fix AER device configuration

 drivers/pci/pcie/aer/aerdrv.c   | 49 ++++-------------------------------------
 drivers/pci/pcie/portdrv_core.c |  4 +++-
 2 files changed, 7 insertions(+), 46 deletions(-)

-- 
1.9.1

[PATCH V2 1/2] PCI/portdrv: Fix switch devctrl error report enable

[Dongdong Liu]

Current code has a bug, switch upstream/downstream port error report
is disabled.
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-

We call aer_probe() for a root port, and it enables AER reporting for
the root port and any downstream devices:

aer_probe(root port)                        # only binds to root ports
  aer_enable_rootport
    set_downstream_devices_error_reporting(root, true)
      set_device_error_reporting(root, true)
        pci_enable_pcie_error_reporting
          pcie_capability_set_word(root, PCI_EXP_DEVCTL, PCI_EXP_AER_FLAGS)
        pci_walk_bus(root->subordinate set_device_error_reporting, true)
          set_device_error_reporting(dev, true)
            pci_enable_pcie_error_reporting
              pcie_capability_set_word(dev, PCI_EXP_DEVCTL,
PCI_EXP_AER_FLAGS)

We later call pcie_portdrv_probe() for every downstream bridge (it
matches PCI_CLASS_BRIDGE_PCI devices, then discards any non-PCIe
devices), and it *disables* AER reporting:

pcie_portdrv_probe(switch port)
  pcie_port_device_register
    get_port_device_capability
      pci_disable_pcie_error_reporting
        pcie_capability_clear_word(dev, PCI_EXP_DEVCTL, PCI_EXP_AER_FLAGS)

The result is that we first enable AER for the downstream switch
ports, then we disable it again.

It does not need to disable AER for upstream/downstream ports as
AER driver only binds to root ports.

Fixes: 2bd50dd800b5(PCI: PCIe: Disable PCIe port services during port
initialization)
Signed-off-by: Dongdong Liu <liudongdong3@huawei.com>
CC: stable@vger.kernel.org
---
 drivers/pci/pcie/portdrv_core.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/pci/pcie/portdrv_core.c b/drivers/pci/pcie/portdrv_core.c
index a592103..a0dff44 100644
--- a/drivers/pci/pcie/portdrv_core.c
+++ b/drivers/pci/pcie/portdrv_core.c
@@ -241,7 +241,9 @@ static int get_port_device_capability(struct pci_dev *dev)
 		 * Disable AER on this port in case it's been enabled by the
 		 * BIOS (the AER service driver will enable it when necessary).
 		 */
-		pci_disable_pcie_error_reporting(dev);
+		if ((pci_pcie_type(dev) != PCI_EXP_TYPE_UPSTREAM) &&
+		    (pci_pcie_type(dev) != PCI_EXP_TYPE_DOWNSTREAM))
+			pci_disable_pcie_error_reporting(dev);
 	}
 	/* VC support */
 	if (pci_find_ext_capability(dev, PCI_EXT_CAP_ID_VC))
-- 
1.9.1

Re: [PATCH V2 1/2] PCI/portdrv: Fix switch devctrl error report enable

[Christoph Hellwig]

On Tue, Dec 05, 2017 at 05:50:37PM +0800, Dongdong Liu wrote:
> +		if ((pci_pcie_type(dev) != PCI_EXP_TYPE_UPSTREAM) &&
> +		    (pci_pcie_type(dev) != PCI_EXP_TYPE_DOWNSTREAM))
> +			pci_disable_pcie_error_reporting(dev);

No need for the inner braces here.

Re: [PATCH V2 1/2] PCI/portdrv: Fix switch devctrl error report enable

[Dongdong Liu]

Hi Christoph
Many thanks for your reveiw.
在 2017/12/6 3:15, Christoph Hellwig 写道:
> On Tue, Dec 05, 2017 at 05:50:37PM +0800, Dongdong Liu wrote:
>> +		if ((pci_pcie_type(dev) != PCI_EXP_TYPE_UPSTREAM) &&
>> +		    (pci_pcie_type(dev) != PCI_EXP_TYPE_DOWNSTREAM))
>> +			pci_disable_pcie_error_reporting(dev);
>
> No need for the inner braces here.
will fix.

Thanks,
Dongdong
>
>

Re: [PATCH V2 1/2] PCI/portdrv: Fix switch devctrl error report enable

[Bjorn Helgaas]

On Tue, Dec 05, 2017 at 05:50:37PM +0800, Dongdong Liu wrote:
> Current code has a bug, switch upstream/downstream port error report
> is disabled.
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
> 
> We call aer_probe() for a root port, and it enables AER reporting for
> the root port and any downstream devices:
> 
> aer_probe(root port)                        # only binds to root ports
>   aer_enable_rootport
>     set_downstream_devices_error_reporting(root, true)
>       set_device_error_reporting(root, true)
>         pci_enable_pcie_error_reporting
>           pcie_capability_set_word(root, PCI_EXP_DEVCTL, PCI_EXP_AER_FLAGS)
>         pci_walk_bus(root->subordinate set_device_error_reporting, true)
>           set_device_error_reporting(dev, true)
>             pci_enable_pcie_error_reporting
>               pcie_capability_set_word(dev, PCI_EXP_DEVCTL,
> PCI_EXP_AER_FLAGS)
> 
> We later call pcie_portdrv_probe() for every downstream bridge (it
> matches PCI_CLASS_BRIDGE_PCI devices, then discards any non-PCIe
> devices), and it *disables* AER reporting:
> 
> pcie_portdrv_probe(switch port)
>   pcie_port_device_register
>     get_port_device_capability
>       pci_disable_pcie_error_reporting
>         pcie_capability_clear_word(dev, PCI_EXP_DEVCTL, PCI_EXP_AER_FLAGS)
> 
> The result is that we first enable AER for the downstream switch
> ports, then we disable it again.
> 
> It does not need to disable AER for upstream/downstream ports as
> AER driver only binds to root ports.
> 
> Fixes: 2bd50dd800b5(PCI: PCIe: Disable PCIe port services during port
> initialization)

While you're correcting nits, use the conventional style here:

  Fixes: 2bd50dd800b5 ("PCI: PCIe: Disable PCIe port services during port initialization")

all on one line for greppability.

> Signed-off-by: Dongdong Liu <liudongdong3@huawei.com>
> CC: stable@vger.kernel.org
> ---
>  drivers/pci/pcie/portdrv_core.c | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/pci/pcie/portdrv_core.c b/drivers/pci/pcie/portdrv_core.c
> index a592103..a0dff44 100644
> --- a/drivers/pci/pcie/portdrv_core.c
> +++ b/drivers/pci/pcie/portdrv_core.c
> @@ -241,7 +241,9 @@ static int get_port_device_capability(struct pci_dev *dev)
>  		 * Disable AER on this port in case it's been enabled by the
>  		 * BIOS (the AER service driver will enable it when necessary).
>  		 */
> -		pci_disable_pcie_error_reporting(dev);
> +		if ((pci_pcie_type(dev) != PCI_EXP_TYPE_UPSTREAM) &&
> +		    (pci_pcie_type(dev) != PCI_EXP_TYPE_DOWNSTREAM))
> +			pci_disable_pcie_error_reporting(dev);

I'm not sure this code is in the right place.  This is
get_port_device_capability(); we should be *getting* information, not
*configuring* the device here.

If we're not prepared to handle AER events, I think it's probably
a good idea to disable them, but I'd rather do it in the
pci_init_capabilities() path, e.g., in pci_aer_init().

pciehp is not a capability, but I think we should also move the
disabling of PCI_EXP_SLTCTL_CCIE | PCI_EXP_SLTCTL_HPIE interrupts out
of get_port_device_capability().  Maybe to pci_configure_device()?

I also do not think we should check for upstream/downstream ports.  If
we're going to disable AER (and I think that probably does make
sense), we should do it for every device until we're ready to handle
AER events.

>  	}
>  	/* VC support */
>  	if (pci_find_ext_capability(dev, PCI_EXT_CAP_ID_VC))
> -- 
> 1.9.1
> 

Re: [PATCH V2 1/2] PCI/portdrv: Fix switch devctrl error report enable

[Dongdong Liu]

Hi Bjorn

Many thanks for your review.
在 2017/12/14 0:29, Bjorn Helgaas 写道:
> On Tue, Dec 05, 2017 at 05:50:37PM +0800, Dongdong Liu wrote:
>> Current code has a bug, switch upstream/downstream port error report
>> is disabled.
>> DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
>>
>> We call aer_probe() for a root port, and it enables AER reporting for
>> the root port and any downstream devices:
>>
>> aer_probe(root port)                        # only binds to root ports
>>   aer_enable_rootport
>>     set_downstream_devices_error_reporting(root, true)
>>       set_device_error_reporting(root, true)
>>         pci_enable_pcie_error_reporting
>>           pcie_capability_set_word(root, PCI_EXP_DEVCTL, PCI_EXP_AER_FLAGS)
>>         pci_walk_bus(root->subordinate set_device_error_reporting, true)
>>           set_device_error_reporting(dev, true)
>>             pci_enable_pcie_error_reporting
>>               pcie_capability_set_word(dev, PCI_EXP_DEVCTL,
>> PCI_EXP_AER_FLAGS)
>>
>> We later call pcie_portdrv_probe() for every downstream bridge (it
>> matches PCI_CLASS_BRIDGE_PCI devices, then discards any non-PCIe
>> devices), and it *disables* AER reporting:
>>
>> pcie_portdrv_probe(switch port)
>>   pcie_port_device_register
>>     get_port_device_capability
>>       pci_disable_pcie_error_reporting
>>         pcie_capability_clear_word(dev, PCI_EXP_DEVCTL, PCI_EXP_AER_FLAGS)
>>
>> The result is that we first enable AER for the downstream switch
>> ports, then we disable it again.
>>
>> It does not need to disable AER for upstream/downstream ports as
>> AER driver only binds to root ports.
>>
>> Fixes: 2bd50dd800b5(PCI: PCIe: Disable PCIe port services during port
>> initialization)
>
> While you're correcting nits, use the conventional style here:
>
>   Fixes: 2bd50dd800b5 ("PCI: PCIe: Disable PCIe port services during port initialization")
>
> all on one line for greppability.
Thanks for pointing out that, will fix.
>
>> Signed-off-by: Dongdong Liu <liudongdong3@huawei.com>
>> CC: stable@vger.kernel.org
>> ---
>>  drivers/pci/pcie/portdrv_core.c | 4 +++-
>>  1 file changed, 3 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/pci/pcie/portdrv_core.c b/drivers/pci/pcie/portdrv_core.c
>> index a592103..a0dff44 100644
>> --- a/drivers/pci/pcie/portdrv_core.c
>> +++ b/drivers/pci/pcie/portdrv_core.c
>> @@ -241,7 +241,9 @@ static int get_port_device_capability(struct pci_dev *dev)
>>  		 * Disable AER on this port in case it's been enabled by the
>>  		 * BIOS (the AER service driver will enable it when necessary).
>>  		 */
>> -		pci_disable_pcie_error_reporting(dev);
>> +		if ((pci_pcie_type(dev) != PCI_EXP_TYPE_UPSTREAM) &&
>> +		    (pci_pcie_type(dev) != PCI_EXP_TYPE_DOWNSTREAM))
>> +			pci_disable_pcie_error_reporting(dev);
>
> I'm not sure this code is in the right place.  This is
> get_port_device_capability(); we should be *getting* information, not
> *configuring* the device here.
>
> If we're not prepared to handle AER events, I think it's probably
> a good idea to disable them, but I'd rather do it in the
> pci_init_capabilities() path, e.g., in pci_aer_init().
So disable them in pci_aer_init(), enable them in aer_enable_rootport().
>
> pciehp is not a capability, but I think we should also move the
> disabling of PCI_EXP_SLTCTL_CCIE | PCI_EXP_SLTCTL_HPIE interrupts out
> of get_port_device_capability().  Maybe to pci_configure_device()?
>
> I also do not think we should check for upstream/downstream ports.  If
> we're going to disable AER (and I think that probably does make
> sense), we should do it for every device until we're ready to handle
> AER events.

It seems good to me.

Thanks,
Dongdong

>
>>  	}
>>  	/* VC support */
>>  	if (pci_find_ext_capability(dev, PCI_EXT_CAP_ID_VC))
>> --
>> 1.9.1
>>
>
> .
>

[PATCH V2 2/2] PCI/AER: Fix AER device configuration

[Dongdong Liu]

AER driver only binds to root ports. It binds to one device and it also
configures other downstream devices.  That opens the door to concurrency
issues and makes it really hard to ensure that hotplug works correctly.
The aer_probe() path should only touch the device it is binding, it
should not use pci_walk_bus().  If we need to configure another device,
that should be done in the enumeration path for *that device*.
We can use  _HPX to set PCI_EXP_DEVCTL to enable error report and ensure
that hotplug works correctly.  For more_HPX details information,
we can see ACPI 6.1 section 6.2.9 _HPX (Hot Plug Parameter Extensions).

Signed-off-by: Dongdong Liu <liudongdong3@huawei.com>
---
 drivers/pci/pcie/aer/aerdrv.c | 49 ++++---------------------------------------
 1 file changed, 4 insertions(+), 45 deletions(-)

diff --git a/drivers/pci/pcie/aer/aerdrv.c b/drivers/pci/pcie/aer/aerdrv.c
index 6ff5f5b..cd63025 100644
--- a/drivers/pci/pcie/aer/aerdrv.c
+++ b/drivers/pci/pcie/aer/aerdrv.c
@@ -58,41 +58,6 @@ bool pci_aer_available(void)
 	return !pcie_aer_disable && pci_msi_enabled();
 }
 
-static int set_device_error_reporting(struct pci_dev *dev, void *data)
-{
-	bool enable = *((bool *)data);
-	int type = pci_pcie_type(dev);
-
-	if ((type == PCI_EXP_TYPE_ROOT_PORT) ||
-	    (type == PCI_EXP_TYPE_UPSTREAM) ||
-	    (type == PCI_EXP_TYPE_DOWNSTREAM)) {
-		if (enable)
-			pci_enable_pcie_error_reporting(dev);
-		else
-			pci_disable_pcie_error_reporting(dev);
-	}
-
-	if (enable)
-		pcie_set_ecrc_checking(dev);
-
-	return 0;
-}
-
-/**
- * set_downstream_devices_error_reporting - enable/disable the error reporting  bits on the root port and its downstream ports.
- * @dev: pointer to root port's pci_dev data structure
- * @enable: true = enable error reporting, false = disable error reporting.
- */
-static void set_downstream_devices_error_reporting(struct pci_dev *dev,
-						   bool enable)
-{
-	set_device_error_reporting(dev, &enable);
-
-	if (!dev->subordinate)
-		return;
-	pci_walk_bus(dev->subordinate, set_device_error_reporting, &enable);
-}
-
 /**
  * aer_enable_rootport - enable Root Port's interrupts when receiving messages
  * @rpc: pointer to a Root Port data structure
@@ -123,11 +88,8 @@ static void aer_enable_rootport(struct aer_rpc *rpc)
 	pci_read_config_dword(pdev, aer_pos + PCI_ERR_UNCOR_STATUS, &reg32);
 	pci_write_config_dword(pdev, aer_pos + PCI_ERR_UNCOR_STATUS, reg32);
 
-	/*
-	 * Enable error reporting for the root port device and downstream port
-	 * devices.
-	 */
-	set_downstream_devices_error_reporting(pdev, true);
+	/* Enable error reporting for the root port device */
+	pci_enable_pcie_error_reporting(pdev);
 
 	/* Enable Root Port's interrupt in response to error messages */
 	pci_read_config_dword(pdev, aer_pos + PCI_ERR_ROOT_COMMAND, &reg32);
@@ -147,11 +109,8 @@ static void aer_disable_rootport(struct aer_rpc *rpc)
 	u32 reg32;
 	int pos;
 
-	/*
-	 * Disable error reporting for the root port device and downstream port
-	 * devices.
-	 */
-	set_downstream_devices_error_reporting(pdev, false);
+	/* Disable error reporting for the root port device */
+	pci_disable_pcie_error_reporting(pdev);
 
 	pos = pdev->aer_cap;
 	/* Disable Root's interrupt in response to error messages */
-- 
1.9.1

Re: [PATCH V2 2/2] PCI/AER: Fix AER device configuration

[Bjorn Helgaas]

On Tue, Dec 05, 2017 at 05:50:38PM +0800, Dongdong Liu wrote:
> AER driver only binds to root ports. It binds to one device and it also
> configures other downstream devices.  That opens the door to concurrency
> issues and makes it really hard to ensure that hotplug works correctly.
> The aer_probe() path should only touch the device it is binding, it
> should not use pci_walk_bus().  If we need to configure another device,
> that should be done in the enumeration path for *that device*.
> We can use  _HPX to set PCI_EXP_DEVCTL to enable error report and ensure
> that hotplug works correctly.  For more_HPX details information,
> we can see ACPI 6.1 section 6.2.9 _HPX (Hot Plug Parameter Extensions).

_HPX is for platform-dependent things.  If Linux has generic AER
support, i.e., if CONFIG_PCIEAER=y, we should not rely on _HPX to
enable AER for hot-added devices.

We need some mechanism in Linux for enabling AER on them.  I don't
*like* the pci_walk_bus(), but we might need it for now because we
have this ordering:

  1) Enumerate hierarchy, disabling AER on all devices.  Currently I
  think we only disable AER for Ports, but I propose doing it for all
  devices.

  2) Install AER driver on Root Ports.  Enable AER on Root Port.  We
  also need to enable AER on the hierarchy below the Root Port, and
  pci_walk_bus() seems like the logical way to do it for now.

  3) Hot-add a device.  Currently I think AER will remain disabled on
  the new device *unless* _HPX enables it.  I think this is wrong --
  Linux should not rely on the platform for this.  We could enable AER
  via a device-add notifier, but that seems overly complicated.

  I'd rather add a bit in pci_dev like "aer_hierarchy" that is set
  whenever we turn on AER for the device.  Then pci_aer_init() could
  enable AER if it is enabled in the upstream device.

Since the AER driver is installed after the whole hierarchy is
enumerated, we can't use the "aer_hierarchy" bit (or whatever we call
it) to enable AER on the devices present at boot.

I would ultimately like to enable AER on the Root Ports in
pci_aer_init() during enumeration instead of installing it as a driver
after enumeration.  If we could ever do that, then the aer_hierarchy
bit would work the same way for boot-time and hot-added devices, and
we wouldn't need to do the pci_walk_bus() thing.

> Signed-off-by: Dongdong Liu <liudongdong3@huawei.com>
> ---
>  drivers/pci/pcie/aer/aerdrv.c | 49 ++++---------------------------------------
>  1 file changed, 4 insertions(+), 45 deletions(-)
> 
> diff --git a/drivers/pci/pcie/aer/aerdrv.c b/drivers/pci/pcie/aer/aerdrv.c
> index 6ff5f5b..cd63025 100644
> --- a/drivers/pci/pcie/aer/aerdrv.c
> +++ b/drivers/pci/pcie/aer/aerdrv.c
> @@ -58,41 +58,6 @@ bool pci_aer_available(void)
>  	return !pcie_aer_disable && pci_msi_enabled();
>  }
>  
> -static int set_device_error_reporting(struct pci_dev *dev, void *data)
> -{
> -	bool enable = *((bool *)data);
> -	int type = pci_pcie_type(dev);
> -
> -	if ((type == PCI_EXP_TYPE_ROOT_PORT) ||
> -	    (type == PCI_EXP_TYPE_UPSTREAM) ||
> -	    (type == PCI_EXP_TYPE_DOWNSTREAM)) {
> -		if (enable)
> -			pci_enable_pcie_error_reporting(dev);
> -		else
> -			pci_disable_pcie_error_reporting(dev);
> -	}
> -
> -	if (enable)
> -		pcie_set_ecrc_checking(dev);
> -
> -	return 0;
> -}
> -
> -/**
> - * set_downstream_devices_error_reporting - enable/disable the error reporting  bits on the root port and its downstream ports.
> - * @dev: pointer to root port's pci_dev data structure
> - * @enable: true = enable error reporting, false = disable error reporting.
> - */
> -static void set_downstream_devices_error_reporting(struct pci_dev *dev,
> -						   bool enable)
> -{
> -	set_device_error_reporting(dev, &enable);
> -
> -	if (!dev->subordinate)
> -		return;
> -	pci_walk_bus(dev->subordinate, set_device_error_reporting, &enable);
> -}
> -
>  /**
>   * aer_enable_rootport - enable Root Port's interrupts when receiving messages
>   * @rpc: pointer to a Root Port data structure
> @@ -123,11 +88,8 @@ static void aer_enable_rootport(struct aer_rpc *rpc)
>  	pci_read_config_dword(pdev, aer_pos + PCI_ERR_UNCOR_STATUS, &reg32);
>  	pci_write_config_dword(pdev, aer_pos + PCI_ERR_UNCOR_STATUS, reg32);
>  
> -	/*
> -	 * Enable error reporting for the root port device and downstream port
> -	 * devices.
> -	 */
> -	set_downstream_devices_error_reporting(pdev, true);
> +	/* Enable error reporting for the root port device */
> +	pci_enable_pcie_error_reporting(pdev);
>  
>  	/* Enable Root Port's interrupt in response to error messages */
>  	pci_read_config_dword(pdev, aer_pos + PCI_ERR_ROOT_COMMAND, &reg32);
> @@ -147,11 +109,8 @@ static void aer_disable_rootport(struct aer_rpc *rpc)
>  	u32 reg32;
>  	int pos;
>  
> -	/*
> -	 * Disable error reporting for the root port device and downstream port
> -	 * devices.
> -	 */
> -	set_downstream_devices_error_reporting(pdev, false);
> +	/* Disable error reporting for the root port device */
> +	pci_disable_pcie_error_reporting(pdev);
>  
>  	pos = pdev->aer_cap;
>  	/* Disable Root's interrupt in response to error messages */
> -- 
> 1.9.1
> 

Re: [PATCH V2 2/2] PCI/AER: Fix AER device configuration

[Dongdong Liu]

在 2017/12/14 0:55, Bjorn Helgaas 写道:
> On Tue, Dec 05, 2017 at 05:50:38PM +0800, Dongdong Liu wrote:
>> AER driver only binds to root ports. It binds to one device and it also
>> configures other downstream devices.  That opens the door to concurrency
>> issues and makes it really hard to ensure that hotplug works correctly.
>> The aer_probe() path should only touch the device it is binding, it
>> should not use pci_walk_bus().  If we need to configure another device,
>> that should be done in the enumeration path for *that device*.
>> We can use  _HPX to set PCI_EXP_DEVCTL to enable error report and ensure
>> that hotplug works correctly.  For more_HPX details information,
>> we can see ACPI 6.1 section 6.2.9 _HPX (Hot Plug Parameter Extensions).
>
> _HPX is for platform-dependent things.  If Linux has generic AER
> support, i.e., if CONFIG_PCIEAER=y, we should not rely on _HPX to
> enable AER for hot-added devices.
>
> We need some mechanism in Linux for enabling AER on them.  I don't
> *like* the pci_walk_bus(), but we might need it for now because we
> have this ordering:
>
>   1) Enumerate hierarchy, disabling AER on all devices.  Currently I
>   think we only disable AER for Ports, but I propose doing it for all
>   devices.
>
>   2) Install AER driver on Root Ports.  Enable AER on Root Port.  We
>   also need to enable AER on the hierarchy below the Root Port, and
>   pci_walk_bus() seems like the logical way to do it for now.
>
>   3) Hot-add a device.  Currently I think AER will remain disabled on
>   the new device *unless* _HPX enables it.  I think this is wrong --
Yes, if the EP device driver does not call pci_enable_pcie_error_reporting(),
AER will remain diables  *unless* _HPX enables it.

>   Linux should not rely on the platform for this.  We could enable AER
>   via a device-add notifier, but that seems overly complicated.
>
>   I'd rather add a bit in pci_dev like "aer_hierarchy" that is set
>   whenever we turn on AER for the device.  Then pci_aer_init() could
>   enable AER if it is enabled in the upstream device.
It maybe just check the root port devices as AER driver binds to
Root Ports.
>
> Since the AER driver is installed after the whole hierarchy is
> enumerated, we can't use the "aer_hierarchy" bit (or whatever we call
> it) to enable AER on the devices present at boot.
>
> I would ultimately like to enable AER on the Root Ports in
> pci_aer_init() during enumeration instead of installing it as a driver
> after enumeration.  If we could ever do that, then the aer_hierarchy
> bit would work the same way for boot-time and hot-added devices, and
> we wouldn't need to do the pci_walk_bus() thing.

Please correct me if i am wrong.
For boot time. Disable AER in pci_aer_init().
Enable AER for all devices in aer_enable_rootport(), and set "aer_hierarchy"
bit just for Root Ports. It seems we have to use pci_walk_bus() to
enable all devices AER.
For Hot-add a device. Check it's root port "aer_hierarchy" bit in
pci_aer_init(). if the bit is set, enable the device AER in pci_aer_init().

Or as you said above. Enable AER on the Root Ports in pci_aer_init()
during enumeration and set the Root Ports "aer_hierarchy".
Then enable other deivces AER in pci_aer_init(） if it's root port
"aer_hierarchy" bit has been set. This seems that we would not
use pci_walk_bus() and disble/enable AER during enumeration,
the AER driver and EP deivces driver do not need to call
pci_enable_pcie_error_reporting()/pci_disable_pcie_error_reporting().

Thanks,
Dongdong

>
>> Signed-off-by: Dongdong Liu <liudongdong3@huawei.com>
>> ---
>>  drivers/pci/pcie/aer/aerdrv.c | 49 ++++---------------------------------------
>>  1 file changed, 4 insertions(+), 45 deletions(-)
>>
>> diff --git a/drivers/pci/pcie/aer/aerdrv.c b/drivers/pci/pcie/aer/aerdrv.c
>> index 6ff5f5b..cd63025 100644
>> --- a/drivers/pci/pcie/aer/aerdrv.c
>> +++ b/drivers/pci/pcie/aer/aerdrv.c
>> @@ -58,41 +58,6 @@ bool pci_aer_available(void)
>>  	return !pcie_aer_disable && pci_msi_enabled();
>>  }
>>
>> -static int set_device_error_reporting(struct pci_dev *dev, void *data)
>> -{
>> -	bool enable = *((bool *)data);
>> -	int type = pci_pcie_type(dev);
>> -
>> -	if ((type == PCI_EXP_TYPE_ROOT_PORT) ||
>> -	    (type == PCI_EXP_TYPE_UPSTREAM) ||
>> -	    (type == PCI_EXP_TYPE_DOWNSTREAM)) {
>> -		if (enable)
>> -			pci_enable_pcie_error_reporting(dev);
>> -		else
>> -			pci_disable_pcie_error_reporting(dev);
>> -	}
>> -
>> -	if (enable)
>> -		pcie_set_ecrc_checking(dev);
>> -
>> -	return 0;
>> -}
>> -
>> -/**
>> - * set_downstream_devices_error_reporting - enable/disable the error reporting  bits on the root port and its downstream ports.
>> - * @dev: pointer to root port's pci_dev data structure
>> - * @enable: true = enable error reporting, false = disable error reporting.
>> - */
>> -static void set_downstream_devices_error_reporting(struct pci_dev *dev,
>> -						   bool enable)
>> -{
>> -	set_device_error_reporting(dev, &enable);
>> -
>> -	if (!dev->subordinate)
>> -		return;
>> -	pci_walk_bus(dev->subordinate, set_device_error_reporting, &enable);
>> -}
>> -
>>  /**
>>   * aer_enable_rootport - enable Root Port's interrupts when receiving messages
>>   * @rpc: pointer to a Root Port data structure
>> @@ -123,11 +88,8 @@ static void aer_enable_rootport(struct aer_rpc *rpc)
>>  	pci_read_config_dword(pdev, aer_pos + PCI_ERR_UNCOR_STATUS, &reg32);
>>  	pci_write_config_dword(pdev, aer_pos + PCI_ERR_UNCOR_STATUS, reg32);
>>
>> -	/*
>> -	 * Enable error reporting for the root port device and downstream port
>> -	 * devices.
>> -	 */
>> -	set_downstream_devices_error_reporting(pdev, true);
>> +	/* Enable error reporting for the root port device */
>> +	pci_enable_pcie_error_reporting(pdev);
>>
>>  	/* Enable Root Port's interrupt in response to error messages */
>>  	pci_read_config_dword(pdev, aer_pos + PCI_ERR_ROOT_COMMAND, &reg32);
>> @@ -147,11 +109,8 @@ static void aer_disable_rootport(struct aer_rpc *rpc)
>>  	u32 reg32;
>>  	int pos;
>>
>> -	/*
>> -	 * Disable error reporting for the root port device and downstream port
>> -	 * devices.
>> -	 */
>> -	set_downstream_devices_error_reporting(pdev, false);
>> +	/* Disable error reporting for the root port device */
>> +	pci_disable_pcie_error_reporting(pdev);
>>
>>  	pos = pdev->aer_cap;
>>  	/* Disable Root's interrupt in response to error messages */
>> --
>> 1.9.1
>>
>
> .
>