[PATCH v2 0/2] Fully enable AER

[PATCH v2 0/2] Fully enable AER

[Stefan Roese]

While working on AER support on a ZynqMP based system, which has some
PCIe Device connected via a PCIe switch, problems with AER enabling in
the Device Control registers of all PCIe devices but the Root Port. In
fact, only the Root Port has AER enabled right now. This patch set now
fixes this problem by first fixing the AER enabing in the
interconnected PCIe switches between the Root Port and the PCIe
devices and in a 2nd patch, also enabling AER in the PCIe Endpoints.

Please note that these changes are quite invasie, as with these patches
applied, AER now will be enabled in the Device Control registers of all
available PCIe Endpoints, which currently is not the case.

Stefan Roese (2):
  PCI/portdrv: Don't disable AER reporting in
    get_port_device_capability()
  PCI/AER: Enable AER on Endpoints as well

 drivers/pci/pcie/aer.c          | 3 ++-
 drivers/pci/pcie/portdrv_core.c | 9 +--------
 2 files changed, 3 insertions(+), 9 deletions(-)

-- 
2.34.1

[PATCH v2 1/2] PCI/portdrv: Don't disable AER reporting in get_port_device_capability()

[Stefan Roese]

Testing has shown, that AER reporting is currently disabled in the
DevCtl registers of all non Root Port PCIe devices on systems using
pcie_ports_native || host->native_aer. Practically disabling AER
completely in such systems. This is due to the fact that with commit
2bd50dd800b5 ("PCI: PCIe: Disable PCIe port services during port
initialization"), a call to pci_disable_pcie_error_reporting() was
added *after* the PCIe AER setup was completed for the PCIe device
tree.

Here a longer analysis about the currect status of AER enaling /
disabling upon bootup provided by Bjorn:

  pcie_portdrv_probe
    pcie_port_device_register
      get_port_device_capability
        pci_disable_pcie_error_reporting
          clear CERE NFERE FERE URRE               # <-- disable for RP USP DSP
      pcie_device_init
        device_register                            # new AER service device
          aer_probe
            aer_enable_rootport                    # RP only
              set_downstream_devices_error_reporting
                set_device_error_reporting         # self (RP)
                  if (RP || USP || DSP)
                    pci_enable_pcie_error_reporting
                      set CERE NFERE FERE URRE     # <-- enable for RP
                pci_walk_bus
                  set_device_error_reporting
                    if (RP || USP || DSP)
                      pci_enable_pcie_error_reporting
                        set CERE NFERE FERE URRE   # <-- enable for USP DSP

In a typical Root Port -> Endpoint hierarchy, the above:
  - Disables Error Reporting for the Root Port,
  - Enables Error Reporting for the Root Port,
  - Does NOT enable Error Reporting for the Endpoint because it is not
    a Root Port or Switch Port.

In a deeper Root Port -> Upstream Switch Port -> Downstream Switch
Port -> Endpoint hierarchy:
  - Disables Error Reporting for the Root Port,
  - Enables Error Reporting for the Root Port,
  - Enables Error Reporting for both Switch Ports,
  - Does NOT enable Error Reporting for the Endpoint because it is not
    a Root Port or Switch Port,
  - Disables Error Reporting for the Switch Ports when
    pcie_portdrv_probe() claims them.  AER does not re-enable it
    because these are not Root Ports.

This patch now removes this call to pci_disable_pcie_error_reporting()
from get_port_device_capability(), leaving the already enabled AER
configuration intact. With this change, AER is enabled in the Root Port
and the PCIe switch upstream and downstream ports. Only the PCIe
Endpoints don't have AER enabled yet. A follow-up patch will take
care of this Endpoint enabling.

Fixes: 2bd50dd800b5 ("PCI: PCIe: Disable PCIe port services during port initialization")
Signed-off-by: Stefan Roese <sr@denx.de>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Bjorn Helgaas <helgaas@kernel.org>
Cc: Pali Rohár <pali@kernel.org>
Cc: Bharat Kumar Gogada <bharat.kumar.gogada@xilinx.com>
Cc: Michal Simek <michal.simek@xilinx.com>
Cc: Yao Hongbo <yaohongbo@linux.alibaba.com>
Cc: Naveen Naidu <naveennaidu479@gmail.com>
---
v2:
- Enhance commit message as suggested by Bjorn

 drivers/pci/pcie/portdrv_core.c | 9 +--------
 1 file changed, 1 insertion(+), 8 deletions(-)

diff --git a/drivers/pci/pcie/portdrv_core.c b/drivers/pci/pcie/portdrv_core.c
index f81c7be4d7d8..27b990cedb4c 100644
--- a/drivers/pci/pcie/portdrv_core.c
+++ b/drivers/pci/pcie/portdrv_core.c
@@ -244,15 +244,8 @@ static int get_port_device_capability(struct pci_dev *dev)
 
 #ifdef CONFIG_PCIEAER
 	if (dev->aer_cap && pci_aer_available() &&
-	    (pcie_ports_native || host->native_aer)) {
+	    (pcie_ports_native || host->native_aer))
 		services |= PCIE_PORT_SERVICE_AER;
-
-		/*
-		 * Disable AER on this port in case it's been enabled by the
-		 * BIOS (the AER service driver will enable it when necessary).
-		 */
-		pci_disable_pcie_error_reporting(dev);
-	}
 #endif
 
 	/* Root Ports and Root Complex Event Collectors may generate PMEs */
-- 
2.34.1

[PATCH v2 2/2] PCI/AER: Enable AER on Endpoints as well

[Stefan Roese]

Currently, the PCIe AER subsystem does not enable AER in the PCIe
Endpoints via the Device Control register. It's only done for the
Root Port and all PCIe Ports in between the Root Port and the
Endpoint(s). Some device drivers enable AER in their PCIe device by
directly calling pci_enable_pcie_error_reporting(). But in most
cases, AER is currently disabled in the PCIe Endpoints.

This patch enables AER on PCIe Endpoints now as well in
set_device_error_reporting(). This will make the ad-hoc calls to
pci_enable_pcie_error_reporting() superfluous.

Signed-off-by: Stefan Roese <sr@denx.de>
Cc: Bjorn Helgaas <helgaas@kernel.org>
Cc: Pali Rohár <pali@kernel.org>
Cc: Bharat Kumar Gogada <bharat.kumar.gogada@xilinx.com>
Cc: Michal Simek <michal.simek@xilinx.com>
Cc: Yao Hongbo <yaohongbo@linux.alibaba.com>
Cc: Naveen Naidu <naveennaidu479@gmail.com>
---
v2:
- New patch

 drivers/pci/pcie/aer.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/pci/pcie/aer.c b/drivers/pci/pcie/aer.c
index 9fa1f97e5b27..385e2033d7b5 100644
--- a/drivers/pci/pcie/aer.c
+++ b/drivers/pci/pcie/aer.c
@@ -1216,7 +1216,8 @@ static int set_device_error_reporting(struct pci_dev *dev, void *data)
 	if ((type == PCI_EXP_TYPE_ROOT_PORT) ||
 	    (type == PCI_EXP_TYPE_RC_EC) ||
 	    (type == PCI_EXP_TYPE_UPSTREAM) ||
-	    (type == PCI_EXP_TYPE_DOWNSTREAM)) {
+	    (type == PCI_EXP_TYPE_DOWNSTREAM) ||
+	    (type == PCI_EXP_TYPE_ENDPOINT)) {
 		if (enable)
 			pci_enable_pcie_error_reporting(dev);
 		else
-- 
2.34.1

Re: [PATCH v2 1/2] PCI/portdrv: Don't disable AER reporting in get_port_device_capability()

[Pali Rohár]

On Monday 17 January 2022 09:03:47 Stefan Roese wrote:
> Testing has shown, that AER reporting is currently disabled in the
> DevCtl registers of all non Root Port PCIe devices on systems using
> pcie_ports_native || host->native_aer. Practically disabling AER
> completely in such systems. This is due to the fact that with commit
> 2bd50dd800b5 ("PCI: PCIe: Disable PCIe port services during port
> initialization"), a call to pci_disable_pcie_error_reporting() was
> added *after* the PCIe AER setup was completed for the PCIe device
> tree.
> 
> Here a longer analysis about the currect status of AER enaling /
> disabling upon bootup provided by Bjorn:
> 
>   pcie_portdrv_probe
>     pcie_port_device_register
>       get_port_device_capability
>         pci_disable_pcie_error_reporting
>           clear CERE NFERE FERE URRE               # <-- disable for RP USP DSP
>       pcie_device_init
>         device_register                            # new AER service device
>           aer_probe
>             aer_enable_rootport                    # RP only
>               set_downstream_devices_error_reporting
>                 set_device_error_reporting         # self (RP)
>                   if (RP || USP || DSP)
>                     pci_enable_pcie_error_reporting
>                       set CERE NFERE FERE URRE     # <-- enable for RP
>                 pci_walk_bus
>                   set_device_error_reporting
>                     if (RP || USP || DSP)
>                       pci_enable_pcie_error_reporting
>                         set CERE NFERE FERE URRE   # <-- enable for USP DSP
> 
> In a typical Root Port -> Endpoint hierarchy, the above:
>   - Disables Error Reporting for the Root Port,
>   - Enables Error Reporting for the Root Port,
>   - Does NOT enable Error Reporting for the Endpoint because it is not
>     a Root Port or Switch Port.
> 
> In a deeper Root Port -> Upstream Switch Port -> Downstream Switch
> Port -> Endpoint hierarchy:
>   - Disables Error Reporting for the Root Port,
>   - Enables Error Reporting for the Root Port,
>   - Enables Error Reporting for both Switch Ports,
>   - Does NOT enable Error Reporting for the Endpoint because it is not
>     a Root Port or Switch Port,
>   - Disables Error Reporting for the Switch Ports when
>     pcie_portdrv_probe() claims them.  AER does not re-enable it
>     because these are not Root Ports.
> 
> This patch now removes this call to pci_disable_pcie_error_reporting()
> from get_port_device_capability(), leaving the already enabled AER
> configuration intact. With this change, AER is enabled in the Root Port
> and the PCIe switch upstream and downstream ports. Only the PCIe
> Endpoints don't have AER enabled yet. A follow-up patch will take
> care of this Endpoint enabling.
> 
> Fixes: 2bd50dd800b5 ("PCI: PCIe: Disable PCIe port services during port initialization")
> Signed-off-by: Stefan Roese <sr@denx.de>
> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> Cc: Bjorn Helgaas <helgaas@kernel.org>
> Cc: Pali Rohár <pali@kernel.org>
> Cc: Bharat Kumar Gogada <bharat.kumar.gogada@xilinx.com>
> Cc: Michal Simek <michal.simek@xilinx.com>
> Cc: Yao Hongbo <yaohongbo@linux.alibaba.com>
> Cc: Naveen Naidu <naveennaidu479@gmail.com>

Reviewed-by: Pali Rohár <pali@kernel.org>

> ---
> v2:
> - Enhance commit message as suggested by Bjorn
> 
>  drivers/pci/pcie/portdrv_core.c | 9 +--------
>  1 file changed, 1 insertion(+), 8 deletions(-)
> 
> diff --git a/drivers/pci/pcie/portdrv_core.c b/drivers/pci/pcie/portdrv_core.c
> index f81c7be4d7d8..27b990cedb4c 100644
> --- a/drivers/pci/pcie/portdrv_core.c
> +++ b/drivers/pci/pcie/portdrv_core.c
> @@ -244,15 +244,8 @@ static int get_port_device_capability(struct pci_dev *dev)
>  
>  #ifdef CONFIG_PCIEAER
>  	if (dev->aer_cap && pci_aer_available() &&
> -	    (pcie_ports_native || host->native_aer)) {
> +	    (pcie_ports_native || host->native_aer))
>  		services |= PCIE_PORT_SERVICE_AER;
> -
> -		/*
> -		 * Disable AER on this port in case it's been enabled by the
> -		 * BIOS (the AER service driver will enable it when necessary).
> -		 */
> -		pci_disable_pcie_error_reporting(dev);
> -	}
>  #endif
>  
>  	/* Root Ports and Root Complex Event Collectors may generate PMEs */
> -- 
> 2.34.1
> 

Re: [PATCH v2 2/2] PCI/AER: Enable AER on Endpoints as well

[Pali Rohár]

On Monday 17 January 2022 09:03:48 Stefan Roese wrote:
> Currently, the PCIe AER subsystem does not enable AER in the PCIe
> Endpoints via the Device Control register. It's only done for the
> Root Port and all PCIe Ports in between the Root Port and the
> Endpoint(s). Some device drivers enable AER in their PCIe device by
> directly calling pci_enable_pcie_error_reporting(). But in most
> cases, AER is currently disabled in the PCIe Endpoints.
> 
> This patch enables AER on PCIe Endpoints now as well in
> set_device_error_reporting(). This will make the ad-hoc calls to
> pci_enable_pcie_error_reporting() superfluous.
> 
> Signed-off-by: Stefan Roese <sr@denx.de>
> Cc: Bjorn Helgaas <helgaas@kernel.org>
> Cc: Pali Rohár <pali@kernel.org>
> Cc: Bharat Kumar Gogada <bharat.kumar.gogada@xilinx.com>
> Cc: Michal Simek <michal.simek@xilinx.com>
> Cc: Yao Hongbo <yaohongbo@linux.alibaba.com>
> Cc: Naveen Naidu <naveennaidu479@gmail.com>

Reviewed-by: Pali Rohár <pali@kernel.org>

> ---
> v2:
> - New patch
> 
>  drivers/pci/pcie/aer.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/pci/pcie/aer.c b/drivers/pci/pcie/aer.c
> index 9fa1f97e5b27..385e2033d7b5 100644
> --- a/drivers/pci/pcie/aer.c
> +++ b/drivers/pci/pcie/aer.c
> @@ -1216,7 +1216,8 @@ static int set_device_error_reporting(struct pci_dev *dev, void *data)
>  	if ((type == PCI_EXP_TYPE_ROOT_PORT) ||
>  	    (type == PCI_EXP_TYPE_RC_EC) ||
>  	    (type == PCI_EXP_TYPE_UPSTREAM) ||
> -	    (type == PCI_EXP_TYPE_DOWNSTREAM)) {
> +	    (type == PCI_EXP_TYPE_DOWNSTREAM) ||
> +	    (type == PCI_EXP_TYPE_ENDPOINT)) {

Hm... maybe another question to discussion: Why enabling of AER is
limited just to above PCIe port types? Why we do not want to enable it
for _all_ PCIe devices? Currently in the above list are missing Legacy
endpoints (which probably do not support AER and so do not have AER
capability in config space), Root Complex Integrated Endpoints (these
should provide AER supports too, right?), PCIe to PCI/X Bridges (these
may generate its own AER errors) and PCI to PCIe Bridges (these are
maybe complicated as subtree behind such bridges are regular PCIe
devices and so could fully support AER but on legacy PCI bus there is
probably no access to extended config space where is AER). But in all of
these cases, are there any issues with enabling AER via function
pci_enable_pcie_error_reporting()? For me it looks like that in the
worst case dev just does not have AER capability in config space or
extended config space is not accessible (which is same as no AER
capability).

>  		if (enable)
>  			pci_enable_pcie_error_reporting(dev);
>  		else
> -- 
> 2.34.1
> 

Re: [PATCH v2 2/2] PCI/AER: Enable AER on Endpoints as well

[Stefan Roese]

On 1/17/22 10:30, Pali Rohár wrote:
> On Monday 17 January 2022 09:03:48 Stefan Roese wrote:
>> Currently, the PCIe AER subsystem does not enable AER in the PCIe
>> Endpoints via the Device Control register. It's only done for the
>> Root Port and all PCIe Ports in between the Root Port and the
>> Endpoint(s). Some device drivers enable AER in their PCIe device by
>> directly calling pci_enable_pcie_error_reporting(). But in most
>> cases, AER is currently disabled in the PCIe Endpoints.
>>
>> This patch enables AER on PCIe Endpoints now as well in
>> set_device_error_reporting(). This will make the ad-hoc calls to
>> pci_enable_pcie_error_reporting() superfluous.
>>
>> Signed-off-by: Stefan Roese <sr@denx.de>
>> Cc: Bjorn Helgaas <helgaas@kernel.org>
>> Cc: Pali Rohár <pali@kernel.org>
>> Cc: Bharat Kumar Gogada <bharat.kumar.gogada@xilinx.com>
>> Cc: Michal Simek <michal.simek@xilinx.com>
>> Cc: Yao Hongbo <yaohongbo@linux.alibaba.com>
>> Cc: Naveen Naidu <naveennaidu479@gmail.com>
> 
> Reviewed-by: Pali Rohár <pali@kernel.org>
> 
>> ---
>> v2:
>> - New patch
>>
>>   drivers/pci/pcie/aer.c | 3 ++-
>>   1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/pci/pcie/aer.c b/drivers/pci/pcie/aer.c
>> index 9fa1f97e5b27..385e2033d7b5 100644
>> --- a/drivers/pci/pcie/aer.c
>> +++ b/drivers/pci/pcie/aer.c
>> @@ -1216,7 +1216,8 @@ static int set_device_error_reporting(struct pci_dev *dev, void *data)
>>   	if ((type == PCI_EXP_TYPE_ROOT_PORT) ||
>>   	    (type == PCI_EXP_TYPE_RC_EC) ||
>>   	    (type == PCI_EXP_TYPE_UPSTREAM) ||
>> -	    (type == PCI_EXP_TYPE_DOWNSTREAM)) {
>> +	    (type == PCI_EXP_TYPE_DOWNSTREAM) ||
>> +	    (type == PCI_EXP_TYPE_ENDPOINT)) {
> 
> Hm... maybe another question to discussion: Why enabling of AER is
> limited just to above PCIe port types? Why we do not want to enable it
> for _all_ PCIe devices? Currently in the above list are missing Legacy
> endpoints (which probably do not support AER and so do not have AER
> capability in config space), Root Complex Integrated Endpoints (these
> should provide AER supports too, right?), PCIe to PCI/X Bridges (these
> may generate its own AER errors) and PCI to PCIe Bridges (these are
> maybe complicated as subtree behind such bridges are regular PCIe
> devices and so could fully support AER but on legacy PCI bus there is
> probably no access to extended config space where is AER). But in all of
> these cases, are there any issues with enabling AER via function
> pci_enable_pcie_error_reporting()? For me it looks like that in the
> worst case dev just does not have AER capability in config space or
> extended config space is not accessible (which is same as no AER
> capability).

I also had similar thoughts on this and was a bit unsure here. Perhaps
Bjorn can also comment. But...

... I noticed that AER is still disabled for hot-plugged PCIe devices.
As this code patch will not get called in this HP case. Right now I'm
testing with this patch here, which could be used instead of this one
from this mail:

     PCI/AER: Enable AER on all PCIe devices supporting it

     With this change, AER is now also enabled for hot-plugged PCIe devices
     as pci_aer_init() is also called upon hot-plugging of a PCIe device.
     When "pci=noaer" is selected, AER stays disabled of course.

     Signed-off-by: Stefan Roese <sr@denx.de>

diff --git a/drivers/pci/pcie/aer.c b/drivers/pci/pcie/aer.c
index 385e2033d7b5..94107309ef2d 100644
--- a/drivers/pci/pcie/aer.c
+++ b/drivers/pci/pcie/aer.c
@@ -387,6 +387,10 @@ void pci_aer_init(struct pci_dev *dev)
         pci_add_ext_cap_save_buffer(dev, PCI_EXT_CAP_ID_ERR, 
sizeof(u32) * n);

         pci_aer_clear_status(dev);
+
+       /* Enable AER if requested */
+       if (pci_aer_available())
+               pci_enable_pcie_error_reporting(dev);
  }

This has the same effect of enabling AER in each PCIe Endpoint and
also is available for hot-plugged Endpoints. If nobody objects, I'll
probably use this patch as patch 2/2 in the next patchset version.

Comments welcome as always.

Thanks,
Stefan