[PATCH v5 0/7] Amazon's Annapurna Labs DT-based PCIe host controller driver

[PATCH v5 0/7] Amazon's Annapurna Labs DT-based PCIe host controller driver

[Jonathan Chocron]

This series adds support for Amazon's Annapurna Labs DT-based PCIe host
controller driver.
Additionally, it adds 3 quirks (ACS, VPD and MSI-X) and 2 generic DWC patches.

Changes since v4:
- Moved the HEADER_TYPE validations to after pp->ops->host_init() and
  ep->ops->ep_init()
- Changed to dw_pcie_rd_own_conf() instead of dw_pcie_readb_dbi() for
  reading the HEADER_TYPE
- Used exsitng quirk_blacklist_vpd() instead of quirk_al_vpd_release()
- Added a newline in ACS quirk comment

Changes since v3:
- Removed PATCH 8/8 since the usage of the PCI flags will be discussed
  in the upcoming LPC
- Align commit subject with the folder convention
- Added explanation regarding ECAM "overload" mechanism
- Switched to read/write{_relaxed} APIs
- Modified a dev_err to dev_dbg
- Removed unnecessary variable
- Removed driver details from dt-binding description
- Changed to SoC specific compatibles
- Fixed typo in a commit message
- Added comment regarding MSI in the MSI-X quirk

Changes since v2:
- Added al_pcie_controller_readl/writel() wrappers
- Reorganized local vars in several functions according to reverse
  tree structure
- Removed unnecessary check of ret value
- Changed return type of al_pcie_config_prepare() from int to void
- Removed check if link is up from probe() [done internally in
  dw_pcie_rd/wr_conf()]

Changes since v1:
- Added comment regarding 0x0031 being used as a dev_id for non root-port devices as well
- Fixed different message/comment/print wordings
- Added panic stacktrace to commit message of MSI-x quirk patch
- Changed to pci_warn() instead of dev_warn()
- Added unit_address after node_name in dt-binding
- Updated Kconfig help description
- Used GENMASK and FIELD_PREP/GET where appropriate
- Removed leftover field from struct al_pcie and moved all ptrs to
  the beginning
- Re-wrapped function definitions and invocations to use fewer lines
- Change %p to %px in dbg prints in rd/wr_conf() functions
- Removed validation that the port is configured to RC mode (as this is
  added generically in PATCH 7/8)
- Removed unnecessary variable initializations
- Swtiched to %pR for printing resources


Ali Saidi (1):
  PCI: Add ACS quirk for Amazon Annapurna Labs root ports

Jonathan Chocron (6):
  PCI: Add Amazon's Annapurna Labs vendor ID
  PCI/VPD: Prevent VPD access for Amazon's Annapurna Labs Root Port
  PCI: Add quirk to disable MSI-X support for Amazon's Annapurna Labs
    Root Port
  dt-bindings: PCI: Add Amazon's Annapurna Labs PCIe host bridge binding
  PCI: dwc: al: Add support for DW based driver type
  PCI: dwc: Add validation that PCIe core is set to correct mode

 .../devicetree/bindings/pci/pcie-al.txt       |  46 +++
 MAINTAINERS                                   |   3 +-
 drivers/pci/controller/dwc/Kconfig            |  12 +
 drivers/pci/controller/dwc/pcie-al.c          | 365 ++++++++++++++++++
 .../pci/controller/dwc/pcie-designware-ep.c   |   8 +
 .../pci/controller/dwc/pcie-designware-host.c |  16 +
 drivers/pci/quirks.c                          |  38 ++
 drivers/pci/vpd.c                             |   6 +
 include/linux/pci_ids.h                       |   2 +
 9 files changed, 495 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/devicetree/bindings/pci/pcie-al.txt

-- 
2.17.1

[PATCH v5 0/7] Amazon's Annapurna Labs DT-based PCIe host controller driver

[Jonathan Chocron]

This series adds support for Amazon's Annapurna Labs DT-based PCIe host
controller driver.
Additionally, it adds 3 quirks (ACS, VPD and MSI-X) and 2 generic DWC patches.

Changes since v4:
- Moved the HEADER_TYPE validations to after pp->ops->host_init() and
  ep->ops->ep_init()
- Changed to dw_pcie_rd_own_conf() instead of dw_pcie_readb_dbi() for
  reading the HEADER_TYPE
- Used exsitng quirk_blacklist_vpd() instead of quirk_al_vpd_release()
- Added a newline in ACS quirk comment

Changes since v3:
- Removed PATCH 8/8 since the usage of the PCI flags will be discussed
  in the upcoming LPC
- Align commit subject with the folder convention
- Added explanation regarding ECAM "overload" mechanism
- Switched to read/write{_relaxed} APIs
- Modified a dev_err to dev_dbg
- Removed unnecessary variable
- Removed driver details from dt-binding description
- Changed to SoC specific compatibles
- Fixed typo in a commit message
- Added comment regarding MSI in the MSI-X quirk

Changes since v2:
- Added al_pcie_controller_readl/writel() wrappers
- Reorganized local vars in several functions according to reverse
  tree structure
- Removed unnecessary check of ret value
- Changed return type of al_pcie_config_prepare() from int to void
- Removed check if link is up from probe() [done internally in
  dw_pcie_rd/wr_conf()]

Changes since v1:
- Added comment regarding 0x0031 being used as a dev_id for non root-port devices as well
- Fixed different message/comment/print wordings
- Added panic stacktrace to commit message of MSI-x quirk patch
- Changed to pci_warn() instead of dev_warn()
- Added unit_address after node_name in dt-binding
- Updated Kconfig help description
- Used GENMASK and FIELD_PREP/GET where appropriate
- Removed leftover field from struct al_pcie and moved all ptrs to
  the beginning
- Re-wrapped function definitions and invocations to use fewer lines
- Change %p to %px in dbg prints in rd/wr_conf() functions
- Removed validation that the port is configured to RC mode (as this is
  added generically in PATCH 7/8)
- Removed unnecessary variable initializations
- Swtiched to %pR for printing resources


Ali Saidi (1):
  PCI: Add ACS quirk for Amazon Annapurna Labs root ports

Jonathan Chocron (6):
  PCI: Add Amazon's Annapurna Labs vendor ID
  PCI/VPD: Prevent VPD access for Amazon's Annapurna Labs Root Port
  PCI: Add quirk to disable MSI-X support for Amazon's Annapurna Labs
    Root Port
  dt-bindings: PCI: Add Amazon's Annapurna Labs PCIe host bridge binding
  PCI: dwc: al: Add support for DW based driver type
  PCI: dwc: Add validation that PCIe core is set to correct mode

 .../devicetree/bindings/pci/pcie-al.txt       |  46 +++
 MAINTAINERS                                   |   3 +-
 drivers/pci/controller/dwc/Kconfig            |  12 +
 drivers/pci/controller/dwc/pcie-al.c          | 365 ++++++++++++++++++
 .../pci/controller/dwc/pcie-designware-ep.c   |   8 +
 .../pci/controller/dwc/pcie-designware-host.c |  16 +
 drivers/pci/quirks.c                          |  38 ++
 drivers/pci/vpd.c                             |   6 +
 include/linux/pci_ids.h                       |   2 +
 9 files changed, 495 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/devicetree/bindings/pci/pcie-al.txt

-- 
2.17.1

[PATCH v5 1/7] PCI: Add Amazon's Annapurna Labs vendor ID

[Jonathan Chocron]

Add Amazon's Annapurna Labs vendor ID to pci_ids.h.

Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
Acked-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Andrew Murray <andrew.murray@arm.com>
---
 include/linux/pci_ids.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
index f59a6f98900c..f3130542c752 100644
--- a/include/linux/pci_ids.h
+++ b/include/linux/pci_ids.h
@@ -2571,6 +2571,8 @@
 
 #define PCI_VENDOR_ID_ASMEDIA		0x1b21
 
+#define PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS	0x1c36
+
 #define PCI_VENDOR_ID_CIRCUITCO		0x1cc8
 #define PCI_SUBSYSTEM_ID_CIRCUITCO_MINNOWBOARD	0x0001
 
-- 
2.17.1

[PATCH v5 1/7] PCI: Add Amazon's Annapurna Labs vendor ID

[Jonathan Chocron]

Add Amazon's Annapurna Labs vendor ID to pci_ids.h.

Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
Acked-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Andrew Murray <andrew.murray@arm.com>
---
 include/linux/pci_ids.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
index f59a6f98900c..f3130542c752 100644
--- a/include/linux/pci_ids.h
+++ b/include/linux/pci_ids.h
@@ -2571,6 +2571,8 @@
 
 #define PCI_VENDOR_ID_ASMEDIA		0x1b21
 
+#define PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS	0x1c36
+
 #define PCI_VENDOR_ID_CIRCUITCO		0x1cc8
 #define PCI_SUBSYSTEM_ID_CIRCUITCO_MINNOWBOARD	0x0001
 
-- 
2.17.1

[PATCH v5 2/7] PCI: Add ACS quirk for Amazon Annapurna Labs root ports

[Jonathan Chocron]

From: Ali Saidi <alisaidi@amazon.com>

The Amazon's Annapurna Labs root ports don't advertise an ACS
capability, but they don't allow peer-to-peer transactions and do
validate bus numbers through the SMMU. Additionally, it's not possible
for one RP to pass traffic to another RP.

Signed-off-by: Ali Saidi <alisaidi@amazon.com>
Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
Reviewed-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com>
Reviewed-by: Andrew Murray <andrew.murray@arm.com>
---
 drivers/pci/quirks.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index ded60757a573..8fe765592943 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -4418,6 +4418,24 @@ static int pci_quirk_qcom_rp_acs(struct pci_dev *dev, u16 acs_flags)
 	return ret;
 }
 
+static int pci_quirk_al_acs(struct pci_dev *dev, u16 acs_flags)
+{
+	/*
+	 * Amazon's Annapurna Labs root ports don't include an ACS capability,
+	 * but do include ACS-like functionality. The hardware doesn't support
+	 * peer-to-peer transactions via the root port and each has a unique
+	 * segment number.
+	 *
+	 * Additionally, the root ports cannot send traffic to each other.
+	 */
+	acs_flags &= ~(PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_SV | PCI_ACS_UF);
+
+	if (pci_pcie_type(dev) != PCI_EXP_TYPE_ROOT_PORT)
+		return -ENOTTY;
+
+	return acs_flags ? 0 : 1;
+}
+
 /*
  * Sunrise Point PCH root ports implement ACS, but unfortunately as shown in
  * the datasheet (Intel 100 Series Chipset Family PCH Datasheet, Vol. 2,
@@ -4611,6 +4629,8 @@ static const struct pci_dev_acs_enabled {
 	{ PCI_VENDOR_ID_AMPERE, 0xE00A, pci_quirk_xgene_acs },
 	{ PCI_VENDOR_ID_AMPERE, 0xE00B, pci_quirk_xgene_acs },
 	{ PCI_VENDOR_ID_AMPERE, 0xE00C, pci_quirk_xgene_acs },
+	/* Amazon Annapurna Labs */
+	{ PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS, 0x0031, pci_quirk_al_acs },
 	{ 0 }
 };
 
-- 
2.17.1

[PATCH v5 2/7] PCI: Add ACS quirk for Amazon Annapurna Labs root ports

[Jonathan Chocron]

From: Ali Saidi <alisaidi@amazon.com>

The Amazon's Annapurna Labs root ports don't advertise an ACS
capability, but they don't allow peer-to-peer transactions and do
validate bus numbers through the SMMU. Additionally, it's not possible
for one RP to pass traffic to another RP.

Signed-off-by: Ali Saidi <alisaidi@amazon.com>
Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
Reviewed-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com>
Reviewed-by: Andrew Murray <andrew.murray@arm.com>
---
 drivers/pci/quirks.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index ded60757a573..8fe765592943 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -4418,6 +4418,24 @@ static int pci_quirk_qcom_rp_acs(struct pci_dev *dev, u16 acs_flags)
 	return ret;
 }
 
+static int pci_quirk_al_acs(struct pci_dev *dev, u16 acs_flags)
+{
+	/*
+	 * Amazon's Annapurna Labs root ports don't include an ACS capability,
+	 * but do include ACS-like functionality. The hardware doesn't support
+	 * peer-to-peer transactions via the root port and each has a unique
+	 * segment number.
+	 *
+	 * Additionally, the root ports cannot send traffic to each other.
+	 */
+	acs_flags &= ~(PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_SV | PCI_ACS_UF);
+
+	if (pci_pcie_type(dev) != PCI_EXP_TYPE_ROOT_PORT)
+		return -ENOTTY;
+
+	return acs_flags ? 0 : 1;
+}
+
 /*
  * Sunrise Point PCH root ports implement ACS, but unfortunately as shown in
  * the datasheet (Intel 100 Series Chipset Family PCH Datasheet, Vol. 2,
@@ -4611,6 +4629,8 @@ static const struct pci_dev_acs_enabled {
 	{ PCI_VENDOR_ID_AMPERE, 0xE00A, pci_quirk_xgene_acs },
 	{ PCI_VENDOR_ID_AMPERE, 0xE00B, pci_quirk_xgene_acs },
 	{ PCI_VENDOR_ID_AMPERE, 0xE00C, pci_quirk_xgene_acs },
+	/* Amazon Annapurna Labs */
+	{ PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS, 0x0031, pci_quirk_al_acs },
 	{ 0 }
 };
 
-- 
2.17.1

Re: [PATCH v5 2/7] PCI: Add ACS quirk for Amazon Annapurna Labs root ports

[Bjorn Helgaas]

On Thu, Sep 05, 2019 at 05:00:16PM +0300, Jonathan Chocron wrote:
> From: Ali Saidi <alisaidi@amazon.com>
> 
> The Amazon's Annapurna Labs root ports don't advertise an ACS
> capability, but they don't allow peer-to-peer transactions and do
> validate bus numbers through the SMMU. Additionally, it's not possible
> for one RP to pass traffic to another RP.
> 
> Signed-off-by: Ali Saidi <alisaidi@amazon.com>
> Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
> Reviewed-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com>
> Reviewed-by: Andrew Murray <andrew.murray@arm.com>

Acked-by: Bjorn Helgaas <bhelgaas@google.com>

But please tweak it as below ...

> ---
>  drivers/pci/quirks.c | 20 ++++++++++++++++++++
>  1 file changed, 20 insertions(+)
> 
> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> index ded60757a573..8fe765592943 100644
> --- a/drivers/pci/quirks.c
> +++ b/drivers/pci/quirks.c
> @@ -4418,6 +4418,24 @@ static int pci_quirk_qcom_rp_acs(struct pci_dev *dev, u16 acs_flags)
>  	return ret;
>  }
>  
> +static int pci_quirk_al_acs(struct pci_dev *dev, u16 acs_flags)
> +{
> +	/*
> +	 * Amazon's Annapurna Labs root ports don't include an ACS capability,
> +	 * but do include ACS-like functionality. The hardware doesn't support
> +	 * peer-to-peer transactions via the root port and each has a unique
> +	 * segment number.
> +	 *
> +	 * Additionally, the root ports cannot send traffic to each other.
> +	 */
> +	acs_flags &= ~(PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_SV | PCI_ACS_UF);

There are several quirks that use this same set of bits, but they
don't use the same order, which is a needless difference.

Can you reorder them in the bit 0 ... bit 7 order?  I.e.,

    PCI_ACS_SV | PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_UF

> +	if (pci_pcie_type(dev) != PCI_EXP_TYPE_ROOT_PORT)
> +		return -ENOTTY;

This could go first (above the comment) so all the acs_flags stuff is
together.

> +	return acs_flags ? 0 : 1;
> +}
> +
>  /*
>   * Sunrise Point PCH root ports implement ACS, but unfortunately as shown in
>   * the datasheet (Intel 100 Series Chipset Family PCH Datasheet, Vol. 2,
> @@ -4611,6 +4629,8 @@ static const struct pci_dev_acs_enabled {
>  	{ PCI_VENDOR_ID_AMPERE, 0xE00A, pci_quirk_xgene_acs },
>  	{ PCI_VENDOR_ID_AMPERE, 0xE00B, pci_quirk_xgene_acs },
>  	{ PCI_VENDOR_ID_AMPERE, 0xE00C, pci_quirk_xgene_acs },
> +	/* Amazon Annapurna Labs */
> +	{ PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS, 0x0031, pci_quirk_al_acs },
>  	{ 0 }
>  };
>  
> -- 
> 2.17.1
> 

Re: [PATCH v5 2/7] PCI: Add ACS quirk for Amazon Annapurna Labs root ports

[Chocron, Jonathan]

On Sat, 2019-09-07 at 11:54 -0500, Bjorn Helgaas wrote:
> On Thu, Sep 05, 2019 at 05:00:16PM +0300, Jonathan Chocron wrote:
> > From: Ali Saidi <alisaidi@amazon.com>
> > 
> > The Amazon's Annapurna Labs root ports don't advertise an ACS
> > capability, but they don't allow peer-to-peer transactions and do
> > validate bus numbers through the SMMU. Additionally, it's not
> > possible
> > for one RP to pass traffic to another RP.
> > 
> > Signed-off-by: Ali Saidi <alisaidi@amazon.com>
> > Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
> > Reviewed-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com>
> > Reviewed-by: Andrew Murray <andrew.murray@arm.com>
> 
> Acked-by: Bjorn Helgaas <bhelgaas@google.com>
> 
> But please tweak it as below ...
> 
Will be part of v6.

> > ---
> >  drivers/pci/quirks.c | 20 ++++++++++++++++++++
> >  1 file changed, 20 insertions(+)
> > 
> > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> > index ded60757a573..8fe765592943 100644
> > --- a/drivers/pci/quirks.c
> > +++ b/drivers/pci/quirks.c
> > @@ -4418,6 +4418,24 @@ static int pci_quirk_qcom_rp_acs(struct
> > pci_dev *dev, u16 acs_flags)
> >  	return ret;
> >  }
> >  
> > +static int pci_quirk_al_acs(struct pci_dev *dev, u16 acs_flags)
> > +{
> > +	/*
> > +	 * Amazon's Annapurna Labs root ports don't include an ACS
> > capability,
> > +	 * but do include ACS-like functionality. The hardware doesn't
> > support
> > +	 * peer-to-peer transactions via the root port and each has a
> > unique
> > +	 * segment number.
> > +	 *
> > +	 * Additionally, the root ports cannot send traffic to each
> > other.
> > +	 */
> > +	acs_flags &= ~(PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_SV |
> > PCI_ACS_UF);
> 
> There are several quirks that use this same set of bits, but they
> don't use the same order, which is a needless difference.
> 
> Can you reorder them in the bit 0 ... bit 7 order?  I.e.,
> 
>     PCI_ACS_SV | PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_UF
> 
Done.

> > +	if (pci_pcie_type(dev) != PCI_EXP_TYPE_ROOT_PORT)
> > +		return -ENOTTY;
> 
> This could go first (above the comment) so all the acs_flags stuff is
> together.
> 
Done.

> > +	return acs_flags ? 0 : 1;
> > +}
> > +
> >  /*
> >   * Sunrise Point PCH root ports implement ACS, but unfortunately
> > as shown in
> >   * the datasheet (Intel 100 Series Chipset Family PCH Datasheet,
> > Vol. 2,
> > @@ -4611,6 +4629,8 @@ static const struct pci_dev_acs_enabled {
> >  	{ PCI_VENDOR_ID_AMPERE, 0xE00A, pci_quirk_xgene_acs },
> >  	{ PCI_VENDOR_ID_AMPERE, 0xE00B, pci_quirk_xgene_acs },
> >  	{ PCI_VENDOR_ID_AMPERE, 0xE00C, pci_quirk_xgene_acs },
> > +	/* Amazon Annapurna Labs */
> > +	{ PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS, 0x0031, pci_quirk_al_acs
> > },
> >  	{ 0 }
> >  };
> >  
> > -- 
> > 2.17.1
> > 

Re: [PATCH v5 2/7] PCI: Add ACS quirk for Amazon Annapurna Labs root ports

[Chocron, Jonathan]

On Sat, 2019-09-07 at 11:54 -0500, Bjorn Helgaas wrote:
> On Thu, Sep 05, 2019 at 05:00:16PM +0300, Jonathan Chocron wrote:
> > From: Ali Saidi <alisaidi@amazon.com>
> > 
> > The Amazon's Annapurna Labs root ports don't advertise an ACS
> > capability, but they don't allow peer-to-peer transactions and do
> > validate bus numbers through the SMMU. Additionally, it's not
> > possible
> > for one RP to pass traffic to another RP.
> > 
> > Signed-off-by: Ali Saidi <alisaidi@amazon.com>
> > Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
> > Reviewed-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com>
> > Reviewed-by: Andrew Murray <andrew.murray@arm.com>
> 
> Acked-by: Bjorn Helgaas <bhelgaas@google.com>
> 
> But please tweak it as below ...
> 
Will be part of v6.

> > ---
> >  drivers/pci/quirks.c | 20 ++++++++++++++++++++
> >  1 file changed, 20 insertions(+)
> > 
> > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> > index ded60757a573..8fe765592943 100644
> > --- a/drivers/pci/quirks.c
> > +++ b/drivers/pci/quirks.c
> > @@ -4418,6 +4418,24 @@ static int pci_quirk_qcom_rp_acs(struct
> > pci_dev *dev, u16 acs_flags)
> >  	return ret;
> >  }
> >  
> > +static int pci_quirk_al_acs(struct pci_dev *dev, u16 acs_flags)
> > +{
> > +	/*
> > +	 * Amazon's Annapurna Labs root ports don't include an ACS
> > capability,
> > +	 * but do include ACS-like functionality. The hardware doesn't
> > support
> > +	 * peer-to-peer transactions via the root port and each has a
> > unique
> > +	 * segment number.
> > +	 *
> > +	 * Additionally, the root ports cannot send traffic to each
> > other.
> > +	 */
> > +	acs_flags &= ~(PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_SV |
> > PCI_ACS_UF);
> 
> There are several quirks that use this same set of bits, but they
> don't use the same order, which is a needless difference.
> 
> Can you reorder them in the bit 0 ... bit 7 order?  I.e.,
> 
>     PCI_ACS_SV | PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_UF
> 
Done.

> > +	if (pci_pcie_type(dev) != PCI_EXP_TYPE_ROOT_PORT)
> > +		return -ENOTTY;
> 
> This could go first (above the comment) so all the acs_flags stuff is
> together.
> 
Done.

> > +	return acs_flags ? 0 : 1;
> > +}
> > +
> >  /*
> >   * Sunrise Point PCH root ports implement ACS, but unfortunately
> > as shown in
> >   * the datasheet (Intel 100 Series Chipset Family PCH Datasheet,
> > Vol. 2,
> > @@ -4611,6 +4629,8 @@ static const struct pci_dev_acs_enabled {
> >  	{ PCI_VENDOR_ID_AMPERE, 0xE00A, pci_quirk_xgene_acs },
> >  	{ PCI_VENDOR_ID_AMPERE, 0xE00B, pci_quirk_xgene_acs },
> >  	{ PCI_VENDOR_ID_AMPERE, 0xE00C, pci_quirk_xgene_acs },
> > +	/* Amazon Annapurna Labs */
> > +	{ PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS, 0x0031, pci_quirk_al_acs
> > },
> >  	{ 0 }
> >  };
> >  
> > -- 
> > 2.17.1
> > 

[PATCH v5 3/7] PCI/VPD: Prevent VPD access for Amazon's Annapurna Labs Root Port

[Jonathan Chocron]

The Amazon Annapurna Labs PCIe Root Port exposes the VPD capability,
but there is no actual support for it.

Trying to access the VPD (for example, as part of lspci -vv or when
reading the vpd sysfs file), results in the following warning print:

pcieport 0001:00:00.0: VPD access failed.  This is likely a firmware bug on this device.  Contact the card vendor for a firmware update

Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
Reviewed-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com>
---
 drivers/pci/vpd.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/pci/vpd.c b/drivers/pci/vpd.c
index 4963c2e2bd4c..7915d10f9aa1 100644
--- a/drivers/pci/vpd.c
+++ b/drivers/pci/vpd.c
@@ -571,6 +571,12 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LSI_LOGIC, 0x005f, quirk_blacklist_vpd);
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, PCI_ANY_ID,
 		quirk_blacklist_vpd);
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_QLOGIC, 0x2261, quirk_blacklist_vpd);
+/*
+ * The Amazon Annapurna Labs 0x0031 device id is reused for other non Root Port
+ * device types, so the quirk is registered for the PCI_CLASS_BRIDGE_PCI class.
+ */
+DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS, 0x0031,
+			      PCI_CLASS_BRIDGE_PCI, 8, quirk_blacklist_vpd);
 
 /*
  * For Broadcom 5706, 5708, 5709 rev. A nics, any read beyond the
-- 
2.17.1

[PATCH v5 3/7] PCI/VPD: Prevent VPD access for Amazon's Annapurna Labs Root Port

[Jonathan Chocron]

The Amazon Annapurna Labs PCIe Root Port exposes the VPD capability,
but there is no actual support for it.

Trying to access the VPD (for example, as part of lspci -vv or when
reading the vpd sysfs file), results in the following warning print:

pcieport 0001:00:00.0: VPD access failed.  This is likely a firmware bug on this device.  Contact the card vendor for a firmware update

Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
Reviewed-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com>
---
 drivers/pci/vpd.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/pci/vpd.c b/drivers/pci/vpd.c
index 4963c2e2bd4c..7915d10f9aa1 100644
--- a/drivers/pci/vpd.c
+++ b/drivers/pci/vpd.c
@@ -571,6 +571,12 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LSI_LOGIC, 0x005f, quirk_blacklist_vpd);
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, PCI_ANY_ID,
 		quirk_blacklist_vpd);
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_QLOGIC, 0x2261, quirk_blacklist_vpd);
+/*
+ * The Amazon Annapurna Labs 0x0031 device id is reused for other non Root Port
+ * device types, so the quirk is registered for the PCI_CLASS_BRIDGE_PCI class.
+ */
+DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS, 0x0031,
+			      PCI_CLASS_BRIDGE_PCI, 8, quirk_blacklist_vpd);
 
 /*
  * For Broadcom 5706, 5708, 5709 rev. A nics, any read beyond the
-- 
2.17.1

Re: [PATCH v5 3/7] PCI/VPD: Prevent VPD access for Amazon's Annapurna Labs Root Port

[Andrew Murray]

On Thu, Sep 05, 2019 at 05:00:17PM +0300, Jonathan Chocron wrote:
> The Amazon Annapurna Labs PCIe Root Port exposes the VPD capability,
> but there is no actual support for it.
> 
> Trying to access the VPD (for example, as part of lspci -vv or when
> reading the vpd sysfs file), results in the following warning print:
> 
> pcieport 0001:00:00.0: VPD access failed.  This is likely a firmware bug on this device.  Contact the card vendor for a firmware update
> 
> Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
> Reviewed-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com>

Reviewed-by: Andrew Murray <andrew.murray@arm.com>

> ---
>  drivers/pci/vpd.c | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/drivers/pci/vpd.c b/drivers/pci/vpd.c
> index 4963c2e2bd4c..7915d10f9aa1 100644
> --- a/drivers/pci/vpd.c
> +++ b/drivers/pci/vpd.c
> @@ -571,6 +571,12 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LSI_LOGIC, 0x005f, quirk_blacklist_vpd);
>  DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, PCI_ANY_ID,
>  		quirk_blacklist_vpd);
>  DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_QLOGIC, 0x2261, quirk_blacklist_vpd);
> +/*
> + * The Amazon Annapurna Labs 0x0031 device id is reused for other non Root Port
> + * device types, so the quirk is registered for the PCI_CLASS_BRIDGE_PCI class.
> + */
> +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS, 0x0031,
> +			      PCI_CLASS_BRIDGE_PCI, 8, quirk_blacklist_vpd);
>  
>  /*
>   * For Broadcom 5706, 5708, 5709 rev. A nics, any read beyond the
> -- 
> 2.17.1
> 

Re: [PATCH v5 3/7] PCI/VPD: Prevent VPD access for Amazon's Annapurna Labs Root Port

[Bjorn Helgaas]

On Thu, Sep 05, 2019 at 05:00:17PM +0300, Jonathan Chocron wrote:
> The Amazon Annapurna Labs PCIe Root Port exposes the VPD capability,
> but there is no actual support for it.

Oops.  Another oops for the device ID reuse mentioned below.

> Trying to access the VPD (for example, as part of lspci -vv or when
> reading the vpd sysfs file), results in the following warning print:
> 
> pcieport 0001:00:00.0: VPD access failed.  This is likely a firmware bug on this device.  Contact the card vendor for a firmware update

Thanks for not wrapping the message (keeping it together makes it
easier to grep for).  Maybe indent it two spaces since it's quoted
material.

*Is* this a firmware defect?  E.g., could firmware disable this
capability so it doesn't appear in config space, as it apparently can
for the MSI-X capability?

> Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
> Reviewed-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com>

Acked-by: Bjorn Helgaas <bhelgaas@google.com>

> ---
>  drivers/pci/vpd.c | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/drivers/pci/vpd.c b/drivers/pci/vpd.c
> index 4963c2e2bd4c..7915d10f9aa1 100644
> --- a/drivers/pci/vpd.c
> +++ b/drivers/pci/vpd.c
> @@ -571,6 +571,12 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LSI_LOGIC, 0x005f, quirk_blacklist_vpd);
>  DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, PCI_ANY_ID,
>  		quirk_blacklist_vpd);
>  DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_QLOGIC, 0x2261, quirk_blacklist_vpd);
> +/*
> + * The Amazon Annapurna Labs 0x0031 device id is reused for other non Root Port
> + * device types, so the quirk is registered for the PCI_CLASS_BRIDGE_PCI class.
> + */
> +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS, 0x0031,
> +			      PCI_CLASS_BRIDGE_PCI, 8, quirk_blacklist_vpd);
>  
>  /*
>   * For Broadcom 5706, 5708, 5709 rev. A nics, any read beyond the
> -- 
> 2.17.1
> 

Re: [PATCH v5 3/7] PCI/VPD: Prevent VPD access for Amazon's Annapurna Labs Root Port

[Chocron, Jonathan]

On Sat, 2019-09-07 at 11:55 -0500, Bjorn Helgaas wrote:
> On Thu, Sep 05, 2019 at 05:00:17PM +0300, Jonathan Chocron wrote:
> > The Amazon Annapurna Labs PCIe Root Port exposes the VPD
> > capability,
> > but there is no actual support for it.
> 
> Oops.  Another oops for the device ID reuse mentioned below.
> 
> > Trying to access the VPD (for example, as part of lspci -vv or when
> > reading the vpd sysfs file), results in the following warning
> > print:
> > 
> > pcieport 0001:00:00.0: VPD access failed.  This is likely a
> > firmware bug on this device.  Contact the card vendor for a
> > firmware update
> 
> Thanks for not wrapping the message (keeping it together makes it
> easier to grep for).  

My thoughts exactly :) 

> Maybe indent it two spaces since it's quoted
> material.
> 
Will be done as part of v6.

> *Is* this a firmware defect?  E.g., could firmware disable this
> capability so it doesn't appear in config space, as it apparently can
> for the MSI-X capability?
> 
Yes, the FW could potentially disable it so it doesn't appear in the
config space.

In theory, VPD should be able to work if some other entity would listen
to the VPD requests and respond, but there are no plans to add support
for such an entity.

> > Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
> > Reviewed-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com>
> 
> Acked-by: Bjorn Helgaas <bhelgaas@google.com>
> 
> > ---
> >  drivers/pci/vpd.c | 6 ++++++
> >  1 file changed, 6 insertions(+)
> > 
> > diff --git a/drivers/pci/vpd.c b/drivers/pci/vpd.c
> > index 4963c2e2bd4c..7915d10f9aa1 100644
> > --- a/drivers/pci/vpd.c
> > +++ b/drivers/pci/vpd.c
> > @@ -571,6 +571,12 @@
> > DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LSI_LOGIC, 0x005f,
> > quirk_blacklist_vpd);
> >  DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, PCI_ANY_ID,
> >  		quirk_blacklist_vpd);
> >  DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_QLOGIC, 0x2261,
> > quirk_blacklist_vpd);
> > +/*
> > + * The Amazon Annapurna Labs 0x0031 device id is reused for other
> > non Root Port
> > + * device types, so the quirk is registered for the
> > PCI_CLASS_BRIDGE_PCI class.
> > + */
> > +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS,
> > 0x0031,
> > +			      PCI_CLASS_BRIDGE_PCI, 8,
> > quirk_blacklist_vpd);
> >  
> >  /*
> >   * For Broadcom 5706, 5708, 5709 rev. A nics, any read beyond the
> > -- 
> > 2.17.1
> > 

Re: [PATCH v5 3/7] PCI/VPD: Prevent VPD access for Amazon's Annapurna Labs Root Port

[Chocron, Jonathan]

On Sat, 2019-09-07 at 11:55 -0500, Bjorn Helgaas wrote:
> On Thu, Sep 05, 2019 at 05:00:17PM +0300, Jonathan Chocron wrote:
> > The Amazon Annapurna Labs PCIe Root Port exposes the VPD
> > capability,
> > but there is no actual support for it.
> 
> Oops.  Another oops for the device ID reuse mentioned below.
> 
> > Trying to access the VPD (for example, as part of lspci -vv or when
> > reading the vpd sysfs file), results in the following warning
> > print:
> > 
> > pcieport 0001:00:00.0: VPD access failed.  This is likely a
> > firmware bug on this device.  Contact the card vendor for a
> > firmware update
> 
> Thanks for not wrapping the message (keeping it together makes it
> easier to grep for).  

My thoughts exactly :) 

> Maybe indent it two spaces since it's quoted
> material.
> 
Will be done as part of v6.

> *Is* this a firmware defect?  E.g., could firmware disable this
> capability so it doesn't appear in config space, as it apparently can
> for the MSI-X capability?
> 
Yes, the FW could potentially disable it so it doesn't appear in the
config space.

In theory, VPD should be able to work if some other entity would listen
to the VPD requests and respond, but there are no plans to add support
for such an entity.

> > Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
> > Reviewed-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com>
> 
> Acked-by: Bjorn Helgaas <bhelgaas@google.com>
> 
> > ---
> >  drivers/pci/vpd.c | 6 ++++++
> >  1 file changed, 6 insertions(+)
> > 
> > diff --git a/drivers/pci/vpd.c b/drivers/pci/vpd.c
> > index 4963c2e2bd4c..7915d10f9aa1 100644
> > --- a/drivers/pci/vpd.c
> > +++ b/drivers/pci/vpd.c
> > @@ -571,6 +571,12 @@
> > DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LSI_LOGIC, 0x005f,
> > quirk_blacklist_vpd);
> >  DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, PCI_ANY_ID,
> >  		quirk_blacklist_vpd);
> >  DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_QLOGIC, 0x2261,
> > quirk_blacklist_vpd);
> > +/*
> > + * The Amazon Annapurna Labs 0x0031 device id is reused for other
> > non Root Port
> > + * device types, so the quirk is registered for the
> > PCI_CLASS_BRIDGE_PCI class.
> > + */
> > +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS,
> > 0x0031,
> > +			      PCI_CLASS_BRIDGE_PCI, 8,
> > quirk_blacklist_vpd);
> >  
> >  /*
> >   * For Broadcom 5706, 5708, 5709 rev. A nics, any read beyond the
> > -- 
> > 2.17.1
> > 

[PATCH v5 4/7] PCI: Add quirk to disable MSI-X support for Amazon's Annapurna Labs Root Port

[Jonathan Chocron]

The Root Port (identified by [1c36:0031]) doesn't support MSI-X. On some
platforms it is configured to not advertise the capability at all, while
on others it (mistakenly) does. This causes a panic during
initialization by the pcieport driver, since it tries to configure the
MSI-X capability. Specifically, when trying to access the MSI-X table
a "non-existing addr" exception occurs.

Example stacktrace snippet:

[    1.632363] SError Interrupt on CPU2, code 0xbf000000 -- SError
[    1.632364] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.2.0-rc1-Jonny-14847-ge76f1d4a1828-dirty #33
[    1.632365] Hardware name: Annapurna Labs Alpine V3 EVP (DT)
[    1.632365] pstate: 80000005 (Nzcv daif -PAN -UAO)
[    1.632366] pc : __pci_enable_msix_range+0x4e4/0x608
[    1.632367] lr : __pci_enable_msix_range+0x498/0x608
[    1.632367] sp : ffffff80117db700
[    1.632368] x29: ffffff80117db700 x28: 0000000000000001
[    1.632370] x27: 0000000000000001 x26: 0000000000000000
[    1.632372] x25: ffffffd3e9d8c0b0 x24: 0000000000000000
[    1.632373] x23: 0000000000000000 x22: 0000000000000000
[    1.632375] x21: 0000000000000001 x20: 0000000000000000
[    1.632376] x19: ffffffd3e9d8c000 x18: ffffffffffffffff
[    1.632378] x17: 0000000000000000 x16: 0000000000000000
[    1.632379] x15: ffffff80116496c8 x14: ffffffd3e9844503
[    1.632380] x13: ffffffd3e9844502 x12: 0000000000000038
[    1.632382] x11: ffffffffffffff00 x10: 0000000000000040
[    1.632384] x9 : ffffff801165e270 x8 : ffffff801165e268
[    1.632385] x7 : 0000000000000002 x6 : 00000000000000b2
[    1.632387] x5 : ffffffd3e9d8c2c0 x4 : 0000000000000000
[    1.632388] x3 : 0000000000000000 x2 : 0000000000000000
[    1.632390] x1 : 0000000000000000 x0 : ffffffd3e9844680
[    1.632392] Kernel panic - not syncing: Asynchronous SError Interrupt
[    1.632393] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.2.0-rc1-Jonny-14847-ge76f1d4a1828-dirty #33
[    1.632394] Hardware name: Annapurna Labs Alpine V3 EVP (DT)
[    1.632394] Call trace:
[    1.632395]  dump_backtrace+0x0/0x140
[    1.632395]  show_stack+0x14/0x20
[    1.632396]  dump_stack+0xa8/0xcc
[    1.632396]  panic+0x140/0x334
[    1.632397]  nmi_panic+0x6c/0x70
[    1.632398]  arm64_serror_panic+0x74/0x88
[    1.632398]  __pte_error+0x0/0x28
[    1.632399]  el1_error+0x84/0xf8
[    1.632400]  __pci_enable_msix_range+0x4e4/0x608
[    1.632400]  pci_alloc_irq_vectors_affinity+0xdc/0x150
[    1.632401]  pcie_port_device_register+0x2b8/0x4e0
[    1.632402]  pcie_portdrv_probe+0x34/0xf0

Notice that this quirk also disables MSI (which may work, but hasn't
been tested nor has a current use case), since currently there is no
standard way to disable only MSI-X.

Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
Reviewed-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com>
Reviewed-by: Andrew Murray <andrew.murray@arm.com>
---
 drivers/pci/quirks.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 8fe765592943..5a8ea5fdeae7 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -2977,6 +2977,24 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, 0x10a1,
 			quirk_msi_intx_disable_qca_bug);
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, 0xe091,
 			quirk_msi_intx_disable_qca_bug);
+
+/*
+ * Amazon's Annapurna Labs 1c36:0031 Root Ports don't support MSI-X, so it
+ * should be disabled on platforms where the device (mistakenly) advertises it.
+ *
+ * Notice that this quirk also disables MSI (which may work, but hasn't been
+ * tested), since currently there is no standard way to disable only MSI-X.
+ *
+ * The 0031 device id is reused for other non Root Port device types,
+ * therefore the quirk is registered for the PCI_CLASS_BRIDGE_PCI class.
+ */
+static void quirk_al_msi_disable(struct pci_dev *dev)
+{
+	dev->no_msi = 1;
+	pci_warn(dev, "Disabling MSI/MSI-X\n");
+}
+DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS, 0x0031,
+			      PCI_CLASS_BRIDGE_PCI, 8, quirk_al_msi_disable);
 #endif /* CONFIG_PCI_MSI */
 
 /*
-- 
2.17.1

[PATCH v5 4/7] PCI: Add quirk to disable MSI-X support for Amazon's Annapurna Labs Root Port

[Jonathan Chocron]

The Root Port (identified by [1c36:0031]) doesn't support MSI-X. On some
platforms it is configured to not advertise the capability at all, while
on others it (mistakenly) does. This causes a panic during
initialization by the pcieport driver, since it tries to configure the
MSI-X capability. Specifically, when trying to access the MSI-X table
a "non-existing addr" exception occurs.

Example stacktrace snippet:

[    1.632363] SError Interrupt on CPU2, code 0xbf000000 -- SError
[    1.632364] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.2.0-rc1-Jonny-14847-ge76f1d4a1828-dirty #33
[    1.632365] Hardware name: Annapurna Labs Alpine V3 EVP (DT)
[    1.632365] pstate: 80000005 (Nzcv daif -PAN -UAO)
[    1.632366] pc : __pci_enable_msix_range+0x4e4/0x608
[    1.632367] lr : __pci_enable_msix_range+0x498/0x608
[    1.632367] sp : ffffff80117db700
[    1.632368] x29: ffffff80117db700 x28: 0000000000000001
[    1.632370] x27: 0000000000000001 x26: 0000000000000000
[    1.632372] x25: ffffffd3e9d8c0b0 x24: 0000000000000000
[    1.632373] x23: 0000000000000000 x22: 0000000000000000
[    1.632375] x21: 0000000000000001 x20: 0000000000000000
[    1.632376] x19: ffffffd3e9d8c000 x18: ffffffffffffffff
[    1.632378] x17: 0000000000000000 x16: 0000000000000000
[    1.632379] x15: ffffff80116496c8 x14: ffffffd3e9844503
[    1.632380] x13: ffffffd3e9844502 x12: 0000000000000038
[    1.632382] x11: ffffffffffffff00 x10: 0000000000000040
[    1.632384] x9 : ffffff801165e270 x8 : ffffff801165e268
[    1.632385] x7 : 0000000000000002 x6 : 00000000000000b2
[    1.632387] x5 : ffffffd3e9d8c2c0 x4 : 0000000000000000
[    1.632388] x3 : 0000000000000000 x2 : 0000000000000000
[    1.632390] x1 : 0000000000000000 x0 : ffffffd3e9844680
[    1.632392] Kernel panic - not syncing: Asynchronous SError Interrupt
[    1.632393] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.2.0-rc1-Jonny-14847-ge76f1d4a1828-dirty #33
[    1.632394] Hardware name: Annapurna Labs Alpine V3 EVP (DT)
[    1.632394] Call trace:
[    1.632395]  dump_backtrace+0x0/0x140
[    1.632395]  show_stack+0x14/0x20
[    1.632396]  dump_stack+0xa8/0xcc
[    1.632396]  panic+0x140/0x334
[    1.632397]  nmi_panic+0x6c/0x70
[    1.632398]  arm64_serror_panic+0x74/0x88
[    1.632398]  __pte_error+0x0/0x28
[    1.632399]  el1_error+0x84/0xf8
[    1.632400]  __pci_enable_msix_range+0x4e4/0x608
[    1.632400]  pci_alloc_irq_vectors_affinity+0xdc/0x150
[    1.632401]  pcie_port_device_register+0x2b8/0x4e0
[    1.632402]  pcie_portdrv_probe+0x34/0xf0

Notice that this quirk also disables MSI (which may work, but hasn't
been tested nor has a current use case), since currently there is no
standard way to disable only MSI-X.

Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
Reviewed-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com>
Reviewed-by: Andrew Murray <andrew.murray@arm.com>
---
 drivers/pci/quirks.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 8fe765592943..5a8ea5fdeae7 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -2977,6 +2977,24 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, 0x10a1,
 			quirk_msi_intx_disable_qca_bug);
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, 0xe091,
 			quirk_msi_intx_disable_qca_bug);
+
+/*
+ * Amazon's Annapurna Labs 1c36:0031 Root Ports don't support MSI-X, so it
+ * should be disabled on platforms where the device (mistakenly) advertises it.
+ *
+ * Notice that this quirk also disables MSI (which may work, but hasn't been
+ * tested), since currently there is no standard way to disable only MSI-X.
+ *
+ * The 0031 device id is reused for other non Root Port device types,
+ * therefore the quirk is registered for the PCI_CLASS_BRIDGE_PCI class.
+ */
+static void quirk_al_msi_disable(struct pci_dev *dev)
+{
+	dev->no_msi = 1;
+	pci_warn(dev, "Disabling MSI/MSI-X\n");
+}
+DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS, 0x0031,
+			      PCI_CLASS_BRIDGE_PCI, 8, quirk_al_msi_disable);
 #endif /* CONFIG_PCI_MSI */
 
 /*
-- 
2.17.1

Re: [PATCH v5 4/7] PCI: Add quirk to disable MSI-X support for Amazon's Annapurna Labs Root Port

[Bjorn Helgaas]

s/Add quirk to disable/Disable/ in subject

On Thu, Sep 05, 2019 at 05:00:18PM +0300, Jonathan Chocron wrote:
> The Root Port (identified by [1c36:0031]) doesn't support MSI-X. On some
> platforms it is configured to not advertise the capability at all, while
> on others it (mistakenly) does. This causes a panic during
> initialization by the pcieport driver, since it tries to configure the
> MSI-X capability. Specifically, when trying to access the MSI-X table
> a "non-existing addr" exception occurs.

MSI-X configuration is all in memory space (not config space), so I
guess this device has a BAR (maybe more) and the MSI-X table access
would be to something in that BAR?  Or is there junk in the capability
so we try to access something completely outside the BAR?  [I wonder if
it'd be worth adding some validation to make sure the Table and PBA
areas are contained in the BAR?]

The reason I'm curious about all this is because on the PCI side this
is probably an Unsupported Request or similar error, and on *most*
platforms, this does not cause a panic.  If it was a read, the read
usually gets ~0 data, and writes are dropped.

So my concern is that we'll avoid this panic by disabling MSI, but
we'll see other panics in other places.

> Example stacktrace snippet:

[Remove the timestamps since they're not really relevant and indent
the quote a couple spaces]

> [    1.632363] SError Interrupt on CPU2, code 0xbf000000 -- SError
> [    1.632364] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.2.0-rc1-Jonny-14847-ge76f1d4a1828-dirty #33
> [    1.632365] Hardware name: Annapurna Labs Alpine V3 EVP (DT)
> [    1.632365] pstate: 80000005 (Nzcv daif -PAN -UAO)
> [    1.632366] pc : __pci_enable_msix_range+0x4e4/0x608
> [    1.632367] lr : __pci_enable_msix_range+0x498/0x608
> [    1.632367] sp : ffffff80117db700
> [    1.632368] x29: ffffff80117db700 x28: 0000000000000001
> [    1.632370] x27: 0000000000000001 x26: 0000000000000000
> [    1.632372] x25: ffffffd3e9d8c0b0 x24: 0000000000000000
> [    1.632373] x23: 0000000000000000 x22: 0000000000000000
> [    1.632375] x21: 0000000000000001 x20: 0000000000000000
> [    1.632376] x19: ffffffd3e9d8c000 x18: ffffffffffffffff
> [    1.632378] x17: 0000000000000000 x16: 0000000000000000
> [    1.632379] x15: ffffff80116496c8 x14: ffffffd3e9844503
> [    1.632380] x13: ffffffd3e9844502 x12: 0000000000000038
> [    1.632382] x11: ffffffffffffff00 x10: 0000000000000040
> [    1.632384] x9 : ffffff801165e270 x8 : ffffff801165e268
> [    1.632385] x7 : 0000000000000002 x6 : 00000000000000b2
> [    1.632387] x5 : ffffffd3e9d8c2c0 x4 : 0000000000000000
> [    1.632388] x3 : 0000000000000000 x2 : 0000000000000000
> [    1.632390] x1 : 0000000000000000 x0 : ffffffd3e9844680
> [    1.632392] Kernel panic - not syncing: Asynchronous SError Interrupt
> [    1.632393] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.2.0-rc1-Jonny-14847-ge76f1d4a1828-dirty #33
> [    1.632394] Hardware name: Annapurna Labs Alpine V3 EVP (DT)
> [    1.632394] Call trace:
> [    1.632395]  dump_backtrace+0x0/0x140
> [    1.632395]  show_stack+0x14/0x20
> [    1.632396]  dump_stack+0xa8/0xcc
> [    1.632396]  panic+0x140/0x334
> [    1.632397]  nmi_panic+0x6c/0x70
> [    1.632398]  arm64_serror_panic+0x74/0x88
> [    1.632398]  __pte_error+0x0/0x28
> [    1.632399]  el1_error+0x84/0xf8
> [    1.632400]  __pci_enable_msix_range+0x4e4/0x608
> [    1.632400]  pci_alloc_irq_vectors_affinity+0xdc/0x150
> [    1.632401]  pcie_port_device_register+0x2b8/0x4e0
> [    1.632402]  pcie_portdrv_probe+0x34/0xf0
> 
> Notice that this quirk also disables MSI (which may work, but hasn't
> been tested nor has a current use case), since currently there is no
> standard way to disable only MSI-X.
> 
> Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
> Reviewed-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com>
> Reviewed-by: Andrew Murray <andrew.murray@arm.com>
> ---
>  drivers/pci/quirks.c | 18 ++++++++++++++++++
>  1 file changed, 18 insertions(+)
> 
> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> index 8fe765592943..5a8ea5fdeae7 100644
> --- a/drivers/pci/quirks.c
> +++ b/drivers/pci/quirks.c
> @@ -2977,6 +2977,24 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, 0x10a1,
>  			quirk_msi_intx_disable_qca_bug);
>  DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, 0xe091,
>  			quirk_msi_intx_disable_qca_bug);
> +
> +/*
> + * Amazon's Annapurna Labs 1c36:0031 Root Ports don't support MSI-X, so it
> + * should be disabled on platforms where the device (mistakenly) advertises it.
> + *
> + * Notice that this quirk also disables MSI (which may work, but hasn't been
> + * tested), since currently there is no standard way to disable only MSI-X.
> + *
> + * The 0031 device id is reused for other non Root Port device types,
> + * therefore the quirk is registered for the PCI_CLASS_BRIDGE_PCI class.
> + */
> +static void quirk_al_msi_disable(struct pci_dev *dev)
> +{
> +	dev->no_msi = 1;
> +	pci_warn(dev, "Disabling MSI/MSI-X\n");
> +}
> +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS, 0x0031,
> +			      PCI_CLASS_BRIDGE_PCI, 8, quirk_al_msi_disable);
>  #endif /* CONFIG_PCI_MSI */
>  
>  /*
> -- 
> 2.17.1
> 

Re: [PATCH v5 4/7] PCI: Add quirk to disable MSI-X support for Amazon's Annapurna Labs Root Port

[Lorenzo Pieralisi]

On Sat, Sep 07, 2019 at 11:55:42AM -0500, Bjorn Helgaas wrote:
> s/Add quirk to disable/Disable/ in subject
> 
> On Thu, Sep 05, 2019 at 05:00:18PM +0300, Jonathan Chocron wrote:
> > The Root Port (identified by [1c36:0031]) doesn't support MSI-X. On some
> > platforms it is configured to not advertise the capability at all, while
> > on others it (mistakenly) does. This causes a panic during
> > initialization by the pcieport driver, since it tries to configure the
> > MSI-X capability. Specifically, when trying to access the MSI-X table
> > a "non-existing addr" exception occurs.
> 
> MSI-X configuration is all in memory space (not config space), so I
> guess this device has a BAR (maybe more) and the MSI-X table access
> would be to something in that BAR?  Or is there junk in the capability
> so we try to access something completely outside the BAR?  [I wonder if
> it'd be worth adding some validation to make sure the Table and PBA
> areas are contained in the BAR?]
> 
> The reason I'm curious about all this is because on the PCI side this
> is probably an Unsupported Request or similar error, and on *most*
> platforms, this does not cause a panic.  If it was a read, the read
> usually gets ~0 data, and writes are dropped.
> 
> So my concern is that we'll avoid this panic by disabling MSI, but
> we'll see other panics in other places.
> 
> > Example stacktrace snippet:
> 
> [Remove the timestamps since they're not really relevant and indent
> the quote a couple spaces]

Jonathan, can you please clarify and update the patch so that we
can proceed with the series and try to get it into v5.4 ?

Thanks,
Lorenzo

> 
> > [    1.632363] SError Interrupt on CPU2, code 0xbf000000 -- SError
> > [    1.632364] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.2.0-rc1-Jonny-14847-ge76f1d4a1828-dirty #33
> > [    1.632365] Hardware name: Annapurna Labs Alpine V3 EVP (DT)
> > [    1.632365] pstate: 80000005 (Nzcv daif -PAN -UAO)
> > [    1.632366] pc : __pci_enable_msix_range+0x4e4/0x608
> > [    1.632367] lr : __pci_enable_msix_range+0x498/0x608
> > [    1.632367] sp : ffffff80117db700
> > [    1.632368] x29: ffffff80117db700 x28: 0000000000000001
> > [    1.632370] x27: 0000000000000001 x26: 0000000000000000
> > [    1.632372] x25: ffffffd3e9d8c0b0 x24: 0000000000000000
> > [    1.632373] x23: 0000000000000000 x22: 0000000000000000
> > [    1.632375] x21: 0000000000000001 x20: 0000000000000000
> > [    1.632376] x19: ffffffd3e9d8c000 x18: ffffffffffffffff
> > [    1.632378] x17: 0000000000000000 x16: 0000000000000000
> > [    1.632379] x15: ffffff80116496c8 x14: ffffffd3e9844503
> > [    1.632380] x13: ffffffd3e9844502 x12: 0000000000000038
> > [    1.632382] x11: ffffffffffffff00 x10: 0000000000000040
> > [    1.632384] x9 : ffffff801165e270 x8 : ffffff801165e268
> > [    1.632385] x7 : 0000000000000002 x6 : 00000000000000b2
> > [    1.632387] x5 : ffffffd3e9d8c2c0 x4 : 0000000000000000
> > [    1.632388] x3 : 0000000000000000 x2 : 0000000000000000
> > [    1.632390] x1 : 0000000000000000 x0 : ffffffd3e9844680
> > [    1.632392] Kernel panic - not syncing: Asynchronous SError Interrupt
> > [    1.632393] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.2.0-rc1-Jonny-14847-ge76f1d4a1828-dirty #33
> > [    1.632394] Hardware name: Annapurna Labs Alpine V3 EVP (DT)
> > [    1.632394] Call trace:
> > [    1.632395]  dump_backtrace+0x0/0x140
> > [    1.632395]  show_stack+0x14/0x20
> > [    1.632396]  dump_stack+0xa8/0xcc
> > [    1.632396]  panic+0x140/0x334
> > [    1.632397]  nmi_panic+0x6c/0x70
> > [    1.632398]  arm64_serror_panic+0x74/0x88
> > [    1.632398]  __pte_error+0x0/0x28
> > [    1.632399]  el1_error+0x84/0xf8
> > [    1.632400]  __pci_enable_msix_range+0x4e4/0x608
> > [    1.632400]  pci_alloc_irq_vectors_affinity+0xdc/0x150
> > [    1.632401]  pcie_port_device_register+0x2b8/0x4e0
> > [    1.632402]  pcie_portdrv_probe+0x34/0xf0
> > 
> > Notice that this quirk also disables MSI (which may work, but hasn't
> > been tested nor has a current use case), since currently there is no
> > standard way to disable only MSI-X.
> > 
> > Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
> > Reviewed-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com>
> > Reviewed-by: Andrew Murray <andrew.murray@arm.com>
> > ---
> >  drivers/pci/quirks.c | 18 ++++++++++++++++++
> >  1 file changed, 18 insertions(+)
> > 
> > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> > index 8fe765592943..5a8ea5fdeae7 100644
> > --- a/drivers/pci/quirks.c
> > +++ b/drivers/pci/quirks.c
> > @@ -2977,6 +2977,24 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, 0x10a1,
> >  			quirk_msi_intx_disable_qca_bug);
> >  DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, 0xe091,
> >  			quirk_msi_intx_disable_qca_bug);
> > +
> > +/*
> > + * Amazon's Annapurna Labs 1c36:0031 Root Ports don't support MSI-X, so it
> > + * should be disabled on platforms where the device (mistakenly) advertises it.
> > + *
> > + * Notice that this quirk also disables MSI (which may work, but hasn't been
> > + * tested), since currently there is no standard way to disable only MSI-X.
> > + *
> > + * The 0031 device id is reused for other non Root Port device types,
> > + * therefore the quirk is registered for the PCI_CLASS_BRIDGE_PCI class.
> > + */
> > +static void quirk_al_msi_disable(struct pci_dev *dev)
> > +{
> > +	dev->no_msi = 1;
> > +	pci_warn(dev, "Disabling MSI/MSI-X\n");
> > +}
> > +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS, 0x0031,
> > +			      PCI_CLASS_BRIDGE_PCI, 8, quirk_al_msi_disable);
> >  #endif /* CONFIG_PCI_MSI */
> >  
> >  /*
> > -- 
> > 2.17.1
> > 

Re: [PATCH v5 4/7] PCI: Add quirk to disable MSI-X support for Amazon's Annapurna Labs Root Port

[Chocron, Jonathan]

On Sat, 2019-09-07 at 11:55 -0500, Bjorn Helgaas wrote:
> s/Add quirk to disable/Disable/ in subject
> 
> On Thu, Sep 05, 2019 at 05:00:18PM +0300, Jonathan Chocron wrote:
> > The Root Port (identified by [1c36:0031]) doesn't support MSI-X. On
> > some
> > platforms it is configured to not advertise the capability at all,
> > while
> > on others it (mistakenly) does. This causes a panic during
> > initialization by the pcieport driver, since it tries to configure
> > the
> > MSI-X capability. Specifically, when trying to access the MSI-X
> > table
> > a "non-existing addr" exception occurs.
> 
> MSI-X configuration is all in memory space (not config space), so I
> guess this device has a BAR (maybe more) and the MSI-X table access
> would be to something in that BAR?  Or is there junk in the
> capability
> so we try to access something completely outside the BAR?  [I wonder
> if
> it'd be worth adding some validation to make sure the Table and PBA
> areas are contained in the BAR?]
> 
There is no actual memory allocated for the MSI-X table behind the BAR,
so it attempts to access non existing regs, resulting in the SError.

> The reason I'm curious about all this is because on the PCI side this
> is probably an Unsupported Request or similar error, and on *most*
> platforms, this does not cause a panic.  If it was a read, the read
> usually gets ~0 data, and writes are dropped.
> 
I'm not entirely sure how this is implemented in the HW (i.e. accessing
the Host controller's memory space), but I assume that an AXI error
returns, which translates to the Serror. I can verify with our HW
engineers if you think it is blocking.

> So my concern is that we'll avoid this panic by disabling MSI, but
> we'll see other panics in other places.
> 
> > Example stacktrace snippet:
> 
> [Remove the timestamps since they're not really relevant and indent
> the quote a couple spaces]
> 
Ack.

> > [    1.632363] SError Interrupt on CPU2, code 0xbf000000 -- SError
> > [    1.632364] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.2.0-rc1-
> > Jonny-14847-ge76f1d4a1828-dirty #33
> > [    1.632365] Hardware name: Annapurna Labs Alpine V3 EVP (DT)
> > [    1.632365] pstate: 80000005 (Nzcv daif -PAN -UAO)
> > [    1.632366] pc : __pci_enable_msix_range+0x4e4/0x608
> > [    1.632367] lr : __pci_enable_msix_range+0x498/0x608
> > [    1.632367] sp : ffffff80117db700
> > [    1.632368] x29: ffffff80117db700 x28: 0000000000000001
> > [    1.632370] x27: 0000000000000001 x26: 0000000000000000
> > [    1.632372] x25: ffffffd3e9d8c0b0 x24: 0000000000000000
> > [    1.632373] x23: 0000000000000000 x22: 0000000000000000
> > [    1.632375] x21: 0000000000000001 x20: 0000000000000000
> > [    1.632376] x19: ffffffd3e9d8c000 x18: ffffffffffffffff
> > [    1.632378] x17: 0000000000000000 x16: 0000000000000000
> > [    1.632379] x15: ffffff80116496c8 x14: ffffffd3e9844503
> > [    1.632380] x13: ffffffd3e9844502 x12: 0000000000000038
> > [    1.632382] x11: ffffffffffffff00 x10: 0000000000000040
> > [    1.632384] x9 : ffffff801165e270 x8 : ffffff801165e268
> > [    1.632385] x7 : 0000000000000002 x6 : 00000000000000b2
> > [    1.632387] x5 : ffffffd3e9d8c2c0 x4 : 0000000000000000
> > [    1.632388] x3 : 0000000000000000 x2 : 0000000000000000
> > [    1.632390] x1 : 0000000000000000 x0 : ffffffd3e9844680
> > [    1.632392] Kernel panic - not syncing: Asynchronous SError
> > Interrupt
> > [    1.632393] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.2.0-rc1-
> > Jonny-14847-ge76f1d4a1828-dirty #33
> > [    1.632394] Hardware name: Annapurna Labs Alpine V3 EVP (DT)
> > [    1.632394] Call trace:
> > [    1.632395]  dump_backtrace+0x0/0x140
> > [    1.632395]  show_stack+0x14/0x20
> > [    1.632396]  dump_stack+0xa8/0xcc
> > [    1.632396]  panic+0x140/0x334
> > [    1.632397]  nmi_panic+0x6c/0x70
> > [    1.632398]  arm64_serror_panic+0x74/0x88
> > [    1.632398]  __pte_error+0x0/0x28
> > [    1.632399]  el1_error+0x84/0xf8
> > [    1.632400]  __pci_enable_msix_range+0x4e4/0x608
> > [    1.632400]  pci_alloc_irq_vectors_affinity+0xdc/0x150
> > [    1.632401]  pcie_port_device_register+0x2b8/0x4e0
> > [    1.632402]  pcie_portdrv_probe+0x34/0xf0
> > 
> > Notice that this quirk also disables MSI (which may work, but
> > hasn't
> > been tested nor has a current use case), since currently there is
> > no
> > standard way to disable only MSI-X.
> > 
> > Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
> > Reviewed-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com>
> > Reviewed-by: Andrew Murray <andrew.murray@arm.com>
> > ---
> >  drivers/pci/quirks.c | 18 ++++++++++++++++++
> >  1 file changed, 18 insertions(+)
> > 
> > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> > index 8fe765592943..5a8ea5fdeae7 100644
> > --- a/drivers/pci/quirks.c
> > +++ b/drivers/pci/quirks.c
> > @@ -2977,6 +2977,24 @@
> > DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, 0x10a1,
> >  			quirk_msi_intx_disable_qca_bug);
> >  DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, 0xe091,
> >  			quirk_msi_intx_disable_qca_bug);
> > +
> > +/*
> > + * Amazon's Annapurna Labs 1c36:0031 Root Ports don't support MSI-
> > X, so it
> > + * should be disabled on platforms where the device (mistakenly)
> > advertises it.
> > + *
> > + * Notice that this quirk also disables MSI (which may work, but
> > hasn't been
> > + * tested), since currently there is no standard way to disable
> > only MSI-X.
> > + *
> > + * The 0031 device id is reused for other non Root Port device
> > types,
> > + * therefore the quirk is registered for the PCI_CLASS_BRIDGE_PCI
> > class.
> > + */
> > +static void quirk_al_msi_disable(struct pci_dev *dev)
> > +{
> > +	dev->no_msi = 1;
> > +	pci_warn(dev, "Disabling MSI/MSI-X\n");
> > +}
> > +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS,
> > 0x0031,
> > +			      PCI_CLASS_BRIDGE_PCI, 8,
> > quirk_al_msi_disable);
> >  #endif /* CONFIG_PCI_MSI */
> >  
> >  /*
> > -- 
> > 2.17.1
> > 

Re: [PATCH v5 4/7] PCI: Add quirk to disable MSI-X support for Amazon's Annapurna Labs Root Port

[Chocron, Jonathan]

On Sat, 2019-09-07 at 11:55 -0500, Bjorn Helgaas wrote:
> s/Add quirk to disable/Disable/ in subject
> 
> On Thu, Sep 05, 2019 at 05:00:18PM +0300, Jonathan Chocron wrote:
> > The Root Port (identified by [1c36:0031]) doesn't support MSI-X. On
> > some
> > platforms it is configured to not advertise the capability at all,
> > while
> > on others it (mistakenly) does. This causes a panic during
> > initialization by the pcieport driver, since it tries to configure
> > the
> > MSI-X capability. Specifically, when trying to access the MSI-X
> > table
> > a "non-existing addr" exception occurs.
> 
> MSI-X configuration is all in memory space (not config space), so I
> guess this device has a BAR (maybe more) and the MSI-X table access
> would be to something in that BAR?  Or is there junk in the
> capability
> so we try to access something completely outside the BAR?  [I wonder
> if
> it'd be worth adding some validation to make sure the Table and PBA
> areas are contained in the BAR?]
> 
There is no actual memory allocated for the MSI-X table behind the BAR,
so it attempts to access non existing regs, resulting in the SError.

> The reason I'm curious about all this is because on the PCI side this
> is probably an Unsupported Request or similar error, and on *most*
> platforms, this does not cause a panic.  If it was a read, the read
> usually gets ~0 data, and writes are dropped.
> 
I'm not entirely sure how this is implemented in the HW (i.e. accessing
the Host controller's memory space), but I assume that an AXI error
returns, which translates to the Serror. I can verify with our HW
engineers if you think it is blocking.

> So my concern is that we'll avoid this panic by disabling MSI, but
> we'll see other panics in other places.
> 
> > Example stacktrace snippet:
> 
> [Remove the timestamps since they're not really relevant and indent
> the quote a couple spaces]
> 
Ack.

> > [    1.632363] SError Interrupt on CPU2, code 0xbf000000 -- SError
> > [    1.632364] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.2.0-rc1-
> > Jonny-14847-ge76f1d4a1828-dirty #33
> > [    1.632365] Hardware name: Annapurna Labs Alpine V3 EVP (DT)
> > [    1.632365] pstate: 80000005 (Nzcv daif -PAN -UAO)
> > [    1.632366] pc : __pci_enable_msix_range+0x4e4/0x608
> > [    1.632367] lr : __pci_enable_msix_range+0x498/0x608
> > [    1.632367] sp : ffffff80117db700
> > [    1.632368] x29: ffffff80117db700 x28: 0000000000000001
> > [    1.632370] x27: 0000000000000001 x26: 0000000000000000
> > [    1.632372] x25: ffffffd3e9d8c0b0 x24: 0000000000000000
> > [    1.632373] x23: 0000000000000000 x22: 0000000000000000
> > [    1.632375] x21: 0000000000000001 x20: 0000000000000000
> > [    1.632376] x19: ffffffd3e9d8c000 x18: ffffffffffffffff
> > [    1.632378] x17: 0000000000000000 x16: 0000000000000000
> > [    1.632379] x15: ffffff80116496c8 x14: ffffffd3e9844503
> > [    1.632380] x13: ffffffd3e9844502 x12: 0000000000000038
> > [    1.632382] x11: ffffffffffffff00 x10: 0000000000000040
> > [    1.632384] x9 : ffffff801165e270 x8 : ffffff801165e268
> > [    1.632385] x7 : 0000000000000002 x6 : 00000000000000b2
> > [    1.632387] x5 : ffffffd3e9d8c2c0 x4 : 0000000000000000
> > [    1.632388] x3 : 0000000000000000 x2 : 0000000000000000
> > [    1.632390] x1 : 0000000000000000 x0 : ffffffd3e9844680
> > [    1.632392] Kernel panic - not syncing: Asynchronous SError
> > Interrupt
> > [    1.632393] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.2.0-rc1-
> > Jonny-14847-ge76f1d4a1828-dirty #33
> > [    1.632394] Hardware name: Annapurna Labs Alpine V3 EVP (DT)
> > [    1.632394] Call trace:
> > [    1.632395]  dump_backtrace+0x0/0x140
> > [    1.632395]  show_stack+0x14/0x20
> > [    1.632396]  dump_stack+0xa8/0xcc
> > [    1.632396]  panic+0x140/0x334
> > [    1.632397]  nmi_panic+0x6c/0x70
> > [    1.632398]  arm64_serror_panic+0x74/0x88
> > [    1.632398]  __pte_error+0x0/0x28
> > [    1.632399]  el1_error+0x84/0xf8
> > [    1.632400]  __pci_enable_msix_range+0x4e4/0x608
> > [    1.632400]  pci_alloc_irq_vectors_affinity+0xdc/0x150
> > [    1.632401]  pcie_port_device_register+0x2b8/0x4e0
> > [    1.632402]  pcie_portdrv_probe+0x34/0xf0
> > 
> > Notice that this quirk also disables MSI (which may work, but
> > hasn't
> > been tested nor has a current use case), since currently there is
> > no
> > standard way to disable only MSI-X.
> > 
> > Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
> > Reviewed-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com>
> > Reviewed-by: Andrew Murray <andrew.murray@arm.com>
> > ---
> >  drivers/pci/quirks.c | 18 ++++++++++++++++++
> >  1 file changed, 18 insertions(+)
> > 
> > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> > index 8fe765592943..5a8ea5fdeae7 100644
> > --- a/drivers/pci/quirks.c
> > +++ b/drivers/pci/quirks.c
> > @@ -2977,6 +2977,24 @@
> > DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, 0x10a1,
> >  			quirk_msi_intx_disable_qca_bug);
> >  DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, 0xe091,
> >  			quirk_msi_intx_disable_qca_bug);
> > +
> > +/*
> > + * Amazon's Annapurna Labs 1c36:0031 Root Ports don't support MSI-
> > X, so it
> > + * should be disabled on platforms where the device (mistakenly)
> > advertises it.
> > + *
> > + * Notice that this quirk also disables MSI (which may work, but
> > hasn't been
> > + * tested), since currently there is no standard way to disable
> > only MSI-X.
> > + *
> > + * The 0031 device id is reused for other non Root Port device
> > types,
> > + * therefore the quirk is registered for the PCI_CLASS_BRIDGE_PCI
> > class.
> > + */
> > +static void quirk_al_msi_disable(struct pci_dev *dev)
> > +{
> > +	dev->no_msi = 1;
> > +	pci_warn(dev, "Disabling MSI/MSI-X\n");
> > +}
> > +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS,
> > 0x0031,
> > +			      PCI_CLASS_BRIDGE_PCI, 8,
> > quirk_al_msi_disable);
> >  #endif /* CONFIG_PCI_MSI */
> >  
> >  /*
> > -- 
> > 2.17.1
> > 

[PATCH v5 5/7] dt-bindings: PCI: Add Amazon's Annapurna Labs PCIe host bridge binding

[Jonathan Chocron]

Document Amazon's Annapurna Labs PCIe host bridge.

Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
Reviewed-by: Andrew Murray <andrew.murray@arm.com>
Reviewed-by: Rob Herring <robh@kernel.org>
---
 .../devicetree/bindings/pci/pcie-al.txt       | 46 +++++++++++++++++++
 MAINTAINERS                                   |  3 +-
 2 files changed, 48 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/devicetree/bindings/pci/pcie-al.txt

diff --git a/Documentation/devicetree/bindings/pci/pcie-al.txt b/Documentation/devicetree/bindings/pci/pcie-al.txt
new file mode 100644
index 000000000000..557a5089229d
--- /dev/null
+++ b/Documentation/devicetree/bindings/pci/pcie-al.txt
@@ -0,0 +1,46 @@
+* Amazon Annapurna Labs PCIe host bridge
+
+Amazon's Annapurna Labs PCIe Host Controller is based on the Synopsys DesignWare
+PCI core. It inherits common properties defined in
+Documentation/devicetree/bindings/pci/designware-pcie.txt.
+
+Properties of the host controller node that differ from it are:
+
+- compatible:
+	Usage: required
+	Value type: <stringlist>
+	Definition: Value should contain
+			- "amazon,al-alpine-v2-pcie" for alpine_v2
+			- "amazon,al-alpine-v3-pcie" for alpine_v3
+
+- reg:
+	Usage: required
+	Value type: <prop-encoded-array>
+	Definition: Register ranges as listed in the reg-names property
+
+- reg-names:
+	Usage: required
+	Value type: <stringlist>
+	Definition: Must include the following entries
+			- "config"	PCIe ECAM space
+			- "controller"	AL proprietary registers
+			- "dbi"		Designware PCIe registers
+
+Example:
+
+	pcie-external0: pcie@fb600000 {
+		compatible = "amazon,al-alpine-v3-pcie";
+		reg = <0x0 0xfb600000 0x0 0x00100000
+		       0x0 0xfd800000 0x0 0x00010000
+		       0x0 0xfd810000 0x0 0x00001000>;
+		reg-names = "config", "controller", "dbi";
+		bus-range = <0 255>;
+		device_type = "pci";
+		#address-cells = <3>;
+		#size-cells = <2>;
+		#interrupt-cells = <1>;
+		interrupts = <GIC_SPI 49 IRQ_TYPE_LEVEL_HIGH>;
+		interrupt-map-mask = <0x00 0 0 7>;
+		interrupt-map = <0x0000 0 0 1 &gic GIC_SPI 41 IRQ_TYPE_LEVEL_HIGH>; /* INTa */
+		ranges = <0x02000000 0x0 0xc0010000 0x0 0xc0010000 0x0 0x07ff0000>;
+	};
diff --git a/MAINTAINERS b/MAINTAINERS
index 783569e3c4b4..d200b16fa95c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -12448,10 +12448,11 @@ T:	git git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/pci.git/
 S:	Supported
 F:	drivers/pci/controller/
 
-PCIE DRIVER FOR ANNAPURNA LABS
+PCIE DRIVER FOR AMAZON ANNAPURNA LABS
 M:	Jonathan Chocron <jonnyc@amazon.com>
 L:	linux-pci@vger.kernel.org
 S:	Maintained
+F:	Documentation/devicetree/bindings/pci/pcie-al.txt
 F:	drivers/pci/controller/dwc/pcie-al.c
 
 PCIE DRIVER FOR AMLOGIC MESON
-- 
2.17.1

[PATCH v5 5/7] dt-bindings: PCI: Add Amazon's Annapurna Labs PCIe host bridge binding

[Jonathan Chocron]

Document Amazon's Annapurna Labs PCIe host bridge.

Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
Reviewed-by: Andrew Murray <andrew.murray@arm.com>
Reviewed-by: Rob Herring <robh@kernel.org>
---
 .../devicetree/bindings/pci/pcie-al.txt       | 46 +++++++++++++++++++
 MAINTAINERS                                   |  3 +-
 2 files changed, 48 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/devicetree/bindings/pci/pcie-al.txt

diff --git a/Documentation/devicetree/bindings/pci/pcie-al.txt b/Documentation/devicetree/bindings/pci/pcie-al.txt
new file mode 100644
index 000000000000..557a5089229d
--- /dev/null
+++ b/Documentation/devicetree/bindings/pci/pcie-al.txt
@@ -0,0 +1,46 @@
+* Amazon Annapurna Labs PCIe host bridge
+
+Amazon's Annapurna Labs PCIe Host Controller is based on the Synopsys DesignWare
+PCI core. It inherits common properties defined in
+Documentation/devicetree/bindings/pci/designware-pcie.txt.
+
+Properties of the host controller node that differ from it are:
+
+- compatible:
+	Usage: required
+	Value type: <stringlist>
+	Definition: Value should contain
+			- "amazon,al-alpine-v2-pcie" for alpine_v2
+			- "amazon,al-alpine-v3-pcie" for alpine_v3
+
+- reg:
+	Usage: required
+	Value type: <prop-encoded-array>
+	Definition: Register ranges as listed in the reg-names property
+
+- reg-names:
+	Usage: required
+	Value type: <stringlist>
+	Definition: Must include the following entries
+			- "config"	PCIe ECAM space
+			- "controller"	AL proprietary registers
+			- "dbi"		Designware PCIe registers
+
+Example:
+
+	pcie-external0: pcie@fb600000 {
+		compatible = "amazon,al-alpine-v3-pcie";
+		reg = <0x0 0xfb600000 0x0 0x00100000
+		       0x0 0xfd800000 0x0 0x00010000
+		       0x0 0xfd810000 0x0 0x00001000>;
+		reg-names = "config", "controller", "dbi";
+		bus-range = <0 255>;
+		device_type = "pci";
+		#address-cells = <3>;
+		#size-cells = <2>;
+		#interrupt-cells = <1>;
+		interrupts = <GIC_SPI 49 IRQ_TYPE_LEVEL_HIGH>;
+		interrupt-map-mask = <0x00 0 0 7>;
+		interrupt-map = <0x0000 0 0 1 &gic GIC_SPI 41 IRQ_TYPE_LEVEL_HIGH>; /* INTa */
+		ranges = <0x02000000 0x0 0xc0010000 0x0 0xc0010000 0x0 0x07ff0000>;
+	};
diff --git a/MAINTAINERS b/MAINTAINERS
index 783569e3c4b4..d200b16fa95c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -12448,10 +12448,11 @@ T:	git git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/pci.git/
 S:	Supported
 F:	drivers/pci/controller/
 
-PCIE DRIVER FOR ANNAPURNA LABS
+PCIE DRIVER FOR AMAZON ANNAPURNA LABS
 M:	Jonathan Chocron <jonnyc@amazon.com>
 L:	linux-pci@vger.kernel.org
 S:	Maintained
+F:	Documentation/devicetree/bindings/pci/pcie-al.txt
 F:	drivers/pci/controller/dwc/pcie-al.c
 
 PCIE DRIVER FOR AMLOGIC MESON
-- 
2.17.1

[PATCH v5 6/7] PCI: dwc: al: Add support for DW based driver type

[Jonathan Chocron]

This driver is DT based and utilizes the DesignWare APIs.

It allows using a smaller ECAM range for a larger bus range -
usually an entire bus uses 1MB of address space, but the driver
can use it for a larger number of buses. This is achieved by using a HW
mechanism which allows changing the BUS part of the "final" outgoing
config transaction. There are 2 HW regs, one which is basically a
bitmask determining which bits to take from the AXI transaction itself
and another which holds the complementary part programmed by the
driver.

All link initializations are handled by the boot FW.

Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
Reviewed-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com>
Reviewed-by: Andrew Murray <andrew.murray@arm.com>
---
 drivers/pci/controller/dwc/Kconfig   |  12 +
 drivers/pci/controller/dwc/pcie-al.c | 365 +++++++++++++++++++++++++++
 2 files changed, 377 insertions(+)

diff --git a/drivers/pci/controller/dwc/Kconfig b/drivers/pci/controller/dwc/Kconfig
index 4fada2e93285..0ba988b5b5bc 100644
--- a/drivers/pci/controller/dwc/Kconfig
+++ b/drivers/pci/controller/dwc/Kconfig
@@ -256,4 +256,16 @@ config PCIE_UNIPHIER
 	  Say Y here if you want PCIe controller support on UniPhier SoCs.
 	  This driver supports LD20 and PXs3 SoCs.
 
+config PCIE_AL
+	bool "Amazon Annapurna Labs PCIe controller"
+	depends on OF && (ARM64 || COMPILE_TEST)
+	depends on PCI_MSI_IRQ_DOMAIN
+	select PCIE_DW_HOST
+	help
+	  Say Y here to enable support of the Amazon's Annapurna Labs PCIe
+	  controller IP on Amazon SoCs. The PCIe controller uses the DesignWare
+	  core plus Annapurna Labs proprietary hardware wrappers. This is
+	  required only for DT-based platforms. ACPI platforms with the
+	  Annapurna Labs PCIe controller don't need to enable this.
+
 endmenu
diff --git a/drivers/pci/controller/dwc/pcie-al.c b/drivers/pci/controller/dwc/pcie-al.c
index 3ab58f0584a8..1eeda2f6371f 100644
--- a/drivers/pci/controller/dwc/pcie-al.c
+++ b/drivers/pci/controller/dwc/pcie-al.c
@@ -91,3 +91,368 @@ struct pci_ecam_ops al_pcie_ops = {
 };
 
 #endif /* defined(CONFIG_ACPI) && defined(CONFIG_PCI_QUIRKS) */
+
+#ifdef CONFIG_PCIE_AL
+
+#include <linux/of_pci.h>
+#include "pcie-designware.h"
+
+#define AL_PCIE_REV_ID_2	2
+#define AL_PCIE_REV_ID_3	3
+#define AL_PCIE_REV_ID_4	4
+
+#define AXI_BASE_OFFSET		0x0
+
+#define DEVICE_ID_OFFSET	0x16c
+
+#define DEVICE_REV_ID			0x0
+#define DEVICE_REV_ID_DEV_ID_MASK	GENMASK(31, 16)
+
+#define DEVICE_REV_ID_DEV_ID_X4		0
+#define DEVICE_REV_ID_DEV_ID_X8		2
+#define DEVICE_REV_ID_DEV_ID_X16	4
+
+#define OB_CTRL_REV1_2_OFFSET	0x0040
+#define OB_CTRL_REV3_5_OFFSET	0x0030
+
+#define CFG_TARGET_BUS			0x0
+#define CFG_TARGET_BUS_MASK_MASK	GENMASK(7, 0)
+#define CFG_TARGET_BUS_BUSNUM_MASK	GENMASK(15, 8)
+
+#define CFG_CONTROL			0x4
+#define CFG_CONTROL_SUBBUS_MASK		GENMASK(15, 8)
+#define CFG_CONTROL_SEC_BUS_MASK	GENMASK(23, 16)
+
+struct al_pcie_reg_offsets {
+	unsigned int ob_ctrl;
+};
+
+struct al_pcie_target_bus_cfg {
+	u8 reg_val;
+	u8 reg_mask;
+	u8 ecam_mask;
+};
+
+struct al_pcie {
+	struct dw_pcie *pci;
+	void __iomem *controller_base; /* base of PCIe unit (not DW core) */
+	struct device *dev;
+	resource_size_t ecam_size;
+	unsigned int controller_rev_id;
+	struct al_pcie_reg_offsets reg_offsets;
+	struct al_pcie_target_bus_cfg target_bus_cfg;
+};
+
+#define PCIE_ECAM_DEVFN(x)		(((x) & 0xff) << 12)
+
+#define to_al_pcie(x)		dev_get_drvdata((x)->dev)
+
+static inline u32 al_pcie_controller_readl(struct al_pcie *pcie, u32 offset)
+{
+	return readl_relaxed(pcie->controller_base + offset);
+}
+
+static inline void al_pcie_controller_writel(struct al_pcie *pcie, u32 offset,
+					     u32 val)
+{
+	writel_relaxed(val, pcie->controller_base + offset);
+}
+
+static int al_pcie_rev_id_get(struct al_pcie *pcie, unsigned int *rev_id)
+{
+	u32 dev_rev_id_val;
+	u32 dev_id_val;
+
+	dev_rev_id_val = al_pcie_controller_readl(pcie, AXI_BASE_OFFSET +
+						  DEVICE_ID_OFFSET +
+						  DEVICE_REV_ID);
+	dev_id_val = FIELD_GET(DEVICE_REV_ID_DEV_ID_MASK, dev_rev_id_val);
+
+	switch (dev_id_val) {
+	case DEVICE_REV_ID_DEV_ID_X4:
+		*rev_id = AL_PCIE_REV_ID_2;
+		break;
+	case DEVICE_REV_ID_DEV_ID_X8:
+		*rev_id = AL_PCIE_REV_ID_3;
+		break;
+	case DEVICE_REV_ID_DEV_ID_X16:
+		*rev_id = AL_PCIE_REV_ID_4;
+		break;
+	default:
+		dev_err(pcie->dev, "Unsupported dev_id_val (0x%x)\n",
+			dev_id_val);
+		return -EINVAL;
+	}
+
+	dev_dbg(pcie->dev, "dev_id_val: 0x%x\n", dev_id_val);
+
+	return 0;
+}
+
+static int al_pcie_reg_offsets_set(struct al_pcie *pcie)
+{
+	switch (pcie->controller_rev_id) {
+	case AL_PCIE_REV_ID_2:
+		pcie->reg_offsets.ob_ctrl = OB_CTRL_REV1_2_OFFSET;
+		break;
+	case AL_PCIE_REV_ID_3:
+	case AL_PCIE_REV_ID_4:
+		pcie->reg_offsets.ob_ctrl = OB_CTRL_REV3_5_OFFSET;
+		break;
+	default:
+		dev_err(pcie->dev, "Unsupported controller rev_id: 0x%x\n",
+			pcie->controller_rev_id);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static inline void al_pcie_target_bus_set(struct al_pcie *pcie,
+					  u8 target_bus,
+					  u8 mask_target_bus)
+{
+	u32 reg;
+
+	reg = FIELD_PREP(CFG_TARGET_BUS_MASK_MASK, mask_target_bus) |
+	      FIELD_PREP(CFG_TARGET_BUS_BUSNUM_MASK, target_bus);
+
+	al_pcie_controller_writel(pcie, AXI_BASE_OFFSET +
+				  pcie->reg_offsets.ob_ctrl + CFG_TARGET_BUS,
+				  reg);
+}
+
+static void __iomem *al_pcie_conf_addr_map(struct al_pcie *pcie,
+					   unsigned int busnr,
+					   unsigned int devfn)
+{
+	struct al_pcie_target_bus_cfg *target_bus_cfg = &pcie->target_bus_cfg;
+	unsigned int busnr_ecam = busnr & target_bus_cfg->ecam_mask;
+	unsigned int busnr_reg = busnr & target_bus_cfg->reg_mask;
+	struct pcie_port *pp = &pcie->pci->pp;
+	void __iomem *pci_base_addr;
+
+	pci_base_addr = (void __iomem *)((uintptr_t)pp->va_cfg0_base +
+					 (busnr_ecam << 20) +
+					 PCIE_ECAM_DEVFN(devfn));
+
+	if (busnr_reg != target_bus_cfg->reg_val) {
+		dev_dbg(pcie->pci->dev, "Changing target bus busnum val from 0x%x to 0x%x\n",
+			target_bus_cfg->reg_val, busnr_reg);
+		target_bus_cfg->reg_val = busnr_reg;
+		al_pcie_target_bus_set(pcie,
+				       target_bus_cfg->reg_val,
+				       target_bus_cfg->reg_mask);
+	}
+
+	return pci_base_addr;
+}
+
+static int al_pcie_rd_other_conf(struct pcie_port *pp, struct pci_bus *bus,
+				 unsigned int devfn, int where, int size,
+				 u32 *val)
+{
+	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+	struct al_pcie *pcie = to_al_pcie(pci);
+	unsigned int busnr = bus->number;
+	void __iomem *pci_addr;
+	int rc;
+
+	pci_addr = al_pcie_conf_addr_map(pcie, busnr, devfn);
+
+	rc = dw_pcie_read(pci_addr + where, size, val);
+
+	dev_dbg(pci->dev, "%d-byte config read from %04x:%02x:%02x.%d offset 0x%x (pci_addr: 0x%px) - val:0x%x\n",
+		size, pci_domain_nr(bus), bus->number,
+		PCI_SLOT(devfn), PCI_FUNC(devfn), where,
+		(pci_addr + where), *val);
+
+	return rc;
+}
+
+static int al_pcie_wr_other_conf(struct pcie_port *pp, struct pci_bus *bus,
+				 unsigned int devfn, int where, int size,
+				 u32 val)
+{
+	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+	struct al_pcie *pcie = to_al_pcie(pci);
+	unsigned int busnr = bus->number;
+	void __iomem *pci_addr;
+	int rc;
+
+	pci_addr = al_pcie_conf_addr_map(pcie, busnr, devfn);
+
+	rc = dw_pcie_write(pci_addr + where, size, val);
+
+	dev_dbg(pci->dev, "%d-byte config write to %04x:%02x:%02x.%d offset 0x%x (pci_addr: 0x%px) - val:0x%x\n",
+		size, pci_domain_nr(bus), bus->number,
+		PCI_SLOT(devfn), PCI_FUNC(devfn), where,
+		(pci_addr + where), val);
+
+	return rc;
+}
+
+static void al_pcie_config_prepare(struct al_pcie *pcie)
+{
+	struct al_pcie_target_bus_cfg *target_bus_cfg;
+	struct pcie_port *pp = &pcie->pci->pp;
+	unsigned int ecam_bus_mask;
+	u32 cfg_control_offset;
+	u8 subordinate_bus;
+	u8 secondary_bus;
+	u32 cfg_control;
+	u32 reg;
+
+	target_bus_cfg = &pcie->target_bus_cfg;
+
+	ecam_bus_mask = (pcie->ecam_size >> 20) - 1;
+	if (ecam_bus_mask > 255) {
+		dev_warn(pcie->dev, "ECAM window size is larger than 256MB. Cutting off at 256\n");
+		ecam_bus_mask = 255;
+	}
+
+	/* This portion is taken from the transaction address */
+	target_bus_cfg->ecam_mask = ecam_bus_mask;
+	/* This portion is taken from the cfg_target_bus reg */
+	target_bus_cfg->reg_mask = ~target_bus_cfg->ecam_mask;
+	target_bus_cfg->reg_val = pp->busn->start & target_bus_cfg->reg_mask;
+
+	al_pcie_target_bus_set(pcie, target_bus_cfg->reg_val,
+			       target_bus_cfg->reg_mask);
+
+	secondary_bus = pp->busn->start + 1;
+	subordinate_bus = pp->busn->end;
+
+	/* Set the valid values of secondary and subordinate buses */
+	cfg_control_offset = AXI_BASE_OFFSET + pcie->reg_offsets.ob_ctrl +
+			     CFG_CONTROL;
+
+	cfg_control = al_pcie_controller_readl(pcie, cfg_control_offset);
+
+	reg = cfg_control &
+	      ~(CFG_CONTROL_SEC_BUS_MASK | CFG_CONTROL_SUBBUS_MASK);
+
+	reg |= FIELD_PREP(CFG_CONTROL_SUBBUS_MASK, subordinate_bus) |
+	       FIELD_PREP(CFG_CONTROL_SEC_BUS_MASK, secondary_bus);
+
+	al_pcie_controller_writel(pcie, cfg_control_offset, reg);
+}
+
+static int al_pcie_host_init(struct pcie_port *pp)
+{
+	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+	struct al_pcie *pcie = to_al_pcie(pci);
+	int rc;
+
+	rc = al_pcie_rev_id_get(pcie, &pcie->controller_rev_id);
+	if (rc)
+		return rc;
+
+	rc = al_pcie_reg_offsets_set(pcie);
+	if (rc)
+		return rc;
+
+	al_pcie_config_prepare(pcie);
+
+	return 0;
+}
+
+static const struct dw_pcie_host_ops al_pcie_host_ops = {
+	.rd_other_conf = al_pcie_rd_other_conf,
+	.wr_other_conf = al_pcie_wr_other_conf,
+	.host_init = al_pcie_host_init,
+};
+
+static int al_add_pcie_port(struct pcie_port *pp,
+			    struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	int ret;
+
+	pp->ops = &al_pcie_host_ops;
+
+	ret = dw_pcie_host_init(pp);
+	if (ret) {
+		dev_err(dev, "failed to initialize host\n");
+		return ret;
+	}
+
+	return 0;
+}
+
+static const struct dw_pcie_ops dw_pcie_ops = {
+};
+
+static int al_pcie_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct resource *controller_res;
+	struct resource *ecam_res;
+	struct resource *dbi_res;
+	struct al_pcie *al_pcie;
+	struct dw_pcie *pci;
+
+	al_pcie = devm_kzalloc(dev, sizeof(*al_pcie), GFP_KERNEL);
+	if (!al_pcie)
+		return -ENOMEM;
+
+	pci = devm_kzalloc(dev, sizeof(*pci), GFP_KERNEL);
+	if (!pci)
+		return -ENOMEM;
+
+	pci->dev = dev;
+	pci->ops = &dw_pcie_ops;
+
+	al_pcie->pci = pci;
+	al_pcie->dev = dev;
+
+	dbi_res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "dbi");
+	pci->dbi_base = devm_pci_remap_cfg_resource(dev, dbi_res);
+	if (IS_ERR(pci->dbi_base)) {
+		dev_err(dev, "couldn't remap dbi base %pR\n", dbi_res);
+		return PTR_ERR(pci->dbi_base);
+	}
+
+	ecam_res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "config");
+	if (!ecam_res) {
+		dev_err(dev, "couldn't find 'config' reg in DT\n");
+		return -ENOENT;
+	}
+	al_pcie->ecam_size = resource_size(ecam_res);
+
+	controller_res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
+						      "controller");
+	al_pcie->controller_base = devm_ioremap_resource(dev, controller_res);
+	if (IS_ERR(al_pcie->controller_base)) {
+		dev_err(dev, "couldn't remap controller base %pR\n",
+			controller_res);
+		return PTR_ERR(al_pcie->controller_base);
+	}
+
+	dev_dbg(dev, "From DT: dbi_base: %pR, controller_base: %pR\n",
+		dbi_res, controller_res);
+
+	platform_set_drvdata(pdev, al_pcie);
+
+	return al_add_pcie_port(&pci->pp, pdev);
+}
+
+static const struct of_device_id al_pcie_of_match[] = {
+	{ .compatible = "amazon,al-alpine-v2-pcie",
+	},
+	{ .compatible = "amazon,al-alpine-v3-pcie",
+	},
+	{},
+};
+
+static struct platform_driver al_pcie_driver = {
+	.driver = {
+		.name	= "al-pcie",
+		.of_match_table = al_pcie_of_match,
+		.suppress_bind_attrs = true,
+	},
+	.probe = al_pcie_probe,
+};
+builtin_platform_driver(al_pcie_driver);
+
+#endif /* CONFIG_PCIE_AL*/
-- 
2.17.1

[PATCH v5 6/7] PCI: dwc: al: Add support for DW based driver type

[Jonathan Chocron]

This driver is DT based and utilizes the DesignWare APIs.

It allows using a smaller ECAM range for a larger bus range -
usually an entire bus uses 1MB of address space, but the driver
can use it for a larger number of buses. This is achieved by using a HW
mechanism which allows changing the BUS part of the "final" outgoing
config transaction. There are 2 HW regs, one which is basically a
bitmask determining which bits to take from the AXI transaction itself
and another which holds the complementary part programmed by the
driver.

All link initializations are handled by the boot FW.

Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
Reviewed-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com>
Reviewed-by: Andrew Murray <andrew.murray@arm.com>
---
 drivers/pci/controller/dwc/Kconfig   |  12 +
 drivers/pci/controller/dwc/pcie-al.c | 365 +++++++++++++++++++++++++++
 2 files changed, 377 insertions(+)

diff --git a/drivers/pci/controller/dwc/Kconfig b/drivers/pci/controller/dwc/Kconfig
index 4fada2e93285..0ba988b5b5bc 100644
--- a/drivers/pci/controller/dwc/Kconfig
+++ b/drivers/pci/controller/dwc/Kconfig
@@ -256,4 +256,16 @@ config PCIE_UNIPHIER
 	  Say Y here if you want PCIe controller support on UniPhier SoCs.
 	  This driver supports LD20 and PXs3 SoCs.
 
+config PCIE_AL
+	bool "Amazon Annapurna Labs PCIe controller"
+	depends on OF && (ARM64 || COMPILE_TEST)
+	depends on PCI_MSI_IRQ_DOMAIN
+	select PCIE_DW_HOST
+	help
+	  Say Y here to enable support of the Amazon's Annapurna Labs PCIe
+	  controller IP on Amazon SoCs. The PCIe controller uses the DesignWare
+	  core plus Annapurna Labs proprietary hardware wrappers. This is
+	  required only for DT-based platforms. ACPI platforms with the
+	  Annapurna Labs PCIe controller don't need to enable this.
+
 endmenu
diff --git a/drivers/pci/controller/dwc/pcie-al.c b/drivers/pci/controller/dwc/pcie-al.c
index 3ab58f0584a8..1eeda2f6371f 100644
--- a/drivers/pci/controller/dwc/pcie-al.c
+++ b/drivers/pci/controller/dwc/pcie-al.c
@@ -91,3 +91,368 @@ struct pci_ecam_ops al_pcie_ops = {
 };
 
 #endif /* defined(CONFIG_ACPI) && defined(CONFIG_PCI_QUIRKS) */
+
+#ifdef CONFIG_PCIE_AL
+
+#include <linux/of_pci.h>
+#include "pcie-designware.h"
+
+#define AL_PCIE_REV_ID_2	2
+#define AL_PCIE_REV_ID_3	3
+#define AL_PCIE_REV_ID_4	4
+
+#define AXI_BASE_OFFSET		0x0
+
+#define DEVICE_ID_OFFSET	0x16c
+
+#define DEVICE_REV_ID			0x0
+#define DEVICE_REV_ID_DEV_ID_MASK	GENMASK(31, 16)
+
+#define DEVICE_REV_ID_DEV_ID_X4		0
+#define DEVICE_REV_ID_DEV_ID_X8		2
+#define DEVICE_REV_ID_DEV_ID_X16	4
+
+#define OB_CTRL_REV1_2_OFFSET	0x0040
+#define OB_CTRL_REV3_5_OFFSET	0x0030
+
+#define CFG_TARGET_BUS			0x0
+#define CFG_TARGET_BUS_MASK_MASK	GENMASK(7, 0)
+#define CFG_TARGET_BUS_BUSNUM_MASK	GENMASK(15, 8)
+
+#define CFG_CONTROL			0x4
+#define CFG_CONTROL_SUBBUS_MASK		GENMASK(15, 8)
+#define CFG_CONTROL_SEC_BUS_MASK	GENMASK(23, 16)
+
+struct al_pcie_reg_offsets {
+	unsigned int ob_ctrl;
+};
+
+struct al_pcie_target_bus_cfg {
+	u8 reg_val;
+	u8 reg_mask;
+	u8 ecam_mask;
+};
+
+struct al_pcie {
+	struct dw_pcie *pci;
+	void __iomem *controller_base; /* base of PCIe unit (not DW core) */
+	struct device *dev;
+	resource_size_t ecam_size;
+	unsigned int controller_rev_id;
+	struct al_pcie_reg_offsets reg_offsets;
+	struct al_pcie_target_bus_cfg target_bus_cfg;
+};
+
+#define PCIE_ECAM_DEVFN(x)		(((x) & 0xff) << 12)
+
+#define to_al_pcie(x)		dev_get_drvdata((x)->dev)
+
+static inline u32 al_pcie_controller_readl(struct al_pcie *pcie, u32 offset)
+{
+	return readl_relaxed(pcie->controller_base + offset);
+}
+
+static inline void al_pcie_controller_writel(struct al_pcie *pcie, u32 offset,
+					     u32 val)
+{
+	writel_relaxed(val, pcie->controller_base + offset);
+}
+
+static int al_pcie_rev_id_get(struct al_pcie *pcie, unsigned int *rev_id)
+{
+	u32 dev_rev_id_val;
+	u32 dev_id_val;
+
+	dev_rev_id_val = al_pcie_controller_readl(pcie, AXI_BASE_OFFSET +
+						  DEVICE_ID_OFFSET +
+						  DEVICE_REV_ID);
+	dev_id_val = FIELD_GET(DEVICE_REV_ID_DEV_ID_MASK, dev_rev_id_val);
+
+	switch (dev_id_val) {
+	case DEVICE_REV_ID_DEV_ID_X4:
+		*rev_id = AL_PCIE_REV_ID_2;
+		break;
+	case DEVICE_REV_ID_DEV_ID_X8:
+		*rev_id = AL_PCIE_REV_ID_3;
+		break;
+	case DEVICE_REV_ID_DEV_ID_X16:
+		*rev_id = AL_PCIE_REV_ID_4;
+		break;
+	default:
+		dev_err(pcie->dev, "Unsupported dev_id_val (0x%x)\n",
+			dev_id_val);
+		return -EINVAL;
+	}
+
+	dev_dbg(pcie->dev, "dev_id_val: 0x%x\n", dev_id_val);
+
+	return 0;
+}
+
+static int al_pcie_reg_offsets_set(struct al_pcie *pcie)
+{
+	switch (pcie->controller_rev_id) {
+	case AL_PCIE_REV_ID_2:
+		pcie->reg_offsets.ob_ctrl = OB_CTRL_REV1_2_OFFSET;
+		break;
+	case AL_PCIE_REV_ID_3:
+	case AL_PCIE_REV_ID_4:
+		pcie->reg_offsets.ob_ctrl = OB_CTRL_REV3_5_OFFSET;
+		break;
+	default:
+		dev_err(pcie->dev, "Unsupported controller rev_id: 0x%x\n",
+			pcie->controller_rev_id);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static inline void al_pcie_target_bus_set(struct al_pcie *pcie,
+					  u8 target_bus,
+					  u8 mask_target_bus)
+{
+	u32 reg;
+
+	reg = FIELD_PREP(CFG_TARGET_BUS_MASK_MASK, mask_target_bus) |
+	      FIELD_PREP(CFG_TARGET_BUS_BUSNUM_MASK, target_bus);
+
+	al_pcie_controller_writel(pcie, AXI_BASE_OFFSET +
+				  pcie->reg_offsets.ob_ctrl + CFG_TARGET_BUS,
+				  reg);
+}
+
+static void __iomem *al_pcie_conf_addr_map(struct al_pcie *pcie,
+					   unsigned int busnr,
+					   unsigned int devfn)
+{
+	struct al_pcie_target_bus_cfg *target_bus_cfg = &pcie->target_bus_cfg;
+	unsigned int busnr_ecam = busnr & target_bus_cfg->ecam_mask;
+	unsigned int busnr_reg = busnr & target_bus_cfg->reg_mask;
+	struct pcie_port *pp = &pcie->pci->pp;
+	void __iomem *pci_base_addr;
+
+	pci_base_addr = (void __iomem *)((uintptr_t)pp->va_cfg0_base +
+					 (busnr_ecam << 20) +
+					 PCIE_ECAM_DEVFN(devfn));
+
+	if (busnr_reg != target_bus_cfg->reg_val) {
+		dev_dbg(pcie->pci->dev, "Changing target bus busnum val from 0x%x to 0x%x\n",
+			target_bus_cfg->reg_val, busnr_reg);
+		target_bus_cfg->reg_val = busnr_reg;
+		al_pcie_target_bus_set(pcie,
+				       target_bus_cfg->reg_val,
+				       target_bus_cfg->reg_mask);
+	}
+
+	return pci_base_addr;
+}
+
+static int al_pcie_rd_other_conf(struct pcie_port *pp, struct pci_bus *bus,
+				 unsigned int devfn, int where, int size,
+				 u32 *val)
+{
+	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+	struct al_pcie *pcie = to_al_pcie(pci);
+	unsigned int busnr = bus->number;
+	void __iomem *pci_addr;
+	int rc;
+
+	pci_addr = al_pcie_conf_addr_map(pcie, busnr, devfn);
+
+	rc = dw_pcie_read(pci_addr + where, size, val);
+
+	dev_dbg(pci->dev, "%d-byte config read from %04x:%02x:%02x.%d offset 0x%x (pci_addr: 0x%px) - val:0x%x\n",
+		size, pci_domain_nr(bus), bus->number,
+		PCI_SLOT(devfn), PCI_FUNC(devfn), where,
+		(pci_addr + where), *val);
+
+	return rc;
+}
+
+static int al_pcie_wr_other_conf(struct pcie_port *pp, struct pci_bus *bus,
+				 unsigned int devfn, int where, int size,
+				 u32 val)
+{
+	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+	struct al_pcie *pcie = to_al_pcie(pci);
+	unsigned int busnr = bus->number;
+	void __iomem *pci_addr;
+	int rc;
+
+	pci_addr = al_pcie_conf_addr_map(pcie, busnr, devfn);
+
+	rc = dw_pcie_write(pci_addr + where, size, val);
+
+	dev_dbg(pci->dev, "%d-byte config write to %04x:%02x:%02x.%d offset 0x%x (pci_addr: 0x%px) - val:0x%x\n",
+		size, pci_domain_nr(bus), bus->number,
+		PCI_SLOT(devfn), PCI_FUNC(devfn), where,
+		(pci_addr + where), val);
+
+	return rc;
+}
+
+static void al_pcie_config_prepare(struct al_pcie *pcie)
+{
+	struct al_pcie_target_bus_cfg *target_bus_cfg;
+	struct pcie_port *pp = &pcie->pci->pp;
+	unsigned int ecam_bus_mask;
+	u32 cfg_control_offset;
+	u8 subordinate_bus;
+	u8 secondary_bus;
+	u32 cfg_control;
+	u32 reg;
+
+	target_bus_cfg = &pcie->target_bus_cfg;
+
+	ecam_bus_mask = (pcie->ecam_size >> 20) - 1;
+	if (ecam_bus_mask > 255) {
+		dev_warn(pcie->dev, "ECAM window size is larger than 256MB. Cutting off at 256\n");
+		ecam_bus_mask = 255;
+	}
+
+	/* This portion is taken from the transaction address */
+	target_bus_cfg->ecam_mask = ecam_bus_mask;
+	/* This portion is taken from the cfg_target_bus reg */
+	target_bus_cfg->reg_mask = ~target_bus_cfg->ecam_mask;
+	target_bus_cfg->reg_val = pp->busn->start & target_bus_cfg->reg_mask;
+
+	al_pcie_target_bus_set(pcie, target_bus_cfg->reg_val,
+			       target_bus_cfg->reg_mask);
+
+	secondary_bus = pp->busn->start + 1;
+	subordinate_bus = pp->busn->end;
+
+	/* Set the valid values of secondary and subordinate buses */
+	cfg_control_offset = AXI_BASE_OFFSET + pcie->reg_offsets.ob_ctrl +
+			     CFG_CONTROL;
+
+	cfg_control = al_pcie_controller_readl(pcie, cfg_control_offset);
+
+	reg = cfg_control &
+	      ~(CFG_CONTROL_SEC_BUS_MASK | CFG_CONTROL_SUBBUS_MASK);
+
+	reg |= FIELD_PREP(CFG_CONTROL_SUBBUS_MASK, subordinate_bus) |
+	       FIELD_PREP(CFG_CONTROL_SEC_BUS_MASK, secondary_bus);
+
+	al_pcie_controller_writel(pcie, cfg_control_offset, reg);
+}
+
+static int al_pcie_host_init(struct pcie_port *pp)
+{
+	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+	struct al_pcie *pcie = to_al_pcie(pci);
+	int rc;
+
+	rc = al_pcie_rev_id_get(pcie, &pcie->controller_rev_id);
+	if (rc)
+		return rc;
+
+	rc = al_pcie_reg_offsets_set(pcie);
+	if (rc)
+		return rc;
+
+	al_pcie_config_prepare(pcie);
+
+	return 0;
+}
+
+static const struct dw_pcie_host_ops al_pcie_host_ops = {
+	.rd_other_conf = al_pcie_rd_other_conf,
+	.wr_other_conf = al_pcie_wr_other_conf,
+	.host_init = al_pcie_host_init,
+};
+
+static int al_add_pcie_port(struct pcie_port *pp,
+			    struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	int ret;
+
+	pp->ops = &al_pcie_host_ops;
+
+	ret = dw_pcie_host_init(pp);
+	if (ret) {
+		dev_err(dev, "failed to initialize host\n");
+		return ret;
+	}
+
+	return 0;
+}
+
+static const struct dw_pcie_ops dw_pcie_ops = {
+};
+
+static int al_pcie_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct resource *controller_res;
+	struct resource *ecam_res;
+	struct resource *dbi_res;
+	struct al_pcie *al_pcie;
+	struct dw_pcie *pci;
+
+	al_pcie = devm_kzalloc(dev, sizeof(*al_pcie), GFP_KERNEL);
+	if (!al_pcie)
+		return -ENOMEM;
+
+	pci = devm_kzalloc(dev, sizeof(*pci), GFP_KERNEL);
+	if (!pci)
+		return -ENOMEM;
+
+	pci->dev = dev;
+	pci->ops = &dw_pcie_ops;
+
+	al_pcie->pci = pci;
+	al_pcie->dev = dev;
+
+	dbi_res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "dbi");
+	pci->dbi_base = devm_pci_remap_cfg_resource(dev, dbi_res);
+	if (IS_ERR(pci->dbi_base)) {
+		dev_err(dev, "couldn't remap dbi base %pR\n", dbi_res);
+		return PTR_ERR(pci->dbi_base);
+	}
+
+	ecam_res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "config");
+	if (!ecam_res) {
+		dev_err(dev, "couldn't find 'config' reg in DT\n");
+		return -ENOENT;
+	}
+	al_pcie->ecam_size = resource_size(ecam_res);
+
+	controller_res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
+						      "controller");
+	al_pcie->controller_base = devm_ioremap_resource(dev, controller_res);
+	if (IS_ERR(al_pcie->controller_base)) {
+		dev_err(dev, "couldn't remap controller base %pR\n",
+			controller_res);
+		return PTR_ERR(al_pcie->controller_base);
+	}
+
+	dev_dbg(dev, "From DT: dbi_base: %pR, controller_base: %pR\n",
+		dbi_res, controller_res);
+
+	platform_set_drvdata(pdev, al_pcie);
+
+	return al_add_pcie_port(&pci->pp, pdev);
+}
+
+static const struct of_device_id al_pcie_of_match[] = {
+	{ .compatible = "amazon,al-alpine-v2-pcie",
+	},
+	{ .compatible = "amazon,al-alpine-v3-pcie",
+	},
+	{},
+};
+
+static struct platform_driver al_pcie_driver = {
+	.driver = {
+		.name	= "al-pcie",
+		.of_match_table = al_pcie_of_match,
+		.suppress_bind_attrs = true,
+	},
+	.probe = al_pcie_probe,
+};
+builtin_platform_driver(al_pcie_driver);
+
+#endif /* CONFIG_PCIE_AL*/
-- 
2.17.1

Re: [PATCH v5 6/7] PCI: dwc: al: Add support for DW based driver type

[Bjorn Helgaas]

s/Add support for DW based driver type/Add Amazon Annapurna Labs PCIe controller driver/

On Thu, Sep 05, 2019 at 05:01:43PM +0300, Jonathan Chocron wrote:
> This driver is DT based and utilizes the DesignWare APIs.
> 
> It allows using a smaller ECAM range for a larger bus range -
> usually an entire bus uses 1MB of address space, but the driver
> can use it for a larger number of buses. This is achieved by using a HW
> mechanism which allows changing the BUS part of the "final" outgoing
> config transaction. There are 2 HW regs, one which is basically a
> bitmask determining which bits to take from the AXI transaction itself
> and another which holds the complementary part programmed by the
> driver.
> 
> All link initializations are handled by the boot FW.
> 
> Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
> Reviewed-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com>
> Reviewed-by: Andrew Murray <andrew.murray@arm.com>
> ---
>  drivers/pci/controller/dwc/Kconfig   |  12 +
>  drivers/pci/controller/dwc/pcie-al.c | 365 +++++++++++++++++++++++++++
>  2 files changed, 377 insertions(+)
> 
> diff --git a/drivers/pci/controller/dwc/Kconfig b/drivers/pci/controller/dwc/Kconfig
> index 4fada2e93285..0ba988b5b5bc 100644
> --- a/drivers/pci/controller/dwc/Kconfig
> +++ b/drivers/pci/controller/dwc/Kconfig
> @@ -256,4 +256,16 @@ config PCIE_UNIPHIER
>  	  Say Y here if you want PCIe controller support on UniPhier SoCs.
>  	  This driver supports LD20 and PXs3 SoCs.
>  
> +config PCIE_AL
> +	bool "Amazon Annapurna Labs PCIe controller"
> +	depends on OF && (ARM64 || COMPILE_TEST)
> +	depends on PCI_MSI_IRQ_DOMAIN
> +	select PCIE_DW_HOST
> +	help
> +	  Say Y here to enable support of the Amazon's Annapurna Labs PCIe
> +	  controller IP on Amazon SoCs. The PCIe controller uses the DesignWare
> +	  core plus Annapurna Labs proprietary hardware wrappers. This is
> +	  required only for DT-based platforms. ACPI platforms with the
> +	  Annapurna Labs PCIe controller don't need to enable this.

Interesting.  How do you deal with the funky ECAM space on ACPI
platforms?  Oh, never mind, I see, it's the pcie-al.c ECAM ops quirk
that's already in the tree.

Bjorn

Re: [PATCH v5 6/7] PCI: dwc: al: Add support for DW based driver type

[Chocron, Jonathan]

On Sat, 2019-09-07 at 11:55 -0500, Bjorn Helgaas wrote:
> s/Add support for DW based driver type/Add Amazon Annapurna Labs PCIe
> controller driver/
> 
Ack.

> On Thu, Sep 05, 2019 at 05:01:43PM +0300, Jonathan Chocron wrote:
> > This driver is DT based and utilizes the DesignWare APIs.
> > 
> > It allows using a smaller ECAM range for a larger bus range -
> > usually an entire bus uses 1MB of address space, but the driver
> > can use it for a larger number of buses. This is achieved by using
> > a HW
> > mechanism which allows changing the BUS part of the "final"
> > outgoing
> > config transaction. There are 2 HW regs, one which is basically a
> > bitmask determining which bits to take from the AXI transaction
> > itself
> > and another which holds the complementary part programmed by the
> > driver.
> > 
> > All link initializations are handled by the boot FW.
> > 
> > Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
> > Reviewed-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com>
> > Reviewed-by: Andrew Murray <andrew.murray@arm.com>
> > ---
> >  drivers/pci/controller/dwc/Kconfig   |  12 +
> >  drivers/pci/controller/dwc/pcie-al.c | 365
> > +++++++++++++++++++++++++++
> >  2 files changed, 377 insertions(+)
> > 
> > diff --git a/drivers/pci/controller/dwc/Kconfig
> > b/drivers/pci/controller/dwc/Kconfig
> > index 4fada2e93285..0ba988b5b5bc 100644
> > --- a/drivers/pci/controller/dwc/Kconfig
> > +++ b/drivers/pci/controller/dwc/Kconfig
> > @@ -256,4 +256,16 @@ config PCIE_UNIPHIER
> >  	  Say Y here if you want PCIe controller support on UniPhier
> > SoCs.
> >  	  This driver supports LD20 and PXs3 SoCs.
> >  
> > +config PCIE_AL
> > +	bool "Amazon Annapurna Labs PCIe controller"
> > +	depends on OF && (ARM64 || COMPILE_TEST)
> > +	depends on PCI_MSI_IRQ_DOMAIN
> > +	select PCIE_DW_HOST
> > +	help
> > +	  Say Y here to enable support of the Amazon's Annapurna Labs
> > PCIe
> > +	  controller IP on Amazon SoCs. The PCIe controller uses the
> > DesignWare
> > +	  core plus Annapurna Labs proprietary hardware wrappers. This
> > is
> > +	  required only for DT-based platforms. ACPI platforms with the
> > +	  Annapurna Labs PCIe controller don't need to enable this.
> 
> Interesting.  How do you deal with the funky ECAM space on ACPI
> platforms?  Oh, never mind, I see, it's the pcie-al.c ECAM ops quirk
> that's already in the tree.
> 
> Bjorn

Re: [PATCH v5 6/7] PCI: dwc: al: Add support for DW based driver type

[Chocron, Jonathan]

On Sat, 2019-09-07 at 11:55 -0500, Bjorn Helgaas wrote:
> s/Add support for DW based driver type/Add Amazon Annapurna Labs PCIe
> controller driver/
> 
Ack.

> On Thu, Sep 05, 2019 at 05:01:43PM +0300, Jonathan Chocron wrote:
> > This driver is DT based and utilizes the DesignWare APIs.
> > 
> > It allows using a smaller ECAM range for a larger bus range -
> > usually an entire bus uses 1MB of address space, but the driver
> > can use it for a larger number of buses. This is achieved by using
> > a HW
> > mechanism which allows changing the BUS part of the "final"
> > outgoing
> > config transaction. There are 2 HW regs, one which is basically a
> > bitmask determining which bits to take from the AXI transaction
> > itself
> > and another which holds the complementary part programmed by the
> > driver.
> > 
> > All link initializations are handled by the boot FW.
> > 
> > Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
> > Reviewed-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com>
> > Reviewed-by: Andrew Murray <andrew.murray@arm.com>
> > ---
> >  drivers/pci/controller/dwc/Kconfig   |  12 +
> >  drivers/pci/controller/dwc/pcie-al.c | 365
> > +++++++++++++++++++++++++++
> >  2 files changed, 377 insertions(+)
> > 
> > diff --git a/drivers/pci/controller/dwc/Kconfig
> > b/drivers/pci/controller/dwc/Kconfig
> > index 4fada2e93285..0ba988b5b5bc 100644
> > --- a/drivers/pci/controller/dwc/Kconfig
> > +++ b/drivers/pci/controller/dwc/Kconfig
> > @@ -256,4 +256,16 @@ config PCIE_UNIPHIER
> >  	  Say Y here if you want PCIe controller support on UniPhier
> > SoCs.
> >  	  This driver supports LD20 and PXs3 SoCs.
> >  
> > +config PCIE_AL
> > +	bool "Amazon Annapurna Labs PCIe controller"
> > +	depends on OF && (ARM64 || COMPILE_TEST)
> > +	depends on PCI_MSI_IRQ_DOMAIN
> > +	select PCIE_DW_HOST
> > +	help
> > +	  Say Y here to enable support of the Amazon's Annapurna Labs
> > PCIe
> > +	  controller IP on Amazon SoCs. The PCIe controller uses the
> > DesignWare
> > +	  core plus Annapurna Labs proprietary hardware wrappers. This
> > is
> > +	  required only for DT-based platforms. ACPI platforms with the
> > +	  Annapurna Labs PCIe controller don't need to enable this.
> 
> Interesting.  How do you deal with the funky ECAM space on ACPI
> platforms?  Oh, never mind, I see, it's the pcie-al.c ECAM ops quirk
> that's already in the tree.
> 
> Bjorn

[PATCH v5 7/7] PCI: dwc: Add validation that PCIe core is set to correct mode

[Jonathan Chocron]

Some PCIe controllers can be set to either Host or EP according to some
early boot FW. To make sure there is no discrepancy (e.g. FW configured
the port to EP mode while the DT specifies it as a host bridge or vice
versa), a check has been added for each mode.

Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
Acked-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com>
---
 drivers/pci/controller/dwc/pcie-designware-ep.c  |  8 ++++++++
 .../pci/controller/dwc/pcie-designware-host.c    | 16 ++++++++++++++++
 2 files changed, 24 insertions(+)

diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
index 65f479250087..3dd2e2697294 100644
--- a/drivers/pci/controller/dwc/pcie-designware-ep.c
+++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
@@ -498,6 +498,7 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
 	int ret;
 	u32 reg;
 	void *addr;
+	u8 hdr_type;
 	unsigned int nbars;
 	unsigned int offset;
 	struct pci_epc *epc;
@@ -562,6 +563,13 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
 	if (ep->ops->ep_init)
 		ep->ops->ep_init(ep);
 
+	hdr_type = dw_pcie_readb_dbi(pci, PCI_HEADER_TYPE);
+	if (hdr_type != PCI_HEADER_TYPE_NORMAL) {
+		dev_err(pci->dev, "PCIe controller is not set to EP mode (hdr_type:0x%x)!\n",
+			hdr_type);
+		return -EIO;
+	}
+
 	ret = of_property_read_u8(np, "max-functions", &epc->max_functions);
 	if (ret < 0)
 		epc->max_functions = 1;
diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
index d3156446ff27..0f36a926059a 100644
--- a/drivers/pci/controller/dwc/pcie-designware-host.c
+++ b/drivers/pci/controller/dwc/pcie-designware-host.c
@@ -323,6 +323,7 @@ int dw_pcie_host_init(struct pcie_port *pp)
 	struct pci_bus *child;
 	struct pci_host_bridge *bridge;
 	struct resource *cfg_res;
+	u32 hdr_type;
 	int ret;
 
 	raw_spin_lock_init(&pci->pp.lock);
@@ -464,6 +465,21 @@ int dw_pcie_host_init(struct pcie_port *pp)
 			goto err_free_msi;
 	}
 
+	ret = dw_pcie_rd_own_conf(pp, PCI_HEADER_TYPE, 1, &hdr_type);
+	if (ret != PCIBIOS_SUCCESSFUL) {
+		dev_err(pci->dev, "Failed reading PCI_HEADER_TYPE cfg space reg (ret: 0x%x)\n",
+			ret);
+		ret = pcibios_err_to_errno(ret);
+		goto err_free_msi;
+	}
+	if (hdr_type != PCI_HEADER_TYPE_BRIDGE) {
+		dev_err(pci->dev,
+			"PCIe controller is not set to bridge type (hdr_type: 0x%x)!\n",
+			hdr_type);
+		ret = -EIO;
+		goto err_free_msi;
+	}
+
 	pp->root_bus_nr = pp->busn->start;
 
 	bridge->dev.parent = dev;
-- 
2.17.1

[PATCH v5 7/7] PCI: dwc: Add validation that PCIe core is set to correct mode

[Jonathan Chocron]

Some PCIe controllers can be set to either Host or EP according to some
early boot FW. To make sure there is no discrepancy (e.g. FW configured
the port to EP mode while the DT specifies it as a host bridge or vice
versa), a check has been added for each mode.

Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
Acked-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com>
---
 drivers/pci/controller/dwc/pcie-designware-ep.c  |  8 ++++++++
 .../pci/controller/dwc/pcie-designware-host.c    | 16 ++++++++++++++++
 2 files changed, 24 insertions(+)

diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
index 65f479250087..3dd2e2697294 100644
--- a/drivers/pci/controller/dwc/pcie-designware-ep.c
+++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
@@ -498,6 +498,7 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
 	int ret;
 	u32 reg;
 	void *addr;
+	u8 hdr_type;
 	unsigned int nbars;
 	unsigned int offset;
 	struct pci_epc *epc;
@@ -562,6 +563,13 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
 	if (ep->ops->ep_init)
 		ep->ops->ep_init(ep);
 
+	hdr_type = dw_pcie_readb_dbi(pci, PCI_HEADER_TYPE);
+	if (hdr_type != PCI_HEADER_TYPE_NORMAL) {
+		dev_err(pci->dev, "PCIe controller is not set to EP mode (hdr_type:0x%x)!\n",
+			hdr_type);
+		return -EIO;
+	}
+
 	ret = of_property_read_u8(np, "max-functions", &epc->max_functions);
 	if (ret < 0)
 		epc->max_functions = 1;
diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
index d3156446ff27..0f36a926059a 100644
--- a/drivers/pci/controller/dwc/pcie-designware-host.c
+++ b/drivers/pci/controller/dwc/pcie-designware-host.c
@@ -323,6 +323,7 @@ int dw_pcie_host_init(struct pcie_port *pp)
 	struct pci_bus *child;
 	struct pci_host_bridge *bridge;
 	struct resource *cfg_res;
+	u32 hdr_type;
 	int ret;
 
 	raw_spin_lock_init(&pci->pp.lock);
@@ -464,6 +465,21 @@ int dw_pcie_host_init(struct pcie_port *pp)
 			goto err_free_msi;
 	}
 
+	ret = dw_pcie_rd_own_conf(pp, PCI_HEADER_TYPE, 1, &hdr_type);
+	if (ret != PCIBIOS_SUCCESSFUL) {
+		dev_err(pci->dev, "Failed reading PCI_HEADER_TYPE cfg space reg (ret: 0x%x)\n",
+			ret);
+		ret = pcibios_err_to_errno(ret);
+		goto err_free_msi;
+	}
+	if (hdr_type != PCI_HEADER_TYPE_BRIDGE) {
+		dev_err(pci->dev,
+			"PCIe controller is not set to bridge type (hdr_type: 0x%x)!\n",
+			hdr_type);
+		ret = -EIO;
+		goto err_free_msi;
+	}
+
 	pp->root_bus_nr = pp->busn->start;
 
 	bridge->dev.parent = dev;
-- 
2.17.1

Re: [PATCH v5 7/7] PCI: dwc: Add validation that PCIe core is set to correct mode

[Andrew Murray]

On Thu, Sep 05, 2019 at 05:01:44PM +0300, Jonathan Chocron wrote:
> Some PCIe controllers can be set to either Host or EP according to some
> early boot FW. To make sure there is no discrepancy (e.g. FW configured
> the port to EP mode while the DT specifies it as a host bridge or vice
> versa), a check has been added for each mode.
> 
> Signed-off-by: Jonathan Chocron <jonnyc@amazon.com>
> Acked-by: Gustavo Pimentel <gustavo.pimentel@synopsys.com>
> ---
>  drivers/pci/controller/dwc/pcie-designware-ep.c  |  8 ++++++++
>  .../pci/controller/dwc/pcie-designware-host.c    | 16 ++++++++++++++++
>  2 files changed, 24 insertions(+)
> 
> diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
> index 65f479250087..3dd2e2697294 100644
> --- a/drivers/pci/controller/dwc/pcie-designware-ep.c
> +++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
> @@ -498,6 +498,7 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
>  	int ret;
>  	u32 reg;
>  	void *addr;
> +	u8 hdr_type;
>  	unsigned int nbars;
>  	unsigned int offset;
>  	struct pci_epc *epc;
> @@ -562,6 +563,13 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
>  	if (ep->ops->ep_init)
>  		ep->ops->ep_init(ep);
>  
> +	hdr_type = dw_pcie_readb_dbi(pci, PCI_HEADER_TYPE);
> +	if (hdr_type != PCI_HEADER_TYPE_NORMAL) {
> +		dev_err(pci->dev, "PCIe controller is not set to EP mode (hdr_type:0x%x)!\n",
> +			hdr_type);
> +		return -EIO;
> +	}
> +
>  	ret = of_property_read_u8(np, "max-functions", &epc->max_functions);
>  	if (ret < 0)
>  		epc->max_functions = 1;
> diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
> index d3156446ff27..0f36a926059a 100644
> --- a/drivers/pci/controller/dwc/pcie-designware-host.c
> +++ b/drivers/pci/controller/dwc/pcie-designware-host.c
> @@ -323,6 +323,7 @@ int dw_pcie_host_init(struct pcie_port *pp)
>  	struct pci_bus *child;
>  	struct pci_host_bridge *bridge;
>  	struct resource *cfg_res;
> +	u32 hdr_type;
>  	int ret;
>  
>  	raw_spin_lock_init(&pci->pp.lock);
> @@ -464,6 +465,21 @@ int dw_pcie_host_init(struct pcie_port *pp)
>  			goto err_free_msi;
>  	}
>  
> +	ret = dw_pcie_rd_own_conf(pp, PCI_HEADER_TYPE, 1, &hdr_type);
> +	if (ret != PCIBIOS_SUCCESSFUL) {
> +		dev_err(pci->dev, "Failed reading PCI_HEADER_TYPE cfg space reg (ret: 0x%x)\n",
> +			ret);
> +		ret = pcibios_err_to_errno(ret);
> +		goto err_free_msi;
> +	}
> +	if (hdr_type != PCI_HEADER_TYPE_BRIDGE) {
> +		dev_err(pci->dev,
> +			"PCIe controller is not set to bridge type (hdr_type: 0x%x)!\n",
> +			hdr_type);
> +		ret = -EIO;
> +		goto err_free_msi;
> +	}
> +

Reviewed-by: Andrew Murray <andrew.murray@arm.com>

>  	pp->root_bus_nr = pp->busn->start;
>  
>  	bridge->dev.parent = dev;
> -- 
> 2.17.1
> 

Re: [PATCH v5 0/7] Amazon's Annapurna Labs DT-based PCIe host controller driver

[Lorenzo Pieralisi]

On Thu, Sep 05, 2019 at 05:00:14PM +0300, Jonathan Chocron wrote:
> This series adds support for Amazon's Annapurna Labs DT-based PCIe host
> controller driver.
> Additionally, it adds 3 quirks (ACS, VPD and MSI-X) and 2 generic DWC patches.
> 
> Changes since v4:
> - Moved the HEADER_TYPE validations to after pp->ops->host_init() and
>   ep->ops->ep_init()
> - Changed to dw_pcie_rd_own_conf() instead of dw_pcie_readb_dbi() for
>   reading the HEADER_TYPE
> - Used exsitng quirk_blacklist_vpd() instead of quirk_al_vpd_release()
> - Added a newline in ACS quirk comment
> 
> Changes since v3:
> - Removed PATCH 8/8 since the usage of the PCI flags will be discussed
>   in the upcoming LPC
> - Align commit subject with the folder convention
> - Added explanation regarding ECAM "overload" mechanism
> - Switched to read/write{_relaxed} APIs
> - Modified a dev_err to dev_dbg
> - Removed unnecessary variable
> - Removed driver details from dt-binding description
> - Changed to SoC specific compatibles
> - Fixed typo in a commit message
> - Added comment regarding MSI in the MSI-X quirk
> 
> Changes since v2:
> - Added al_pcie_controller_readl/writel() wrappers
> - Reorganized local vars in several functions according to reverse
>   tree structure
> - Removed unnecessary check of ret value
> - Changed return type of al_pcie_config_prepare() from int to void
> - Removed check if link is up from probe() [done internally in
>   dw_pcie_rd/wr_conf()]
> 
> Changes since v1:
> - Added comment regarding 0x0031 being used as a dev_id for non root-port devices as well
> - Fixed different message/comment/print wordings
> - Added panic stacktrace to commit message of MSI-x quirk patch
> - Changed to pci_warn() instead of dev_warn()
> - Added unit_address after node_name in dt-binding
> - Updated Kconfig help description
> - Used GENMASK and FIELD_PREP/GET where appropriate
> - Removed leftover field from struct al_pcie and moved all ptrs to
>   the beginning
> - Re-wrapped function definitions and invocations to use fewer lines
> - Change %p to %px in dbg prints in rd/wr_conf() functions
> - Removed validation that the port is configured to RC mode (as this is
>   added generically in PATCH 7/8)
> - Removed unnecessary variable initializations
> - Swtiched to %pR for printing resources
> 
> 
> Ali Saidi (1):
>   PCI: Add ACS quirk for Amazon Annapurna Labs root ports
> 
> Jonathan Chocron (6):
>   PCI: Add Amazon's Annapurna Labs vendor ID
>   PCI/VPD: Prevent VPD access for Amazon's Annapurna Labs Root Port
>   PCI: Add quirk to disable MSI-X support for Amazon's Annapurna Labs
>     Root Port
>   dt-bindings: PCI: Add Amazon's Annapurna Labs PCIe host bridge binding
>   PCI: dwc: al: Add support for DW based driver type
>   PCI: dwc: Add validation that PCIe core is set to correct mode
> 
>  .../devicetree/bindings/pci/pcie-al.txt       |  46 +++
>  MAINTAINERS                                   |   3 +-
>  drivers/pci/controller/dwc/Kconfig            |  12 +
>  drivers/pci/controller/dwc/pcie-al.c          | 365 ++++++++++++++++++
>  .../pci/controller/dwc/pcie-designware-ep.c   |   8 +
>  .../pci/controller/dwc/pcie-designware-host.c |  16 +
>  drivers/pci/quirks.c                          |  38 ++
>  drivers/pci/vpd.c                             |   6 +
>  include/linux/pci_ids.h                       |   2 +
>  9 files changed, 495 insertions(+), 1 deletion(-)
>  create mode 100644 Documentation/devicetree/bindings/pci/pcie-al.txt

Hi Bjorn,

I would like to queue this series for v5.4 but I need your ACK on
patches 2/3/4 for that to happen, please let me know.

Thanks,
Lorenzo