[PATCH 0/3] Add support to handle prefetchable memory

[PATCH 0/3] Add support to handle prefetchable memory

[Vidya Sagar]

This patch series adds support for configuring the DesignWare IP's ATU
region for prefetchable memory translations.
It first starts by flagging a warning if the size of non-prefetchable
aperture goes beyond 32-bit as PCIe spec doesn't allow it.
And then adds required support for programming the ATU to handle higher
(i.e. >4GB) sizes and then finally adds support for differentiating
between prefetchable and non-prefetchable regions and configuring one of
the ATU regions for prefetchable memory translations purpose.

Vidya Sagar (3):
  PCI: of: Warn if non-prefetchable memory aperture size is > 32-bit
  PCI: dwc: Add support to program ATU for >4GB memory aperture sizes
  PCI: dwc: Add support to handle prefetchable memory mapping

 .../pci/controller/dwc/pcie-designware-host.c | 39 ++++++++++++++++---
 drivers/pci/controller/dwc/pcie-designware.c  | 12 +++---
 drivers/pci/controller/dwc/pcie-designware.h  |  4 +-
 drivers/pci/of.c                              |  5 +++
 4 files changed, 48 insertions(+), 12 deletions(-)

-- 
2.17.1

[PATCH 1/3] PCI: of: Warn if non-prefetchable memory aperture size is > 32-bit

[Vidya Sagar]

As per PCIe spec r5.0, sec 7.5.1.3.8 only 32-bit BAR registers are defined
for non-prefetchable memory and hence a warning should be reported when
the size of them go beyond 32-bits.

Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
---
 drivers/pci/of.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/pci/of.c b/drivers/pci/of.c
index ac24cd5439a9..5ea472ae22ac 100644
--- a/drivers/pci/of.c
+++ b/drivers/pci/of.c
@@ -556,6 +556,11 @@ static int pci_parse_request_of_pci_ranges(struct device *dev,
 			break;
 		case IORESOURCE_MEM:
 			res_valid |= !(res->flags & IORESOURCE_PREFETCH);
+
+			if (!(res->flags & IORESOURCE_PREFETCH))
+				if (upper_32_bits(resource_size(res)))
+					dev_warn(dev, "Memory resource size exceeds max for 32 bits\n");
+
 			break;
 		}
 	}
-- 
2.17.1

[PATCH 2/3] PCI: dwc: Add support to program ATU for >4GB memory aperture sizes

[Vidya Sagar]

Add support to program the ATU to enable translations for >4GB sizes of
the prefetchable memory apertures.

Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
---
 drivers/pci/controller/dwc/pcie-designware.c | 12 +++++++-----
 drivers/pci/controller/dwc/pcie-designware.h |  3 ++-
 2 files changed, 9 insertions(+), 6 deletions(-)

diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
index c2dea8fc97c8..b5e438b70cd5 100644
--- a/drivers/pci/controller/dwc/pcie-designware.c
+++ b/drivers/pci/controller/dwc/pcie-designware.c
@@ -228,7 +228,7 @@ static void dw_pcie_writel_ob_unroll(struct dw_pcie *pci, u32 index, u32 reg,
 static void dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, u8 func_no,
 					     int index, int type,
 					     u64 cpu_addr, u64 pci_addr,
-					     u32 size)
+					     u64 size)
 {
 	u32 retries, val;
 	u64 limit_addr = cpu_addr + size - 1;
@@ -245,8 +245,10 @@ static void dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, u8 func_no,
 				 lower_32_bits(pci_addr));
 	dw_pcie_writel_ob_unroll(pci, index, PCIE_ATU_UNR_UPPER_TARGET,
 				 upper_32_bits(pci_addr));
-	dw_pcie_writel_ob_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL1,
-				 type | PCIE_ATU_FUNC_NUM(func_no));
+	val = type | PCIE_ATU_FUNC_NUM(func_no);
+	val = upper_32_bits(size - 1) ?
+		val | PCIE_ATU_INCREASE_REGION_SIZE : val;
+	dw_pcie_writel_ob_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL1, val);
 	dw_pcie_writel_ob_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL2,
 				 PCIE_ATU_ENABLE);
 
@@ -267,7 +269,7 @@ static void dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, u8 func_no,
 
 static void __dw_pcie_prog_outbound_atu(struct dw_pcie *pci, u8 func_no,
 					int index, int type, u64 cpu_addr,
-					u64 pci_addr, u32 size)
+					u64 pci_addr, u64 size)
 {
 	u32 retries, val;
 
@@ -311,7 +313,7 @@ static void __dw_pcie_prog_outbound_atu(struct dw_pcie *pci, u8 func_no,
 }
 
 void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
-			       u64 cpu_addr, u64 pci_addr, u32 size)
+			       u64 cpu_addr, u64 pci_addr, u64 size)
 {
 	__dw_pcie_prog_outbound_atu(pci, 0, index, type,
 				    cpu_addr, pci_addr, size);
diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
index 9d2f511f13fa..e7f441441db2 100644
--- a/drivers/pci/controller/dwc/pcie-designware.h
+++ b/drivers/pci/controller/dwc/pcie-designware.h
@@ -84,6 +84,7 @@
 #define PCIE_ATU_REGION_INDEX1		0x1
 #define PCIE_ATU_REGION_INDEX0		0x0
 #define PCIE_ATU_CR1			0x904
+#define PCIE_ATU_INCREASE_REGION_SIZE	BIT(13)
 #define PCIE_ATU_TYPE_MEM		0x0
 #define PCIE_ATU_TYPE_IO		0x2
 #define PCIE_ATU_TYPE_CFG0		0x4
@@ -295,7 +296,7 @@ void dw_pcie_upconfig_setup(struct dw_pcie *pci);
 int dw_pcie_wait_for_link(struct dw_pcie *pci);
 void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index,
 			       int type, u64 cpu_addr, u64 pci_addr,
-			       u32 size);
+			       u64 size);
 void dw_pcie_prog_ep_outbound_atu(struct dw_pcie *pci, u8 func_no, int index,
 				  int type, u64 cpu_addr, u64 pci_addr,
 				  u32 size);
-- 
2.17.1

[PATCH 3/3] PCI: dwc: Add support to handle prefetchable memory mapping

[Vidya Sagar]

DWC sub-system currently doesn't differentiate between prefetchable and
non-prefetchable memory aperture entries in the 'ranges' property and
provides ATU mapping only for the first memory aperture entry out of all
the entries present. This was introduced by the
commit 0f71c60ffd26 ("PCI: dwc: Remove storing of PCI resources").
Mapping for a memory apreture is required if its CPU address and the bus
address are different and the current mechanism works only if the memory
aperture which needs mapping happens to be the first entry. It doesn't
work either if the memory aperture that needs mapping is not the first
entry or if both prefetchable and non-prefetchable apertures need mapping.

This patch fixes this issue by differentiating between prefetchable and
non-prefetchable apertures in the 'ranges' property there by removing the
dependency on the order in which they are specified and adds support for
mapping prefetchable aperture using ATU region-3 if required.

Fixes: 0f71c60ffd26 ("PCI: dwc: Remove storing of PCI resources")
Link: http://patchwork.ozlabs.org/project/linux-pci/patch/20200513190855.23318-1-vidyas@nvidia.com/

Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
---
Changes from previous versions:
* Addressed Rob's comments and as part of that split the patch into three sub-patches
* Rewrote commit subject and description
* Addressed review comments from Lorenzo

 .../pci/controller/dwc/pcie-designware-host.c | 39 ++++++++++++++++---
 drivers/pci/controller/dwc/pcie-designware.h  |  1 +
 2 files changed, 34 insertions(+), 6 deletions(-)

diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
index 674f32db85ca..a1f319ccd816 100644
--- a/drivers/pci/controller/dwc/pcie-designware-host.c
+++ b/drivers/pci/controller/dwc/pcie-designware-host.c
@@ -529,9 +529,39 @@ static struct pci_ops dw_pcie_ops = {
 	.write = pci_generic_config_write,
 };
 
+static void dw_pcie_setup_mem_atu(struct pcie_port *pp,
+				  struct resource_entry *win)
+{
+	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+
+	/* Check for prefetchable memory aperture */
+	if (win->res->flags & IORESOURCE_PREFETCH) {
+		/* Number of view ports must at least be 4 to enable mapping */
+		if (pci->num_viewport < 4) {
+			dev_warn(pci->dev,
+				 "Insufficient ATU regions to map Prefetchable memory\n");
+		} else {
+			dw_pcie_prog_outbound_atu(pci,
+						  PCIE_ATU_REGION_INDEX3,
+						  PCIE_ATU_TYPE_MEM,
+						  win->res->start,
+						  win->res->start - win->offset,
+						  resource_size(win->res));
+		}
+	} else { /* Non-prefetchable memory aperture */
+		dw_pcie_prog_outbound_atu(pci,
+					  PCIE_ATU_REGION_INDEX0,
+					  PCIE_ATU_TYPE_MEM,
+					  win->res->start,
+					  win->res->start - win->offset,
+					  resource_size(win->res));
+	}
+}
+
 void dw_pcie_setup_rc(struct pcie_port *pp)
 {
 	u32 val, ctrl, num_ctrls;
+	struct resource_entry *win;
 	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
 
 	/*
@@ -586,13 +616,10 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
 	 * ATU, so we should not program the ATU here.
 	 */
 	if (pp->bridge->child_ops == &dw_child_pcie_ops) {
-		struct resource_entry *entry =
-			resource_list_first_type(&pp->bridge->windows, IORESOURCE_MEM);
+		resource_list_for_each_entry(win, &pp->bridge->windows)
+			if (resource_type(win->res) == IORESOURCE_MEM)
+				dw_pcie_setup_mem_atu(pp, win);
 
-		dw_pcie_prog_outbound_atu(pci, PCIE_ATU_REGION_INDEX0,
-					  PCIE_ATU_TYPE_MEM, entry->res->start,
-					  entry->res->start - entry->offset,
-					  resource_size(entry->res));
 		if (pci->num_viewport > 2)
 			dw_pcie_prog_outbound_atu(pci, PCIE_ATU_REGION_INDEX2,
 						  PCIE_ATU_TYPE_IO, pp->io_base,
diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
index e7f441441db2..21dd06831b50 100644
--- a/drivers/pci/controller/dwc/pcie-designware.h
+++ b/drivers/pci/controller/dwc/pcie-designware.h
@@ -80,6 +80,7 @@
 #define PCIE_ATU_VIEWPORT		0x900
 #define PCIE_ATU_REGION_INBOUND		BIT(31)
 #define PCIE_ATU_REGION_OUTBOUND	0
+#define PCIE_ATU_REGION_INDEX3		0x3
 #define PCIE_ATU_REGION_INDEX2		0x2
 #define PCIE_ATU_REGION_INDEX1		0x1
 #define PCIE_ATU_REGION_INDEX0		0x0
-- 
2.17.1

Re: [PATCH 0/3] Add support to handle prefetchable memory

[Jingoo Han]

On 10/23/20, 3:57 PM, Vidya Sagar wrote:
> 
> This patch series adds support for configuring the DesignWare IP's ATU
> region for prefetchable memory translations.
> It first starts by flagging a warning if the size of non-prefetchable
> aperture goes beyond 32-bit as PCIe spec doesn't allow it.
> And then adds required support for programming the ATU to handle higher
> (i.e. >4GB) sizes and then finally adds support for differentiating
> between prefetchable and non-prefetchable regions and configuring one of
> the ATU regions for prefetchable memory translations purpose.
>
> Vidya Sagar (3):
>   PCI: of: Warn if non-prefetchable memory aperture size is > 32-bit
>   PCI: dwc: Add support to program ATU for >4GB memory aperture sizes
>   PCI: dwc: Add support to handle prefetchable memory mapping

For 2nd & 3rd,
Acked-by: Jingoo <jingoohan1@gmail.com>
But, I still want someone to ack 1st patch, not me.

To Vidya,
If possible, can you ask your coworker to give 'Tested-by'? It will be very helpful.
Thank you.

Best regards,
Jingoo Han


>
>  .../pci/controller/dwc/pcie-designware-host.c | 39 ++++++++++++++++---
>  drivers/pci/controller/dwc/pcie-designware.c  | 12 +++---
>  drivers/pci/controller/dwc/pcie-designware.h  |  4 +-
>  drivers/pci/of.c                              |  5 +++
>  4 files changed, 48 insertions(+), 12 deletions(-)
>
> -- 
> 2.17.1

Re: [PATCH 0/3] Add support to handle prefetchable memory

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 1977 bytes --]

On Sat, Oct 24, 2020 at 04:03:41AM +0000, Jingoo Han wrote:
> On 10/23/20, 3:57 PM, Vidya Sagar wrote:
> > 
> > This patch series adds support for configuring the DesignWare IP's ATU
> > region for prefetchable memory translations.
> > It first starts by flagging a warning if the size of non-prefetchable
> > aperture goes beyond 32-bit as PCIe spec doesn't allow it.
> > And then adds required support for programming the ATU to handle higher
> > (i.e. >4GB) sizes and then finally adds support for differentiating
> > between prefetchable and non-prefetchable regions and configuring one of
> > the ATU regions for prefetchable memory translations purpose.
> >
> > Vidya Sagar (3):
> >   PCI: of: Warn if non-prefetchable memory aperture size is > 32-bit
> >   PCI: dwc: Add support to program ATU for >4GB memory aperture sizes
> >   PCI: dwc: Add support to handle prefetchable memory mapping
> 
> For 2nd & 3rd,
> Acked-by: Jingoo <jingoohan1@gmail.com>
> But, I still want someone to ack 1st patch, not me.
> 
> To Vidya,
> If possible, can you ask your coworker to give 'Tested-by'? It will be very helpful.
> Thank you.

On next-20201026 (but also going back quite a while) I'm seeing this
during boot on Jetson AGX Xavier (Tegra194):

[    3.493382] ahci 0001:01:00.0: version 3.0
[    3.493889] ahci 0001:01:00.0: SSS flag set, parallel bus scan disabled
[    4.497706] ahci 0001:01:00.0: controller reset failed (0xffffffff)
[    4.498114] ahci: probe of 0001:01:00.0 failed with error -5

After applying this series, AHCI over PCI is back to normal:

[    3.543230] ahci 0001:01:00.0: AHCI 0001.0000 32 slots 1 ports 6 Gbps 0x1 impl SATA mode
[    3.550841] ahci 0001:01:00.0: flags: 64bit ncq sntf led only pmp fbs pio slum part sxs
[    3.559747] scsi host0: ahci
[    3.561998] ata1: SATA max UDMA/133 abar m512@0x1230010000 port 0x1230010100 irq 63

So for the series:

Tested-by: Thierry Reding <treding@nvidia.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 3/3] PCI: dwc: Add support to handle prefetchable memory mapping

[Rob Herring]

On Fri, Oct 23, 2020 at 2:57 PM Vidya Sagar <vidyas@nvidia.com> wrote:
>
> DWC sub-system currently doesn't differentiate between prefetchable and
> non-prefetchable memory aperture entries in the 'ranges' property and
> provides ATU mapping only for the first memory aperture entry out of all
> the entries present. This was introduced by the
> commit 0f71c60ffd26 ("PCI: dwc: Remove storing of PCI resources").
> Mapping for a memory apreture is required if its CPU address and the bus
> address are different and the current mechanism works only if the memory
> aperture which needs mapping happens to be the first entry. It doesn't
> work either if the memory aperture that needs mapping is not the first
> entry or if both prefetchable and non-prefetchable apertures need mapping.
>
> This patch fixes this issue by differentiating between prefetchable and
> non-prefetchable apertures in the 'ranges' property there by removing the
> dependency on the order in which they are specified and adds support for
> mapping prefetchable aperture using ATU region-3 if required.
>
> Fixes: 0f71c60ffd26 ("PCI: dwc: Remove storing of PCI resources")

Fixes should come first, then new features.

> Link: http://patchwork.ozlabs.org/project/linux-pci/patch/20200513190855.23318-1-vidyas@nvidia.com/

'Link' is the link for this message and should be a lore.kernel.org
link. Maintainers will add it.

>
> Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
> ---
> Changes from previous versions:
> * Addressed Rob's comments and as part of that split the patch into three sub-patches
> * Rewrote commit subject and description
> * Addressed review comments from Lorenzo
>
>  .../pci/controller/dwc/pcie-designware-host.c | 39 ++++++++++++++++---
>  drivers/pci/controller/dwc/pcie-designware.h  |  1 +
>  2 files changed, 34 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
> index 674f32db85ca..a1f319ccd816 100644
> --- a/drivers/pci/controller/dwc/pcie-designware-host.c
> +++ b/drivers/pci/controller/dwc/pcie-designware-host.c
> @@ -529,9 +529,39 @@ static struct pci_ops dw_pcie_ops = {
>         .write = pci_generic_config_write,
>  };
>
> +static void dw_pcie_setup_mem_atu(struct pcie_port *pp,
> +                                 struct resource_entry *win)
> +{
> +       struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> +
> +       /* Check for prefetchable memory aperture */
> +       if (win->res->flags & IORESOURCE_PREFETCH) {
> +               /* Number of view ports must at least be 4 to enable mapping */
> +               if (pci->num_viewport < 4) {
> +                       dev_warn(pci->dev,
> +                                "Insufficient ATU regions to map Prefetchable memory\n");
> +               } else {
> +                       dw_pcie_prog_outbound_atu(pci,
> +                                                 PCIE_ATU_REGION_INDEX3,
> +                                                 PCIE_ATU_TYPE_MEM,
> +                                                 win->res->start,
> +                                                 win->res->start - win->offset,
> +                                                 resource_size(win->res));
> +               }
> +       } else { /* Non-prefetchable memory aperture */
> +               dw_pcie_prog_outbound_atu(pci,
> +                                         PCIE_ATU_REGION_INDEX0,
> +                                         PCIE_ATU_TYPE_MEM,
> +                                         win->res->start,
> +                                         win->res->start - win->offset,
> +                                         resource_size(win->res));
> +       }
> +}
> +

This is in no way a minimal fix. I'll send my proposed fix.

>  void dw_pcie_setup_rc(struct pcie_port *pp)
>  {
>         u32 val, ctrl, num_ctrls;
> +       struct resource_entry *win;
>         struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
>
>         /*
> @@ -586,13 +616,10 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
>          * ATU, so we should not program the ATU here.
>          */
>         if (pp->bridge->child_ops == &dw_child_pcie_ops) {
> -               struct resource_entry *entry =
> -                       resource_list_first_type(&pp->bridge->windows, IORESOURCE_MEM);
> +               resource_list_for_each_entry(win, &pp->bridge->windows)
> +                       if (resource_type(win->res) == IORESOURCE_MEM)
> +                               dw_pcie_setup_mem_atu(pp, win);
>
> -               dw_pcie_prog_outbound_atu(pci, PCIE_ATU_REGION_INDEX0,
> -                                         PCIE_ATU_TYPE_MEM, entry->res->start,
> -                                         entry->res->start - entry->offset,
> -                                         resource_size(entry->res));
>                 if (pci->num_viewport > 2)
>                         dw_pcie_prog_outbound_atu(pci, PCIE_ATU_REGION_INDEX2,
>                                                   PCIE_ATU_TYPE_IO, pp->io_base,
> diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
> index e7f441441db2..21dd06831b50 100644
> --- a/drivers/pci/controller/dwc/pcie-designware.h
> +++ b/drivers/pci/controller/dwc/pcie-designware.h
> @@ -80,6 +80,7 @@
>  #define PCIE_ATU_VIEWPORT              0x900
>  #define PCIE_ATU_REGION_INBOUND                BIT(31)
>  #define PCIE_ATU_REGION_OUTBOUND       0
> +#define PCIE_ATU_REGION_INDEX3         0x3
>  #define PCIE_ATU_REGION_INDEX2         0x2
>  #define PCIE_ATU_REGION_INDEX1         0x1
>  #define PCIE_ATU_REGION_INDEX0         0x0
> --
> 2.17.1
>

Re: [PATCH 2/3] PCI: dwc: Add support to program ATU for >4GB memory aperture sizes

[Rob Herring]

On Fri, Oct 23, 2020 at 2:57 PM Vidya Sagar <vidyas@nvidia.com> wrote:
>
> Add support to program the ATU to enable translations for >4GB sizes of
> the prefetchable memory apertures.
>
> Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
> ---
>  drivers/pci/controller/dwc/pcie-designware.c | 12 +++++++-----
>  drivers/pci/controller/dwc/pcie-designware.h |  3 ++-
>  2 files changed, 9 insertions(+), 6 deletions(-)

Reviewed-by: Rob Herring <robh@kernel.org>

Re: [PATCH 1/3] PCI: of: Warn if non-prefetchable memory aperture size is > 32-bit

[Rob Herring]

On Fri, Oct 23, 2020 at 2:57 PM Vidya Sagar <vidyas@nvidia.com> wrote:
>
> As per PCIe spec r5.0, sec 7.5.1.3.8 only 32-bit BAR registers are defined
> for non-prefetchable memory and hence a warning should be reported when
> the size of them go beyond 32-bits.
>
> Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
> ---
>  drivers/pci/of.c | 5 +++++
>  1 file changed, 5 insertions(+)

Reviewed-by: Rob Herring <robh@kernel.org>

Re: [PATCH 0/3] Add support to handle prefetchable memory

[Vidya Sagar]

Lorenzo / Bjorn,
Could you please review patches-1 & 2 in this series?
For the third patch, we already went with Rob's patch @ 
http://patchwork.ozlabs.org/project/linux-pci/patch/20201026154852.221483-1-robh@kernel.org/

Thanks,
Vidya Sagar

On 10/26/2020 6:02 PM, Thierry Reding wrote:
> On Sat, Oct 24, 2020 at 04:03:41AM +0000, Jingoo Han wrote:
>> On 10/23/20, 3:57 PM, Vidya Sagar wrote:
>>>
>>> This patch series adds support for configuring the DesignWare IP's ATU
>>> region for prefetchable memory translations.
>>> It first starts by flagging a warning if the size of non-prefetchable
>>> aperture goes beyond 32-bit as PCIe spec doesn't allow it.
>>> And then adds required support for programming the ATU to handle higher
>>> (i.e. >4GB) sizes and then finally adds support for differentiating
>>> between prefetchable and non-prefetchable regions and configuring one of
>>> the ATU regions for prefetchable memory translations purpose.
>>>
>>> Vidya Sagar (3):
>>>    PCI: of: Warn if non-prefetchable memory aperture size is > 32-bit
>>>    PCI: dwc: Add support to program ATU for >4GB memory aperture sizes
>>>    PCI: dwc: Add support to handle prefetchable memory mapping
>>
>> For 2nd & 3rd,
>> Acked-by: Jingoo <jingoohan1@gmail.com>
>> But, I still want someone to ack 1st patch, not me.
>>
>> To Vidya,
>> If possible, can you ask your coworker to give 'Tested-by'? It will be very helpful.
>> Thank you.
> 
> On next-20201026 (but also going back quite a while) I'm seeing this
> during boot on Jetson AGX Xavier (Tegra194):
> 
> [    3.493382] ahci 0001:01:00.0: version 3.0
> [    3.493889] ahci 0001:01:00.0: SSS flag set, parallel bus scan disabled
> [    4.497706] ahci 0001:01:00.0: controller reset failed (0xffffffff)
> [    4.498114] ahci: probe of 0001:01:00.0 failed with error -5
> 
> After applying this series, AHCI over PCI is back to normal:
> 
> [    3.543230] ahci 0001:01:00.0: AHCI 0001.0000 32 slots 1 ports 6 Gbps 0x1 impl SATA mode
> [    3.550841] ahci 0001:01:00.0: flags: 64bit ncq sntf led only pmp fbs pio slum part sxs
> [    3.559747] scsi host0: ahci
> [    3.561998] ata1: SATA max UDMA/133 abar m512@0x1230010000 port 0x1230010100 irq 63
> 
> So for the series:
> 
> Tested-by: Thierry Reding <treding@nvidia.com>
> 

Re: [PATCH 0/3] Add support to handle prefetchable memory

[Jon Hunter]

On 26/10/2020 12:32, Thierry Reding wrote:
> On Sat, Oct 24, 2020 at 04:03:41AM +0000, Jingoo Han wrote:
>> On 10/23/20, 3:57 PM, Vidya Sagar wrote:
>>>
>>> This patch series adds support for configuring the DesignWare IP's ATU
>>> region for prefetchable memory translations.
>>> It first starts by flagging a warning if the size of non-prefetchable
>>> aperture goes beyond 32-bit as PCIe spec doesn't allow it.
>>> And then adds required support for programming the ATU to handle higher
>>> (i.e. >4GB) sizes and then finally adds support for differentiating
>>> between prefetchable and non-prefetchable regions and configuring one of
>>> the ATU regions for prefetchable memory translations purpose.
>>>
>>> Vidya Sagar (3):
>>>   PCI: of: Warn if non-prefetchable memory aperture size is > 32-bit
>>>   PCI: dwc: Add support to program ATU for >4GB memory aperture sizes
>>>   PCI: dwc: Add support to handle prefetchable memory mapping
>>
>> For 2nd & 3rd,
>> Acked-by: Jingoo <jingoohan1@gmail.com>
>> But, I still want someone to ack 1st patch, not me.
>>
>> To Vidya,
>> If possible, can you ask your coworker to give 'Tested-by'? It will be very helpful.
>> Thank you.
> 
> On next-20201026 (but also going back quite a while) I'm seeing this
> during boot on Jetson AGX Xavier (Tegra194):
> 
> [    3.493382] ahci 0001:01:00.0: version 3.0
> [    3.493889] ahci 0001:01:00.0: SSS flag set, parallel bus scan disabled
> [    4.497706] ahci 0001:01:00.0: controller reset failed (0xffffffff)
> [    4.498114] ahci: probe of 0001:01:00.0 failed with error -5
> 
> After applying this series, AHCI over PCI is back to normal:
> 
> [    3.543230] ahci 0001:01:00.0: AHCI 0001.0000 32 slots 1 ports 6 Gbps 0x1 impl SATA mode
> [    3.550841] ahci 0001:01:00.0: flags: 64bit ncq sntf led only pmp fbs pio slum part sxs
> [    3.559747] scsi host0: ahci
> [    3.561998] ata1: SATA max UDMA/133 abar m512@0x1230010000 port 0x1230010100 irq 63
> 
> So for the series:
> 
> Tested-by: Thierry Reding <treding@nvidia.com>

FWIW ...

Tested-by: Jon Hunter <jonathanh@nvidia.com>

Cheers
Jon

-- 
nvpublic

Re: [PATCH 0/3] Add support to handle prefetchable memory

[Vidya Sagar]

Hi Lorenzo & Bjorn,
Sorry to bother you.
Could you please take a look at the patches-1 & 2 from this series?

Thanks,
Vidya Sagar

On 11/4/2020 1:16 PM, Vidya Sagar wrote:
> External email: Use caution opening links or attachments
> 
> 
> Lorenzo / Bjorn,
> Could you please review patches-1 & 2 in this series?
> For the third patch, we already went with Rob's patch @
> http://patchwork.ozlabs.org/project/linux-pci/patch/20201026154852.221483-1-robh@kernel.org/ 
> 
> 
> Thanks,
> Vidya Sagar
> 
> On 10/26/2020 6:02 PM, Thierry Reding wrote:
>> On Sat, Oct 24, 2020 at 04:03:41AM +0000, Jingoo Han wrote:
>>> On 10/23/20, 3:57 PM, Vidya Sagar wrote:
>>>>
>>>> This patch series adds support for configuring the DesignWare IP's ATU
>>>> region for prefetchable memory translations.
>>>> It first starts by flagging a warning if the size of non-prefetchable
>>>> aperture goes beyond 32-bit as PCIe spec doesn't allow it.
>>>> And then adds required support for programming the ATU to handle higher
>>>> (i.e. >4GB) sizes and then finally adds support for differentiating
>>>> between prefetchable and non-prefetchable regions and configuring 
>>>> one of
>>>> the ATU regions for prefetchable memory translations purpose.
>>>>
>>>> Vidya Sagar (3):
>>>>    PCI: of: Warn if non-prefetchable memory aperture size is > 32-bit
>>>>    PCI: dwc: Add support to program ATU for >4GB memory aperture sizes
>>>>    PCI: dwc: Add support to handle prefetchable memory mapping
>>>
>>> For 2nd & 3rd,
>>> Acked-by: Jingoo <jingoohan1@gmail.com>
>>> But, I still want someone to ack 1st patch, not me.
>>>
>>> To Vidya,
>>> If possible, can you ask your coworker to give 'Tested-by'? It will 
>>> be very helpful.
>>> Thank you.
>>
>> On next-20201026 (but also going back quite a while) I'm seeing this
>> during boot on Jetson AGX Xavier (Tegra194):
>>
>> [    3.493382] ahci 0001:01:00.0: version 3.0
>> [    3.493889] ahci 0001:01:00.0: SSS flag set, parallel bus scan 
>> disabled
>> [    4.497706] ahci 0001:01:00.0: controller reset failed (0xffffffff)
>> [    4.498114] ahci: probe of 0001:01:00.0 failed with error -5
>>
>> After applying this series, AHCI over PCI is back to normal:
>>
>> [    3.543230] ahci 0001:01:00.0: AHCI 0001.0000 32 slots 1 ports 6 
>> Gbps 0x1 impl SATA mode
>> [    3.550841] ahci 0001:01:00.0: flags: 64bit ncq sntf led only pmp 
>> fbs pio slum part sxs
>> [    3.559747] scsi host0: ahci
>> [    3.561998] ata1: SATA max UDMA/133 abar m512@0x1230010000 port 
>> 0x1230010100 irq 63
>>
>> So for the series:
>>
>> Tested-by: Thierry Reding <treding@nvidia.com>
>>

Re: [PATCH 0/3] Add support to handle prefetchable memoryg

[Lorenzo Pieralisi]

On Tue, Nov 17, 2020 at 10:08:35AM +0530, Vidya Sagar wrote:
> Hi Lorenzo & Bjorn,
> Sorry to bother you.
> Could you please take a look at the patches-1 & 2 from this series?

IIUC we should:

(1) apply https://patchwork.kernel.org/project/linux-pci/patch/20201026181652.418729-1-robh@kernel.org
(2) apply [1,2] from this series

For (2), are they rebased against v5.10-rc3 with (1) applied ? I need to
check but I will probably have to use v5.10-rc3 as baseline owing to
commit:

9fff3256f93d

(1) depends on it.

Lorenzo

> Thanks,
> Vidya Sagar
> 
> On 11/4/2020 1:16 PM, Vidya Sagar wrote:
> > External email: Use caution opening links or attachments
> > 
> > 
> > Lorenzo / Bjorn,
> > Could you please review patches-1 & 2 in this series?
> > For the third patch, we already went with Rob's patch @
> > http://patchwork.ozlabs.org/project/linux-pci/patch/20201026154852.221483-1-robh@kernel.org/
> > 
> > 
> > Thanks,
> > Vidya Sagar
> > 
> > On 10/26/2020 6:02 PM, Thierry Reding wrote:
> > > On Sat, Oct 24, 2020 at 04:03:41AM +0000, Jingoo Han wrote:
> > > > On 10/23/20, 3:57 PM, Vidya Sagar wrote:
> > > > > 
> > > > > This patch series adds support for configuring the DesignWare IP's ATU
> > > > > region for prefetchable memory translations.
> > > > > It first starts by flagging a warning if the size of non-prefetchable
> > > > > aperture goes beyond 32-bit as PCIe spec doesn't allow it.
> > > > > And then adds required support for programming the ATU to handle higher
> > > > > (i.e. >4GB) sizes and then finally adds support for differentiating
> > > > > between prefetchable and non-prefetchable regions and
> > > > > configuring one of
> > > > > the ATU regions for prefetchable memory translations purpose.
> > > > > 
> > > > > Vidya Sagar (3):
> > > > >    PCI: of: Warn if non-prefetchable memory aperture size is > 32-bit
> > > > >    PCI: dwc: Add support to program ATU for >4GB memory aperture sizes
> > > > >    PCI: dwc: Add support to handle prefetchable memory mapping
> > > > 
> > > > For 2nd & 3rd,
> > > > Acked-by: Jingoo <jingoohan1@gmail.com>
> > > > But, I still want someone to ack 1st patch, not me.
> > > > 
> > > > To Vidya,
> > > > If possible, can you ask your coworker to give 'Tested-by'? It
> > > > will be very helpful.
> > > > Thank you.
> > > 
> > > On next-20201026 (but also going back quite a while) I'm seeing this
> > > during boot on Jetson AGX Xavier (Tegra194):
> > > 
> > > [    3.493382] ahci 0001:01:00.0: version 3.0
> > > [    3.493889] ahci 0001:01:00.0: SSS flag set, parallel bus scan
> > > disabled
> > > [    4.497706] ahci 0001:01:00.0: controller reset failed (0xffffffff)
> > > [    4.498114] ahci: probe of 0001:01:00.0 failed with error -5
> > > 
> > > After applying this series, AHCI over PCI is back to normal:
> > > 
> > > [    3.543230] ahci 0001:01:00.0: AHCI 0001.0000 32 slots 1 ports 6
> > > Gbps 0x1 impl SATA mode
> > > [    3.550841] ahci 0001:01:00.0: flags: 64bit ncq sntf led only pmp
> > > fbs pio slum part sxs
> > > [    3.559747] scsi host0: ahci
> > > [    3.561998] ata1: SATA max UDMA/133 abar m512@0x1230010000 port
> > > 0x1230010100 irq 63
> > > 
> > > So for the series:
> > > 
> > > Tested-by: Thierry Reding <treding@nvidia.com>
> > > 

Re: [PATCH 0/3] Add support to handle prefetchable memoryg

[Vidya Sagar]

On 11/17/2020 5:40 PM, Lorenzo Pieralisi wrote:
> External email: Use caution opening links or attachments
> 
> 
> On Tue, Nov 17, 2020 at 10:08:35AM +0530, Vidya Sagar wrote:
>> Hi Lorenzo & Bjorn,
>> Sorry to bother you.
>> Could you please take a look at the patches-1 & 2 from this series?
> 
> IIUC we should:
> 
> (1) apply https://patchwork.kernel.org/project/linux-pci/patch/20201026181652.418729-1-robh@kernel.org
> (2) apply [1,2] from this series
> 
> For (2), are they rebased against v5.10-rc3 with (1) applied ? I need to
> check but I will probably have to use v5.10-rc3 as baseline owing to
> commit:
> 
> 9fff3256f93d
> 
> (1) depends on it.
> 
> Lorenzo
My patches [1,2] from this series apply cleanly on v5.10-rc3. But with 
(1) applied first, there is a trivial rebase required. Let me know if 
you want me to send the trivial rebased version (of patch-2 particularly).

Thanks,
Vidya Sagar
> 
>> Thanks,
>> Vidya Sagar
>>
>> On 11/4/2020 1:16 PM, Vidya Sagar wrote:
>>> External email: Use caution opening links or attachments
>>>
>>>
>>> Lorenzo / Bjorn,
>>> Could you please review patches-1 & 2 in this series?
>>> For the third patch, we already went with Rob's patch @
>>> http://patchwork.ozlabs.org/project/linux-pci/patch/20201026154852.221483-1-robh@kernel.org/
>>>
>>>
>>> Thanks,
>>> Vidya Sagar
>>>
>>> On 10/26/2020 6:02 PM, Thierry Reding wrote:
>>>> On Sat, Oct 24, 2020 at 04:03:41AM +0000, Jingoo Han wrote:
>>>>> On 10/23/20, 3:57 PM, Vidya Sagar wrote:
>>>>>>
>>>>>> This patch series adds support for configuring the DesignWare IP's ATU
>>>>>> region for prefetchable memory translations.
>>>>>> It first starts by flagging a warning if the size of non-prefetchable
>>>>>> aperture goes beyond 32-bit as PCIe spec doesn't allow it.
>>>>>> And then adds required support for programming the ATU to handle higher
>>>>>> (i.e. >4GB) sizes and then finally adds support for differentiating
>>>>>> between prefetchable and non-prefetchable regions and
>>>>>> configuring one of
>>>>>> the ATU regions for prefetchable memory translations purpose.
>>>>>>
>>>>>> Vidya Sagar (3):
>>>>>>     PCI: of: Warn if non-prefetchable memory aperture size is > 32-bit
>>>>>>     PCI: dwc: Add support to program ATU for >4GB memory aperture sizes
>>>>>>     PCI: dwc: Add support to handle prefetchable memory mapping
>>>>>
>>>>> For 2nd & 3rd,
>>>>> Acked-by: Jingoo <jingoohan1@gmail.com>
>>>>> But, I still want someone to ack 1st patch, not me.
>>>>>
>>>>> To Vidya,
>>>>> If possible, can you ask your coworker to give 'Tested-by'? It
>>>>> will be very helpful.
>>>>> Thank you.
>>>>
>>>> On next-20201026 (but also going back quite a while) I'm seeing this
>>>> during boot on Jetson AGX Xavier (Tegra194):
>>>>
>>>> [    3.493382] ahci 0001:01:00.0: version 3.0
>>>> [    3.493889] ahci 0001:01:00.0: SSS flag set, parallel bus scan
>>>> disabled
>>>> [    4.497706] ahci 0001:01:00.0: controller reset failed (0xffffffff)
>>>> [    4.498114] ahci: probe of 0001:01:00.0 failed with error -5
>>>>
>>>> After applying this series, AHCI over PCI is back to normal:
>>>>
>>>> [    3.543230] ahci 0001:01:00.0: AHCI 0001.0000 32 slots 1 ports 6
>>>> Gbps 0x1 impl SATA mode
>>>> [    3.550841] ahci 0001:01:00.0: flags: 64bit ncq sntf led only pmp
>>>> fbs pio slum part sxs
>>>> [    3.559747] scsi host0: ahci
>>>> [    3.561998] ata1: SATA max UDMA/133 abar m512@0x1230010000 port
>>>> 0x1230010100 irq 63
>>>>
>>>> So for the series:
>>>>
>>>> Tested-by: Thierry Reding <treding@nvidia.com>
>>>>

Re: [PATCH 0/3] Add support to handle prefetchable memory

[Lorenzo Pieralisi]

On Tue, Nov 17, 2020 at 11:04:57PM +0530, Vidya Sagar wrote:

[...]

> > IIUC we should:
> > 
> > (1) apply https://patchwork.kernel.org/project/linux-pci/patch/20201026181652.418729-1-robh@kernel.org
> > (2) apply [1,2] from this series
> > 
> > For (2), are they rebased against v5.10-rc3 with (1) applied ? I need to
> > check but I will probably have to use v5.10-rc3 as baseline owing to
> > commit:
> > 
> > 9fff3256f93d
> > 
> > (1) depends on it.
> > 
> > Lorenzo
> My patches [1,2] from this series apply cleanly on v5.10-rc3. But with (1)
> applied first, there is a trivial rebase required. Let me know if you want
> me to send the trivial rebased version (of patch-2 particularly).

Please do - I shall apply (1) first (on top of v5.10-rc3).

Lorenzo