[PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Jisheng Zhang]

dw_pcie_setup_rc(), as its name indicates, setups the RC. But current
dw_pcie_host_init() also contains some necessary rc setup code.

Another reason: the host may lost power during suspend to ram, the RC
need to be re-setup after resume. The rc can't be correctly resumed
without the rc setup code in dw_pcie_host_init().

So this patch moves the code to dw_pcie_setup_rc() to address the above
two issues. After this patch, each pcie designware driver users could
call dw_pcie_setup_rc() to re-setup rc when resume back.

Signed-off-by: Jisheng Zhang <jszhang@marvell.com>
---
Since v1:
 - fix gcc warning found by lkp, thanks

 drivers/pci/host/pcie-designware.c | 39 +++++++++++++++++++-------------------
 1 file changed, 19 insertions(+), 20 deletions(-)

diff --git a/drivers/pci/host/pcie-designware.c b/drivers/pci/host/pcie-designware.c
index a4cccd3..261e4a11 100644
--- a/drivers/pci/host/pcie-designware.c
+++ b/drivers/pci/host/pcie-designware.c
@@ -434,7 +434,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
 	struct platform_device *pdev = to_platform_device(pp->dev);
 	struct pci_bus *bus, *child;
 	struct resource *cfg_res;
-	u32 val;
 	int i, ret;
 	LIST_HEAD(res);
 	struct resource_entry *win;
@@ -544,25 +543,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
 	if (pp->ops->host_init)
 		pp->ops->host_init(pp);
 
-	/*
-	 * If the platform provides ->rd_other_conf, it means the platform
-	 * uses its own address translation component rather than ATU, so
-	 * we should not program the ATU here.
-	 */
-	if (!pp->ops->rd_other_conf)
-		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
-					  PCIE_ATU_TYPE_MEM, pp->mem_base,
-					  pp->mem_bus_addr, pp->mem_size);
-
-	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
-
-	/* program correct class for RC */
-	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
-
-	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
-	val |= PORT_LOGIC_SPEED_CHANGE;
-	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
-
 	pp->root_bus_nr = pp->busn->start;
 	if (IS_ENABLED(CONFIG_PCI_MSI)) {
 		bus = pci_scan_root_bus_msi(pp->dev, pp->root_bus_nr,
@@ -800,6 +780,25 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
 	val |= PCI_COMMAND_IO | PCI_COMMAND_MEMORY |
 		PCI_COMMAND_MASTER | PCI_COMMAND_SERR;
 	dw_pcie_writel_rc(pp, val, PCI_COMMAND);
+
+	/*
+	 * If the platform provides ->rd_other_conf, it means the platform
+	 * uses its own address translation component rather than ATU, so
+	 * we should not program the ATU here.
+	 */
+	if (!pp->ops->rd_other_conf)
+		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
+					  PCIE_ATU_TYPE_MEM, pp->mem_base,
+					  pp->mem_bus_addr, pp->mem_size);
+
+	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
+
+	/* program correct class for RC */
+	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
+
+	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
+	val |= PORT_LOGIC_SPEED_CHANGE;
+	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
 }
 
 MODULE_AUTHOR("Jingoo Han <jg1.han@samsung.com>");
-- 
2.7.0

[PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Jisheng Zhang]

dw_pcie_setup_rc(), as its name indicates, setups the RC. But current
dw_pcie_host_init() also contains some necessary rc setup code.

Another reason: the host may lost power during suspend to ram, the RC
need to be re-setup after resume. The rc can't be correctly resumed
without the rc setup code in dw_pcie_host_init().

So this patch moves the code to dw_pcie_setup_rc() to address the above
two issues. After this patch, each pcie designware driver users could
call dw_pcie_setup_rc() to re-setup rc when resume back.

Signed-off-by: Jisheng Zhang <jszhang@marvell.com>
---
Since v1:
 - fix gcc warning found by lkp, thanks

 drivers/pci/host/pcie-designware.c | 39 +++++++++++++++++++-------------------
 1 file changed, 19 insertions(+), 20 deletions(-)

diff --git a/drivers/pci/host/pcie-designware.c b/drivers/pci/host/pcie-designware.c
index a4cccd3..261e4a11 100644
--- a/drivers/pci/host/pcie-designware.c
+++ b/drivers/pci/host/pcie-designware.c
@@ -434,7 +434,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
 	struct platform_device *pdev = to_platform_device(pp->dev);
 	struct pci_bus *bus, *child;
 	struct resource *cfg_res;
-	u32 val;
 	int i, ret;
 	LIST_HEAD(res);
 	struct resource_entry *win;
@@ -544,25 +543,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
 	if (pp->ops->host_init)
 		pp->ops->host_init(pp);
 
-	/*
-	 * If the platform provides ->rd_other_conf, it means the platform
-	 * uses its own address translation component rather than ATU, so
-	 * we should not program the ATU here.
-	 */
-	if (!pp->ops->rd_other_conf)
-		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
-					  PCIE_ATU_TYPE_MEM, pp->mem_base,
-					  pp->mem_bus_addr, pp->mem_size);
-
-	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
-
-	/* program correct class for RC */
-	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
-
-	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
-	val |= PORT_LOGIC_SPEED_CHANGE;
-	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
-
 	pp->root_bus_nr = pp->busn->start;
 	if (IS_ENABLED(CONFIG_PCI_MSI)) {
 		bus = pci_scan_root_bus_msi(pp->dev, pp->root_bus_nr,
@@ -800,6 +780,25 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
 	val |= PCI_COMMAND_IO | PCI_COMMAND_MEMORY |
 		PCI_COMMAND_MASTER | PCI_COMMAND_SERR;
 	dw_pcie_writel_rc(pp, val, PCI_COMMAND);
+
+	/*
+	 * If the platform provides ->rd_other_conf, it means the platform
+	 * uses its own address translation component rather than ATU, so
+	 * we should not program the ATU here.
+	 */
+	if (!pp->ops->rd_other_conf)
+		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
+					  PCIE_ATU_TYPE_MEM, pp->mem_base,
+					  pp->mem_bus_addr, pp->mem_size);
+
+	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
+
+	/* program correct class for RC */
+	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
+
+	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
+	val |= PORT_LOGIC_SPEED_CHANGE;
+	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
 }
 
 MODULE_AUTHOR("Jingoo Han <jg1.han@samsung.com>");
-- 
2.7.0

Re: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Pratyush Anand]

On Wed, Mar 16, 2016 at 5:10 PM, Jisheng Zhang <jszhang@marvell.com> wrote:
> dw_pcie_setup_rc(), as its name indicates, setups the RC. But current
> dw_pcie_host_init() also contains some necessary rc setup code.
>
> Another reason: the host may lost power during suspend to ram, the RC
> need to be re-setup after resume. The rc can't be correctly resumed
> without the rc setup code in dw_pcie_host_init().
>
> So this patch moves the code to dw_pcie_setup_rc() to address the above
> two issues. After this patch, each pcie designware driver users could
> call dw_pcie_setup_rc() to re-setup rc when resume back.
>
> Signed-off-by: Jisheng Zhang <jszhang@marvell.com>

Seems reasonable. Thanks for the cleanup. :-)

Acked-by: Pratyush Anand <pratyush.anand@gmail.com>

Re: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Pratyush Anand]

On Wed, Mar 16, 2016 at 5:10 PM, Jisheng Zhang <jszhang@marvell.com> wrote:
> dw_pcie_setup_rc(), as its name indicates, setups the RC. But current
> dw_pcie_host_init() also contains some necessary rc setup code.
>
> Another reason: the host may lost power during suspend to ram, the RC
> need to be re-setup after resume. The rc can't be correctly resumed
> without the rc setup code in dw_pcie_host_init().
>
> So this patch moves the code to dw_pcie_setup_rc() to address the above
> two issues. After this patch, each pcie designware driver users could
> call dw_pcie_setup_rc() to re-setup rc when resume back.
>
> Signed-off-by: Jisheng Zhang <jszhang@marvell.com>

Seems reasonable. Thanks for the cleanup. :-)

Acked-by: Pratyush Anand <pratyush.anand@gmail.com>

[PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Pratyush Anand]

On Wed, Mar 16, 2016 at 5:10 PM, Jisheng Zhang <jszhang@marvell.com> wrote:
> dw_pcie_setup_rc(), as its name indicates, setups the RC. But current
> dw_pcie_host_init() also contains some necessary rc setup code.
>
> Another reason: the host may lost power during suspend to ram, the RC
> need to be re-setup after resume. The rc can't be correctly resumed
> without the rc setup code in dw_pcie_host_init().
>
> So this patch moves the code to dw_pcie_setup_rc() to address the above
> two issues. After this patch, each pcie designware driver users could
> call dw_pcie_setup_rc() to re-setup rc when resume back.
>
> Signed-off-by: Jisheng Zhang <jszhang@marvell.com>

Seems reasonable. Thanks for the cleanup. :-)

Acked-by: Pratyush Anand <pratyush.anand@gmail.com>

Re: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Bjorn Helgaas]

On Wed, Mar 16, 2016 at 07:40:33PM +0800, Jisheng Zhang wrote:
> dw_pcie_setup_rc(), as its name indicates, setups the RC. But current
> dw_pcie_host_init() also contains some necessary rc setup code.
> 
> Another reason: the host may lost power during suspend to ram, the RC
> need to be re-setup after resume. The rc can't be correctly resumed
> without the rc setup code in dw_pcie_host_init().
> 
> So this patch moves the code to dw_pcie_setup_rc() to address the above
> two issues. After this patch, each pcie designware driver users could
> call dw_pcie_setup_rc() to re-setup rc when resume back.
> 
> Signed-off-by: Jisheng Zhang <jszhang@marvell.com>

Very nice, applied with Pratyush's ack to pci/host-designware for
v4.7, thanks!

> ---
> Since v1:
>  - fix gcc warning found by lkp, thanks
> 
>  drivers/pci/host/pcie-designware.c | 39 +++++++++++++++++++-------------------
>  1 file changed, 19 insertions(+), 20 deletions(-)
> 
> diff --git a/drivers/pci/host/pcie-designware.c b/drivers/pci/host/pcie-designware.c
> index a4cccd3..261e4a11 100644
> --- a/drivers/pci/host/pcie-designware.c
> +++ b/drivers/pci/host/pcie-designware.c
> @@ -434,7 +434,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
>  	struct platform_device *pdev = to_platform_device(pp->dev);
>  	struct pci_bus *bus, *child;
>  	struct resource *cfg_res;
> -	u32 val;
>  	int i, ret;
>  	LIST_HEAD(res);
>  	struct resource_entry *win;
> @@ -544,25 +543,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
>  	if (pp->ops->host_init)
>  		pp->ops->host_init(pp);
>  
> -	/*
> -	 * If the platform provides ->rd_other_conf, it means the platform
> -	 * uses its own address translation component rather than ATU, so
> -	 * we should not program the ATU here.
> -	 */
> -	if (!pp->ops->rd_other_conf)
> -		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> -					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> -					  pp->mem_bus_addr, pp->mem_size);
> -
> -	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> -
> -	/* program correct class for RC */
> -	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
> -
> -	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> -	val |= PORT_LOGIC_SPEED_CHANGE;
> -	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
> -
>  	pp->root_bus_nr = pp->busn->start;
>  	if (IS_ENABLED(CONFIG_PCI_MSI)) {
>  		bus = pci_scan_root_bus_msi(pp->dev, pp->root_bus_nr,
> @@ -800,6 +780,25 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
>  	val |= PCI_COMMAND_IO | PCI_COMMAND_MEMORY |
>  		PCI_COMMAND_MASTER | PCI_COMMAND_SERR;
>  	dw_pcie_writel_rc(pp, val, PCI_COMMAND);
> +
> +	/*
> +	 * If the platform provides ->rd_other_conf, it means the platform
> +	 * uses its own address translation component rather than ATU, so
> +	 * we should not program the ATU here.
> +	 */
> +	if (!pp->ops->rd_other_conf)
> +		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> +					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> +					  pp->mem_bus_addr, pp->mem_size);
> +
> +	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> +
> +	/* program correct class for RC */
> +	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
> +
> +	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> +	val |= PORT_LOGIC_SPEED_CHANGE;
> +	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
>  }
>  
>  MODULE_AUTHOR("Jingoo Han <jg1.han@samsung.com>");
> -- 
> 2.7.0
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Bjorn Helgaas]

On Wed, Mar 16, 2016 at 07:40:33PM +0800, Jisheng Zhang wrote:
> dw_pcie_setup_rc(), as its name indicates, setups the RC. But current
> dw_pcie_host_init() also contains some necessary rc setup code.
> 
> Another reason: the host may lost power during suspend to ram, the RC
> need to be re-setup after resume. The rc can't be correctly resumed
> without the rc setup code in dw_pcie_host_init().
> 
> So this patch moves the code to dw_pcie_setup_rc() to address the above
> two issues. After this patch, each pcie designware driver users could
> call dw_pcie_setup_rc() to re-setup rc when resume back.
> 
> Signed-off-by: Jisheng Zhang <jszhang@marvell.com>

Very nice, applied with Pratyush's ack to pci/host-designware for
v4.7, thanks!

> ---
> Since v1:
>  - fix gcc warning found by lkp, thanks
> 
>  drivers/pci/host/pcie-designware.c | 39 +++++++++++++++++++-------------------
>  1 file changed, 19 insertions(+), 20 deletions(-)
> 
> diff --git a/drivers/pci/host/pcie-designware.c b/drivers/pci/host/pcie-designware.c
> index a4cccd3..261e4a11 100644
> --- a/drivers/pci/host/pcie-designware.c
> +++ b/drivers/pci/host/pcie-designware.c
> @@ -434,7 +434,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
>  	struct platform_device *pdev = to_platform_device(pp->dev);
>  	struct pci_bus *bus, *child;
>  	struct resource *cfg_res;
> -	u32 val;
>  	int i, ret;
>  	LIST_HEAD(res);
>  	struct resource_entry *win;
> @@ -544,25 +543,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
>  	if (pp->ops->host_init)
>  		pp->ops->host_init(pp);
>  
> -	/*
> -	 * If the platform provides ->rd_other_conf, it means the platform
> -	 * uses its own address translation component rather than ATU, so
> -	 * we should not program the ATU here.
> -	 */
> -	if (!pp->ops->rd_other_conf)
> -		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> -					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> -					  pp->mem_bus_addr, pp->mem_size);
> -
> -	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> -
> -	/* program correct class for RC */
> -	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
> -
> -	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> -	val |= PORT_LOGIC_SPEED_CHANGE;
> -	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
> -
>  	pp->root_bus_nr = pp->busn->start;
>  	if (IS_ENABLED(CONFIG_PCI_MSI)) {
>  		bus = pci_scan_root_bus_msi(pp->dev, pp->root_bus_nr,
> @@ -800,6 +780,25 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
>  	val |= PCI_COMMAND_IO | PCI_COMMAND_MEMORY |
>  		PCI_COMMAND_MASTER | PCI_COMMAND_SERR;
>  	dw_pcie_writel_rc(pp, val, PCI_COMMAND);
> +
> +	/*
> +	 * If the platform provides ->rd_other_conf, it means the platform
> +	 * uses its own address translation component rather than ATU, so
> +	 * we should not program the ATU here.
> +	 */
> +	if (!pp->ops->rd_other_conf)
> +		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> +					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> +					  pp->mem_bus_addr, pp->mem_size);
> +
> +	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> +
> +	/* program correct class for RC */
> +	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
> +
> +	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> +	val |= PORT_LOGIC_SPEED_CHANGE;
> +	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
>  }
>  
>  MODULE_AUTHOR("Jingoo Han <jg1.han@samsung.com>");
> -- 
> 2.7.0
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Gabriele Paoloni]

Hi, sorry to be late on this

> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-
> owner@vger.kernel.org] On Behalf Of Jisheng Zhang
> Sent: 16 March 2016 11:41
> To: jingoohan1@gmail.com; pratyush.anand@gmail.com; bhelgaas@google.com
> Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org; Jisheng Zhang
> Subject: [PATCH v2] PCI: designware: move remaining rc setup code to
> dw_pcie_setup_rc()
> 
> dw_pcie_setup_rc(), as its name indicates, setups the RC. But current
> dw_pcie_host_init() also contains some necessary rc setup code.
> 
> Another reason: the host may lost power during suspend to ram, the RC
> need to be re-setup after resume. The rc can't be correctly resumed
> without the rc setup code in dw_pcie_host_init().
> 
> So this patch moves the code to dw_pcie_setup_rc() to address the above
> two issues. After this patch, each pcie designware driver users could
> call dw_pcie_setup_rc() to re-setup rc when resume back.

I think this patch breaks the Hisilicon driver...

Our driver performs linkup setup in UEFI therefore we do not call
dw_pcie_setup_rc(), we only call dw_pcie_host_init().

Maybe better to group the part of code to be moved in as separate
function...

Thanks and sorry for late reply.

Gab


> 
> Signed-off-by: Jisheng Zhang <jszhang@marvell.com>
> ---
> Since v1:
>  - fix gcc warning found by lkp, thanks
> 
>  drivers/pci/host/pcie-designware.c | 39 +++++++++++++++++++-----------
> --------
>  1 file changed, 19 insertions(+), 20 deletions(-)
> 
> diff --git a/drivers/pci/host/pcie-designware.c
> b/drivers/pci/host/pcie-designware.c
> index a4cccd3..261e4a11 100644
> --- a/drivers/pci/host/pcie-designware.c
> +++ b/drivers/pci/host/pcie-designware.c
> @@ -434,7 +434,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
>  	struct platform_device *pdev = to_platform_device(pp->dev);
>  	struct pci_bus *bus, *child;
>  	struct resource *cfg_res;
> -	u32 val;
>  	int i, ret;
>  	LIST_HEAD(res);
>  	struct resource_entry *win;
> @@ -544,25 +543,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
>  	if (pp->ops->host_init)
>  		pp->ops->host_init(pp);
> 
> -	/*
> -	 * If the platform provides ->rd_other_conf, it means the
> platform
> -	 * uses its own address translation component rather than ATU, so
> -	 * we should not program the ATU here.
> -	 */
> -	if (!pp->ops->rd_other_conf)
> -		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> -					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> -					  pp->mem_bus_addr, pp->mem_size);
> -
> -	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> -
> -	/* program correct class for RC */
> -	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2,
> PCI_CLASS_BRIDGE_PCI);
> -
> -	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> -	val |= PORT_LOGIC_SPEED_CHANGE;
> -	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
> -
>  	pp->root_bus_nr = pp->busn->start;
>  	if (IS_ENABLED(CONFIG_PCI_MSI)) {
>  		bus = pci_scan_root_bus_msi(pp->dev, pp->root_bus_nr,
> @@ -800,6 +780,25 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
>  	val |= PCI_COMMAND_IO | PCI_COMMAND_MEMORY |
>  		PCI_COMMAND_MASTER | PCI_COMMAND_SERR;
>  	dw_pcie_writel_rc(pp, val, PCI_COMMAND);
> +
> +	/*
> +	 * If the platform provides ->rd_other_conf, it means the
> platform
> +	 * uses its own address translation component rather than ATU, so
> +	 * we should not program the ATU here.
> +	 */
> +	if (!pp->ops->rd_other_conf)
> +		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> +					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> +					  pp->mem_bus_addr, pp->mem_size);
> +
> +	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> +
> +	/* program correct class for RC */
> +	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2,
> PCI_CLASS_BRIDGE_PCI);
> +
> +	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> +	val |= PORT_LOGIC_SPEED_CHANGE;
> +	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
>  }
> 
>  MODULE_AUTHOR("Jingoo Han <jg1.han@samsung.com>");
> --
> 2.7.0

RE: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Gabriele Paoloni]

Hi, sorry to be late on this

> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-
> owner@vger.kernel.org] On Behalf Of Jisheng Zhang
> Sent: 16 March 2016 11:41
> To: jingoohan1@gmail.com; pratyush.anand@gmail.com; bhelgaas@google.com
> Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org; Jisheng Zhang
> Subject: [PATCH v2] PCI: designware: move remaining rc setup code to
> dw_pcie_setup_rc()
> 
> dw_pcie_setup_rc(), as its name indicates, setups the RC. But current
> dw_pcie_host_init() also contains some necessary rc setup code.
> 
> Another reason: the host may lost power during suspend to ram, the RC
> need to be re-setup after resume. The rc can't be correctly resumed
> without the rc setup code in dw_pcie_host_init().
> 
> So this patch moves the code to dw_pcie_setup_rc() to address the above
> two issues. After this patch, each pcie designware driver users could
> call dw_pcie_setup_rc() to re-setup rc when resume back.

I think this patch breaks the Hisilicon driver...

Our driver performs linkup setup in UEFI therefore we do not call
dw_pcie_setup_rc(), we only call dw_pcie_host_init().

Maybe better to group the part of code to be moved in as separate
function...

Thanks and sorry for late reply.

Gab


> 
> Signed-off-by: Jisheng Zhang <jszhang@marvell.com>
> ---
> Since v1:
>  - fix gcc warning found by lkp, thanks
> 
>  drivers/pci/host/pcie-designware.c | 39 +++++++++++++++++++-----------
> --------
>  1 file changed, 19 insertions(+), 20 deletions(-)
> 
> diff --git a/drivers/pci/host/pcie-designware.c
> b/drivers/pci/host/pcie-designware.c
> index a4cccd3..261e4a11 100644
> --- a/drivers/pci/host/pcie-designware.c
> +++ b/drivers/pci/host/pcie-designware.c
> @@ -434,7 +434,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
>  	struct platform_device *pdev = to_platform_device(pp->dev);
>  	struct pci_bus *bus, *child;
>  	struct resource *cfg_res;
> -	u32 val;
>  	int i, ret;
>  	LIST_HEAD(res);
>  	struct resource_entry *win;
> @@ -544,25 +543,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
>  	if (pp->ops->host_init)
>  		pp->ops->host_init(pp);
> 
> -	/*
> -	 * If the platform provides ->rd_other_conf, it means the
> platform
> -	 * uses its own address translation component rather than ATU, so
> -	 * we should not program the ATU here.
> -	 */
> -	if (!pp->ops->rd_other_conf)
> -		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> -					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> -					  pp->mem_bus_addr, pp->mem_size);
> -
> -	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> -
> -	/* program correct class for RC */
> -	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2,
> PCI_CLASS_BRIDGE_PCI);
> -
> -	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> -	val |= PORT_LOGIC_SPEED_CHANGE;
> -	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
> -
>  	pp->root_bus_nr = pp->busn->start;
>  	if (IS_ENABLED(CONFIG_PCI_MSI)) {
>  		bus = pci_scan_root_bus_msi(pp->dev, pp->root_bus_nr,
> @@ -800,6 +780,25 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
>  	val |= PCI_COMMAND_IO | PCI_COMMAND_MEMORY |
>  		PCI_COMMAND_MASTER | PCI_COMMAND_SERR;
>  	dw_pcie_writel_rc(pp, val, PCI_COMMAND);
> +
> +	/*
> +	 * If the platform provides ->rd_other_conf, it means the
> platform
> +	 * uses its own address translation component rather than ATU, so
> +	 * we should not program the ATU here.
> +	 */
> +	if (!pp->ops->rd_other_conf)
> +		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> +					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> +					  pp->mem_bus_addr, pp->mem_size);
> +
> +	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> +
> +	/* program correct class for RC */
> +	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2,
> PCI_CLASS_BRIDGE_PCI);
> +
> +	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> +	val |= PORT_LOGIC_SPEED_CHANGE;
> +	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
>  }
> 
>  MODULE_AUTHOR("Jingoo Han <jg1.han@samsung.com>");
> --
> 2.7.0

[PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Gabriele Paoloni]

Hi, sorry to be late on this

> -----Original Message-----
> From: linux-kernel-owner at vger.kernel.org [mailto:linux-kernel-
> owner at vger.kernel.org] On Behalf Of Jisheng Zhang
> Sent: 16 March 2016 11:41
> To: jingoohan1 at gmail.com; pratyush.anand at gmail.com; bhelgaas at google.com
> Cc: linux-pci at vger.kernel.org; linux-kernel at vger.kernel.org; linux-arm-
> kernel at lists.infradead.org; Jisheng Zhang
> Subject: [PATCH v2] PCI: designware: move remaining rc setup code to
> dw_pcie_setup_rc()
> 
> dw_pcie_setup_rc(), as its name indicates, setups the RC. But current
> dw_pcie_host_init() also contains some necessary rc setup code.
> 
> Another reason: the host may lost power during suspend to ram, the RC
> need to be re-setup after resume. The rc can't be correctly resumed
> without the rc setup code in dw_pcie_host_init().
> 
> So this patch moves the code to dw_pcie_setup_rc() to address the above
> two issues. After this patch, each pcie designware driver users could
> call dw_pcie_setup_rc() to re-setup rc when resume back.

I think this patch breaks the Hisilicon driver...

Our driver performs linkup setup in UEFI therefore we do not call
dw_pcie_setup_rc(), we only call dw_pcie_host_init().

Maybe better to group the part of code to be moved in as separate
function...

Thanks and sorry for late reply.

Gab


> 
> Signed-off-by: Jisheng Zhang <jszhang@marvell.com>
> ---
> Since v1:
>  - fix gcc warning found by lkp, thanks
> 
>  drivers/pci/host/pcie-designware.c | 39 +++++++++++++++++++-----------
> --------
>  1 file changed, 19 insertions(+), 20 deletions(-)
> 
> diff --git a/drivers/pci/host/pcie-designware.c
> b/drivers/pci/host/pcie-designware.c
> index a4cccd3..261e4a11 100644
> --- a/drivers/pci/host/pcie-designware.c
> +++ b/drivers/pci/host/pcie-designware.c
> @@ -434,7 +434,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
>  	struct platform_device *pdev = to_platform_device(pp->dev);
>  	struct pci_bus *bus, *child;
>  	struct resource *cfg_res;
> -	u32 val;
>  	int i, ret;
>  	LIST_HEAD(res);
>  	struct resource_entry *win;
> @@ -544,25 +543,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
>  	if (pp->ops->host_init)
>  		pp->ops->host_init(pp);
> 
> -	/*
> -	 * If the platform provides ->rd_other_conf, it means the
> platform
> -	 * uses its own address translation component rather than ATU, so
> -	 * we should not program the ATU here.
> -	 */
> -	if (!pp->ops->rd_other_conf)
> -		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> -					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> -					  pp->mem_bus_addr, pp->mem_size);
> -
> -	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> -
> -	/* program correct class for RC */
> -	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2,
> PCI_CLASS_BRIDGE_PCI);
> -
> -	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> -	val |= PORT_LOGIC_SPEED_CHANGE;
> -	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
> -
>  	pp->root_bus_nr = pp->busn->start;
>  	if (IS_ENABLED(CONFIG_PCI_MSI)) {
>  		bus = pci_scan_root_bus_msi(pp->dev, pp->root_bus_nr,
> @@ -800,6 +780,25 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
>  	val |= PCI_COMMAND_IO | PCI_COMMAND_MEMORY |
>  		PCI_COMMAND_MASTER | PCI_COMMAND_SERR;
>  	dw_pcie_writel_rc(pp, val, PCI_COMMAND);
> +
> +	/*
> +	 * If the platform provides ->rd_other_conf, it means the
> platform
> +	 * uses its own address translation component rather than ATU, so
> +	 * we should not program the ATU here.
> +	 */
> +	if (!pp->ops->rd_other_conf)
> +		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> +					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> +					  pp->mem_bus_addr, pp->mem_size);
> +
> +	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> +
> +	/* program correct class for RC */
> +	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2,
> PCI_CLASS_BRIDGE_PCI);
> +
> +	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> +	val |= PORT_LOGIC_SPEED_CHANGE;
> +	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
>  }
> 
>  MODULE_AUTHOR("Jingoo Han <jg1.han@samsung.com>");
> --
> 2.7.0

Re: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Jisheng Zhang]

Hi Gabriele,

On Wed, 6 Apr 2016 14:50:29 +0000 Gabriele Paoloni wrote:

> Hi, sorry to be late on this
> 
> > -----Original Message-----
> > From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-
> > owner@vger.kernel.org] On Behalf Of Jisheng Zhang
> > Sent: 16 March 2016 11:41
> > To: jingoohan1@gmail.com; pratyush.anand@gmail.com; bhelgaas@google.com
> > Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; linux-arm-
> > kernel@lists.infradead.org; Jisheng Zhang
> > Subject: [PATCH v2] PCI: designware: move remaining rc setup code to
> > dw_pcie_setup_rc()
> > 
> > dw_pcie_setup_rc(), as its name indicates, setups the RC. But current
> > dw_pcie_host_init() also contains some necessary rc setup code.
> > 
> > Another reason: the host may lost power during suspend to ram, the RC
> > need to be re-setup after resume. The rc can't be correctly resumed
> > without the rc setup code in dw_pcie_host_init().
> > 
> > So this patch moves the code to dw_pcie_setup_rc() to address the above
> > two issues. After this patch, each pcie designware driver users could
> > call dw_pcie_setup_rc() to re-setup rc when resume back.  
> 
> I think this patch breaks the Hisilicon driver...
> 
> Our driver performs linkup setup in UEFI therefore we do not call
> dw_pcie_setup_rc(), we only call dw_pcie_host_init().

Thanks for the information. So pcie-hisi rely on UEFI to do something similar
in dw_pcie_setup_rc(), this comes to a common driver implement question: should
linux device driver rely on bootloader to configure HW device?

Is it acceptable that pcie-hisi adds a call to dw_pcie_setup_rc() in
hisi_add_pcie_port()?

Thanks,
Jisheng

> 
> Maybe better to group the part of code to be moved in as separate
> function...
> 
> Thanks and sorry for late reply.
> 
> Gab
> 
> 

Re: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Jisheng Zhang]

Hi Gabriele,

On Wed, 6 Apr 2016 14:50:29 +0000 Gabriele Paoloni wrote:

> Hi, sorry to be late on this
> 
> > -----Original Message-----
> > From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-
> > owner@vger.kernel.org] On Behalf Of Jisheng Zhang
> > Sent: 16 March 2016 11:41
> > To: jingoohan1@gmail.com; pratyush.anand@gmail.com; bhelgaas@google.com
> > Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; linux-arm-
> > kernel@lists.infradead.org; Jisheng Zhang
> > Subject: [PATCH v2] PCI: designware: move remaining rc setup code to
> > dw_pcie_setup_rc()
> > 
> > dw_pcie_setup_rc(), as its name indicates, setups the RC. But current
> > dw_pcie_host_init() also contains some necessary rc setup code.
> > 
> > Another reason: the host may lost power during suspend to ram, the RC
> > need to be re-setup after resume. The rc can't be correctly resumed
> > without the rc setup code in dw_pcie_host_init().
> > 
> > So this patch moves the code to dw_pcie_setup_rc() to address the above
> > two issues. After this patch, each pcie designware driver users could
> > call dw_pcie_setup_rc() to re-setup rc when resume back.  
> 
> I think this patch breaks the Hisilicon driver...
> 
> Our driver performs linkup setup in UEFI therefore we do not call
> dw_pcie_setup_rc(), we only call dw_pcie_host_init().

Thanks for the information. So pcie-hisi rely on UEFI to do something similar
in dw_pcie_setup_rc(), this comes to a common driver implement question: should
linux device driver rely on bootloader to configure HW device?

Is it acceptable that pcie-hisi adds a call to dw_pcie_setup_rc() in
hisi_add_pcie_port()?

Thanks,
Jisheng

> 
> Maybe better to group the part of code to be moved in as separate
> function...
> 
> Thanks and sorry for late reply.
> 
> Gab
> 
> 

[PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Jisheng Zhang]

Hi Gabriele,

On Wed, 6 Apr 2016 14:50:29 +0000 Gabriele Paoloni wrote:

> Hi, sorry to be late on this
> 
> > -----Original Message-----
> > From: linux-kernel-owner at vger.kernel.org [mailto:linux-kernel-
> > owner at vger.kernel.org] On Behalf Of Jisheng Zhang
> > Sent: 16 March 2016 11:41
> > To: jingoohan1 at gmail.com; pratyush.anand at gmail.com; bhelgaas at google.com
> > Cc: linux-pci at vger.kernel.org; linux-kernel at vger.kernel.org; linux-arm-
> > kernel at lists.infradead.org; Jisheng Zhang
> > Subject: [PATCH v2] PCI: designware: move remaining rc setup code to
> > dw_pcie_setup_rc()
> > 
> > dw_pcie_setup_rc(), as its name indicates, setups the RC. But current
> > dw_pcie_host_init() also contains some necessary rc setup code.
> > 
> > Another reason: the host may lost power during suspend to ram, the RC
> > need to be re-setup after resume. The rc can't be correctly resumed
> > without the rc setup code in dw_pcie_host_init().
> > 
> > So this patch moves the code to dw_pcie_setup_rc() to address the above
> > two issues. After this patch, each pcie designware driver users could
> > call dw_pcie_setup_rc() to re-setup rc when resume back.  
> 
> I think this patch breaks the Hisilicon driver...
> 
> Our driver performs linkup setup in UEFI therefore we do not call
> dw_pcie_setup_rc(), we only call dw_pcie_host_init().

Thanks for the information. So pcie-hisi rely on UEFI to do something similar
in dw_pcie_setup_rc(), this comes to a common driver implement question: should
linux device driver rely on bootloader to configure HW device?

Is it acceptable that pcie-hisi adds a call to dw_pcie_setup_rc() in
hisi_add_pcie_port()?

Thanks,
Jisheng

> 
> Maybe better to group the part of code to be moved in as separate
> function...
> 
> Thanks and sorry for late reply.
> 
> Gab
> 
> 

Re: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Pratyush Anand]

Hi Gab,

Thanks for bringing it.


On Wed, Apr 6, 2016 at 8:20 PM, Gabriele Paoloni
<gabriele.paoloni@huawei.com> wrote:
> Hi, sorry to be late on this
>
>> -----Original Message-----
>> From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-
>> owner@vger.kernel.org] On Behalf Of Jisheng Zhang
>> Sent: 16 March 2016 11:41
>> To: jingoohan1@gmail.com; pratyush.anand@gmail.com; bhelgaas@google.com
>> Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; linux-arm-
>> kernel@lists.infradead.org; Jisheng Zhang
>> Subject: [PATCH v2] PCI: designware: move remaining rc setup code to
>> dw_pcie_setup_rc()
>>
>> dw_pcie_setup_rc(), as its name indicates, setups the RC. But current
>> dw_pcie_host_init() also contains some necessary rc setup code.
>>
>> Another reason: the host may lost power during suspend to ram, the RC
>> need to be re-setup after resume. The rc can't be correctly resumed
>> without the rc setup code in dw_pcie_host_init().
>>
>> So this patch moves the code to dw_pcie_setup_rc() to address the above
>> two issues. After this patch, each pcie designware driver users could
>> call dw_pcie_setup_rc() to re-setup rc when resume back.
>
> I think this patch breaks the Hisilicon driver...
>
> Our driver performs linkup setup in UEFI therefore we do not call
> dw_pcie_setup_rc(), we only call dw_pcie_host_init().
>
> Maybe better to group the part of code to be moved in as separate
> function...
>
> Thanks and sorry for late reply.
>

I am just wondering, should n't then what ever we do in
dw_pcie_setup_rc() be done in your boot loader and not just link up.

~Pratyush

Re: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Pratyush Anand]

Hi Gab,

Thanks for bringing it.


On Wed, Apr 6, 2016 at 8:20 PM, Gabriele Paoloni
<gabriele.paoloni@huawei.com> wrote:
> Hi, sorry to be late on this
>
>> -----Original Message-----
>> From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-
>> owner@vger.kernel.org] On Behalf Of Jisheng Zhang
>> Sent: 16 March 2016 11:41
>> To: jingoohan1@gmail.com; pratyush.anand@gmail.com; bhelgaas@google.com
>> Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; linux-arm-
>> kernel@lists.infradead.org; Jisheng Zhang
>> Subject: [PATCH v2] PCI: designware: move remaining rc setup code to
>> dw_pcie_setup_rc()
>>
>> dw_pcie_setup_rc(), as its name indicates, setups the RC. But current
>> dw_pcie_host_init() also contains some necessary rc setup code.
>>
>> Another reason: the host may lost power during suspend to ram, the RC
>> need to be re-setup after resume. The rc can't be correctly resumed
>> without the rc setup code in dw_pcie_host_init().
>>
>> So this patch moves the code to dw_pcie_setup_rc() to address the above
>> two issues. After this patch, each pcie designware driver users could
>> call dw_pcie_setup_rc() to re-setup rc when resume back.
>
> I think this patch breaks the Hisilicon driver...
>
> Our driver performs linkup setup in UEFI therefore we do not call
> dw_pcie_setup_rc(), we only call dw_pcie_host_init().
>
> Maybe better to group the part of code to be moved in as separate
> function...
>
> Thanks and sorry for late reply.
>

I am just wondering, should n't then what ever we do in
dw_pcie_setup_rc() be done in your boot loader and not just link up.

~Pratyush

[PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Pratyush Anand]

Hi Gab,

Thanks for bringing it.


On Wed, Apr 6, 2016 at 8:20 PM, Gabriele Paoloni
<gabriele.paoloni@huawei.com> wrote:
> Hi, sorry to be late on this
>
>> -----Original Message-----
>> From: linux-kernel-owner at vger.kernel.org [mailto:linux-kernel-
>> owner at vger.kernel.org] On Behalf Of Jisheng Zhang
>> Sent: 16 March 2016 11:41
>> To: jingoohan1 at gmail.com; pratyush.anand at gmail.com; bhelgaas at google.com
>> Cc: linux-pci at vger.kernel.org; linux-kernel at vger.kernel.org; linux-arm-
>> kernel at lists.infradead.org; Jisheng Zhang
>> Subject: [PATCH v2] PCI: designware: move remaining rc setup code to
>> dw_pcie_setup_rc()
>>
>> dw_pcie_setup_rc(), as its name indicates, setups the RC. But current
>> dw_pcie_host_init() also contains some necessary rc setup code.
>>
>> Another reason: the host may lost power during suspend to ram, the RC
>> need to be re-setup after resume. The rc can't be correctly resumed
>> without the rc setup code in dw_pcie_host_init().
>>
>> So this patch moves the code to dw_pcie_setup_rc() to address the above
>> two issues. After this patch, each pcie designware driver users could
>> call dw_pcie_setup_rc() to re-setup rc when resume back.
>
> I think this patch breaks the Hisilicon driver...
>
> Our driver performs linkup setup in UEFI therefore we do not call
> dw_pcie_setup_rc(), we only call dw_pcie_host_init().
>
> Maybe better to group the part of code to be moved in as separate
> function...
>
> Thanks and sorry for late reply.
>

I am just wondering, should n't then what ever we do in
dw_pcie_setup_rc() be done in your boot loader and not just link up.

~Pratyush

RE: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Gabriele Paoloni]

Hi Pratyush

Many thanks for quick replying

> -----Original Message-----
> From: Pratyush Anand [mailto:pratyush.anand@gmail.com]
> Sent: 07 April 2016 07:59
> To: Gabriele Paoloni
> Cc: Jisheng Zhang; jingoohan1@gmail.com; bhelgaas@google.com; linux-
> pci@vger.kernel.org; linux-kernel@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org
> Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> to dw_pcie_setup_rc()
> 
> Hi Gab,
> 
> Thanks for bringing it.
> 
> 
> On Wed, Apr 6, 2016 at 8:20 PM, Gabriele Paoloni
> <gabriele.paoloni@huawei.com> wrote:
> > Hi, sorry to be late on this
> >
> >> -----Original Message-----
> >> From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-
> >> owner@vger.kernel.org] On Behalf Of Jisheng Zhang
> >> Sent: 16 March 2016 11:41
> >> To: jingoohan1@gmail.com; pratyush.anand@gmail.com;
> bhelgaas@google.com
> >> Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; linux-
> arm-
> >> kernel@lists.infradead.org; Jisheng Zhang
> >> Subject: [PATCH v2] PCI: designware: move remaining rc setup code to
> >> dw_pcie_setup_rc()
> >>
> >> dw_pcie_setup_rc(), as its name indicates, setups the RC. But
> current
> >> dw_pcie_host_init() also contains some necessary rc setup code.
> >>
> >> Another reason: the host may lost power during suspend to ram, the
> RC
> >> need to be re-setup after resume. The rc can't be correctly resumed
> >> without the rc setup code in dw_pcie_host_init().
> >>
> >> So this patch moves the code to dw_pcie_setup_rc() to address the
> above
> >> two issues. After this patch, each pcie designware driver users
> could
> >> call dw_pcie_setup_rc() to re-setup rc when resume back.
> >
> > I think this patch breaks the Hisilicon driver...
> >
> > Our driver performs linkup setup in UEFI therefore we do not call
> > dw_pcie_setup_rc(), we only call dw_pcie_host_init().
> >
> > Maybe better to group the part of code to be moved in as separate
> > function...
> >
> > Thanks and sorry for late reply.
> >
> 
> I am just wondering, should n't then what ever we do in
> dw_pcie_setup_rc() be done in your boot loader and not just link up.

Currently the HiSilicon driver does not call dw_pcie_setup_rc() at all;
so everything is done in dw_pcie_setup_rc() is done in bootloader.

I guess your question is if we can execute in bootloader the part of
code the this patch has moved to in "dw_pcie_setup_rc()". I think the
problem here is that even if it was possible we would break backward
compatibility with previous bootloaders...

Thanks

Gab 


> 
> ~Pratyush

RE: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Gabriele Paoloni]

SGkgUHJhdHl1c2gNCg0KTWFueSB0aGFua3MgZm9yIHF1aWNrIHJlcGx5aW5nDQoNCj4gLS0tLS1P
cmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gRnJvbTogUHJhdHl1c2ggQW5hbmQgW21haWx0bzpwcmF0
eXVzaC5hbmFuZEBnbWFpbC5jb21dDQo+IFNlbnQ6IDA3IEFwcmlsIDIwMTYgMDc6NTkNCj4gVG86
IEdhYnJpZWxlIFBhb2xvbmkNCj4gQ2M6IEppc2hlbmcgWmhhbmc7IGppbmdvb2hhbjFAZ21haWwu
Y29tOyBiaGVsZ2Fhc0Bnb29nbGUuY29tOyBsaW51eC0NCj4gcGNpQHZnZXIua2VybmVsLm9yZzsg
bGludXgta2VybmVsQHZnZXIua2VybmVsLm9yZzsgbGludXgtYXJtLQ0KPiBrZXJuZWxAbGlzdHMu
aW5mcmFkZWFkLm9yZw0KPiBTdWJqZWN0OiBSZTogW1BBVENIIHYyXSBQQ0k6IGRlc2lnbndhcmU6
IG1vdmUgcmVtYWluaW5nIHJjIHNldHVwIGNvZGUNCj4gdG8gZHdfcGNpZV9zZXR1cF9yYygpDQo+
IA0KPiBIaSBHYWIsDQo+IA0KPiBUaGFua3MgZm9yIGJyaW5naW5nIGl0Lg0KPiANCj4gDQo+IE9u
IFdlZCwgQXByIDYsIDIwMTYgYXQgODoyMCBQTSwgR2FicmllbGUgUGFvbG9uaQ0KPiA8Z2Ficmll
bGUucGFvbG9uaUBodWF3ZWkuY29tPiB3cm90ZToNCj4gPiBIaSwgc29ycnkgdG8gYmUgbGF0ZSBv
biB0aGlzDQo+ID4NCj4gPj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gPj4gRnJvbTog
bGludXgta2VybmVsLW93bmVyQHZnZXIua2VybmVsLm9yZyBbbWFpbHRvOmxpbnV4LWtlcm5lbC0N
Cj4gPj4gb3duZXJAdmdlci5rZXJuZWwub3JnXSBPbiBCZWhhbGYgT2YgSmlzaGVuZyBaaGFuZw0K
PiA+PiBTZW50OiAxNiBNYXJjaCAyMDE2IDExOjQxDQo+ID4+IFRvOiBqaW5nb29oYW4xQGdtYWls
LmNvbTsgcHJhdHl1c2guYW5hbmRAZ21haWwuY29tOw0KPiBiaGVsZ2Fhc0Bnb29nbGUuY29tDQo+
ID4+IENjOiBsaW51eC1wY2lAdmdlci5rZXJuZWwub3JnOyBsaW51eC1rZXJuZWxAdmdlci5rZXJu
ZWwub3JnOyBsaW51eC0NCj4gYXJtLQ0KPiA+PiBrZXJuZWxAbGlzdHMuaW5mcmFkZWFkLm9yZzsg
SmlzaGVuZyBaaGFuZw0KPiA+PiBTdWJqZWN0OiBbUEFUQ0ggdjJdIFBDSTogZGVzaWdud2FyZTog
bW92ZSByZW1haW5pbmcgcmMgc2V0dXAgY29kZSB0bw0KPiA+PiBkd19wY2llX3NldHVwX3JjKCkN
Cj4gPj4NCj4gPj4gZHdfcGNpZV9zZXR1cF9yYygpLCBhcyBpdHMgbmFtZSBpbmRpY2F0ZXMsIHNl
dHVwcyB0aGUgUkMuIEJ1dA0KPiBjdXJyZW50DQo+ID4+IGR3X3BjaWVfaG9zdF9pbml0KCkgYWxz
byBjb250YWlucyBzb21lIG5lY2Vzc2FyeSByYyBzZXR1cCBjb2RlLg0KPiA+Pg0KPiA+PiBBbm90
aGVyIHJlYXNvbjogdGhlIGhvc3QgbWF5IGxvc3QgcG93ZXIgZHVyaW5nIHN1c3BlbmQgdG8gcmFt
LCB0aGUNCj4gUkMNCj4gPj4gbmVlZCB0byBiZSByZS1zZXR1cCBhZnRlciByZXN1bWUuIFRoZSBy
YyBjYW4ndCBiZSBjb3JyZWN0bHkgcmVzdW1lZA0KPiA+PiB3aXRob3V0IHRoZSByYyBzZXR1cCBj
b2RlIGluIGR3X3BjaWVfaG9zdF9pbml0KCkuDQo+ID4+DQo+ID4+IFNvIHRoaXMgcGF0Y2ggbW92
ZXMgdGhlIGNvZGUgdG8gZHdfcGNpZV9zZXR1cF9yYygpIHRvIGFkZHJlc3MgdGhlDQo+IGFib3Zl
DQo+ID4+IHR3byBpc3N1ZXMuIEFmdGVyIHRoaXMgcGF0Y2gsIGVhY2ggcGNpZSBkZXNpZ253YXJl
IGRyaXZlciB1c2Vycw0KPiBjb3VsZA0KPiA+PiBjYWxsIGR3X3BjaWVfc2V0dXBfcmMoKSB0byBy
ZS1zZXR1cCByYyB3aGVuIHJlc3VtZSBiYWNrLg0KPiA+DQo+ID4gSSB0aGluayB0aGlzIHBhdGNo
IGJyZWFrcyB0aGUgSGlzaWxpY29uIGRyaXZlci4uLg0KPiA+DQo+ID4gT3VyIGRyaXZlciBwZXJm
b3JtcyBsaW5rdXAgc2V0dXAgaW4gVUVGSSB0aGVyZWZvcmUgd2UgZG8gbm90IGNhbGwNCj4gPiBk
d19wY2llX3NldHVwX3JjKCksIHdlIG9ubHkgY2FsbCBkd19wY2llX2hvc3RfaW5pdCgpLg0KPiA+
DQo+ID4gTWF5YmUgYmV0dGVyIHRvIGdyb3VwIHRoZSBwYXJ0IG9mIGNvZGUgdG8gYmUgbW92ZWQg
aW4gYXMgc2VwYXJhdGUNCj4gPiBmdW5jdGlvbi4uLg0KPiA+DQo+ID4gVGhhbmtzIGFuZCBzb3Jy
eSBmb3IgbGF0ZSByZXBseS4NCj4gPg0KPiANCj4gSSBhbSBqdXN0IHdvbmRlcmluZywgc2hvdWxk
IG4ndCB0aGVuIHdoYXQgZXZlciB3ZSBkbyBpbg0KPiBkd19wY2llX3NldHVwX3JjKCkgYmUgZG9u
ZSBpbiB5b3VyIGJvb3QgbG9hZGVyIGFuZCBub3QganVzdCBsaW5rIHVwLg0KDQpDdXJyZW50bHkg
dGhlIEhpU2lsaWNvbiBkcml2ZXIgZG9lcyBub3QgY2FsbCBkd19wY2llX3NldHVwX3JjKCkgYXQg
YWxsOw0Kc28gZXZlcnl0aGluZyBpcyBkb25lIGluIGR3X3BjaWVfc2V0dXBfcmMoKSBpcyBkb25l
IGluIGJvb3Rsb2FkZXIuDQoNCkkgZ3Vlc3MgeW91ciBxdWVzdGlvbiBpcyBpZiB3ZSBjYW4gZXhl
Y3V0ZSBpbiBib290bG9hZGVyIHRoZSBwYXJ0IG9mDQpjb2RlIHRoZSB0aGlzIHBhdGNoIGhhcyBt
b3ZlZCB0byBpbiAiZHdfcGNpZV9zZXR1cF9yYygpIi4gSSB0aGluayB0aGUNCnByb2JsZW0gaGVy
ZSBpcyB0aGF0IGV2ZW4gaWYgaXQgd2FzIHBvc3NpYmxlIHdlIHdvdWxkIGJyZWFrIGJhY2t3YXJk
DQpjb21wYXRpYmlsaXR5IHdpdGggcHJldmlvdXMgYm9vdGxvYWRlcnMuLi4NCg0KVGhhbmtzDQoN
CkdhYiANCg0KDQo+IA0KPiB+UHJhdHl1c2gNCg==

[PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Gabriele Paoloni]

Hi Pratyush

Many thanks for quick replying

> -----Original Message-----
> From: Pratyush Anand [mailto:pratyush.anand at gmail.com]
> Sent: 07 April 2016 07:59
> To: Gabriele Paoloni
> Cc: Jisheng Zhang; jingoohan1 at gmail.com; bhelgaas at google.com; linux-
> pci at vger.kernel.org; linux-kernel at vger.kernel.org; linux-arm-
> kernel at lists.infradead.org
> Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> to dw_pcie_setup_rc()
> 
> Hi Gab,
> 
> Thanks for bringing it.
> 
> 
> On Wed, Apr 6, 2016 at 8:20 PM, Gabriele Paoloni
> <gabriele.paoloni@huawei.com> wrote:
> > Hi, sorry to be late on this
> >
> >> -----Original Message-----
> >> From: linux-kernel-owner at vger.kernel.org [mailto:linux-kernel-
> >> owner at vger.kernel.org] On Behalf Of Jisheng Zhang
> >> Sent: 16 March 2016 11:41
> >> To: jingoohan1 at gmail.com; pratyush.anand at gmail.com;
> bhelgaas at google.com
> >> Cc: linux-pci at vger.kernel.org; linux-kernel at vger.kernel.org; linux-
> arm-
> >> kernel at lists.infradead.org; Jisheng Zhang
> >> Subject: [PATCH v2] PCI: designware: move remaining rc setup code to
> >> dw_pcie_setup_rc()
> >>
> >> dw_pcie_setup_rc(), as its name indicates, setups the RC. But
> current
> >> dw_pcie_host_init() also contains some necessary rc setup code.
> >>
> >> Another reason: the host may lost power during suspend to ram, the
> RC
> >> need to be re-setup after resume. The rc can't be correctly resumed
> >> without the rc setup code in dw_pcie_host_init().
> >>
> >> So this patch moves the code to dw_pcie_setup_rc() to address the
> above
> >> two issues. After this patch, each pcie designware driver users
> could
> >> call dw_pcie_setup_rc() to re-setup rc when resume back.
> >
> > I think this patch breaks the Hisilicon driver...
> >
> > Our driver performs linkup setup in UEFI therefore we do not call
> > dw_pcie_setup_rc(), we only call dw_pcie_host_init().
> >
> > Maybe better to group the part of code to be moved in as separate
> > function...
> >
> > Thanks and sorry for late reply.
> >
> 
> I am just wondering, should n't then what ever we do in
> dw_pcie_setup_rc() be done in your boot loader and not just link up.

Currently the HiSilicon driver does not call dw_pcie_setup_rc() at all;
so everything is done in dw_pcie_setup_rc() is done in bootloader.

I guess your question is if we can execute in bootloader the part of
code the this patch has moved to in "dw_pcie_setup_rc()". I think the
problem here is that even if it was possible we would break backward
compatibility with previous bootloaders...

Thanks

Gab 


> 
> ~Pratyush

RE: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Gabriele Paoloni]

Hi Jisheng

Thanks for your reply

> -----Original Message-----
> From: Jisheng Zhang [mailto:jszhang@marvell.com]
> Sent: 07 April 2016 03:38
> To: Gabriele Paoloni; jingoohan1@gmail.com; pratyush.anand@gmail.com;
> bhelgaas@google.com
> Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org
> Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> to dw_pcie_setup_rc()
> 
> Hi Gabriele,
> 
> On Wed, 6 Apr 2016 14:50:29 +0000 Gabriele Paoloni wrote:
> 
> > Hi, sorry to be late on this
> >
> > > -----Original Message-----
> > > From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-
> > > owner@vger.kernel.org] On Behalf Of Jisheng Zhang
> > > Sent: 16 March 2016 11:41
> > > To: jingoohan1@gmail.com; pratyush.anand@gmail.com;
> bhelgaas@google.com
> > > Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; linux-
> arm-
> > > kernel@lists.infradead.org; Jisheng Zhang
> > > Subject: [PATCH v2] PCI: designware: move remaining rc setup code
> to
> > > dw_pcie_setup_rc()
> > >
> > > dw_pcie_setup_rc(), as its name indicates, setups the RC. But
> current
> > > dw_pcie_host_init() also contains some necessary rc setup code.
> > >
> > > Another reason: the host may lost power during suspend to ram, the
> RC
> > > need to be re-setup after resume. The rc can't be correctly resumed
> > > without the rc setup code in dw_pcie_host_init().
> > >
> > > So this patch moves the code to dw_pcie_setup_rc() to address the
> above
> > > two issues. After this patch, each pcie designware driver users
> could
> > > call dw_pcie_setup_rc() to re-setup rc when resume back.
> >
> > I think this patch breaks the Hisilicon driver...
> >
> > Our driver performs linkup setup in UEFI therefore we do not call
> > dw_pcie_setup_rc(), we only call dw_pcie_host_init().
> 
> Thanks for the information. So pcie-hisi rely on UEFI to do something
> similar
> in dw_pcie_setup_rc(), this comes to a common driver implement
> question: should
> linux device driver rely on bootloader to configure HW device?

I don't see any issue with this...

> 
> Is it acceptable that pcie-hisi adds a call to dw_pcie_setup_rc() in
> hisi_add_pcie_port()?

I don't think so...that would try to overwrite what is already set by
the bootloader; so it is wrong in principle and maybe it can lead to
undefined behaviours...

Thanks

Gab

> 
> Thanks,
> Jisheng
> 
> >
> > Maybe better to group the part of code to be moved in as separate
> > function...
> >
> > Thanks and sorry for late reply.
> >
> > Gab
> >
> >

RE: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Gabriele Paoloni]

Hi Jisheng

Thanks for your reply

> -----Original Message-----
> From: Jisheng Zhang [mailto:jszhang@marvell.com]
> Sent: 07 April 2016 03:38
> To: Gabriele Paoloni; jingoohan1@gmail.com; pratyush.anand@gmail.com;
> bhelgaas@google.com
> Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org
> Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> to dw_pcie_setup_rc()
> 
> Hi Gabriele,
> 
> On Wed, 6 Apr 2016 14:50:29 +0000 Gabriele Paoloni wrote:
> 
> > Hi, sorry to be late on this
> >
> > > -----Original Message-----
> > > From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-
> > > owner@vger.kernel.org] On Behalf Of Jisheng Zhang
> > > Sent: 16 March 2016 11:41
> > > To: jingoohan1@gmail.com; pratyush.anand@gmail.com;
> bhelgaas@google.com
> > > Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; linux-
> arm-
> > > kernel@lists.infradead.org; Jisheng Zhang
> > > Subject: [PATCH v2] PCI: designware: move remaining rc setup code
> to
> > > dw_pcie_setup_rc()
> > >
> > > dw_pcie_setup_rc(), as its name indicates, setups the RC. But
> current
> > > dw_pcie_host_init() also contains some necessary rc setup code.
> > >
> > > Another reason: the host may lost power during suspend to ram, the
> RC
> > > need to be re-setup after resume. The rc can't be correctly resumed
> > > without the rc setup code in dw_pcie_host_init().
> > >
> > > So this patch moves the code to dw_pcie_setup_rc() to address the
> above
> > > two issues. After this patch, each pcie designware driver users
> could
> > > call dw_pcie_setup_rc() to re-setup rc when resume back.
> >
> > I think this patch breaks the Hisilicon driver...
> >
> > Our driver performs linkup setup in UEFI therefore we do not call
> > dw_pcie_setup_rc(), we only call dw_pcie_host_init().
> 
> Thanks for the information. So pcie-hisi rely on UEFI to do something
> similar
> in dw_pcie_setup_rc(), this comes to a common driver implement
> question: should
> linux device driver rely on bootloader to configure HW device?

I don't see any issue with this...

> 
> Is it acceptable that pcie-hisi adds a call to dw_pcie_setup_rc() in
> hisi_add_pcie_port()?

I don't think so...that would try to overwrite what is already set by
the bootloader; so it is wrong in principle and maybe it can lead to
undefined behaviours...

Thanks

Gab

> 
> Thanks,
> Jisheng
> 
> >
> > Maybe better to group the part of code to be moved in as separate
> > function...
> >
> > Thanks and sorry for late reply.
> >
> > Gab
> >
> >

[PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Gabriele Paoloni]

Hi Jisheng

Thanks for your reply

> -----Original Message-----
> From: Jisheng Zhang [mailto:jszhang at marvell.com]
> Sent: 07 April 2016 03:38
> To: Gabriele Paoloni; jingoohan1 at gmail.com; pratyush.anand at gmail.com;
> bhelgaas at google.com
> Cc: linux-pci at vger.kernel.org; linux-kernel at vger.kernel.org; linux-arm-
> kernel at lists.infradead.org
> Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> to dw_pcie_setup_rc()
> 
> Hi Gabriele,
> 
> On Wed, 6 Apr 2016 14:50:29 +0000 Gabriele Paoloni wrote:
> 
> > Hi, sorry to be late on this
> >
> > > -----Original Message-----
> > > From: linux-kernel-owner at vger.kernel.org [mailto:linux-kernel-
> > > owner at vger.kernel.org] On Behalf Of Jisheng Zhang
> > > Sent: 16 March 2016 11:41
> > > To: jingoohan1 at gmail.com; pratyush.anand at gmail.com;
> bhelgaas at google.com
> > > Cc: linux-pci at vger.kernel.org; linux-kernel at vger.kernel.org; linux-
> arm-
> > > kernel at lists.infradead.org; Jisheng Zhang
> > > Subject: [PATCH v2] PCI: designware: move remaining rc setup code
> to
> > > dw_pcie_setup_rc()
> > >
> > > dw_pcie_setup_rc(), as its name indicates, setups the RC. But
> current
> > > dw_pcie_host_init() also contains some necessary rc setup code.
> > >
> > > Another reason: the host may lost power during suspend to ram, the
> RC
> > > need to be re-setup after resume. The rc can't be correctly resumed
> > > without the rc setup code in dw_pcie_host_init().
> > >
> > > So this patch moves the code to dw_pcie_setup_rc() to address the
> above
> > > two issues. After this patch, each pcie designware driver users
> could
> > > call dw_pcie_setup_rc() to re-setup rc when resume back.
> >
> > I think this patch breaks the Hisilicon driver...
> >
> > Our driver performs linkup setup in UEFI therefore we do not call
> > dw_pcie_setup_rc(), we only call dw_pcie_host_init().
> 
> Thanks for the information. So pcie-hisi rely on UEFI to do something
> similar
> in dw_pcie_setup_rc(), this comes to a common driver implement
> question: should
> linux device driver rely on bootloader to configure HW device?

I don't see any issue with this...

> 
> Is it acceptable that pcie-hisi adds a call to dw_pcie_setup_rc() in
> hisi_add_pcie_port()?

I don't think so...that would try to overwrite what is already set by
the bootloader; so it is wrong in principle and maybe it can lead to
undefined behaviours...

Thanks

Gab

> 
> Thanks,
> Jisheng
> 
> >
> > Maybe better to group the part of code to be moved in as separate
> > function...
> >
> > Thanks and sorry for late reply.
> >
> > Gab
> >
> >

Re: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Jisheng Zhang]

Hi Gabriele,

On Thu, 7 Apr 2016 08:20:28 +0000 Gabriele Paoloni wrote:

> Hi Jisheng
> 
> Thanks for your reply
> 
> > -----Original Message-----
> > From: Jisheng Zhang [mailto:jszhang@marvell.com]
> > Sent: 07 April 2016 03:38
> > To: Gabriele Paoloni; jingoohan1@gmail.com; pratyush.anand@gmail.com;
> > bhelgaas@google.com
> > Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; linux-arm-
> > kernel@lists.infradead.org
> > Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> > to dw_pcie_setup_rc()
> > 
> > Hi Gabriele,
> > 
> > On Wed, 6 Apr 2016 14:50:29 +0000 Gabriele Paoloni wrote:
> >   
> > > Hi, sorry to be late on this
> > >  
> > > > -----Original Message-----
> > > > From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-
> > > > owner@vger.kernel.org] On Behalf Of Jisheng Zhang
> > > > Sent: 16 March 2016 11:41
> > > > To: jingoohan1@gmail.com; pratyush.anand@gmail.com;  
> > bhelgaas@google.com  
> > > > Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; linux-  
> > arm-  
> > > > kernel@lists.infradead.org; Jisheng Zhang
> > > > Subject: [PATCH v2] PCI: designware: move remaining rc setup code  
> > to  
> > > > dw_pcie_setup_rc()
> > > >
> > > > dw_pcie_setup_rc(), as its name indicates, setups the RC. But  
> > current  
> > > > dw_pcie_host_init() also contains some necessary rc setup code.
> > > >
> > > > Another reason: the host may lost power during suspend to ram, the  
> > RC  
> > > > need to be re-setup after resume. The rc can't be correctly resumed
> > > > without the rc setup code in dw_pcie_host_init().
> > > >
> > > > So this patch moves the code to dw_pcie_setup_rc() to address the  
> > above  
> > > > two issues. After this patch, each pcie designware driver users  
> > could  
> > > > call dw_pcie_setup_rc() to re-setup rc when resume back.  
> > >
> > > I think this patch breaks the Hisilicon driver...
> > >
> > > Our driver performs linkup setup in UEFI therefore we do not call
> > > dw_pcie_setup_rc(), we only call dw_pcie_host_init().  
> > 
> > Thanks for the information. So pcie-hisi rely on UEFI to do something
> > similar
> > in dw_pcie_setup_rc(), this comes to a common driver implement
> > question: should
> > linux device driver rely on bootloader to configure HW device?  
> 
> I don't see any issue with this...
> 
> > 
> > Is it acceptable that pcie-hisi adds a call to dw_pcie_setup_rc() in
> > hisi_add_pcie_port()?  
> 
> I don't think so...that would try to overwrite what is already set by
> the bootloader; so it is wrong in principle and maybe it can lead to
> undefined behaviours...

make sense! This commit is intend to re-setup the rc when waken from s2ram (in
s2ram state, the host lost power)

I have no good solution but to introduce one function e.g 
dw_pcie_setup_rc_after_linkup(), then move related code from dw_pcie_host_init
to it, then let my host driver resume hook to call.

Hi Pratyush, Jingoo and Bjorn etc.

any suggestions are appreciated!

Thanks,
Jisheng

Re: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Jisheng Zhang]

Hi Gabriele,

On Thu, 7 Apr 2016 08:20:28 +0000 Gabriele Paoloni wrote:

> Hi Jisheng
> 
> Thanks for your reply
> 
> > -----Original Message-----
> > From: Jisheng Zhang [mailto:jszhang@marvell.com]
> > Sent: 07 April 2016 03:38
> > To: Gabriele Paoloni; jingoohan1@gmail.com; pratyush.anand@gmail.com;
> > bhelgaas@google.com
> > Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; linux-arm-
> > kernel@lists.infradead.org
> > Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> > to dw_pcie_setup_rc()
> > 
> > Hi Gabriele,
> > 
> > On Wed, 6 Apr 2016 14:50:29 +0000 Gabriele Paoloni wrote:
> >   
> > > Hi, sorry to be late on this
> > >  
> > > > -----Original Message-----
> > > > From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-
> > > > owner@vger.kernel.org] On Behalf Of Jisheng Zhang
> > > > Sent: 16 March 2016 11:41
> > > > To: jingoohan1@gmail.com; pratyush.anand@gmail.com;  
> > bhelgaas@google.com  
> > > > Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; linux-  
> > arm-  
> > > > kernel@lists.infradead.org; Jisheng Zhang
> > > > Subject: [PATCH v2] PCI: designware: move remaining rc setup code  
> > to  
> > > > dw_pcie_setup_rc()
> > > >
> > > > dw_pcie_setup_rc(), as its name indicates, setups the RC. But  
> > current  
> > > > dw_pcie_host_init() also contains some necessary rc setup code.
> > > >
> > > > Another reason: the host may lost power during suspend to ram, the  
> > RC  
> > > > need to be re-setup after resume. The rc can't be correctly resumed
> > > > without the rc setup code in dw_pcie_host_init().
> > > >
> > > > So this patch moves the code to dw_pcie_setup_rc() to address the  
> > above  
> > > > two issues. After this patch, each pcie designware driver users  
> > could  
> > > > call dw_pcie_setup_rc() to re-setup rc when resume back.  
> > >
> > > I think this patch breaks the Hisilicon driver...
> > >
> > > Our driver performs linkup setup in UEFI therefore we do not call
> > > dw_pcie_setup_rc(), we only call dw_pcie_host_init().  
> > 
> > Thanks for the information. So pcie-hisi rely on UEFI to do something
> > similar
> > in dw_pcie_setup_rc(), this comes to a common driver implement
> > question: should
> > linux device driver rely on bootloader to configure HW device?  
> 
> I don't see any issue with this...
> 
> > 
> > Is it acceptable that pcie-hisi adds a call to dw_pcie_setup_rc() in
> > hisi_add_pcie_port()?  
> 
> I don't think so...that would try to overwrite what is already set by
> the bootloader; so it is wrong in principle and maybe it can lead to
> undefined behaviours...

make sense! This commit is intend to re-setup the rc when waken from s2ram (in
s2ram state, the host lost power)

I have no good solution but to introduce one function e.g 
dw_pcie_setup_rc_after_linkup(), then move related code from dw_pcie_host_init
to it, then let my host driver resume hook to call.

Hi Pratyush, Jingoo and Bjorn etc.

any suggestions are appreciated!

Thanks,
Jisheng

[PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Jisheng Zhang]

Hi Gabriele,

On Thu, 7 Apr 2016 08:20:28 +0000 Gabriele Paoloni wrote:

> Hi Jisheng
> 
> Thanks for your reply
> 
> > -----Original Message-----
> > From: Jisheng Zhang [mailto:jszhang at marvell.com]
> > Sent: 07 April 2016 03:38
> > To: Gabriele Paoloni; jingoohan1 at gmail.com; pratyush.anand at gmail.com;
> > bhelgaas at google.com
> > Cc: linux-pci at vger.kernel.org; linux-kernel at vger.kernel.org; linux-arm-
> > kernel at lists.infradead.org
> > Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> > to dw_pcie_setup_rc()
> > 
> > Hi Gabriele,
> > 
> > On Wed, 6 Apr 2016 14:50:29 +0000 Gabriele Paoloni wrote:
> >   
> > > Hi, sorry to be late on this
> > >  
> > > > -----Original Message-----
> > > > From: linux-kernel-owner at vger.kernel.org [mailto:linux-kernel-
> > > > owner at vger.kernel.org] On Behalf Of Jisheng Zhang
> > > > Sent: 16 March 2016 11:41
> > > > To: jingoohan1 at gmail.com; pratyush.anand at gmail.com;  
> > bhelgaas at google.com  
> > > > Cc: linux-pci at vger.kernel.org; linux-kernel at vger.kernel.org; linux-  
> > arm-  
> > > > kernel at lists.infradead.org; Jisheng Zhang
> > > > Subject: [PATCH v2] PCI: designware: move remaining rc setup code  
> > to  
> > > > dw_pcie_setup_rc()
> > > >
> > > > dw_pcie_setup_rc(), as its name indicates, setups the RC. But  
> > current  
> > > > dw_pcie_host_init() also contains some necessary rc setup code.
> > > >
> > > > Another reason: the host may lost power during suspend to ram, the  
> > RC  
> > > > need to be re-setup after resume. The rc can't be correctly resumed
> > > > without the rc setup code in dw_pcie_host_init().
> > > >
> > > > So this patch moves the code to dw_pcie_setup_rc() to address the  
> > above  
> > > > two issues. After this patch, each pcie designware driver users  
> > could  
> > > > call dw_pcie_setup_rc() to re-setup rc when resume back.  
> > >
> > > I think this patch breaks the Hisilicon driver...
> > >
> > > Our driver performs linkup setup in UEFI therefore we do not call
> > > dw_pcie_setup_rc(), we only call dw_pcie_host_init().  
> > 
> > Thanks for the information. So pcie-hisi rely on UEFI to do something
> > similar
> > in dw_pcie_setup_rc(), this comes to a common driver implement
> > question: should
> > linux device driver rely on bootloader to configure HW device?  
> 
> I don't see any issue with this...
> 
> > 
> > Is it acceptable that pcie-hisi adds a call to dw_pcie_setup_rc() in
> > hisi_add_pcie_port()?  
> 
> I don't think so...that would try to overwrite what is already set by
> the bootloader; so it is wrong in principle and maybe it can lead to
> undefined behaviours...

make sense! This commit is intend to re-setup the rc when waken from s2ram (in
s2ram state, the host lost power)

I have no good solution but to introduce one function e.g 
dw_pcie_setup_rc_after_linkup(), then move related code from dw_pcie_host_init
to it, then let my host driver resume hook to call.

Hi Pratyush, Jingoo and Bjorn etc.

any suggestions are appreciated!

Thanks,
Jisheng

RE: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Gabriele Paoloni]

Hi Jisheng

> -----Original Message-----
> From: Jisheng Zhang [mailto:jszhang@marvell.com]
> Sent: 07 April 2016 09:35
> To: Gabriele Paoloni
> Cc: jingoohan1@gmail.com; pratyush.anand@gmail.com;
> bhelgaas@google.com; linux-pci@vger.kernel.org; linux-
> kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
> Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> to dw_pcie_setup_rc()
> 
> Hi Gabriele,
> 
> On Thu, 7 Apr 2016 08:20:28 +0000 Gabriele Paoloni wrote:
> 
> > Hi Jisheng
> >
> > Thanks for your reply
> >
> > > -----Original Message-----
> > > From: Jisheng Zhang [mailto:jszhang@marvell.com]
> > > Sent: 07 April 2016 03:38
> > > To: Gabriele Paoloni; jingoohan1@gmail.com;
> pratyush.anand@gmail.com;
> > > bhelgaas@google.com
> > > Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; linux-
> arm-
> > > kernel@lists.infradead.org
> > > Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup
> code
> > > to dw_pcie_setup_rc()
> > >
> > > Hi Gabriele,
> > >
> > > On Wed, 6 Apr 2016 14:50:29 +0000 Gabriele Paoloni wrote:
> > >
> > > > Hi, sorry to be late on this
> > > >
> > > > > -----Original Message-----
> > > > > From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-
> > > > > owner@vger.kernel.org] On Behalf Of Jisheng Zhang
> > > > > Sent: 16 March 2016 11:41
> > > > > To: jingoohan1@gmail.com; pratyush.anand@gmail.com;
> > > bhelgaas@google.com
> > > > > Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-
> > > arm-
> > > > > kernel@lists.infradead.org; Jisheng Zhang
> > > > > Subject: [PATCH v2] PCI: designware: move remaining rc setup
> code
> > > to
> > > > > dw_pcie_setup_rc()
> > > > >
> > > > > dw_pcie_setup_rc(), as its name indicates, setups the RC. But
> > > current
> > > > > dw_pcie_host_init() also contains some necessary rc setup code.
> > > > >
> > > > > Another reason: the host may lost power during suspend to ram,
> the
> > > RC
> > > > > need to be re-setup after resume. The rc can't be correctly
> resumed
> > > > > without the rc setup code in dw_pcie_host_init().
> > > > >
> > > > > So this patch moves the code to dw_pcie_setup_rc() to address
> the
> > > above
> > > > > two issues. After this patch, each pcie designware driver users
> > > could
> > > > > call dw_pcie_setup_rc() to re-setup rc when resume back.
> > > >
> > > > I think this patch breaks the Hisilicon driver...
> > > >
> > > > Our driver performs linkup setup in UEFI therefore we do not call
> > > > dw_pcie_setup_rc(), we only call dw_pcie_host_init().
> > >
> > > Thanks for the information. So pcie-hisi rely on UEFI to do
> something
> > > similar
> > > in dw_pcie_setup_rc(), this comes to a common driver implement
> > > question: should
> > > linux device driver rely on bootloader to configure HW device?
> >
> > I don't see any issue with this...
> >
> > >
> > > Is it acceptable that pcie-hisi adds a call to dw_pcie_setup_rc()
> in
> > > hisi_add_pcie_port()?
> >
> > I don't think so...that would try to overwrite what is already set by
> > the bootloader; so it is wrong in principle and maybe it can lead to
> > undefined behaviours...
> 
> make sense! This commit is intend to re-setup the rc when waken from
> s2ram (in
> s2ram state, the host lost power)
> 
> I have no good solution but to introduce one function e.g
> dw_pcie_setup_rc_after_linkup(), then move related code from
> dw_pcie_host_init
> to it, then let my host driver resume hook to call.
> 
> Hi Pratyush, Jingoo and Bjorn etc.
> 
> any suggestions are appreciated!

What about:

diff --git a/drivers/pci/host/pcie-designware.c b/drivers/pci/host/pcie-designware.c
index a4cccd3..e461f5d 100644
--- a/drivers/pci/host/pcie-designware.c
+++ b/drivers/pci/host/pcie-designware.c
@@ -434,7 +434,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
 	struct platform_device *pdev = to_platform_device(pp->dev);
 	struct pci_bus *bus, *child;
 	struct resource *cfg_res;
-	u32 val;
 	int i, ret;
 	LIST_HEAD(res);
 	struct resource_entry *win;
@@ -544,25 +543,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
 	if (pp->ops->host_init)
 		pp->ops->host_init(pp);
 
-	/*
-	 * If the platform provides ->rd_other_conf, it means the platform
-	 * uses its own address translation component rather than ATU, so
-	 * we should not program the ATU here.
-	 */
-	if (!pp->ops->rd_other_conf)
-		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
-					  PCIE_ATU_TYPE_MEM, pp->mem_base,
-					  pp->mem_bus_addr, pp->mem_size);
-
-	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
-
-	/* program correct class for RC */
-	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
-
-	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
-	val |= PORT_LOGIC_SPEED_CHANGE;
-	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
-
 	pp->root_bus_nr = pp->busn->start;
 	if (IS_ENABLED(CONFIG_PCI_MSI)) {
 		bus = pci_scan_root_bus_msi(pp->dev, pp->root_bus_nr,
@@ -725,6 +705,29 @@ static struct pci_ops dw_pcie_ops = {
 	.write = dw_pcie_wr_conf,
 };
 
+void dw_pcie_setup_own_cfg(struct pcie_port *pp)
+{
+	u32 val;
+	/*
+	 * If the platform provides ->rd_other_conf, it means the platform
+	 * uses its own address translation component rather than ATU, so
+	 * we should not program the ATU here.
+	 */
+	if (!pp->ops->rd_other_conf)
+		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
+					  PCIE_ATU_TYPE_MEM, pp->mem_base,
+					  pp->mem_bus_addr, pp->mem_size);
+
+	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
+
+	/* program correct class for RC */
+	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
+
+	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
+	val |= PORT_LOGIC_SPEED_CHANGE;
+	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
+}
+
 void dw_pcie_setup_rc(struct pcie_port *pp)
 {
 	u32 val;
@@ -800,6 +803,8 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
 	val |= PCI_COMMAND_IO | PCI_COMMAND_MEMORY |
 		PCI_COMMAND_MASTER | PCI_COMMAND_SERR;
 	dw_pcie_writel_rc(pp, val, PCI_COMMAND);
+
+	dw_pcie_setup_own_cfg(pp);
 }
 
 MODULE_AUTHOR("Jingoo Han <jg1.han@samsung.com>");
diff --git a/drivers/pci/host/pcie-designware.h b/drivers/pci/host/pcie-designware.h
index f437f9b..caf0f5d 100644
--- a/drivers/pci/host/pcie-designware.h
+++ b/drivers/pci/host/pcie-designware.h
@@ -85,5 +85,6 @@ int dw_pcie_wait_for_link(struct pcie_port *pp);
 int dw_pcie_link_up(struct pcie_port *pp);
 void dw_pcie_setup_rc(struct pcie_port *pp);
 int dw_pcie_host_init(struct pcie_port *pp);
+void dw_pcie_setup_own_cfg(struct pcie_port *pp);
 
 #endif /* _PCIE_DESIGNWARE_H */
diff --git a/drivers/pci/host/pcie-hisi.c b/drivers/pci/host/pcie-hisi.c
index 3e98d4e..8da29b2 100644
--- a/drivers/pci/host/pcie-hisi.c
+++ b/drivers/pci/host/pcie-hisi.c
@@ -164,6 +164,7 @@ static int hisi_add_pcie_port(struct pcie_port *pp,
 		dev_err(&pdev->dev, "failed to initialize host\n");
 		return ret;
 	}
+	dw_pcie_setup_own_cfg(pp);
 
 	return 0;
 }

> 
> Thanks,
> Jisheng

RE: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Gabriele Paoloni]

Hi Jisheng

> -----Original Message-----
> From: Jisheng Zhang [mailto:jszhang@marvell.com]
> Sent: 07 April 2016 09:35
> To: Gabriele Paoloni
> Cc: jingoohan1@gmail.com; pratyush.anand@gmail.com;
> bhelgaas@google.com; linux-pci@vger.kernel.org; linux-
> kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
> Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> to dw_pcie_setup_rc()
> 
> Hi Gabriele,
> 
> On Thu, 7 Apr 2016 08:20:28 +0000 Gabriele Paoloni wrote:
> 
> > Hi Jisheng
> >
> > Thanks for your reply
> >
> > > -----Original Message-----
> > > From: Jisheng Zhang [mailto:jszhang@marvell.com]
> > > Sent: 07 April 2016 03:38
> > > To: Gabriele Paoloni; jingoohan1@gmail.com;
> pratyush.anand@gmail.com;
> > > bhelgaas@google.com
> > > Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; linux-
> arm-
> > > kernel@lists.infradead.org
> > > Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup
> code
> > > to dw_pcie_setup_rc()
> > >
> > > Hi Gabriele,
> > >
> > > On Wed, 6 Apr 2016 14:50:29 +0000 Gabriele Paoloni wrote:
> > >
> > > > Hi, sorry to be late on this
> > > >
> > > > > -----Original Message-----
> > > > > From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-
> > > > > owner@vger.kernel.org] On Behalf Of Jisheng Zhang
> > > > > Sent: 16 March 2016 11:41
> > > > > To: jingoohan1@gmail.com; pratyush.anand@gmail.com;
> > > bhelgaas@google.com
> > > > > Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-
> > > arm-
> > > > > kernel@lists.infradead.org; Jisheng Zhang
> > > > > Subject: [PATCH v2] PCI: designware: move remaining rc setup
> code
> > > to
> > > > > dw_pcie_setup_rc()
> > > > >
> > > > > dw_pcie_setup_rc(), as its name indicates, setups the RC. But
> > > current
> > > > > dw_pcie_host_init() also contains some necessary rc setup code.
> > > > >
> > > > > Another reason: the host may lost power during suspend to ram,
> the
> > > RC
> > > > > need to be re-setup after resume. The rc can't be correctly
> resumed
> > > > > without the rc setup code in dw_pcie_host_init().
> > > > >
> > > > > So this patch moves the code to dw_pcie_setup_rc() to address
> the
> > > above
> > > > > two issues. After this patch, each pcie designware driver users
> > > could
> > > > > call dw_pcie_setup_rc() to re-setup rc when resume back.
> > > >
> > > > I think this patch breaks the Hisilicon driver...
> > > >
> > > > Our driver performs linkup setup in UEFI therefore we do not call
> > > > dw_pcie_setup_rc(), we only call dw_pcie_host_init().
> > >
> > > Thanks for the information. So pcie-hisi rely on UEFI to do
> something
> > > similar
> > > in dw_pcie_setup_rc(), this comes to a common driver implement
> > > question: should
> > > linux device driver rely on bootloader to configure HW device?
> >
> > I don't see any issue with this...
> >
> > >
> > > Is it acceptable that pcie-hisi adds a call to dw_pcie_setup_rc()
> in
> > > hisi_add_pcie_port()?
> >
> > I don't think so...that would try to overwrite what is already set by
> > the bootloader; so it is wrong in principle and maybe it can lead to
> > undefined behaviours...
> 
> make sense! This commit is intend to re-setup the rc when waken from
> s2ram (in
> s2ram state, the host lost power)
> 
> I have no good solution but to introduce one function e.g
> dw_pcie_setup_rc_after_linkup(), then move related code from
> dw_pcie_host_init
> to it, then let my host driver resume hook to call.
> 
> Hi Pratyush, Jingoo and Bjorn etc.
> 
> any suggestions are appreciated!

What about:

diff --git a/drivers/pci/host/pcie-designware.c b/drivers/pci/host/pcie-designware.c
index a4cccd3..e461f5d 100644
--- a/drivers/pci/host/pcie-designware.c
+++ b/drivers/pci/host/pcie-designware.c
@@ -434,7 +434,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
 	struct platform_device *pdev = to_platform_device(pp->dev);
 	struct pci_bus *bus, *child;
 	struct resource *cfg_res;
-	u32 val;
 	int i, ret;
 	LIST_HEAD(res);
 	struct resource_entry *win;
@@ -544,25 +543,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
 	if (pp->ops->host_init)
 		pp->ops->host_init(pp);
 
-	/*
-	 * If the platform provides ->rd_other_conf, it means the platform
-	 * uses its own address translation component rather than ATU, so
-	 * we should not program the ATU here.
-	 */
-	if (!pp->ops->rd_other_conf)
-		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
-					  PCIE_ATU_TYPE_MEM, pp->mem_base,
-					  pp->mem_bus_addr, pp->mem_size);
-
-	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
-
-	/* program correct class for RC */
-	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
-
-	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
-	val |= PORT_LOGIC_SPEED_CHANGE;
-	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
-
 	pp->root_bus_nr = pp->busn->start;
 	if (IS_ENABLED(CONFIG_PCI_MSI)) {
 		bus = pci_scan_root_bus_msi(pp->dev, pp->root_bus_nr,
@@ -725,6 +705,29 @@ static struct pci_ops dw_pcie_ops = {
 	.write = dw_pcie_wr_conf,
 };
 
+void dw_pcie_setup_own_cfg(struct pcie_port *pp)
+{
+	u32 val;
+	/*
+	 * If the platform provides ->rd_other_conf, it means the platform
+	 * uses its own address translation component rather than ATU, so
+	 * we should not program the ATU here.
+	 */
+	if (!pp->ops->rd_other_conf)
+		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
+					  PCIE_ATU_TYPE_MEM, pp->mem_base,
+					  pp->mem_bus_addr, pp->mem_size);
+
+	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
+
+	/* program correct class for RC */
+	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
+
+	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
+	val |= PORT_LOGIC_SPEED_CHANGE;
+	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
+}
+
 void dw_pcie_setup_rc(struct pcie_port *pp)
 {
 	u32 val;
@@ -800,6 +803,8 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
 	val |= PCI_COMMAND_IO | PCI_COMMAND_MEMORY |
 		PCI_COMMAND_MASTER | PCI_COMMAND_SERR;
 	dw_pcie_writel_rc(pp, val, PCI_COMMAND);
+
+	dw_pcie_setup_own_cfg(pp);
 }
 
 MODULE_AUTHOR("Jingoo Han <jg1.han@samsung.com>");
diff --git a/drivers/pci/host/pcie-designware.h b/drivers/pci/host/pcie-designware.h
index f437f9b..caf0f5d 100644
--- a/drivers/pci/host/pcie-designware.h
+++ b/drivers/pci/host/pcie-designware.h
@@ -85,5 +85,6 @@ int dw_pcie_wait_for_link(struct pcie_port *pp);
 int dw_pcie_link_up(struct pcie_port *pp);
 void dw_pcie_setup_rc(struct pcie_port *pp);
 int dw_pcie_host_init(struct pcie_port *pp);
+void dw_pcie_setup_own_cfg(struct pcie_port *pp);
 
 #endif /* _PCIE_DESIGNWARE_H */
diff --git a/drivers/pci/host/pcie-hisi.c b/drivers/pci/host/pcie-hisi.c
index 3e98d4e..8da29b2 100644
--- a/drivers/pci/host/pcie-hisi.c
+++ b/drivers/pci/host/pcie-hisi.c
@@ -164,6 +164,7 @@ static int hisi_add_pcie_port(struct pcie_port *pp,
 		dev_err(&pdev->dev, "failed to initialize host\n");
 		return ret;
 	}
+	dw_pcie_setup_own_cfg(pp);
 
 	return 0;
 }

> 
> Thanks,
> Jisheng

[PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Gabriele Paoloni]

Hi Jisheng

> -----Original Message-----
> From: Jisheng Zhang [mailto:jszhang at marvell.com]
> Sent: 07 April 2016 09:35
> To: Gabriele Paoloni
> Cc: jingoohan1 at gmail.com; pratyush.anand at gmail.com;
> bhelgaas at google.com; linux-pci at vger.kernel.org; linux-
> kernel at vger.kernel.org; linux-arm-kernel at lists.infradead.org
> Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> to dw_pcie_setup_rc()
> 
> Hi Gabriele,
> 
> On Thu, 7 Apr 2016 08:20:28 +0000 Gabriele Paoloni wrote:
> 
> > Hi Jisheng
> >
> > Thanks for your reply
> >
> > > -----Original Message-----
> > > From: Jisheng Zhang [mailto:jszhang at marvell.com]
> > > Sent: 07 April 2016 03:38
> > > To: Gabriele Paoloni; jingoohan1 at gmail.com;
> pratyush.anand at gmail.com;
> > > bhelgaas at google.com
> > > Cc: linux-pci at vger.kernel.org; linux-kernel at vger.kernel.org; linux-
> arm-
> > > kernel at lists.infradead.org
> > > Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup
> code
> > > to dw_pcie_setup_rc()
> > >
> > > Hi Gabriele,
> > >
> > > On Wed, 6 Apr 2016 14:50:29 +0000 Gabriele Paoloni wrote:
> > >
> > > > Hi, sorry to be late on this
> > > >
> > > > > -----Original Message-----
> > > > > From: linux-kernel-owner at vger.kernel.org [mailto:linux-kernel-
> > > > > owner at vger.kernel.org] On Behalf Of Jisheng Zhang
> > > > > Sent: 16 March 2016 11:41
> > > > > To: jingoohan1 at gmail.com; pratyush.anand at gmail.com;
> > > bhelgaas at google.com
> > > > > Cc: linux-pci at vger.kernel.org; linux-kernel at vger.kernel.org;
> linux-
> > > arm-
> > > > > kernel at lists.infradead.org; Jisheng Zhang
> > > > > Subject: [PATCH v2] PCI: designware: move remaining rc setup
> code
> > > to
> > > > > dw_pcie_setup_rc()
> > > > >
> > > > > dw_pcie_setup_rc(), as its name indicates, setups the RC. But
> > > current
> > > > > dw_pcie_host_init() also contains some necessary rc setup code.
> > > > >
> > > > > Another reason: the host may lost power during suspend to ram,
> the
> > > RC
> > > > > need to be re-setup after resume. The rc can't be correctly
> resumed
> > > > > without the rc setup code in dw_pcie_host_init().
> > > > >
> > > > > So this patch moves the code to dw_pcie_setup_rc() to address
> the
> > > above
> > > > > two issues. After this patch, each pcie designware driver users
> > > could
> > > > > call dw_pcie_setup_rc() to re-setup rc when resume back.
> > > >
> > > > I think this patch breaks the Hisilicon driver...
> > > >
> > > > Our driver performs linkup setup in UEFI therefore we do not call
> > > > dw_pcie_setup_rc(), we only call dw_pcie_host_init().
> > >
> > > Thanks for the information. So pcie-hisi rely on UEFI to do
> something
> > > similar
> > > in dw_pcie_setup_rc(), this comes to a common driver implement
> > > question: should
> > > linux device driver rely on bootloader to configure HW device?
> >
> > I don't see any issue with this...
> >
> > >
> > > Is it acceptable that pcie-hisi adds a call to dw_pcie_setup_rc()
> in
> > > hisi_add_pcie_port()?
> >
> > I don't think so...that would try to overwrite what is already set by
> > the bootloader; so it is wrong in principle and maybe it can lead to
> > undefined behaviours...
> 
> make sense! This commit is intend to re-setup the rc when waken from
> s2ram (in
> s2ram state, the host lost power)
> 
> I have no good solution but to introduce one function e.g
> dw_pcie_setup_rc_after_linkup(), then move related code from
> dw_pcie_host_init
> to it, then let my host driver resume hook to call.
> 
> Hi Pratyush, Jingoo and Bjorn etc.
> 
> any suggestions are appreciated!

What about:

diff --git a/drivers/pci/host/pcie-designware.c b/drivers/pci/host/pcie-designware.c
index a4cccd3..e461f5d 100644
--- a/drivers/pci/host/pcie-designware.c
+++ b/drivers/pci/host/pcie-designware.c
@@ -434,7 +434,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
 	struct platform_device *pdev = to_platform_device(pp->dev);
 	struct pci_bus *bus, *child;
 	struct resource *cfg_res;
-	u32 val;
 	int i, ret;
 	LIST_HEAD(res);
 	struct resource_entry *win;
@@ -544,25 +543,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
 	if (pp->ops->host_init)
 		pp->ops->host_init(pp);
 
-	/*
-	 * If the platform provides ->rd_other_conf, it means the platform
-	 * uses its own address translation component rather than ATU, so
-	 * we should not program the ATU here.
-	 */
-	if (!pp->ops->rd_other_conf)
-		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
-					  PCIE_ATU_TYPE_MEM, pp->mem_base,
-					  pp->mem_bus_addr, pp->mem_size);
-
-	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
-
-	/* program correct class for RC */
-	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
-
-	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
-	val |= PORT_LOGIC_SPEED_CHANGE;
-	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
-
 	pp->root_bus_nr = pp->busn->start;
 	if (IS_ENABLED(CONFIG_PCI_MSI)) {
 		bus = pci_scan_root_bus_msi(pp->dev, pp->root_bus_nr,
@@ -725,6 +705,29 @@ static struct pci_ops dw_pcie_ops = {
 	.write = dw_pcie_wr_conf,
 };
 
+void dw_pcie_setup_own_cfg(struct pcie_port *pp)
+{
+	u32 val;
+	/*
+	 * If the platform provides ->rd_other_conf, it means the platform
+	 * uses its own address translation component rather than ATU, so
+	 * we should not program the ATU here.
+	 */
+	if (!pp->ops->rd_other_conf)
+		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
+					  PCIE_ATU_TYPE_MEM, pp->mem_base,
+					  pp->mem_bus_addr, pp->mem_size);
+
+	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
+
+	/* program correct class for RC */
+	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
+
+	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
+	val |= PORT_LOGIC_SPEED_CHANGE;
+	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
+}
+
 void dw_pcie_setup_rc(struct pcie_port *pp)
 {
 	u32 val;
@@ -800,6 +803,8 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
 	val |= PCI_COMMAND_IO | PCI_COMMAND_MEMORY |
 		PCI_COMMAND_MASTER | PCI_COMMAND_SERR;
 	dw_pcie_writel_rc(pp, val, PCI_COMMAND);
+
+	dw_pcie_setup_own_cfg(pp);
 }
 
 MODULE_AUTHOR("Jingoo Han <jg1.han@samsung.com>");
diff --git a/drivers/pci/host/pcie-designware.h b/drivers/pci/host/pcie-designware.h
index f437f9b..caf0f5d 100644
--- a/drivers/pci/host/pcie-designware.h
+++ b/drivers/pci/host/pcie-designware.h
@@ -85,5 +85,6 @@ int dw_pcie_wait_for_link(struct pcie_port *pp);
 int dw_pcie_link_up(struct pcie_port *pp);
 void dw_pcie_setup_rc(struct pcie_port *pp);
 int dw_pcie_host_init(struct pcie_port *pp);
+void dw_pcie_setup_own_cfg(struct pcie_port *pp);
 
 #endif /* _PCIE_DESIGNWARE_H */
diff --git a/drivers/pci/host/pcie-hisi.c b/drivers/pci/host/pcie-hisi.c
index 3e98d4e..8da29b2 100644
--- a/drivers/pci/host/pcie-hisi.c
+++ b/drivers/pci/host/pcie-hisi.c
@@ -164,6 +164,7 @@ static int hisi_add_pcie_port(struct pcie_port *pp,
 		dev_err(&pdev->dev, "failed to initialize host\n");
 		return ret;
 	}
+	dw_pcie_setup_own_cfg(pp);
 
 	return 0;
 }

> 
> Thanks,
> Jisheng

Re: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Jisheng Zhang]

On Thu, 7 Apr 2016 10:06:45 +0000 Gabriele Paoloni wrote:

> Hi Jisheng
> 
..
> > >  
> > > >
> > > > Is it acceptable that pcie-hisi adds a call to dw_pcie_setup_rc()  
> > in  
> > > > hisi_add_pcie_port()?  
> > >
> > > I don't think so...that would try to overwrite what is already set by
> > > the bootloader; so it is wrong in principle and maybe it can lead to
> > > undefined behaviours...  
> > 
> > make sense! This commit is intend to re-setup the rc when waken from
> > s2ram (in
> > s2ram state, the host lost power)
> > 
> > I have no good solution but to introduce one function e.g
> > dw_pcie_setup_rc_after_linkup(), then move related code from
> > dw_pcie_host_init
> > to it, then let my host driver resume hook to call.
> > 
> > Hi Pratyush, Jingoo and Bjorn etc.
> > 
> > any suggestions are appreciated!  
> 
> What about:
> 
> diff --git a/drivers/pci/host/pcie-designware.c b/drivers/pci/host/pcie-designware.c
> index a4cccd3..e461f5d 100644
> --- a/drivers/pci/host/pcie-designware.c
> +++ b/drivers/pci/host/pcie-designware.c
> @@ -434,7 +434,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
>  	struct platform_device *pdev = to_platform_device(pp->dev);
>  	struct pci_bus *bus, *child;
>  	struct resource *cfg_res;
> -	u32 val;
>  	int i, ret;
>  	LIST_HEAD(res);
>  	struct resource_entry *win;
> @@ -544,25 +543,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
>  	if (pp->ops->host_init)
>  		pp->ops->host_init(pp);
>  
> -	/*
> -	 * If the platform provides ->rd_other_conf, it means the platform
> -	 * uses its own address translation component rather than ATU, so
> -	 * we should not program the ATU here.
> -	 */
> -	if (!pp->ops->rd_other_conf)
> -		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> -					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> -					  pp->mem_bus_addr, pp->mem_size);
> -
> -	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> -
> -	/* program correct class for RC */
> -	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
> -
> -	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> -	val |= PORT_LOGIC_SPEED_CHANGE;
> -	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);

If we introduced one global function, is it better to keep the RC registers
program sequence not changed at all? And ask each pcie dw users to call this
new function in its resume hook if necessary. Either is fine, and can provide
what I want, let's wait maintainers' comments.

Something like:


diff --git a/drivers/pci/host/pcie-designware.c b/drivers/pci/host/pcie-designware.c
index a4cccd3..53c6176 100644
--- a/drivers/pci/host/pcie-designware.c
+++ b/drivers/pci/host/pcie-designware.c
@@ -434,7 +434,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
 	struct platform_device *pdev = to_platform_device(pp->dev);
 	struct pci_bus *bus, *child;
 	struct resource *cfg_res;
-	u32 val;
 	int i, ret;
 	LIST_HEAD(res);
 	struct resource_entry *win;
@@ -544,24 +543,7 @@ int dw_pcie_host_init(struct pcie_port *pp)
 	if (pp->ops->host_init)
 		pp->ops->host_init(pp);
 
-	/*
-	 * If the platform provides ->rd_other_conf, it means the platform
-	 * uses its own address translation component rather than ATU, so
-	 * we should not program the ATU here.
-	 */
-	if (!pp->ops->rd_other_conf)
-		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
-					  PCIE_ATU_TYPE_MEM, pp->mem_base,
-					  pp->mem_bus_addr, pp->mem_size);
-
-	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
-
-	/* program correct class for RC */
-	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
-
-	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
-	val |= PORT_LOGIC_SPEED_CHANGE;
-	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
+	dw_pcie_setup_rc_after_linkup(pp);
 
 	pp->root_bus_nr = pp->busn->start;
 	if (IS_ENABLED(CONFIG_PCI_MSI)) {
@@ -725,6 +707,30 @@ static struct pci_ops dw_pcie_ops = {
 	.write = dw_pcie_wr_conf,
 };
 
+void dw_pcie_setup_rc_after_linkup(struct pcie_port *pp)
+{
+	u32 val;
+
+	/*
+	 * If the platform provides ->rd_other_conf, it means the platform
+	 * uses its own address translation component rather than ATU, so
+	 * we should not program the ATU here.
+	 */
+	if (!pp->ops->rd_other_conf)
+		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
+					  PCIE_ATU_TYPE_MEM, pp->mem_base,
+					  pp->mem_bus_addr, pp->mem_size);
+
+	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
+
+	/* program correct class for RC */
+	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
+
+	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
+	val |= PORT_LOGIC_SPEED_CHANGE;
+	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
+}
+
 void dw_pcie_setup_rc(struct pcie_port *pp)
 {
 	u32 val;
diff --git a/drivers/pci/host/pcie-designware.h b/drivers/pci/host/pcie-designware.h
index f437f9b..f7746bf 100644
--- a/drivers/pci/host/pcie-designware.h
+++ b/drivers/pci/host/pcie-designware.h
@@ -85,5 +85,6 @@ int dw_pcie_wait_for_link(struct pcie_port *pp);
 int dw_pcie_link_up(struct pcie_port *pp);
 void dw_pcie_setup_rc(struct pcie_port *pp);
 int dw_pcie_host_init(struct pcie_port *pp);
+void dw_pcie_setup_rc_after_linkup(struct pcie_port *pp);
 
 #endif /* _PCIE_DESIGNWARE_H */

Re: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Jisheng Zhang]

On Thu, 7 Apr 2016 10:06:45 +0000 Gabriele Paoloni wrote:

> Hi Jisheng
> 
..
> > >  
> > > >
> > > > Is it acceptable that pcie-hisi adds a call to dw_pcie_setup_rc()  
> > in  
> > > > hisi_add_pcie_port()?  
> > >
> > > I don't think so...that would try to overwrite what is already set by
> > > the bootloader; so it is wrong in principle and maybe it can lead to
> > > undefined behaviours...  
> > 
> > make sense! This commit is intend to re-setup the rc when waken from
> > s2ram (in
> > s2ram state, the host lost power)
> > 
> > I have no good solution but to introduce one function e.g
> > dw_pcie_setup_rc_after_linkup(), then move related code from
> > dw_pcie_host_init
> > to it, then let my host driver resume hook to call.
> > 
> > Hi Pratyush, Jingoo and Bjorn etc.
> > 
> > any suggestions are appreciated!  
> 
> What about:
> 
> diff --git a/drivers/pci/host/pcie-designware.c b/drivers/pci/host/pcie-designware.c
> index a4cccd3..e461f5d 100644
> --- a/drivers/pci/host/pcie-designware.c
> +++ b/drivers/pci/host/pcie-designware.c
> @@ -434,7 +434,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
>  	struct platform_device *pdev = to_platform_device(pp->dev);
>  	struct pci_bus *bus, *child;
>  	struct resource *cfg_res;
> -	u32 val;
>  	int i, ret;
>  	LIST_HEAD(res);
>  	struct resource_entry *win;
> @@ -544,25 +543,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
>  	if (pp->ops->host_init)
>  		pp->ops->host_init(pp);
>  
> -	/*
> -	 * If the platform provides ->rd_other_conf, it means the platform
> -	 * uses its own address translation component rather than ATU, so
> -	 * we should not program the ATU here.
> -	 */
> -	if (!pp->ops->rd_other_conf)
> -		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> -					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> -					  pp->mem_bus_addr, pp->mem_size);
> -
> -	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> -
> -	/* program correct class for RC */
> -	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
> -
> -	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> -	val |= PORT_LOGIC_SPEED_CHANGE;
> -	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);

If we introduced one global function, is it better to keep the RC registers
program sequence not changed at all? And ask each pcie dw users to call this
new function in its resume hook if necessary. Either is fine, and can provide
what I want, let's wait maintainers' comments.

Something like:


diff --git a/drivers/pci/host/pcie-designware.c b/drivers/pci/host/pcie-designware.c
index a4cccd3..53c6176 100644
--- a/drivers/pci/host/pcie-designware.c
+++ b/drivers/pci/host/pcie-designware.c
@@ -434,7 +434,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
 	struct platform_device *pdev = to_platform_device(pp->dev);
 	struct pci_bus *bus, *child;
 	struct resource *cfg_res;
-	u32 val;
 	int i, ret;
 	LIST_HEAD(res);
 	struct resource_entry *win;
@@ -544,24 +543,7 @@ int dw_pcie_host_init(struct pcie_port *pp)
 	if (pp->ops->host_init)
 		pp->ops->host_init(pp);
 
-	/*
-	 * If the platform provides ->rd_other_conf, it means the platform
-	 * uses its own address translation component rather than ATU, so
-	 * we should not program the ATU here.
-	 */
-	if (!pp->ops->rd_other_conf)
-		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
-					  PCIE_ATU_TYPE_MEM, pp->mem_base,
-					  pp->mem_bus_addr, pp->mem_size);
-
-	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
-
-	/* program correct class for RC */
-	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
-
-	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
-	val |= PORT_LOGIC_SPEED_CHANGE;
-	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
+	dw_pcie_setup_rc_after_linkup(pp);
 
 	pp->root_bus_nr = pp->busn->start;
 	if (IS_ENABLED(CONFIG_PCI_MSI)) {
@@ -725,6 +707,30 @@ static struct pci_ops dw_pcie_ops = {
 	.write = dw_pcie_wr_conf,
 };
 
+void dw_pcie_setup_rc_after_linkup(struct pcie_port *pp)
+{
+	u32 val;
+
+	/*
+	 * If the platform provides ->rd_other_conf, it means the platform
+	 * uses its own address translation component rather than ATU, so
+	 * we should not program the ATU here.
+	 */
+	if (!pp->ops->rd_other_conf)
+		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
+					  PCIE_ATU_TYPE_MEM, pp->mem_base,
+					  pp->mem_bus_addr, pp->mem_size);
+
+	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
+
+	/* program correct class for RC */
+	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
+
+	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
+	val |= PORT_LOGIC_SPEED_CHANGE;
+	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
+}
+
 void dw_pcie_setup_rc(struct pcie_port *pp)
 {
 	u32 val;
diff --git a/drivers/pci/host/pcie-designware.h b/drivers/pci/host/pcie-designware.h
index f437f9b..f7746bf 100644
--- a/drivers/pci/host/pcie-designware.h
+++ b/drivers/pci/host/pcie-designware.h
@@ -85,5 +85,6 @@ int dw_pcie_wait_for_link(struct pcie_port *pp);
 int dw_pcie_link_up(struct pcie_port *pp);
 void dw_pcie_setup_rc(struct pcie_port *pp);
 int dw_pcie_host_init(struct pcie_port *pp);
+void dw_pcie_setup_rc_after_linkup(struct pcie_port *pp);
 
 #endif /* _PCIE_DESIGNWARE_H */

[PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Jisheng Zhang]

On Thu, 7 Apr 2016 10:06:45 +0000 Gabriele Paoloni wrote:

> Hi Jisheng
> 
..
> > >  
> > > >
> > > > Is it acceptable that pcie-hisi adds a call to dw_pcie_setup_rc()  
> > in  
> > > > hisi_add_pcie_port()?  
> > >
> > > I don't think so...that would try to overwrite what is already set by
> > > the bootloader; so it is wrong in principle and maybe it can lead to
> > > undefined behaviours...  
> > 
> > make sense! This commit is intend to re-setup the rc when waken from
> > s2ram (in
> > s2ram state, the host lost power)
> > 
> > I have no good solution but to introduce one function e.g
> > dw_pcie_setup_rc_after_linkup(), then move related code from
> > dw_pcie_host_init
> > to it, then let my host driver resume hook to call.
> > 
> > Hi Pratyush, Jingoo and Bjorn etc.
> > 
> > any suggestions are appreciated!  
> 
> What about:
> 
> diff --git a/drivers/pci/host/pcie-designware.c b/drivers/pci/host/pcie-designware.c
> index a4cccd3..e461f5d 100644
> --- a/drivers/pci/host/pcie-designware.c
> +++ b/drivers/pci/host/pcie-designware.c
> @@ -434,7 +434,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
>  	struct platform_device *pdev = to_platform_device(pp->dev);
>  	struct pci_bus *bus, *child;
>  	struct resource *cfg_res;
> -	u32 val;
>  	int i, ret;
>  	LIST_HEAD(res);
>  	struct resource_entry *win;
> @@ -544,25 +543,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
>  	if (pp->ops->host_init)
>  		pp->ops->host_init(pp);
>  
> -	/*
> -	 * If the platform provides ->rd_other_conf, it means the platform
> -	 * uses its own address translation component rather than ATU, so
> -	 * we should not program the ATU here.
> -	 */
> -	if (!pp->ops->rd_other_conf)
> -		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> -					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> -					  pp->mem_bus_addr, pp->mem_size);
> -
> -	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> -
> -	/* program correct class for RC */
> -	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
> -
> -	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> -	val |= PORT_LOGIC_SPEED_CHANGE;
> -	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);

If we introduced one global function, is it better to keep the RC registers
program sequence not changed at all? And ask each pcie dw users to call this
new function in its resume hook if necessary. Either is fine, and can provide
what I want, let's wait maintainers' comments.

Something like:


diff --git a/drivers/pci/host/pcie-designware.c b/drivers/pci/host/pcie-designware.c
index a4cccd3..53c6176 100644
--- a/drivers/pci/host/pcie-designware.c
+++ b/drivers/pci/host/pcie-designware.c
@@ -434,7 +434,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
 	struct platform_device *pdev = to_platform_device(pp->dev);
 	struct pci_bus *bus, *child;
 	struct resource *cfg_res;
-	u32 val;
 	int i, ret;
 	LIST_HEAD(res);
 	struct resource_entry *win;
@@ -544,24 +543,7 @@ int dw_pcie_host_init(struct pcie_port *pp)
 	if (pp->ops->host_init)
 		pp->ops->host_init(pp);
 
-	/*
-	 * If the platform provides ->rd_other_conf, it means the platform
-	 * uses its own address translation component rather than ATU, so
-	 * we should not program the ATU here.
-	 */
-	if (!pp->ops->rd_other_conf)
-		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
-					  PCIE_ATU_TYPE_MEM, pp->mem_base,
-					  pp->mem_bus_addr, pp->mem_size);
-
-	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
-
-	/* program correct class for RC */
-	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
-
-	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
-	val |= PORT_LOGIC_SPEED_CHANGE;
-	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
+	dw_pcie_setup_rc_after_linkup(pp);
 
 	pp->root_bus_nr = pp->busn->start;
 	if (IS_ENABLED(CONFIG_PCI_MSI)) {
@@ -725,6 +707,30 @@ static struct pci_ops dw_pcie_ops = {
 	.write = dw_pcie_wr_conf,
 };
 
+void dw_pcie_setup_rc_after_linkup(struct pcie_port *pp)
+{
+	u32 val;
+
+	/*
+	 * If the platform provides ->rd_other_conf, it means the platform
+	 * uses its own address translation component rather than ATU, so
+	 * we should not program the ATU here.
+	 */
+	if (!pp->ops->rd_other_conf)
+		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
+					  PCIE_ATU_TYPE_MEM, pp->mem_base,
+					  pp->mem_bus_addr, pp->mem_size);
+
+	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
+
+	/* program correct class for RC */
+	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
+
+	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
+	val |= PORT_LOGIC_SPEED_CHANGE;
+	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
+}
+
 void dw_pcie_setup_rc(struct pcie_port *pp)
 {
 	u32 val;
diff --git a/drivers/pci/host/pcie-designware.h b/drivers/pci/host/pcie-designware.h
index f437f9b..f7746bf 100644
--- a/drivers/pci/host/pcie-designware.h
+++ b/drivers/pci/host/pcie-designware.h
@@ -85,5 +85,6 @@ int dw_pcie_wait_for_link(struct pcie_port *pp);
 int dw_pcie_link_up(struct pcie_port *pp);
 void dw_pcie_setup_rc(struct pcie_port *pp);
 int dw_pcie_host_init(struct pcie_port *pp);
+void dw_pcie_setup_rc_after_linkup(struct pcie_port *pp);
 
 #endif /* _PCIE_DESIGNWARE_H */

Re: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Bjorn Helgaas]

On Thu, Apr 07, 2016 at 10:06:45AM +0000, Gabriele Paoloni wrote:
> Hi Jisheng
> 
> > -----Original Message-----
> > From: Jisheng Zhang [mailto:jszhang@marvell.com]
> > Sent: 07 April 2016 09:35
> > To: Gabriele Paoloni
> > Cc: jingoohan1@gmail.com; pratyush.anand@gmail.com;
> > bhelgaas@google.com; linux-pci@vger.kernel.org; linux-
> > kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
> > Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> > to dw_pcie_setup_rc()
> > 
> > Hi Gabriele,
> > 
> > On Thu, 7 Apr 2016 08:20:28 +0000 Gabriele Paoloni wrote:
> > 
> > > Hi Jisheng
> > >
> > > Thanks for your reply
> > >
> > > > -----Original Message-----
> > > > From: Jisheng Zhang [mailto:jszhang@marvell.com]
> > > > Sent: 07 April 2016 03:38
> > > > To: Gabriele Paoloni; jingoohan1@gmail.com;
> > pratyush.anand@gmail.com;
> > > > bhelgaas@google.com
> > > > Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; linux-
> > arm-
> > > > kernel@lists.infradead.org
> > > > Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup
> > code
> > > > to dw_pcie_setup_rc()
> > > >
> > > > Hi Gabriele,
> > > >
> > > > On Wed, 6 Apr 2016 14:50:29 +0000 Gabriele Paoloni wrote:
> > > >
> > > > > Hi, sorry to be late on this
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-
> > > > > > owner@vger.kernel.org] On Behalf Of Jisheng Zhang
> > > > > > Sent: 16 March 2016 11:41
> > > > > > To: jingoohan1@gmail.com; pratyush.anand@gmail.com;
> > > > bhelgaas@google.com
> > > > > > Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org;
> > linux-
> > > > arm-
> > > > > > kernel@lists.infradead.org; Jisheng Zhang
> > > > > > Subject: [PATCH v2] PCI: designware: move remaining rc setup
> > code
> > > > to
> > > > > > dw_pcie_setup_rc()
> > > > > >
> > > > > > dw_pcie_setup_rc(), as its name indicates, setups the RC. But
> > > > current
> > > > > > dw_pcie_host_init() also contains some necessary rc setup code.
> > > > > >
> > > > > > Another reason: the host may lost power during suspend to ram,
> > the
> > > > RC
> > > > > > need to be re-setup after resume. The rc can't be correctly
> > resumed
> > > > > > without the rc setup code in dw_pcie_host_init().
> > > > > >
> > > > > > So this patch moves the code to dw_pcie_setup_rc() to address
> > the
> > > > above
> > > > > > two issues. After this patch, each pcie designware driver users
> > > > could
> > > > > > call dw_pcie_setup_rc() to re-setup rc when resume back.
> > > > >
> > > > > I think this patch breaks the Hisilicon driver...
> > > > >
> > > > > Our driver performs linkup setup in UEFI therefore we do not call
> > > > > dw_pcie_setup_rc(), we only call dw_pcie_host_init().
> > > >
> > > > Thanks for the information. So pcie-hisi rely on UEFI to do
> > something
> > > > similar
> > > > in dw_pcie_setup_rc(), this comes to a common driver implement
> > > > question: should
> > > > linux device driver rely on bootloader to configure HW device?
> > >
> > > I don't see any issue with this...
> > >
> > > >
> > > > Is it acceptable that pcie-hisi adds a call to dw_pcie_setup_rc()
> > in
> > > > hisi_add_pcie_port()?
> > >
> > > I don't think so...that would try to overwrite what is already set by
> > > the bootloader; so it is wrong in principle and maybe it can lead to
> > > undefined behaviours...
> > 
> > make sense! This commit is intend to re-setup the rc when waken from
> > s2ram (in
> > s2ram state, the host lost power)
> > 
> > I have no good solution but to introduce one function e.g
> > dw_pcie_setup_rc_after_linkup(), then move related code from
> > dw_pcie_host_init
> > to it, then let my host driver resume hook to call.
> > 
> > Hi Pratyush, Jingoo and Bjorn etc.
> > 
> > any suggestions are appreciated!
> 
> What about:
> 
> diff --git a/drivers/pci/host/pcie-designware.c b/drivers/pci/host/pcie-designware.c
> index a4cccd3..e461f5d 100644
> --- a/drivers/pci/host/pcie-designware.c
> +++ b/drivers/pci/host/pcie-designware.c
> @@ -434,7 +434,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
>  	struct platform_device *pdev = to_platform_device(pp->dev);
>  	struct pci_bus *bus, *child;
>  	struct resource *cfg_res;
> -	u32 val;
>  	int i, ret;
>  	LIST_HEAD(res);
>  	struct resource_entry *win;
> @@ -544,25 +543,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
>  	if (pp->ops->host_init)
>  		pp->ops->host_init(pp);
>  
> -	/*
> -	 * If the platform provides ->rd_other_conf, it means the platform
> -	 * uses its own address translation component rather than ATU, so
> -	 * we should not program the ATU here.
> -	 */
> -	if (!pp->ops->rd_other_conf)
> -		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> -					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> -					  pp->mem_bus_addr, pp->mem_size);
> -
> -	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> -
> -	/* program correct class for RC */
> -	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
> -
> -	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> -	val |= PORT_LOGIC_SPEED_CHANGE;
> -	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
> -
>  	pp->root_bus_nr = pp->busn->start;
>  	if (IS_ENABLED(CONFIG_PCI_MSI)) {
>  		bus = pci_scan_root_bus_msi(pp->dev, pp->root_bus_nr,
> @@ -725,6 +705,29 @@ static struct pci_ops dw_pcie_ops = {
>  	.write = dw_pcie_wr_conf,
>  };
>  
> +void dw_pcie_setup_own_cfg(struct pcie_port *pp)
> +{
> +	u32 val;
> +	/*
> +	 * If the platform provides ->rd_other_conf, it means the platform
> +	 * uses its own address translation component rather than ATU, so
> +	 * we should not program the ATU here.
> +	 */
> +	if (!pp->ops->rd_other_conf)
> +		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> +					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> +					  pp->mem_bus_addr, pp->mem_size);
> +
> +	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> +
> +	/* program correct class for RC */
> +	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
> +
> +	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> +	val |= PORT_LOGIC_SPEED_CHANGE;
> +	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
> +}
> +
>  void dw_pcie_setup_rc(struct pcie_port *pp)
>  {
>  	u32 val;
> @@ -800,6 +803,8 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
>  	val |= PCI_COMMAND_IO | PCI_COMMAND_MEMORY |
>  		PCI_COMMAND_MASTER | PCI_COMMAND_SERR;
>  	dw_pcie_writel_rc(pp, val, PCI_COMMAND);
> +
> +	dw_pcie_setup_own_cfg(pp);
>  }
>  
>  MODULE_AUTHOR("Jingoo Han <jg1.han@samsung.com>");
> diff --git a/drivers/pci/host/pcie-designware.h b/drivers/pci/host/pcie-designware.h
> index f437f9b..caf0f5d 100644
> --- a/drivers/pci/host/pcie-designware.h
> +++ b/drivers/pci/host/pcie-designware.h
> @@ -85,5 +85,6 @@ int dw_pcie_wait_for_link(struct pcie_port *pp);
>  int dw_pcie_link_up(struct pcie_port *pp);
>  void dw_pcie_setup_rc(struct pcie_port *pp);
>  int dw_pcie_host_init(struct pcie_port *pp);
> +void dw_pcie_setup_own_cfg(struct pcie_port *pp);
>  
>  #endif /* _PCIE_DESIGNWARE_H */
> diff --git a/drivers/pci/host/pcie-hisi.c b/drivers/pci/host/pcie-hisi.c
> index 3e98d4e..8da29b2 100644
> --- a/drivers/pci/host/pcie-hisi.c
> +++ b/drivers/pci/host/pcie-hisi.c
> @@ -164,6 +164,7 @@ static int hisi_add_pcie_port(struct pcie_port *pp,
>  		dev_err(&pdev->dev, "failed to initialize host\n");
>  		return ret;
>  	}
> +	dw_pcie_setup_own_cfg(pp);
>  
>  	return 0;
>  }

What's the hisi plan for resuming after suspend-to-RAM?  How does the
RC get reprogrammed after it loses all its state?

What would break if hisi did call dw_pcie_setup_rc()?  I know you said
it would overwrite what the bootloader already did, which is true.

But hisi does call dw_pcie_host_init(), so it reads pp->mem (which
determines pp->mem_base) and pp->lanes from the DT.  Other drivers
then call dw_pcie_setup_rc() which programs the RC based on
pp->mem_base and pp->lanes.  So hisi assumes UEFI programmed the RC to
match the DT, while the other drivers read the DT and program the RC
to match.  The latter seems more robust because it enforces the
consistency rather than relying on it.

Bjorn

Re: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Bjorn Helgaas]

On Thu, Apr 07, 2016 at 10:06:45AM +0000, Gabriele Paoloni wrote:
> Hi Jisheng
> 
> > -----Original Message-----
> > From: Jisheng Zhang [mailto:jszhang@marvell.com]
> > Sent: 07 April 2016 09:35
> > To: Gabriele Paoloni
> > Cc: jingoohan1@gmail.com; pratyush.anand@gmail.com;
> > bhelgaas@google.com; linux-pci@vger.kernel.org; linux-
> > kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
> > Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> > to dw_pcie_setup_rc()
> > 
> > Hi Gabriele,
> > 
> > On Thu, 7 Apr 2016 08:20:28 +0000 Gabriele Paoloni wrote:
> > 
> > > Hi Jisheng
> > >
> > > Thanks for your reply
> > >
> > > > -----Original Message-----
> > > > From: Jisheng Zhang [mailto:jszhang@marvell.com]
> > > > Sent: 07 April 2016 03:38
> > > > To: Gabriele Paoloni; jingoohan1@gmail.com;
> > pratyush.anand@gmail.com;
> > > > bhelgaas@google.com
> > > > Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; linux-
> > arm-
> > > > kernel@lists.infradead.org
> > > > Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup
> > code
> > > > to dw_pcie_setup_rc()
> > > >
> > > > Hi Gabriele,
> > > >
> > > > On Wed, 6 Apr 2016 14:50:29 +0000 Gabriele Paoloni wrote:
> > > >
> > > > > Hi, sorry to be late on this
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-
> > > > > > owner@vger.kernel.org] On Behalf Of Jisheng Zhang
> > > > > > Sent: 16 March 2016 11:41
> > > > > > To: jingoohan1@gmail.com; pratyush.anand@gmail.com;
> > > > bhelgaas@google.com
> > > > > > Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org;
> > linux-
> > > > arm-
> > > > > > kernel@lists.infradead.org; Jisheng Zhang
> > > > > > Subject: [PATCH v2] PCI: designware: move remaining rc setup
> > code
> > > > to
> > > > > > dw_pcie_setup_rc()
> > > > > >
> > > > > > dw_pcie_setup_rc(), as its name indicates, setups the RC. But
> > > > current
> > > > > > dw_pcie_host_init() also contains some necessary rc setup code.
> > > > > >
> > > > > > Another reason: the host may lost power during suspend to ram,
> > the
> > > > RC
> > > > > > need to be re-setup after resume. The rc can't be correctly
> > resumed
> > > > > > without the rc setup code in dw_pcie_host_init().
> > > > > >
> > > > > > So this patch moves the code to dw_pcie_setup_rc() to address
> > the
> > > > above
> > > > > > two issues. After this patch, each pcie designware driver users
> > > > could
> > > > > > call dw_pcie_setup_rc() to re-setup rc when resume back.
> > > > >
> > > > > I think this patch breaks the Hisilicon driver...
> > > > >
> > > > > Our driver performs linkup setup in UEFI therefore we do not call
> > > > > dw_pcie_setup_rc(), we only call dw_pcie_host_init().
> > > >
> > > > Thanks for the information. So pcie-hisi rely on UEFI to do
> > something
> > > > similar
> > > > in dw_pcie_setup_rc(), this comes to a common driver implement
> > > > question: should
> > > > linux device driver rely on bootloader to configure HW device?
> > >
> > > I don't see any issue with this...
> > >
> > > >
> > > > Is it acceptable that pcie-hisi adds a call to dw_pcie_setup_rc()
> > in
> > > > hisi_add_pcie_port()?
> > >
> > > I don't think so...that would try to overwrite what is already set by
> > > the bootloader; so it is wrong in principle and maybe it can lead to
> > > undefined behaviours...
> > 
> > make sense! This commit is intend to re-setup the rc when waken from
> > s2ram (in
> > s2ram state, the host lost power)
> > 
> > I have no good solution but to introduce one function e.g
> > dw_pcie_setup_rc_after_linkup(), then move related code from
> > dw_pcie_host_init
> > to it, then let my host driver resume hook to call.
> > 
> > Hi Pratyush, Jingoo and Bjorn etc.
> > 
> > any suggestions are appreciated!
> 
> What about:
> 
> diff --git a/drivers/pci/host/pcie-designware.c b/drivers/pci/host/pcie-designware.c
> index a4cccd3..e461f5d 100644
> --- a/drivers/pci/host/pcie-designware.c
> +++ b/drivers/pci/host/pcie-designware.c
> @@ -434,7 +434,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
>  	struct platform_device *pdev = to_platform_device(pp->dev);
>  	struct pci_bus *bus, *child;
>  	struct resource *cfg_res;
> -	u32 val;
>  	int i, ret;
>  	LIST_HEAD(res);
>  	struct resource_entry *win;
> @@ -544,25 +543,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
>  	if (pp->ops->host_init)
>  		pp->ops->host_init(pp);
>  
> -	/*
> -	 * If the platform provides ->rd_other_conf, it means the platform
> -	 * uses its own address translation component rather than ATU, so
> -	 * we should not program the ATU here.
> -	 */
> -	if (!pp->ops->rd_other_conf)
> -		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> -					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> -					  pp->mem_bus_addr, pp->mem_size);
> -
> -	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> -
> -	/* program correct class for RC */
> -	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
> -
> -	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> -	val |= PORT_LOGIC_SPEED_CHANGE;
> -	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
> -
>  	pp->root_bus_nr = pp->busn->start;
>  	if (IS_ENABLED(CONFIG_PCI_MSI)) {
>  		bus = pci_scan_root_bus_msi(pp->dev, pp->root_bus_nr,
> @@ -725,6 +705,29 @@ static struct pci_ops dw_pcie_ops = {
>  	.write = dw_pcie_wr_conf,
>  };
>  
> +void dw_pcie_setup_own_cfg(struct pcie_port *pp)
> +{
> +	u32 val;
> +	/*
> +	 * If the platform provides ->rd_other_conf, it means the platform
> +	 * uses its own address translation component rather than ATU, so
> +	 * we should not program the ATU here.
> +	 */
> +	if (!pp->ops->rd_other_conf)
> +		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> +					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> +					  pp->mem_bus_addr, pp->mem_size);
> +
> +	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> +
> +	/* program correct class for RC */
> +	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
> +
> +	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> +	val |= PORT_LOGIC_SPEED_CHANGE;
> +	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
> +}
> +
>  void dw_pcie_setup_rc(struct pcie_port *pp)
>  {
>  	u32 val;
> @@ -800,6 +803,8 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
>  	val |= PCI_COMMAND_IO | PCI_COMMAND_MEMORY |
>  		PCI_COMMAND_MASTER | PCI_COMMAND_SERR;
>  	dw_pcie_writel_rc(pp, val, PCI_COMMAND);
> +
> +	dw_pcie_setup_own_cfg(pp);
>  }
>  
>  MODULE_AUTHOR("Jingoo Han <jg1.han@samsung.com>");
> diff --git a/drivers/pci/host/pcie-designware.h b/drivers/pci/host/pcie-designware.h
> index f437f9b..caf0f5d 100644
> --- a/drivers/pci/host/pcie-designware.h
> +++ b/drivers/pci/host/pcie-designware.h
> @@ -85,5 +85,6 @@ int dw_pcie_wait_for_link(struct pcie_port *pp);
>  int dw_pcie_link_up(struct pcie_port *pp);
>  void dw_pcie_setup_rc(struct pcie_port *pp);
>  int dw_pcie_host_init(struct pcie_port *pp);
> +void dw_pcie_setup_own_cfg(struct pcie_port *pp);
>  
>  #endif /* _PCIE_DESIGNWARE_H */
> diff --git a/drivers/pci/host/pcie-hisi.c b/drivers/pci/host/pcie-hisi.c
> index 3e98d4e..8da29b2 100644
> --- a/drivers/pci/host/pcie-hisi.c
> +++ b/drivers/pci/host/pcie-hisi.c
> @@ -164,6 +164,7 @@ static int hisi_add_pcie_port(struct pcie_port *pp,
>  		dev_err(&pdev->dev, "failed to initialize host\n");
>  		return ret;
>  	}
> +	dw_pcie_setup_own_cfg(pp);
>  
>  	return 0;
>  }

What's the hisi plan for resuming after suspend-to-RAM?  How does the
RC get reprogrammed after it loses all its state?

What would break if hisi did call dw_pcie_setup_rc()?  I know you said
it would overwrite what the bootloader already did, which is true.

But hisi does call dw_pcie_host_init(), so it reads pp->mem (which
determines pp->mem_base) and pp->lanes from the DT.  Other drivers
then call dw_pcie_setup_rc() which programs the RC based on
pp->mem_base and pp->lanes.  So hisi assumes UEFI programmed the RC to
match the DT, while the other drivers read the DT and program the RC
to match.  The latter seems more robust because it enforces the
consistency rather than relying on it.

Bjorn

[PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Bjorn Helgaas]

On Thu, Apr 07, 2016 at 10:06:45AM +0000, Gabriele Paoloni wrote:
> Hi Jisheng
> 
> > -----Original Message-----
> > From: Jisheng Zhang [mailto:jszhang at marvell.com]
> > Sent: 07 April 2016 09:35
> > To: Gabriele Paoloni
> > Cc: jingoohan1 at gmail.com; pratyush.anand at gmail.com;
> > bhelgaas at google.com; linux-pci at vger.kernel.org; linux-
> > kernel at vger.kernel.org; linux-arm-kernel at lists.infradead.org
> > Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> > to dw_pcie_setup_rc()
> > 
> > Hi Gabriele,
> > 
> > On Thu, 7 Apr 2016 08:20:28 +0000 Gabriele Paoloni wrote:
> > 
> > > Hi Jisheng
> > >
> > > Thanks for your reply
> > >
> > > > -----Original Message-----
> > > > From: Jisheng Zhang [mailto:jszhang at marvell.com]
> > > > Sent: 07 April 2016 03:38
> > > > To: Gabriele Paoloni; jingoohan1 at gmail.com;
> > pratyush.anand at gmail.com;
> > > > bhelgaas at google.com
> > > > Cc: linux-pci at vger.kernel.org; linux-kernel at vger.kernel.org; linux-
> > arm-
> > > > kernel at lists.infradead.org
> > > > Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup
> > code
> > > > to dw_pcie_setup_rc()
> > > >
> > > > Hi Gabriele,
> > > >
> > > > On Wed, 6 Apr 2016 14:50:29 +0000 Gabriele Paoloni wrote:
> > > >
> > > > > Hi, sorry to be late on this
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: linux-kernel-owner at vger.kernel.org [mailto:linux-kernel-
> > > > > > owner at vger.kernel.org] On Behalf Of Jisheng Zhang
> > > > > > Sent: 16 March 2016 11:41
> > > > > > To: jingoohan1 at gmail.com; pratyush.anand at gmail.com;
> > > > bhelgaas at google.com
> > > > > > Cc: linux-pci at vger.kernel.org; linux-kernel at vger.kernel.org;
> > linux-
> > > > arm-
> > > > > > kernel at lists.infradead.org; Jisheng Zhang
> > > > > > Subject: [PATCH v2] PCI: designware: move remaining rc setup
> > code
> > > > to
> > > > > > dw_pcie_setup_rc()
> > > > > >
> > > > > > dw_pcie_setup_rc(), as its name indicates, setups the RC. But
> > > > current
> > > > > > dw_pcie_host_init() also contains some necessary rc setup code.
> > > > > >
> > > > > > Another reason: the host may lost power during suspend to ram,
> > the
> > > > RC
> > > > > > need to be re-setup after resume. The rc can't be correctly
> > resumed
> > > > > > without the rc setup code in dw_pcie_host_init().
> > > > > >
> > > > > > So this patch moves the code to dw_pcie_setup_rc() to address
> > the
> > > > above
> > > > > > two issues. After this patch, each pcie designware driver users
> > > > could
> > > > > > call dw_pcie_setup_rc() to re-setup rc when resume back.
> > > > >
> > > > > I think this patch breaks the Hisilicon driver...
> > > > >
> > > > > Our driver performs linkup setup in UEFI therefore we do not call
> > > > > dw_pcie_setup_rc(), we only call dw_pcie_host_init().
> > > >
> > > > Thanks for the information. So pcie-hisi rely on UEFI to do
> > something
> > > > similar
> > > > in dw_pcie_setup_rc(), this comes to a common driver implement
> > > > question: should
> > > > linux device driver rely on bootloader to configure HW device?
> > >
> > > I don't see any issue with this...
> > >
> > > >
> > > > Is it acceptable that pcie-hisi adds a call to dw_pcie_setup_rc()
> > in
> > > > hisi_add_pcie_port()?
> > >
> > > I don't think so...that would try to overwrite what is already set by
> > > the bootloader; so it is wrong in principle and maybe it can lead to
> > > undefined behaviours...
> > 
> > make sense! This commit is intend to re-setup the rc when waken from
> > s2ram (in
> > s2ram state, the host lost power)
> > 
> > I have no good solution but to introduce one function e.g
> > dw_pcie_setup_rc_after_linkup(), then move related code from
> > dw_pcie_host_init
> > to it, then let my host driver resume hook to call.
> > 
> > Hi Pratyush, Jingoo and Bjorn etc.
> > 
> > any suggestions are appreciated!
> 
> What about:
> 
> diff --git a/drivers/pci/host/pcie-designware.c b/drivers/pci/host/pcie-designware.c
> index a4cccd3..e461f5d 100644
> --- a/drivers/pci/host/pcie-designware.c
> +++ b/drivers/pci/host/pcie-designware.c
> @@ -434,7 +434,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
>  	struct platform_device *pdev = to_platform_device(pp->dev);
>  	struct pci_bus *bus, *child;
>  	struct resource *cfg_res;
> -	u32 val;
>  	int i, ret;
>  	LIST_HEAD(res);
>  	struct resource_entry *win;
> @@ -544,25 +543,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
>  	if (pp->ops->host_init)
>  		pp->ops->host_init(pp);
>  
> -	/*
> -	 * If the platform provides ->rd_other_conf, it means the platform
> -	 * uses its own address translation component rather than ATU, so
> -	 * we should not program the ATU here.
> -	 */
> -	if (!pp->ops->rd_other_conf)
> -		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> -					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> -					  pp->mem_bus_addr, pp->mem_size);
> -
> -	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> -
> -	/* program correct class for RC */
> -	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
> -
> -	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> -	val |= PORT_LOGIC_SPEED_CHANGE;
> -	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
> -
>  	pp->root_bus_nr = pp->busn->start;
>  	if (IS_ENABLED(CONFIG_PCI_MSI)) {
>  		bus = pci_scan_root_bus_msi(pp->dev, pp->root_bus_nr,
> @@ -725,6 +705,29 @@ static struct pci_ops dw_pcie_ops = {
>  	.write = dw_pcie_wr_conf,
>  };
>  
> +void dw_pcie_setup_own_cfg(struct pcie_port *pp)
> +{
> +	u32 val;
> +	/*
> +	 * If the platform provides ->rd_other_conf, it means the platform
> +	 * uses its own address translation component rather than ATU, so
> +	 * we should not program the ATU here.
> +	 */
> +	if (!pp->ops->rd_other_conf)
> +		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> +					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> +					  pp->mem_bus_addr, pp->mem_size);
> +
> +	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> +
> +	/* program correct class for RC */
> +	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2, PCI_CLASS_BRIDGE_PCI);
> +
> +	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> +	val |= PORT_LOGIC_SPEED_CHANGE;
> +	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
> +}
> +
>  void dw_pcie_setup_rc(struct pcie_port *pp)
>  {
>  	u32 val;
> @@ -800,6 +803,8 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
>  	val |= PCI_COMMAND_IO | PCI_COMMAND_MEMORY |
>  		PCI_COMMAND_MASTER | PCI_COMMAND_SERR;
>  	dw_pcie_writel_rc(pp, val, PCI_COMMAND);
> +
> +	dw_pcie_setup_own_cfg(pp);
>  }
>  
>  MODULE_AUTHOR("Jingoo Han <jg1.han@samsung.com>");
> diff --git a/drivers/pci/host/pcie-designware.h b/drivers/pci/host/pcie-designware.h
> index f437f9b..caf0f5d 100644
> --- a/drivers/pci/host/pcie-designware.h
> +++ b/drivers/pci/host/pcie-designware.h
> @@ -85,5 +85,6 @@ int dw_pcie_wait_for_link(struct pcie_port *pp);
>  int dw_pcie_link_up(struct pcie_port *pp);
>  void dw_pcie_setup_rc(struct pcie_port *pp);
>  int dw_pcie_host_init(struct pcie_port *pp);
> +void dw_pcie_setup_own_cfg(struct pcie_port *pp);
>  
>  #endif /* _PCIE_DESIGNWARE_H */
> diff --git a/drivers/pci/host/pcie-hisi.c b/drivers/pci/host/pcie-hisi.c
> index 3e98d4e..8da29b2 100644
> --- a/drivers/pci/host/pcie-hisi.c
> +++ b/drivers/pci/host/pcie-hisi.c
> @@ -164,6 +164,7 @@ static int hisi_add_pcie_port(struct pcie_port *pp,
>  		dev_err(&pdev->dev, "failed to initialize host\n");
>  		return ret;
>  	}
> +	dw_pcie_setup_own_cfg(pp);
>  
>  	return 0;
>  }

What's the hisi plan for resuming after suspend-to-RAM?  How does the
RC get reprogrammed after it loses all its state?

What would break if hisi did call dw_pcie_setup_rc()?  I know you said
it would overwrite what the bootloader already did, which is true.

But hisi does call dw_pcie_host_init(), so it reads pp->mem (which
determines pp->mem_base) and pp->lanes from the DT.  Other drivers
then call dw_pcie_setup_rc() which programs the RC based on
pp->mem_base and pp->lanes.  So hisi assumes UEFI programmed the RC to
match the DT, while the other drivers read the DT and program the RC
to match.  The latter seems more robust because it enforces the
consistency rather than relying on it.

Bjorn

RE: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Gabriele Paoloni]

Hi Bjorn

Many thanks for your reply

> -----Original Message-----
> From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> Sent: 07 April 2016 15:06
> To: Gabriele Paoloni
> Cc: Jisheng Zhang; jingoohan1@gmail.com; pratyush.anand@gmail.com;
> bhelgaas@google.com; linux-pci@vger.kernel.org; linux-
> kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
> Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> to dw_pcie_setup_rc()
> 
> On Thu, Apr 07, 2016 at 10:06:45AM +0000, Gabriele Paoloni wrote:
> > Hi Jisheng
> >
> > > -----Original Message-----
> > > From: Jisheng Zhang [mailto:jszhang@marvell.com]
> > > Sent: 07 April 2016 09:35
> > > To: Gabriele Paoloni
> > > Cc: jingoohan1@gmail.com; pratyush.anand@gmail.com;
> > > bhelgaas@google.com; linux-pci@vger.kernel.org; linux-
> > > kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
> > > Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup
> code
> > > to dw_pcie_setup_rc()
> > >
> > > Hi Gabriele,
> > >
> > > On Thu, 7 Apr 2016 08:20:28 +0000 Gabriele Paoloni wrote:
> > >
> > > > Hi Jisheng
> > > >
> > > > Thanks for your reply
> > > >
> > > > > -----Original Message-----
> > > > > From: Jisheng Zhang [mailto:jszhang@marvell.com]
> > > > > Sent: 07 April 2016 03:38
> > > > > To: Gabriele Paoloni; jingoohan1@gmail.com;
> > > pratyush.anand@gmail.com;
> > > > > bhelgaas@google.com
> > > > > Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-
> > > arm-
> > > > > kernel@lists.infradead.org
> > > > > Subject: Re: [PATCH v2] PCI: designware: move remaining rc
> setup
> > > code
> > > > > to dw_pcie_setup_rc()
> > > > >
> > > > > Hi Gabriele,
> > > > >
> > > > > On Wed, 6 Apr 2016 14:50:29 +0000 Gabriele Paoloni wrote:
> > > > >
> > > > > > Hi, sorry to be late on this
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: linux-kernel-owner@vger.kernel.org [mailto:linux-
> kernel-
> > > > > > > owner@vger.kernel.org] On Behalf Of Jisheng Zhang
> > > > > > > Sent: 16 March 2016 11:41
> > > > > > > To: jingoohan1@gmail.com; pratyush.anand@gmail.com;
> > > > > bhelgaas@google.com
> > > > > > > Cc: linux-pci@vger.kernel.org; linux-
> kernel@vger.kernel.org;
> > > linux-
> > > > > arm-
> > > > > > > kernel@lists.infradead.org; Jisheng Zhang
> > > > > > > Subject: [PATCH v2] PCI: designware: move remaining rc
> setup
> > > code
> > > > > to
> > > > > > > dw_pcie_setup_rc()
> > > > > > >
> > > > > > > dw_pcie_setup_rc(), as its name indicates, setups the RC.
> But
> > > > > current
> > > > > > > dw_pcie_host_init() also contains some necessary rc setup
> code.
> > > > > > >
> > > > > > > Another reason: the host may lost power during suspend to
> ram,
> > > the
> > > > > RC
> > > > > > > need to be re-setup after resume. The rc can't be correctly
> > > resumed
> > > > > > > without the rc setup code in dw_pcie_host_init().
> > > > > > >
> > > > > > > So this patch moves the code to dw_pcie_setup_rc() to
> address
> > > the
> > > > > above
> > > > > > > two issues. After this patch, each pcie designware driver
> users
> > > > > could
> > > > > > > call dw_pcie_setup_rc() to re-setup rc when resume back.
> > > > > >
> > > > > > I think this patch breaks the Hisilicon driver...
> > > > > >
> > > > > > Our driver performs linkup setup in UEFI therefore we do not
> call
> > > > > > dw_pcie_setup_rc(), we only call dw_pcie_host_init().
> > > > >
> > > > > Thanks for the information. So pcie-hisi rely on UEFI to do
> > > something
> > > > > similar
> > > > > in dw_pcie_setup_rc(), this comes to a common driver implement
> > > > > question: should
> > > > > linux device driver rely on bootloader to configure HW device?
> > > >
> > > > I don't see any issue with this...
> > > >
> > > > >
> > > > > Is it acceptable that pcie-hisi adds a call to
> dw_pcie_setup_rc()
> > > in
> > > > > hisi_add_pcie_port()?
> > > >
> > > > I don't think so...that would try to overwrite what is already
> set by
> > > > the bootloader; so it is wrong in principle and maybe it can lead
> to
> > > > undefined behaviours...
> > >
> > > make sense! This commit is intend to re-setup the rc when waken
> from
> > > s2ram (in
> > > s2ram state, the host lost power)
> > >
> > > I have no good solution but to introduce one function e.g
> > > dw_pcie_setup_rc_after_linkup(), then move related code from
> > > dw_pcie_host_init
> > > to it, then let my host driver resume hook to call.
> > >
> > > Hi Pratyush, Jingoo and Bjorn etc.
> > >
> > > any suggestions are appreciated!
> >
> > What about:
> >
> > diff --git a/drivers/pci/host/pcie-designware.c
> b/drivers/pci/host/pcie-designware.c
> > index a4cccd3..e461f5d 100644
> > --- a/drivers/pci/host/pcie-designware.c
> > +++ b/drivers/pci/host/pcie-designware.c
> > @@ -434,7 +434,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
> >  	struct platform_device *pdev = to_platform_device(pp->dev);
> >  	struct pci_bus *bus, *child;
> >  	struct resource *cfg_res;
> > -	u32 val;
> >  	int i, ret;
> >  	LIST_HEAD(res);
> >  	struct resource_entry *win;
> > @@ -544,25 +543,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
> >  	if (pp->ops->host_init)
> >  		pp->ops->host_init(pp);
> >
> > -	/*
> > -	 * If the platform provides ->rd_other_conf, it means the
> platform
> > -	 * uses its own address translation component rather than ATU, so
> > -	 * we should not program the ATU here.
> > -	 */
> > -	if (!pp->ops->rd_other_conf)
> > -		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> > -					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> > -					  pp->mem_bus_addr, pp->mem_size);
> > -
> > -	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> > -
> > -	/* program correct class for RC */
> > -	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2,
> PCI_CLASS_BRIDGE_PCI);
> > -
> > -	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> > -	val |= PORT_LOGIC_SPEED_CHANGE;
> > -	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
> > -
> >  	pp->root_bus_nr = pp->busn->start;
> >  	if (IS_ENABLED(CONFIG_PCI_MSI)) {
> >  		bus = pci_scan_root_bus_msi(pp->dev, pp->root_bus_nr,
> > @@ -725,6 +705,29 @@ static struct pci_ops dw_pcie_ops = {
> >  	.write = dw_pcie_wr_conf,
> >  };
> >
> > +void dw_pcie_setup_own_cfg(struct pcie_port *pp)
> > +{
> > +	u32 val;
> > +	/*
> > +	 * If the platform provides ->rd_other_conf, it means the
> platform
> > +	 * uses its own address translation component rather than ATU, so
> > +	 * we should not program the ATU here.
> > +	 */
> > +	if (!pp->ops->rd_other_conf)
> > +		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> > +					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> > +					  pp->mem_bus_addr, pp->mem_size);
> > +
> > +	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> > +
> > +	/* program correct class for RC */
> > +	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2,
> PCI_CLASS_BRIDGE_PCI);
> > +
> > +	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> > +	val |= PORT_LOGIC_SPEED_CHANGE;
> > +	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
> > +}
> > +
> >  void dw_pcie_setup_rc(struct pcie_port *pp)
> >  {
> >  	u32 val;
> > @@ -800,6 +803,8 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
> >  	val |= PCI_COMMAND_IO | PCI_COMMAND_MEMORY |
> >  		PCI_COMMAND_MASTER | PCI_COMMAND_SERR;
> >  	dw_pcie_writel_rc(pp, val, PCI_COMMAND);
> > +
> > +	dw_pcie_setup_own_cfg(pp);
> >  }
> >
> >  MODULE_AUTHOR("Jingoo Han <jg1.han@samsung.com>");
> > diff --git a/drivers/pci/host/pcie-designware.h
> b/drivers/pci/host/pcie-designware.h
> > index f437f9b..caf0f5d 100644
> > --- a/drivers/pci/host/pcie-designware.h
> > +++ b/drivers/pci/host/pcie-designware.h
> > @@ -85,5 +85,6 @@ int dw_pcie_wait_for_link(struct pcie_port *pp);
> >  int dw_pcie_link_up(struct pcie_port *pp);
> >  void dw_pcie_setup_rc(struct pcie_port *pp);
> >  int dw_pcie_host_init(struct pcie_port *pp);
> > +void dw_pcie_setup_own_cfg(struct pcie_port *pp);
> >
> >  #endif /* _PCIE_DESIGNWARE_H */
> > diff --git a/drivers/pci/host/pcie-hisi.c b/drivers/pci/host/pcie-
> hisi.c
> > index 3e98d4e..8da29b2 100644
> > --- a/drivers/pci/host/pcie-hisi.c
> > +++ b/drivers/pci/host/pcie-hisi.c
> > @@ -164,6 +164,7 @@ static int hisi_add_pcie_port(struct pcie_port
> *pp,
> >  		dev_err(&pdev->dev, "failed to initialize host\n");
> >  		return ret;
> >  	}
> > +	dw_pcie_setup_own_cfg(pp);
> >
> >  	return 0;
> >  }
> 
> What's the hisi plan for resuming after suspend-to-RAM?  How does the
> RC get reprogrammed after it loses all its state?

PM is not part of the driver yet. This is planned for near
future release so haven't made such considerations yet

> 
> What would break if hisi did call dw_pcie_setup_rc()?  I know you said
> it would overwrite what the bootloader already did, which is true.

I am try to figure this out now with our HW team.

> 
> But hisi does call dw_pcie_host_init(), so it reads pp->mem (which
> determines pp->mem_base) and pp->lanes from the DT.  Other drivers
> then call dw_pcie_setup_rc() which programs the RC based on
> pp->mem_base and pp->lanes.  So hisi assumes UEFI programmed the RC to
> match the DT, while the other drivers read the DT and program the RC
> to match.  The latter seems more robust because it enforces the
> consistency rather than relying on it.

Yes I agree with you, however we have preferred to move RC config to
BIOS to have a single driver to support multiple versions of the
same SoC.

The patch I proposed above does the same job as the original patch
proposed by Jisheng and also allows hisi driver to call the moved
code.

Do you see anything wrong with it?

Thanks

Gab

> 
> Bjorn

RE: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Gabriele Paoloni]

Hi Bjorn

Many thanks for your reply

> -----Original Message-----
> From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> Sent: 07 April 2016 15:06
> To: Gabriele Paoloni
> Cc: Jisheng Zhang; jingoohan1@gmail.com; pratyush.anand@gmail.com;
> bhelgaas@google.com; linux-pci@vger.kernel.org; linux-
> kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
> Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> to dw_pcie_setup_rc()
> 
> On Thu, Apr 07, 2016 at 10:06:45AM +0000, Gabriele Paoloni wrote:
> > Hi Jisheng
> >
> > > -----Original Message-----
> > > From: Jisheng Zhang [mailto:jszhang@marvell.com]
> > > Sent: 07 April 2016 09:35
> > > To: Gabriele Paoloni
> > > Cc: jingoohan1@gmail.com; pratyush.anand@gmail.com;
> > > bhelgaas@google.com; linux-pci@vger.kernel.org; linux-
> > > kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
> > > Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup
> code
> > > to dw_pcie_setup_rc()
> > >
> > > Hi Gabriele,
> > >
> > > On Thu, 7 Apr 2016 08:20:28 +0000 Gabriele Paoloni wrote:
> > >
> > > > Hi Jisheng
> > > >
> > > > Thanks for your reply
> > > >
> > > > > -----Original Message-----
> > > > > From: Jisheng Zhang [mailto:jszhang@marvell.com]
> > > > > Sent: 07 April 2016 03:38
> > > > > To: Gabriele Paoloni; jingoohan1@gmail.com;
> > > pratyush.anand@gmail.com;
> > > > > bhelgaas@google.com
> > > > > Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-
> > > arm-
> > > > > kernel@lists.infradead.org
> > > > > Subject: Re: [PATCH v2] PCI: designware: move remaining rc
> setup
> > > code
> > > > > to dw_pcie_setup_rc()
> > > > >
> > > > > Hi Gabriele,
> > > > >
> > > > > On Wed, 6 Apr 2016 14:50:29 +0000 Gabriele Paoloni wrote:
> > > > >
> > > > > > Hi, sorry to be late on this
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: linux-kernel-owner@vger.kernel.org [mailto:linux-
> kernel-
> > > > > > > owner@vger.kernel.org] On Behalf Of Jisheng Zhang
> > > > > > > Sent: 16 March 2016 11:41
> > > > > > > To: jingoohan1@gmail.com; pratyush.anand@gmail.com;
> > > > > bhelgaas@google.com
> > > > > > > Cc: linux-pci@vger.kernel.org; linux-
> kernel@vger.kernel.org;
> > > linux-
> > > > > arm-
> > > > > > > kernel@lists.infradead.org; Jisheng Zhang
> > > > > > > Subject: [PATCH v2] PCI: designware: move remaining rc
> setup
> > > code
> > > > > to
> > > > > > > dw_pcie_setup_rc()
> > > > > > >
> > > > > > > dw_pcie_setup_rc(), as its name indicates, setups the RC.
> But
> > > > > current
> > > > > > > dw_pcie_host_init() also contains some necessary rc setup
> code.
> > > > > > >
> > > > > > > Another reason: the host may lost power during suspend to
> ram,
> > > the
> > > > > RC
> > > > > > > need to be re-setup after resume. The rc can't be correctly
> > > resumed
> > > > > > > without the rc setup code in dw_pcie_host_init().
> > > > > > >
> > > > > > > So this patch moves the code to dw_pcie_setup_rc() to
> address
> > > the
> > > > > above
> > > > > > > two issues. After this patch, each pcie designware driver
> users
> > > > > could
> > > > > > > call dw_pcie_setup_rc() to re-setup rc when resume back.
> > > > > >
> > > > > > I think this patch breaks the Hisilicon driver...
> > > > > >
> > > > > > Our driver performs linkup setup in UEFI therefore we do not
> call
> > > > > > dw_pcie_setup_rc(), we only call dw_pcie_host_init().
> > > > >
> > > > > Thanks for the information. So pcie-hisi rely on UEFI to do
> > > something
> > > > > similar
> > > > > in dw_pcie_setup_rc(), this comes to a common driver implement
> > > > > question: should
> > > > > linux device driver rely on bootloader to configure HW device?
> > > >
> > > > I don't see any issue with this...
> > > >
> > > > >
> > > > > Is it acceptable that pcie-hisi adds a call to
> dw_pcie_setup_rc()
> > > in
> > > > > hisi_add_pcie_port()?
> > > >
> > > > I don't think so...that would try to overwrite what is already
> set by
> > > > the bootloader; so it is wrong in principle and maybe it can lead
> to
> > > > undefined behaviours...
> > >
> > > make sense! This commit is intend to re-setup the rc when waken
> from
> > > s2ram (in
> > > s2ram state, the host lost power)
> > >
> > > I have no good solution but to introduce one function e.g
> > > dw_pcie_setup_rc_after_linkup(), then move related code from
> > > dw_pcie_host_init
> > > to it, then let my host driver resume hook to call.
> > >
> > > Hi Pratyush, Jingoo and Bjorn etc.
> > >
> > > any suggestions are appreciated!
> >
> > What about:
> >
> > diff --git a/drivers/pci/host/pcie-designware.c
> b/drivers/pci/host/pcie-designware.c
> > index a4cccd3..e461f5d 100644
> > --- a/drivers/pci/host/pcie-designware.c
> > +++ b/drivers/pci/host/pcie-designware.c
> > @@ -434,7 +434,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
> >  	struct platform_device *pdev = to_platform_device(pp->dev);
> >  	struct pci_bus *bus, *child;
> >  	struct resource *cfg_res;
> > -	u32 val;
> >  	int i, ret;
> >  	LIST_HEAD(res);
> >  	struct resource_entry *win;
> > @@ -544,25 +543,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
> >  	if (pp->ops->host_init)
> >  		pp->ops->host_init(pp);
> >
> > -	/*
> > -	 * If the platform provides ->rd_other_conf, it means the
> platform
> > -	 * uses its own address translation component rather than ATU, so
> > -	 * we should not program the ATU here.
> > -	 */
> > -	if (!pp->ops->rd_other_conf)
> > -		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> > -					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> > -					  pp->mem_bus_addr, pp->mem_size);
> > -
> > -	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> > -
> > -	/* program correct class for RC */
> > -	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2,
> PCI_CLASS_BRIDGE_PCI);
> > -
> > -	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> > -	val |= PORT_LOGIC_SPEED_CHANGE;
> > -	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
> > -
> >  	pp->root_bus_nr = pp->busn->start;
> >  	if (IS_ENABLED(CONFIG_PCI_MSI)) {
> >  		bus = pci_scan_root_bus_msi(pp->dev, pp->root_bus_nr,
> > @@ -725,6 +705,29 @@ static struct pci_ops dw_pcie_ops = {
> >  	.write = dw_pcie_wr_conf,
> >  };
> >
> > +void dw_pcie_setup_own_cfg(struct pcie_port *pp)
> > +{
> > +	u32 val;
> > +	/*
> > +	 * If the platform provides ->rd_other_conf, it means the
> platform
> > +	 * uses its own address translation component rather than ATU, so
> > +	 * we should not program the ATU here.
> > +	 */
> > +	if (!pp->ops->rd_other_conf)
> > +		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> > +					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> > +					  pp->mem_bus_addr, pp->mem_size);
> > +
> > +	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> > +
> > +	/* program correct class for RC */
> > +	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2,
> PCI_CLASS_BRIDGE_PCI);
> > +
> > +	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> > +	val |= PORT_LOGIC_SPEED_CHANGE;
> > +	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
> > +}
> > +
> >  void dw_pcie_setup_rc(struct pcie_port *pp)
> >  {
> >  	u32 val;
> > @@ -800,6 +803,8 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
> >  	val |= PCI_COMMAND_IO | PCI_COMMAND_MEMORY |
> >  		PCI_COMMAND_MASTER | PCI_COMMAND_SERR;
> >  	dw_pcie_writel_rc(pp, val, PCI_COMMAND);
> > +
> > +	dw_pcie_setup_own_cfg(pp);
> >  }
> >
> >  MODULE_AUTHOR("Jingoo Han <jg1.han@samsung.com>");
> > diff --git a/drivers/pci/host/pcie-designware.h
> b/drivers/pci/host/pcie-designware.h
> > index f437f9b..caf0f5d 100644
> > --- a/drivers/pci/host/pcie-designware.h
> > +++ b/drivers/pci/host/pcie-designware.h
> > @@ -85,5 +85,6 @@ int dw_pcie_wait_for_link(struct pcie_port *pp);
> >  int dw_pcie_link_up(struct pcie_port *pp);
> >  void dw_pcie_setup_rc(struct pcie_port *pp);
> >  int dw_pcie_host_init(struct pcie_port *pp);
> > +void dw_pcie_setup_own_cfg(struct pcie_port *pp);
> >
> >  #endif /* _PCIE_DESIGNWARE_H */
> > diff --git a/drivers/pci/host/pcie-hisi.c b/drivers/pci/host/pcie-
> hisi.c
> > index 3e98d4e..8da29b2 100644
> > --- a/drivers/pci/host/pcie-hisi.c
> > +++ b/drivers/pci/host/pcie-hisi.c
> > @@ -164,6 +164,7 @@ static int hisi_add_pcie_port(struct pcie_port
> *pp,
> >  		dev_err(&pdev->dev, "failed to initialize host\n");
> >  		return ret;
> >  	}
> > +	dw_pcie_setup_own_cfg(pp);
> >
> >  	return 0;
> >  }
> 
> What's the hisi plan for resuming after suspend-to-RAM?  How does the
> RC get reprogrammed after it loses all its state?

PM is not part of the driver yet. This is planned for near
future release so haven't made such considerations yet

> 
> What would break if hisi did call dw_pcie_setup_rc()?  I know you said
> it would overwrite what the bootloader already did, which is true.

I am try to figure this out now with our HW team.

> 
> But hisi does call dw_pcie_host_init(), so it reads pp->mem (which
> determines pp->mem_base) and pp->lanes from the DT.  Other drivers
> then call dw_pcie_setup_rc() which programs the RC based on
> pp->mem_base and pp->lanes.  So hisi assumes UEFI programmed the RC to
> match the DT, while the other drivers read the DT and program the RC
> to match.  The latter seems more robust because it enforces the
> consistency rather than relying on it.

Yes I agree with you, however we have preferred to move RC config to
BIOS to have a single driver to support multiple versions of the
same SoC.

The patch I proposed above does the same job as the original patch
proposed by Jisheng and also allows hisi driver to call the moved
code.

Do you see anything wrong with it?

Thanks

Gab

> 
> Bjorn

[PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Gabriele Paoloni]

Hi Bjorn

Many thanks for your reply

> -----Original Message-----
> From: Bjorn Helgaas [mailto:helgaas at kernel.org]
> Sent: 07 April 2016 15:06
> To: Gabriele Paoloni
> Cc: Jisheng Zhang; jingoohan1 at gmail.com; pratyush.anand at gmail.com;
> bhelgaas at google.com; linux-pci at vger.kernel.org; linux-
> kernel at vger.kernel.org; linux-arm-kernel at lists.infradead.org
> Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> to dw_pcie_setup_rc()
> 
> On Thu, Apr 07, 2016 at 10:06:45AM +0000, Gabriele Paoloni wrote:
> > Hi Jisheng
> >
> > > -----Original Message-----
> > > From: Jisheng Zhang [mailto:jszhang at marvell.com]
> > > Sent: 07 April 2016 09:35
> > > To: Gabriele Paoloni
> > > Cc: jingoohan1 at gmail.com; pratyush.anand at gmail.com;
> > > bhelgaas at google.com; linux-pci at vger.kernel.org; linux-
> > > kernel at vger.kernel.org; linux-arm-kernel at lists.infradead.org
> > > Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup
> code
> > > to dw_pcie_setup_rc()
> > >
> > > Hi Gabriele,
> > >
> > > On Thu, 7 Apr 2016 08:20:28 +0000 Gabriele Paoloni wrote:
> > >
> > > > Hi Jisheng
> > > >
> > > > Thanks for your reply
> > > >
> > > > > -----Original Message-----
> > > > > From: Jisheng Zhang [mailto:jszhang at marvell.com]
> > > > > Sent: 07 April 2016 03:38
> > > > > To: Gabriele Paoloni; jingoohan1 at gmail.com;
> > > pratyush.anand at gmail.com;
> > > > > bhelgaas at google.com
> > > > > Cc: linux-pci at vger.kernel.org; linux-kernel at vger.kernel.org;
> linux-
> > > arm-
> > > > > kernel at lists.infradead.org
> > > > > Subject: Re: [PATCH v2] PCI: designware: move remaining rc
> setup
> > > code
> > > > > to dw_pcie_setup_rc()
> > > > >
> > > > > Hi Gabriele,
> > > > >
> > > > > On Wed, 6 Apr 2016 14:50:29 +0000 Gabriele Paoloni wrote:
> > > > >
> > > > > > Hi, sorry to be late on this
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: linux-kernel-owner at vger.kernel.org [mailto:linux-
> kernel-
> > > > > > > owner at vger.kernel.org] On Behalf Of Jisheng Zhang
> > > > > > > Sent: 16 March 2016 11:41
> > > > > > > To: jingoohan1 at gmail.com; pratyush.anand at gmail.com;
> > > > > bhelgaas at google.com
> > > > > > > Cc: linux-pci at vger.kernel.org; linux-
> kernel at vger.kernel.org;
> > > linux-
> > > > > arm-
> > > > > > > kernel at lists.infradead.org; Jisheng Zhang
> > > > > > > Subject: [PATCH v2] PCI: designware: move remaining rc
> setup
> > > code
> > > > > to
> > > > > > > dw_pcie_setup_rc()
> > > > > > >
> > > > > > > dw_pcie_setup_rc(), as its name indicates, setups the RC.
> But
> > > > > current
> > > > > > > dw_pcie_host_init() also contains some necessary rc setup
> code.
> > > > > > >
> > > > > > > Another reason: the host may lost power during suspend to
> ram,
> > > the
> > > > > RC
> > > > > > > need to be re-setup after resume. The rc can't be correctly
> > > resumed
> > > > > > > without the rc setup code in dw_pcie_host_init().
> > > > > > >
> > > > > > > So this patch moves the code to dw_pcie_setup_rc() to
> address
> > > the
> > > > > above
> > > > > > > two issues. After this patch, each pcie designware driver
> users
> > > > > could
> > > > > > > call dw_pcie_setup_rc() to re-setup rc when resume back.
> > > > > >
> > > > > > I think this patch breaks the Hisilicon driver...
> > > > > >
> > > > > > Our driver performs linkup setup in UEFI therefore we do not
> call
> > > > > > dw_pcie_setup_rc(), we only call dw_pcie_host_init().
> > > > >
> > > > > Thanks for the information. So pcie-hisi rely on UEFI to do
> > > something
> > > > > similar
> > > > > in dw_pcie_setup_rc(), this comes to a common driver implement
> > > > > question: should
> > > > > linux device driver rely on bootloader to configure HW device?
> > > >
> > > > I don't see any issue with this...
> > > >
> > > > >
> > > > > Is it acceptable that pcie-hisi adds a call to
> dw_pcie_setup_rc()
> > > in
> > > > > hisi_add_pcie_port()?
> > > >
> > > > I don't think so...that would try to overwrite what is already
> set by
> > > > the bootloader; so it is wrong in principle and maybe it can lead
> to
> > > > undefined behaviours...
> > >
> > > make sense! This commit is intend to re-setup the rc when waken
> from
> > > s2ram (in
> > > s2ram state, the host lost power)
> > >
> > > I have no good solution but to introduce one function e.g
> > > dw_pcie_setup_rc_after_linkup(), then move related code from
> > > dw_pcie_host_init
> > > to it, then let my host driver resume hook to call.
> > >
> > > Hi Pratyush, Jingoo and Bjorn etc.
> > >
> > > any suggestions are appreciated!
> >
> > What about:
> >
> > diff --git a/drivers/pci/host/pcie-designware.c
> b/drivers/pci/host/pcie-designware.c
> > index a4cccd3..e461f5d 100644
> > --- a/drivers/pci/host/pcie-designware.c
> > +++ b/drivers/pci/host/pcie-designware.c
> > @@ -434,7 +434,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
> >  	struct platform_device *pdev = to_platform_device(pp->dev);
> >  	struct pci_bus *bus, *child;
> >  	struct resource *cfg_res;
> > -	u32 val;
> >  	int i, ret;
> >  	LIST_HEAD(res);
> >  	struct resource_entry *win;
> > @@ -544,25 +543,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
> >  	if (pp->ops->host_init)
> >  		pp->ops->host_init(pp);
> >
> > -	/*
> > -	 * If the platform provides ->rd_other_conf, it means the
> platform
> > -	 * uses its own address translation component rather than ATU, so
> > -	 * we should not program the ATU here.
> > -	 */
> > -	if (!pp->ops->rd_other_conf)
> > -		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> > -					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> > -					  pp->mem_bus_addr, pp->mem_size);
> > -
> > -	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> > -
> > -	/* program correct class for RC */
> > -	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2,
> PCI_CLASS_BRIDGE_PCI);
> > -
> > -	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> > -	val |= PORT_LOGIC_SPEED_CHANGE;
> > -	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
> > -
> >  	pp->root_bus_nr = pp->busn->start;
> >  	if (IS_ENABLED(CONFIG_PCI_MSI)) {
> >  		bus = pci_scan_root_bus_msi(pp->dev, pp->root_bus_nr,
> > @@ -725,6 +705,29 @@ static struct pci_ops dw_pcie_ops = {
> >  	.write = dw_pcie_wr_conf,
> >  };
> >
> > +void dw_pcie_setup_own_cfg(struct pcie_port *pp)
> > +{
> > +	u32 val;
> > +	/*
> > +	 * If the platform provides ->rd_other_conf, it means the
> platform
> > +	 * uses its own address translation component rather than ATU, so
> > +	 * we should not program the ATU here.
> > +	 */
> > +	if (!pp->ops->rd_other_conf)
> > +		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> > +					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> > +					  pp->mem_bus_addr, pp->mem_size);
> > +
> > +	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> > +
> > +	/* program correct class for RC */
> > +	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2,
> PCI_CLASS_BRIDGE_PCI);
> > +
> > +	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> > +	val |= PORT_LOGIC_SPEED_CHANGE;
> > +	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
> > +}
> > +
> >  void dw_pcie_setup_rc(struct pcie_port *pp)
> >  {
> >  	u32 val;
> > @@ -800,6 +803,8 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
> >  	val |= PCI_COMMAND_IO | PCI_COMMAND_MEMORY |
> >  		PCI_COMMAND_MASTER | PCI_COMMAND_SERR;
> >  	dw_pcie_writel_rc(pp, val, PCI_COMMAND);
> > +
> > +	dw_pcie_setup_own_cfg(pp);
> >  }
> >
> >  MODULE_AUTHOR("Jingoo Han <jg1.han@samsung.com>");
> > diff --git a/drivers/pci/host/pcie-designware.h
> b/drivers/pci/host/pcie-designware.h
> > index f437f9b..caf0f5d 100644
> > --- a/drivers/pci/host/pcie-designware.h
> > +++ b/drivers/pci/host/pcie-designware.h
> > @@ -85,5 +85,6 @@ int dw_pcie_wait_for_link(struct pcie_port *pp);
> >  int dw_pcie_link_up(struct pcie_port *pp);
> >  void dw_pcie_setup_rc(struct pcie_port *pp);
> >  int dw_pcie_host_init(struct pcie_port *pp);
> > +void dw_pcie_setup_own_cfg(struct pcie_port *pp);
> >
> >  #endif /* _PCIE_DESIGNWARE_H */
> > diff --git a/drivers/pci/host/pcie-hisi.c b/drivers/pci/host/pcie-
> hisi.c
> > index 3e98d4e..8da29b2 100644
> > --- a/drivers/pci/host/pcie-hisi.c
> > +++ b/drivers/pci/host/pcie-hisi.c
> > @@ -164,6 +164,7 @@ static int hisi_add_pcie_port(struct pcie_port
> *pp,
> >  		dev_err(&pdev->dev, "failed to initialize host\n");
> >  		return ret;
> >  	}
> > +	dw_pcie_setup_own_cfg(pp);
> >
> >  	return 0;
> >  }
> 
> What's the hisi plan for resuming after suspend-to-RAM?  How does the
> RC get reprogrammed after it loses all its state?

PM is not part of the driver yet. This is planned for near
future release so haven't made such considerations yet

> 
> What would break if hisi did call dw_pcie_setup_rc()?  I know you said
> it would overwrite what the bootloader already did, which is true.

I am try to figure this out now with our HW team.

> 
> But hisi does call dw_pcie_host_init(), so it reads pp->mem (which
> determines pp->mem_base) and pp->lanes from the DT.  Other drivers
> then call dw_pcie_setup_rc() which programs the RC based on
> pp->mem_base and pp->lanes.  So hisi assumes UEFI programmed the RC to
> match the DT, while the other drivers read the DT and program the RC
> to match.  The latter seems more robust because it enforces the
> consistency rather than relying on it.

Yes I agree with you, however we have preferred to move RC config to
BIOS to have a single driver to support multiple versions of the
same SoC.

The patch I proposed above does the same job as the original patch
proposed by Jisheng and also allows hisi driver to call the moved
code.

Do you see anything wrong with it?

Thanks

Gab

> 
> Bjorn

Re: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Bjorn Helgaas]

On Fri, Apr 08, 2016 at 01:33:28PM +0000, Gabriele Paoloni wrote:
> Hi Bjorn
> 
> Many thanks for your reply
> 
> > -----Original Message-----
> > From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> > Sent: 07 April 2016 15:06
> > To: Gabriele Paoloni
> > Cc: Jisheng Zhang; jingoohan1@gmail.com; pratyush.anand@gmail.com;
> > bhelgaas@google.com; linux-pci@vger.kernel.org; linux-
> > kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
> > Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> > to dw_pcie_setup_rc()
> > 
> > On Thu, Apr 07, 2016 at 10:06:45AM +0000, Gabriele Paoloni wrote:
> > > Hi Jisheng
> > >
> > > > -----Original Message-----
> > > > From: Jisheng Zhang [mailto:jszhang@marvell.com]
> > > > Sent: 07 April 2016 09:35
> > > > To: Gabriele Paoloni
> > > > Cc: jingoohan1@gmail.com; pratyush.anand@gmail.com;
> > > > bhelgaas@google.com; linux-pci@vger.kernel.org; linux-
> > > > kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
> > > > Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup
> > code
> > > > to dw_pcie_setup_rc()
> > > >
> > > > Hi Gabriele,
> > > >
> > > > On Thu, 7 Apr 2016 08:20:28 +0000 Gabriele Paoloni wrote:
> > > >
> > > > > Hi Jisheng
> > > > >
> > > > > Thanks for your reply
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Jisheng Zhang [mailto:jszhang@marvell.com]
> > > > > > Sent: 07 April 2016 03:38
> > > > > > To: Gabriele Paoloni; jingoohan1@gmail.com;
> > > > pratyush.anand@gmail.com;
> > > > > > bhelgaas@google.com
> > > > > > Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org;
> > linux-
> > > > arm-
> > > > > > kernel@lists.infradead.org
> > > > > > Subject: Re: [PATCH v2] PCI: designware: move remaining rc
> > setup
> > > > code
> > > > > > to dw_pcie_setup_rc()
> > > > > >
> > > > > > Hi Gabriele,
> > > > > >
> > > > > > On Wed, 6 Apr 2016 14:50:29 +0000 Gabriele Paoloni wrote:
> > > > > >
> > > > > > > Hi, sorry to be late on this
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: linux-kernel-owner@vger.kernel.org [mailto:linux-
> > kernel-
> > > > > > > > owner@vger.kernel.org] On Behalf Of Jisheng Zhang
> > > > > > > > Sent: 16 March 2016 11:41
> > > > > > > > To: jingoohan1@gmail.com; pratyush.anand@gmail.com;
> > > > > > bhelgaas@google.com
> > > > > > > > Cc: linux-pci@vger.kernel.org; linux-
> > kernel@vger.kernel.org;
> > > > linux-
> > > > > > arm-
> > > > > > > > kernel@lists.infradead.org; Jisheng Zhang
> > > > > > > > Subject: [PATCH v2] PCI: designware: move remaining rc
> > setup
> > > > code
> > > > > > to
> > > > > > > > dw_pcie_setup_rc()
> > > > > > > >
> > > > > > > > dw_pcie_setup_rc(), as its name indicates, setups the RC.
> > But
> > > > > > current
> > > > > > > > dw_pcie_host_init() also contains some necessary rc setup
> > code.
> > > > > > > >
> > > > > > > > Another reason: the host may lost power during suspend to
> > ram,
> > > > the
> > > > > > RC
> > > > > > > > need to be re-setup after resume. The rc can't be correctly
> > > > resumed
> > > > > > > > without the rc setup code in dw_pcie_host_init().
> > > > > > > >
> > > > > > > > So this patch moves the code to dw_pcie_setup_rc() to
> > address
> > > > the
> > > > > > above
> > > > > > > > two issues. After this patch, each pcie designware driver
> > users
> > > > > > could
> > > > > > > > call dw_pcie_setup_rc() to re-setup rc when resume back.
> > > > > > >
> > > > > > > I think this patch breaks the Hisilicon driver...
> > > > > > >
> > > > > > > Our driver performs linkup setup in UEFI therefore we do not
> > call
> > > > > > > dw_pcie_setup_rc(), we only call dw_pcie_host_init().
> > > > > >
> > > > > > Thanks for the information. So pcie-hisi rely on UEFI to do
> > > > something
> > > > > > similar
> > > > > > in dw_pcie_setup_rc(), this comes to a common driver implement
> > > > > > question: should
> > > > > > linux device driver rely on bootloader to configure HW device?
> > > > >
> > > > > I don't see any issue with this...
> > > > >
> > > > > >
> > > > > > Is it acceptable that pcie-hisi adds a call to
> > dw_pcie_setup_rc()
> > > > in
> > > > > > hisi_add_pcie_port()?
> > > > >
> > > > > I don't think so...that would try to overwrite what is already
> > set by
> > > > > the bootloader; so it is wrong in principle and maybe it can lead
> > to
> > > > > undefined behaviours...
> > > >
> > > > make sense! This commit is intend to re-setup the rc when waken
> > from
> > > > s2ram (in
> > > > s2ram state, the host lost power)
> > > >
> > > > I have no good solution but to introduce one function e.g
> > > > dw_pcie_setup_rc_after_linkup(), then move related code from
> > > > dw_pcie_host_init
> > > > to it, then let my host driver resume hook to call.
> > > >
> > > > Hi Pratyush, Jingoo and Bjorn etc.
> > > >
> > > > any suggestions are appreciated!
> > >
> > > What about:
> > >
> > > diff --git a/drivers/pci/host/pcie-designware.c
> > b/drivers/pci/host/pcie-designware.c
> > > index a4cccd3..e461f5d 100644
> > > --- a/drivers/pci/host/pcie-designware.c
> > > +++ b/drivers/pci/host/pcie-designware.c
> > > @@ -434,7 +434,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
> > >  	struct platform_device *pdev = to_platform_device(pp->dev);
> > >  	struct pci_bus *bus, *child;
> > >  	struct resource *cfg_res;
> > > -	u32 val;
> > >  	int i, ret;
> > >  	LIST_HEAD(res);
> > >  	struct resource_entry *win;
> > > @@ -544,25 +543,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
> > >  	if (pp->ops->host_init)
> > >  		pp->ops->host_init(pp);
> > >
> > > -	/*
> > > -	 * If the platform provides ->rd_other_conf, it means the
> > platform
> > > -	 * uses its own address translation component rather than ATU, so
> > > -	 * we should not program the ATU here.
> > > -	 */
> > > -	if (!pp->ops->rd_other_conf)
> > > -		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> > > -					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> > > -					  pp->mem_bus_addr, pp->mem_size);
> > > -
> > > -	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> > > -
> > > -	/* program correct class for RC */
> > > -	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2,
> > PCI_CLASS_BRIDGE_PCI);
> > > -
> > > -	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> > > -	val |= PORT_LOGIC_SPEED_CHANGE;
> > > -	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
> > > -
> > >  	pp->root_bus_nr = pp->busn->start;
> > >  	if (IS_ENABLED(CONFIG_PCI_MSI)) {
> > >  		bus = pci_scan_root_bus_msi(pp->dev, pp->root_bus_nr,
> > > @@ -725,6 +705,29 @@ static struct pci_ops dw_pcie_ops = {
> > >  	.write = dw_pcie_wr_conf,
> > >  };
> > >
> > > +void dw_pcie_setup_own_cfg(struct pcie_port *pp)
> > > +{
> > > +	u32 val;
> > > +	/*
> > > +	 * If the platform provides ->rd_other_conf, it means the
> > platform
> > > +	 * uses its own address translation component rather than ATU, so
> > > +	 * we should not program the ATU here.
> > > +	 */
> > > +	if (!pp->ops->rd_other_conf)
> > > +		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> > > +					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> > > +					  pp->mem_bus_addr, pp->mem_size);
> > > +
> > > +	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> > > +
> > > +	/* program correct class for RC */
> > > +	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2,
> > PCI_CLASS_BRIDGE_PCI);
> > > +
> > > +	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> > > +	val |= PORT_LOGIC_SPEED_CHANGE;
> > > +	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
> > > +}
> > > +
> > >  void dw_pcie_setup_rc(struct pcie_port *pp)
> > >  {
> > >  	u32 val;
> > > @@ -800,6 +803,8 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
> > >  	val |= PCI_COMMAND_IO | PCI_COMMAND_MEMORY |
> > >  		PCI_COMMAND_MASTER | PCI_COMMAND_SERR;
> > >  	dw_pcie_writel_rc(pp, val, PCI_COMMAND);
> > > +
> > > +	dw_pcie_setup_own_cfg(pp);
> > >  }
> > >
> > >  MODULE_AUTHOR("Jingoo Han <jg1.han@samsung.com>");
> > > diff --git a/drivers/pci/host/pcie-designware.h
> > b/drivers/pci/host/pcie-designware.h
> > > index f437f9b..caf0f5d 100644
> > > --- a/drivers/pci/host/pcie-designware.h
> > > +++ b/drivers/pci/host/pcie-designware.h
> > > @@ -85,5 +85,6 @@ int dw_pcie_wait_for_link(struct pcie_port *pp);
> > >  int dw_pcie_link_up(struct pcie_port *pp);
> > >  void dw_pcie_setup_rc(struct pcie_port *pp);
> > >  int dw_pcie_host_init(struct pcie_port *pp);
> > > +void dw_pcie_setup_own_cfg(struct pcie_port *pp);
> > >
> > >  #endif /* _PCIE_DESIGNWARE_H */
> > > diff --git a/drivers/pci/host/pcie-hisi.c b/drivers/pci/host/pcie-
> > hisi.c
> > > index 3e98d4e..8da29b2 100644
> > > --- a/drivers/pci/host/pcie-hisi.c
> > > +++ b/drivers/pci/host/pcie-hisi.c
> > > @@ -164,6 +164,7 @@ static int hisi_add_pcie_port(struct pcie_port
> > *pp,
> > >  		dev_err(&pdev->dev, "failed to initialize host\n");
> > >  		return ret;
> > >  	}
> > > +	dw_pcie_setup_own_cfg(pp);
> > >
> > >  	return 0;
> > >  }
> > 
> > What's the hisi plan for resuming after suspend-to-RAM?  How does the
> > RC get reprogrammed after it loses all its state?
> 
> PM is not part of the driver yet. This is planned for near
> future release so haven't made such considerations yet
> > 
> > What would break if hisi did call dw_pcie_setup_rc()?  I know you said
> > it would overwrite what the bootloader already did, which is true.
> 
> I am try to figure this out now with our HW team.
> 
> > 
> > But hisi does call dw_pcie_host_init(), so it reads pp->mem (which
> > determines pp->mem_base) and pp->lanes from the DT.  Other drivers
> > then call dw_pcie_setup_rc() which programs the RC based on
> > pp->mem_base and pp->lanes.  So hisi assumes UEFI programmed the RC to
> > match the DT, while the other drivers read the DT and program the RC
> > to match.  The latter seems more robust because it enforces the
> > consistency rather than relying on it.
> 
> Yes I agree with you, however we have preferred to move RC config to
> BIOS to have a single driver to support multiple versions of the
> same SoC.

I think there are two reasonable approaches:

  1) A single generic driver that doesn't have any knowledge about the
  chipset registers; it uses run-time firmware interfaces to manage
  the bridge.  The ACPI pci_root.c driver is the best example so far
  and works very well.  It supports basically all x86 and ia64
  chipsets and requires no kernel work for new ones.

  2) Native drivers specific to each chipset.  These may get
  configuration information from DT, but they do their own
  register-level programming of the device without run-time help from
  firmware.

I think hisi is a native driver because it uses hip05/hip06 registers
to check link state and perform config operations.  And apparently you
rely on the ATU, BAR, class, and link width programming currently done
in dw_pcie_host_init().  But you want to rely on pre-boot firmware to
set up the link.  That doesn't make sense to me -- if the driver wants
to twiddle the registers, it should know how to do it all.  I don't
see how you can reasonably manage this half-way approach.

> The patch I proposed above does the same job as the original patch
> proposed by Jisheng and also allows hisi driver to call the moved
> code.
> 
> Do you see anything wrong with it?

Only that it makes the structure more complicated and we haven't
identified a corresponding benefit yet.

Bjorn

Re: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Bjorn Helgaas]

On Fri, Apr 08, 2016 at 01:33:28PM +0000, Gabriele Paoloni wrote:
> Hi Bjorn
> 
> Many thanks for your reply
> 
> > -----Original Message-----
> > From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> > Sent: 07 April 2016 15:06
> > To: Gabriele Paoloni
> > Cc: Jisheng Zhang; jingoohan1@gmail.com; pratyush.anand@gmail.com;
> > bhelgaas@google.com; linux-pci@vger.kernel.org; linux-
> > kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
> > Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> > to dw_pcie_setup_rc()
> > 
> > On Thu, Apr 07, 2016 at 10:06:45AM +0000, Gabriele Paoloni wrote:
> > > Hi Jisheng
> > >
> > > > -----Original Message-----
> > > > From: Jisheng Zhang [mailto:jszhang@marvell.com]
> > > > Sent: 07 April 2016 09:35
> > > > To: Gabriele Paoloni
> > > > Cc: jingoohan1@gmail.com; pratyush.anand@gmail.com;
> > > > bhelgaas@google.com; linux-pci@vger.kernel.org; linux-
> > > > kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
> > > > Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup
> > code
> > > > to dw_pcie_setup_rc()
> > > >
> > > > Hi Gabriele,
> > > >
> > > > On Thu, 7 Apr 2016 08:20:28 +0000 Gabriele Paoloni wrote:
> > > >
> > > > > Hi Jisheng
> > > > >
> > > > > Thanks for your reply
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Jisheng Zhang [mailto:jszhang@marvell.com]
> > > > > > Sent: 07 April 2016 03:38
> > > > > > To: Gabriele Paoloni; jingoohan1@gmail.com;
> > > > pratyush.anand@gmail.com;
> > > > > > bhelgaas@google.com
> > > > > > Cc: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org;
> > linux-
> > > > arm-
> > > > > > kernel@lists.infradead.org
> > > > > > Subject: Re: [PATCH v2] PCI: designware: move remaining rc
> > setup
> > > > code
> > > > > > to dw_pcie_setup_rc()
> > > > > >
> > > > > > Hi Gabriele,
> > > > > >
> > > > > > On Wed, 6 Apr 2016 14:50:29 +0000 Gabriele Paoloni wrote:
> > > > > >
> > > > > > > Hi, sorry to be late on this
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: linux-kernel-owner@vger.kernel.org [mailto:linux-
> > kernel-
> > > > > > > > owner@vger.kernel.org] On Behalf Of Jisheng Zhang
> > > > > > > > Sent: 16 March 2016 11:41
> > > > > > > > To: jingoohan1@gmail.com; pratyush.anand@gmail.com;
> > > > > > bhelgaas@google.com
> > > > > > > > Cc: linux-pci@vger.kernel.org; linux-
> > kernel@vger.kernel.org;
> > > > linux-
> > > > > > arm-
> > > > > > > > kernel@lists.infradead.org; Jisheng Zhang
> > > > > > > > Subject: [PATCH v2] PCI: designware: move remaining rc
> > setup
> > > > code
> > > > > > to
> > > > > > > > dw_pcie_setup_rc()
> > > > > > > >
> > > > > > > > dw_pcie_setup_rc(), as its name indicates, setups the RC.
> > But
> > > > > > current
> > > > > > > > dw_pcie_host_init() also contains some necessary rc setup
> > code.
> > > > > > > >
> > > > > > > > Another reason: the host may lost power during suspend to
> > ram,
> > > > the
> > > > > > RC
> > > > > > > > need to be re-setup after resume. The rc can't be correctly
> > > > resumed
> > > > > > > > without the rc setup code in dw_pcie_host_init().
> > > > > > > >
> > > > > > > > So this patch moves the code to dw_pcie_setup_rc() to
> > address
> > > > the
> > > > > > above
> > > > > > > > two issues. After this patch, each pcie designware driver
> > users
> > > > > > could
> > > > > > > > call dw_pcie_setup_rc() to re-setup rc when resume back.
> > > > > > >
> > > > > > > I think this patch breaks the Hisilicon driver...
> > > > > > >
> > > > > > > Our driver performs linkup setup in UEFI therefore we do not
> > call
> > > > > > > dw_pcie_setup_rc(), we only call dw_pcie_host_init().
> > > > > >
> > > > > > Thanks for the information. So pcie-hisi rely on UEFI to do
> > > > something
> > > > > > similar
> > > > > > in dw_pcie_setup_rc(), this comes to a common driver implement
> > > > > > question: should
> > > > > > linux device driver rely on bootloader to configure HW device?
> > > > >
> > > > > I don't see any issue with this...
> > > > >
> > > > > >
> > > > > > Is it acceptable that pcie-hisi adds a call to
> > dw_pcie_setup_rc()
> > > > in
> > > > > > hisi_add_pcie_port()?
> > > > >
> > > > > I don't think so...that would try to overwrite what is already
> > set by
> > > > > the bootloader; so it is wrong in principle and maybe it can lead
> > to
> > > > > undefined behaviours...
> > > >
> > > > make sense! This commit is intend to re-setup the rc when waken
> > from
> > > > s2ram (in
> > > > s2ram state, the host lost power)
> > > >
> > > > I have no good solution but to introduce one function e.g
> > > > dw_pcie_setup_rc_after_linkup(), then move related code from
> > > > dw_pcie_host_init
> > > > to it, then let my host driver resume hook to call.
> > > >
> > > > Hi Pratyush, Jingoo and Bjorn etc.
> > > >
> > > > any suggestions are appreciated!
> > >
> > > What about:
> > >
> > > diff --git a/drivers/pci/host/pcie-designware.c
> > b/drivers/pci/host/pcie-designware.c
> > > index a4cccd3..e461f5d 100644
> > > --- a/drivers/pci/host/pcie-designware.c
> > > +++ b/drivers/pci/host/pcie-designware.c
> > > @@ -434,7 +434,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
> > >  	struct platform_device *pdev = to_platform_device(pp->dev);
> > >  	struct pci_bus *bus, *child;
> > >  	struct resource *cfg_res;
> > > -	u32 val;
> > >  	int i, ret;
> > >  	LIST_HEAD(res);
> > >  	struct resource_entry *win;
> > > @@ -544,25 +543,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
> > >  	if (pp->ops->host_init)
> > >  		pp->ops->host_init(pp);
> > >
> > > -	/*
> > > -	 * If the platform provides ->rd_other_conf, it means the
> > platform
> > > -	 * uses its own address translation component rather than ATU, so
> > > -	 * we should not program the ATU here.
> > > -	 */
> > > -	if (!pp->ops->rd_other_conf)
> > > -		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> > > -					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> > > -					  pp->mem_bus_addr, pp->mem_size);
> > > -
> > > -	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> > > -
> > > -	/* program correct class for RC */
> > > -	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2,
> > PCI_CLASS_BRIDGE_PCI);
> > > -
> > > -	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> > > -	val |= PORT_LOGIC_SPEED_CHANGE;
> > > -	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
> > > -
> > >  	pp->root_bus_nr = pp->busn->start;
> > >  	if (IS_ENABLED(CONFIG_PCI_MSI)) {
> > >  		bus = pci_scan_root_bus_msi(pp->dev, pp->root_bus_nr,
> > > @@ -725,6 +705,29 @@ static struct pci_ops dw_pcie_ops = {
> > >  	.write = dw_pcie_wr_conf,
> > >  };
> > >
> > > +void dw_pcie_setup_own_cfg(struct pcie_port *pp)
> > > +{
> > > +	u32 val;
> > > +	/*
> > > +	 * If the platform provides ->rd_other_conf, it means the
> > platform
> > > +	 * uses its own address translation component rather than ATU, so
> > > +	 * we should not program the ATU here.
> > > +	 */
> > > +	if (!pp->ops->rd_other_conf)
> > > +		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> > > +					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> > > +					  pp->mem_bus_addr, pp->mem_size);
> > > +
> > > +	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> > > +
> > > +	/* program correct class for RC */
> > > +	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2,
> > PCI_CLASS_BRIDGE_PCI);
> > > +
> > > +	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> > > +	val |= PORT_LOGIC_SPEED_CHANGE;
> > > +	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
> > > +}
> > > +
> > >  void dw_pcie_setup_rc(struct pcie_port *pp)
> > >  {
> > >  	u32 val;
> > > @@ -800,6 +803,8 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
> > >  	val |= PCI_COMMAND_IO | PCI_COMMAND_MEMORY |
> > >  		PCI_COMMAND_MASTER | PCI_COMMAND_SERR;
> > >  	dw_pcie_writel_rc(pp, val, PCI_COMMAND);
> > > +
> > > +	dw_pcie_setup_own_cfg(pp);
> > >  }
> > >
> > >  MODULE_AUTHOR("Jingoo Han <jg1.han@samsung.com>");
> > > diff --git a/drivers/pci/host/pcie-designware.h
> > b/drivers/pci/host/pcie-designware.h
> > > index f437f9b..caf0f5d 100644
> > > --- a/drivers/pci/host/pcie-designware.h
> > > +++ b/drivers/pci/host/pcie-designware.h
> > > @@ -85,5 +85,6 @@ int dw_pcie_wait_for_link(struct pcie_port *pp);
> > >  int dw_pcie_link_up(struct pcie_port *pp);
> > >  void dw_pcie_setup_rc(struct pcie_port *pp);
> > >  int dw_pcie_host_init(struct pcie_port *pp);
> > > +void dw_pcie_setup_own_cfg(struct pcie_port *pp);
> > >
> > >  #endif /* _PCIE_DESIGNWARE_H */
> > > diff --git a/drivers/pci/host/pcie-hisi.c b/drivers/pci/host/pcie-
> > hisi.c
> > > index 3e98d4e..8da29b2 100644
> > > --- a/drivers/pci/host/pcie-hisi.c
> > > +++ b/drivers/pci/host/pcie-hisi.c
> > > @@ -164,6 +164,7 @@ static int hisi_add_pcie_port(struct pcie_port
> > *pp,
> > >  		dev_err(&pdev->dev, "failed to initialize host\n");
> > >  		return ret;
> > >  	}
> > > +	dw_pcie_setup_own_cfg(pp);
> > >
> > >  	return 0;
> > >  }
> > 
> > What's the hisi plan for resuming after suspend-to-RAM?  How does the
> > RC get reprogrammed after it loses all its state?
> 
> PM is not part of the driver yet. This is planned for near
> future release so haven't made such considerations yet
> > 
> > What would break if hisi did call dw_pcie_setup_rc()?  I know you said
> > it would overwrite what the bootloader already did, which is true.
> 
> I am try to figure this out now with our HW team.
> 
> > 
> > But hisi does call dw_pcie_host_init(), so it reads pp->mem (which
> > determines pp->mem_base) and pp->lanes from the DT.  Other drivers
> > then call dw_pcie_setup_rc() which programs the RC based on
> > pp->mem_base and pp->lanes.  So hisi assumes UEFI programmed the RC to
> > match the DT, while the other drivers read the DT and program the RC
> > to match.  The latter seems more robust because it enforces the
> > consistency rather than relying on it.
> 
> Yes I agree with you, however we have preferred to move RC config to
> BIOS to have a single driver to support multiple versions of the
> same SoC.

I think there are two reasonable approaches:

  1) A single generic driver that doesn't have any knowledge about the
  chipset registers; it uses run-time firmware interfaces to manage
  the bridge.  The ACPI pci_root.c driver is the best example so far
  and works very well.  It supports basically all x86 and ia64
  chipsets and requires no kernel work for new ones.

  2) Native drivers specific to each chipset.  These may get
  configuration information from DT, but they do their own
  register-level programming of the device without run-time help from
  firmware.

I think hisi is a native driver because it uses hip05/hip06 registers
to check link state and perform config operations.  And apparently you
rely on the ATU, BAR, class, and link width programming currently done
in dw_pcie_host_init().  But you want to rely on pre-boot firmware to
set up the link.  That doesn't make sense to me -- if the driver wants
to twiddle the registers, it should know how to do it all.  I don't
see how you can reasonably manage this half-way approach.

> The patch I proposed above does the same job as the original patch
> proposed by Jisheng and also allows hisi driver to call the moved
> code.
> 
> Do you see anything wrong with it?

Only that it makes the structure more complicated and we haven't
identified a corresponding benefit yet.

Bjorn

[PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Bjorn Helgaas]

On Fri, Apr 08, 2016 at 01:33:28PM +0000, Gabriele Paoloni wrote:
> Hi Bjorn
> 
> Many thanks for your reply
> 
> > -----Original Message-----
> > From: Bjorn Helgaas [mailto:helgaas at kernel.org]
> > Sent: 07 April 2016 15:06
> > To: Gabriele Paoloni
> > Cc: Jisheng Zhang; jingoohan1 at gmail.com; pratyush.anand at gmail.com;
> > bhelgaas at google.com; linux-pci at vger.kernel.org; linux-
> > kernel at vger.kernel.org; linux-arm-kernel at lists.infradead.org
> > Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> > to dw_pcie_setup_rc()
> > 
> > On Thu, Apr 07, 2016 at 10:06:45AM +0000, Gabriele Paoloni wrote:
> > > Hi Jisheng
> > >
> > > > -----Original Message-----
> > > > From: Jisheng Zhang [mailto:jszhang at marvell.com]
> > > > Sent: 07 April 2016 09:35
> > > > To: Gabriele Paoloni
> > > > Cc: jingoohan1 at gmail.com; pratyush.anand at gmail.com;
> > > > bhelgaas at google.com; linux-pci at vger.kernel.org; linux-
> > > > kernel at vger.kernel.org; linux-arm-kernel at lists.infradead.org
> > > > Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup
> > code
> > > > to dw_pcie_setup_rc()
> > > >
> > > > Hi Gabriele,
> > > >
> > > > On Thu, 7 Apr 2016 08:20:28 +0000 Gabriele Paoloni wrote:
> > > >
> > > > > Hi Jisheng
> > > > >
> > > > > Thanks for your reply
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Jisheng Zhang [mailto:jszhang at marvell.com]
> > > > > > Sent: 07 April 2016 03:38
> > > > > > To: Gabriele Paoloni; jingoohan1 at gmail.com;
> > > > pratyush.anand at gmail.com;
> > > > > > bhelgaas at google.com
> > > > > > Cc: linux-pci at vger.kernel.org; linux-kernel at vger.kernel.org;
> > linux-
> > > > arm-
> > > > > > kernel at lists.infradead.org
> > > > > > Subject: Re: [PATCH v2] PCI: designware: move remaining rc
> > setup
> > > > code
> > > > > > to dw_pcie_setup_rc()
> > > > > >
> > > > > > Hi Gabriele,
> > > > > >
> > > > > > On Wed, 6 Apr 2016 14:50:29 +0000 Gabriele Paoloni wrote:
> > > > > >
> > > > > > > Hi, sorry to be late on this
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: linux-kernel-owner at vger.kernel.org [mailto:linux-
> > kernel-
> > > > > > > > owner at vger.kernel.org] On Behalf Of Jisheng Zhang
> > > > > > > > Sent: 16 March 2016 11:41
> > > > > > > > To: jingoohan1 at gmail.com; pratyush.anand at gmail.com;
> > > > > > bhelgaas at google.com
> > > > > > > > Cc: linux-pci at vger.kernel.org; linux-
> > kernel at vger.kernel.org;
> > > > linux-
> > > > > > arm-
> > > > > > > > kernel at lists.infradead.org; Jisheng Zhang
> > > > > > > > Subject: [PATCH v2] PCI: designware: move remaining rc
> > setup
> > > > code
> > > > > > to
> > > > > > > > dw_pcie_setup_rc()
> > > > > > > >
> > > > > > > > dw_pcie_setup_rc(), as its name indicates, setups the RC.
> > But
> > > > > > current
> > > > > > > > dw_pcie_host_init() also contains some necessary rc setup
> > code.
> > > > > > > >
> > > > > > > > Another reason: the host may lost power during suspend to
> > ram,
> > > > the
> > > > > > RC
> > > > > > > > need to be re-setup after resume. The rc can't be correctly
> > > > resumed
> > > > > > > > without the rc setup code in dw_pcie_host_init().
> > > > > > > >
> > > > > > > > So this patch moves the code to dw_pcie_setup_rc() to
> > address
> > > > the
> > > > > > above
> > > > > > > > two issues. After this patch, each pcie designware driver
> > users
> > > > > > could
> > > > > > > > call dw_pcie_setup_rc() to re-setup rc when resume back.
> > > > > > >
> > > > > > > I think this patch breaks the Hisilicon driver...
> > > > > > >
> > > > > > > Our driver performs linkup setup in UEFI therefore we do not
> > call
> > > > > > > dw_pcie_setup_rc(), we only call dw_pcie_host_init().
> > > > > >
> > > > > > Thanks for the information. So pcie-hisi rely on UEFI to do
> > > > something
> > > > > > similar
> > > > > > in dw_pcie_setup_rc(), this comes to a common driver implement
> > > > > > question: should
> > > > > > linux device driver rely on bootloader to configure HW device?
> > > > >
> > > > > I don't see any issue with this...
> > > > >
> > > > > >
> > > > > > Is it acceptable that pcie-hisi adds a call to
> > dw_pcie_setup_rc()
> > > > in
> > > > > > hisi_add_pcie_port()?
> > > > >
> > > > > I don't think so...that would try to overwrite what is already
> > set by
> > > > > the bootloader; so it is wrong in principle and maybe it can lead
> > to
> > > > > undefined behaviours...
> > > >
> > > > make sense! This commit is intend to re-setup the rc when waken
> > from
> > > > s2ram (in
> > > > s2ram state, the host lost power)
> > > >
> > > > I have no good solution but to introduce one function e.g
> > > > dw_pcie_setup_rc_after_linkup(), then move related code from
> > > > dw_pcie_host_init
> > > > to it, then let my host driver resume hook to call.
> > > >
> > > > Hi Pratyush, Jingoo and Bjorn etc.
> > > >
> > > > any suggestions are appreciated!
> > >
> > > What about:
> > >
> > > diff --git a/drivers/pci/host/pcie-designware.c
> > b/drivers/pci/host/pcie-designware.c
> > > index a4cccd3..e461f5d 100644
> > > --- a/drivers/pci/host/pcie-designware.c
> > > +++ b/drivers/pci/host/pcie-designware.c
> > > @@ -434,7 +434,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
> > >  	struct platform_device *pdev = to_platform_device(pp->dev);
> > >  	struct pci_bus *bus, *child;
> > >  	struct resource *cfg_res;
> > > -	u32 val;
> > >  	int i, ret;
> > >  	LIST_HEAD(res);
> > >  	struct resource_entry *win;
> > > @@ -544,25 +543,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
> > >  	if (pp->ops->host_init)
> > >  		pp->ops->host_init(pp);
> > >
> > > -	/*
> > > -	 * If the platform provides ->rd_other_conf, it means the
> > platform
> > > -	 * uses its own address translation component rather than ATU, so
> > > -	 * we should not program the ATU here.
> > > -	 */
> > > -	if (!pp->ops->rd_other_conf)
> > > -		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> > > -					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> > > -					  pp->mem_bus_addr, pp->mem_size);
> > > -
> > > -	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> > > -
> > > -	/* program correct class for RC */
> > > -	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2,
> > PCI_CLASS_BRIDGE_PCI);
> > > -
> > > -	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> > > -	val |= PORT_LOGIC_SPEED_CHANGE;
> > > -	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
> > > -
> > >  	pp->root_bus_nr = pp->busn->start;
> > >  	if (IS_ENABLED(CONFIG_PCI_MSI)) {
> > >  		bus = pci_scan_root_bus_msi(pp->dev, pp->root_bus_nr,
> > > @@ -725,6 +705,29 @@ static struct pci_ops dw_pcie_ops = {
> > >  	.write = dw_pcie_wr_conf,
> > >  };
> > >
> > > +void dw_pcie_setup_own_cfg(struct pcie_port *pp)
> > > +{
> > > +	u32 val;
> > > +	/*
> > > +	 * If the platform provides ->rd_other_conf, it means the
> > platform
> > > +	 * uses its own address translation component rather than ATU, so
> > > +	 * we should not program the ATU here.
> > > +	 */
> > > +	if (!pp->ops->rd_other_conf)
> > > +		dw_pcie_prog_outbound_atu(pp, PCIE_ATU_REGION_INDEX1,
> > > +					  PCIE_ATU_TYPE_MEM, pp->mem_base,
> > > +					  pp->mem_bus_addr, pp->mem_size);
> > > +
> > > +	dw_pcie_wr_own_conf(pp, PCI_BASE_ADDRESS_0, 4, 0);
> > > +
> > > +	/* program correct class for RC */
> > > +	dw_pcie_wr_own_conf(pp, PCI_CLASS_DEVICE, 2,
> > PCI_CLASS_BRIDGE_PCI);
> > > +
> > > +	dw_pcie_rd_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, &val);
> > > +	val |= PORT_LOGIC_SPEED_CHANGE;
> > > +	dw_pcie_wr_own_conf(pp, PCIE_LINK_WIDTH_SPEED_CONTROL, 4, val);
> > > +}
> > > +
> > >  void dw_pcie_setup_rc(struct pcie_port *pp)
> > >  {
> > >  	u32 val;
> > > @@ -800,6 +803,8 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
> > >  	val |= PCI_COMMAND_IO | PCI_COMMAND_MEMORY |
> > >  		PCI_COMMAND_MASTER | PCI_COMMAND_SERR;
> > >  	dw_pcie_writel_rc(pp, val, PCI_COMMAND);
> > > +
> > > +	dw_pcie_setup_own_cfg(pp);
> > >  }
> > >
> > >  MODULE_AUTHOR("Jingoo Han <jg1.han@samsung.com>");
> > > diff --git a/drivers/pci/host/pcie-designware.h
> > b/drivers/pci/host/pcie-designware.h
> > > index f437f9b..caf0f5d 100644
> > > --- a/drivers/pci/host/pcie-designware.h
> > > +++ b/drivers/pci/host/pcie-designware.h
> > > @@ -85,5 +85,6 @@ int dw_pcie_wait_for_link(struct pcie_port *pp);
> > >  int dw_pcie_link_up(struct pcie_port *pp);
> > >  void dw_pcie_setup_rc(struct pcie_port *pp);
> > >  int dw_pcie_host_init(struct pcie_port *pp);
> > > +void dw_pcie_setup_own_cfg(struct pcie_port *pp);
> > >
> > >  #endif /* _PCIE_DESIGNWARE_H */
> > > diff --git a/drivers/pci/host/pcie-hisi.c b/drivers/pci/host/pcie-
> > hisi.c
> > > index 3e98d4e..8da29b2 100644
> > > --- a/drivers/pci/host/pcie-hisi.c
> > > +++ b/drivers/pci/host/pcie-hisi.c
> > > @@ -164,6 +164,7 @@ static int hisi_add_pcie_port(struct pcie_port
> > *pp,
> > >  		dev_err(&pdev->dev, "failed to initialize host\n");
> > >  		return ret;
> > >  	}
> > > +	dw_pcie_setup_own_cfg(pp);
> > >
> > >  	return 0;
> > >  }
> > 
> > What's the hisi plan for resuming after suspend-to-RAM?  How does the
> > RC get reprogrammed after it loses all its state?
> 
> PM is not part of the driver yet. This is planned for near
> future release so haven't made such considerations yet
> > 
> > What would break if hisi did call dw_pcie_setup_rc()?  I know you said
> > it would overwrite what the bootloader already did, which is true.
> 
> I am try to figure this out now with our HW team.
> 
> > 
> > But hisi does call dw_pcie_host_init(), so it reads pp->mem (which
> > determines pp->mem_base) and pp->lanes from the DT.  Other drivers
> > then call dw_pcie_setup_rc() which programs the RC based on
> > pp->mem_base and pp->lanes.  So hisi assumes UEFI programmed the RC to
> > match the DT, while the other drivers read the DT and program the RC
> > to match.  The latter seems more robust because it enforces the
> > consistency rather than relying on it.
> 
> Yes I agree with you, however we have preferred to move RC config to
> BIOS to have a single driver to support multiple versions of the
> same SoC.

I think there are two reasonable approaches:

  1) A single generic driver that doesn't have any knowledge about the
  chipset registers; it uses run-time firmware interfaces to manage
  the bridge.  The ACPI pci_root.c driver is the best example so far
  and works very well.  It supports basically all x86 and ia64
  chipsets and requires no kernel work for new ones.

  2) Native drivers specific to each chipset.  These may get
  configuration information from DT, but they do their own
  register-level programming of the device without run-time help from
  firmware.

I think hisi is a native driver because it uses hip05/hip06 registers
to check link state and perform config operations.  And apparently you
rely on the ATU, BAR, class, and link width programming currently done
in dw_pcie_host_init().  But you want to rely on pre-boot firmware to
set up the link.  That doesn't make sense to me -- if the driver wants
to twiddle the registers, it should know how to do it all.  I don't
see how you can reasonably manage this half-way approach.

> The patch I proposed above does the same job as the original patch
> proposed by Jisheng and also allows hisi driver to call the moved
> code.
> 
> Do you see anything wrong with it?

Only that it makes the structure more complicated and we haven't
identified a corresponding benefit yet.

Bjorn

RE: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Gabriele Paoloni]

Hi Bjorn

[...]

> > >
> > > What's the hisi plan for resuming after suspend-to-RAM?  How does
> the
> > > RC get reprogrammed after it loses all its state?
> >
> > PM is not part of the driver yet. This is planned for near
> > future release so haven't made such considerations yet
> > >
> > > What would break if hisi did call dw_pcie_setup_rc()?  I know you
> said
> > > it would overwrite what the bootloader already did, which is true.
> >
> > I am try to figure this out now with our HW team.
> >
> > >
> > > But hisi does call dw_pcie_host_init(), so it reads pp->mem (which
> > > determines pp->mem_base) and pp->lanes from the DT.  Other drivers
> > > then call dw_pcie_setup_rc() which programs the RC based on
> > > pp->mem_base and pp->lanes.  So hisi assumes UEFI programmed the RC
> to
> > > match the DT, while the other drivers read the DT and program the
> RC
> > > to match.  The latter seems more robust because it enforces the
> > > consistency rather than relying on it.
> >
> > Yes I agree with you, however we have preferred to move RC config to
> > BIOS to have a single driver to support multiple versions of the
> > same SoC.
> 
> I think there are two reasonable approaches:
> 
>   1) A single generic driver that doesn't have any knowledge about the
>   chipset registers; it uses run-time firmware interfaces to manage
>   the bridge.  The ACPI pci_root.c driver is the best example so far
>   and works very well.  It supports basically all x86 and ia64
>   chipsets and requires no kernel work for new ones.
> 
>   2) Native drivers specific to each chipset.  These may get
>   configuration information from DT, but they do their own
>   register-level programming of the device without run-time help from
>   firmware.
> 
> I think hisi is a native driver because it uses hip05/hip06 registers
> to check link state and perform config operations.  And apparently you
> rely on the ATU, BAR, class, and link width programming currently done
> in dw_pcie_host_init().  But you want to rely on pre-boot firmware to
> set up the link.  That doesn't make sense to me -- if the driver wants
> to twiddle the registers, it should know how to do it all.  I don't
> see how you can reasonably manage this half-way approach.
> 
> > The patch I proposed above does the same job as the original patch
> > proposed by Jisheng and also allows hisi driver to call the moved
> > code.
> >
> > Do you see anything wrong with it?
> 
> Only that it makes the structure more complicated and we haven't
> identified a corresponding benefit yet.

Finally I have checked that assigning .host_init function pointer
in our driver to call dw_pcie_setup_rc() will not affect the values
already set by BIOS.

Also I agree with you that a hybrid approach is not ideal.

So I will update the driver to call dw_pcie_setup_rc() from
.host_init and ask the BIOS team to update the firmware for next
releases (the driver will be backward compatible anyway). 

Also during my investigation I have noticed that in dw_pcie_setup_rc()
http://lxr.free-electrons.com/source/drivers/pci/host/pcie-designware.c#L762

we use pp->mem_base rather than pp->mem_bus_addr to setup
memory base and memory limit in the Type1 header...I think this
is wrong right?
Also I do not see why this code is needed at all since we overwrite
this register when we call pci_bus_assign_resources(bus) that
will end up in calling pci_setup_bridge() and then
pci_setup_bridge_mmio()...?  

Many Thanks

Gab

> 
> Bjorn
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Gabriele Paoloni]

Hi Bjorn

[...]

> > >
> > > What's the hisi plan for resuming after suspend-to-RAM?  How does
> the
> > > RC get reprogrammed after it loses all its state?
> >
> > PM is not part of the driver yet. This is planned for near
> > future release so haven't made such considerations yet
> > >
> > > What would break if hisi did call dw_pcie_setup_rc()?  I know you
> said
> > > it would overwrite what the bootloader already did, which is true.
> >
> > I am try to figure this out now with our HW team.
> >
> > >
> > > But hisi does call dw_pcie_host_init(), so it reads pp->mem (which
> > > determines pp->mem_base) and pp->lanes from the DT.  Other drivers
> > > then call dw_pcie_setup_rc() which programs the RC based on
> > > pp->mem_base and pp->lanes.  So hisi assumes UEFI programmed the RC
> to
> > > match the DT, while the other drivers read the DT and program the
> RC
> > > to match.  The latter seems more robust because it enforces the
> > > consistency rather than relying on it.
> >
> > Yes I agree with you, however we have preferred to move RC config to
> > BIOS to have a single driver to support multiple versions of the
> > same SoC.
> 
> I think there are two reasonable approaches:
> 
>   1) A single generic driver that doesn't have any knowledge about the
>   chipset registers; it uses run-time firmware interfaces to manage
>   the bridge.  The ACPI pci_root.c driver is the best example so far
>   and works very well.  It supports basically all x86 and ia64
>   chipsets and requires no kernel work for new ones.
> 
>   2) Native drivers specific to each chipset.  These may get
>   configuration information from DT, but they do their own
>   register-level programming of the device without run-time help from
>   firmware.
> 
> I think hisi is a native driver because it uses hip05/hip06 registers
> to check link state and perform config operations.  And apparently you
> rely on the ATU, BAR, class, and link width programming currently done
> in dw_pcie_host_init().  But you want to rely on pre-boot firmware to
> set up the link.  That doesn't make sense to me -- if the driver wants
> to twiddle the registers, it should know how to do it all.  I don't
> see how you can reasonably manage this half-way approach.
> 
> > The patch I proposed above does the same job as the original patch
> > proposed by Jisheng and also allows hisi driver to call the moved
> > code.
> >
> > Do you see anything wrong with it?
> 
> Only that it makes the structure more complicated and we haven't
> identified a corresponding benefit yet.

Finally I have checked that assigning .host_init function pointer
in our driver to call dw_pcie_setup_rc() will not affect the values
already set by BIOS.

Also I agree with you that a hybrid approach is not ideal.

So I will update the driver to call dw_pcie_setup_rc() from
.host_init and ask the BIOS team to update the firmware for next
releases (the driver will be backward compatible anyway). 

Also during my investigation I have noticed that in dw_pcie_setup_rc()
http://lxr.free-electrons.com/source/drivers/pci/host/pcie-designware.c#L762

we use pp->mem_base rather than pp->mem_bus_addr to setup
memory base and memory limit in the Type1 header...I think this
is wrong right?
Also I do not see why this code is needed at all since we overwrite
this register when we call pci_bus_assign_resources(bus) that
will end up in calling pci_setup_bridge() and then
pci_setup_bridge_mmio()...?  

Many Thanks

Gab

> 
> Bjorn
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Gabriele Paoloni]

Hi Bjorn

[...]

> > >
> > > What's the hisi plan for resuming after suspend-to-RAM?  How does
> the
> > > RC get reprogrammed after it loses all its state?
> >
> > PM is not part of the driver yet. This is planned for near
> > future release so haven't made such considerations yet
> > >
> > > What would break if hisi did call dw_pcie_setup_rc()?  I know you
> said
> > > it would overwrite what the bootloader already did, which is true.
> >
> > I am try to figure this out now with our HW team.
> >
> > >
> > > But hisi does call dw_pcie_host_init(), so it reads pp->mem (which
> > > determines pp->mem_base) and pp->lanes from the DT.  Other drivers
> > > then call dw_pcie_setup_rc() which programs the RC based on
> > > pp->mem_base and pp->lanes.  So hisi assumes UEFI programmed the RC
> to
> > > match the DT, while the other drivers read the DT and program the
> RC
> > > to match.  The latter seems more robust because it enforces the
> > > consistency rather than relying on it.
> >
> > Yes I agree with you, however we have preferred to move RC config to
> > BIOS to have a single driver to support multiple versions of the
> > same SoC.
> 
> I think there are two reasonable approaches:
> 
>   1) A single generic driver that doesn't have any knowledge about the
>   chipset registers; it uses run-time firmware interfaces to manage
>   the bridge.  The ACPI pci_root.c driver is the best example so far
>   and works very well.  It supports basically all x86 and ia64
>   chipsets and requires no kernel work for new ones.
> 
>   2) Native drivers specific to each chipset.  These may get
>   configuration information from DT, but they do their own
>   register-level programming of the device without run-time help from
>   firmware.
> 
> I think hisi is a native driver because it uses hip05/hip06 registers
> to check link state and perform config operations.  And apparently you
> rely on the ATU, BAR, class, and link width programming currently done
> in dw_pcie_host_init().  But you want to rely on pre-boot firmware to
> set up the link.  That doesn't make sense to me -- if the driver wants
> to twiddle the registers, it should know how to do it all.  I don't
> see how you can reasonably manage this half-way approach.
> 
> > The patch I proposed above does the same job as the original patch
> > proposed by Jisheng and also allows hisi driver to call the moved
> > code.
> >
> > Do you see anything wrong with it?
> 
> Only that it makes the structure more complicated and we haven't
> identified a corresponding benefit yet.

Finally I have checked that assigning .host_init function pointer
in our driver to call dw_pcie_setup_rc() will not affect the values
already set by BIOS.

Also I agree with you that a hybrid approach is not ideal.

So I will update the driver to call dw_pcie_setup_rc() from
.host_init and ask the BIOS team to update the firmware for next
releases (the driver will be backward compatible anyway). 

Also during my investigation I have noticed that in dw_pcie_setup_rc()
http://lxr.free-electrons.com/source/drivers/pci/host/pcie-designware.c#L762

we use pp->mem_base rather than pp->mem_bus_addr to setup
memory base and memory limit in the Type1 header...I think this
is wrong right?
Also I do not see why this code is needed at all since we overwrite
this register when we call pci_bus_assign_resources(bus) that
will end up in calling pci_setup_bridge() and then
pci_setup_bridge_mmio()...?  

Many Thanks

Gab

> 
> Bjorn
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Jingoo Han]

On Tuesday, April 12, 2016 6:44 PM, Gabriele Paoloni wrote:
> 
> Hi Bjorn
> 
> [...]
> 
> > > >
> > > > What's the hisi plan for resuming after suspend-to-RAM?  How does
> > the
> > > > RC get reprogrammed after it loses all its state?
> > >
> > > PM is not part of the driver yet. This is planned for near
> > > future release so haven't made such considerations yet
> > > >
> > > > What would break if hisi did call dw_pcie_setup_rc()?  I know you
> > said
> > > > it would overwrite what the bootloader already did, which is true.
> > >
> > > I am try to figure this out now with our HW team.
> > >
> > > >
> > > > But hisi does call dw_pcie_host_init(), so it reads pp->mem (which
> > > > determines pp->mem_base) and pp->lanes from the DT.  Other drivers
> > > > then call dw_pcie_setup_rc() which programs the RC based on
> > > > pp->mem_base and pp->lanes.  So hisi assumes UEFI programmed the RC
> > to
> > > > match the DT, while the other drivers read the DT and program the
> > RC
> > > > to match.  The latter seems more robust because it enforces the
> > > > consistency rather than relying on it.
> > >
> > > Yes I agree with you, however we have preferred to move RC config to
> > > BIOS to have a single driver to support multiple versions of the
> > > same SoC.
> >
> > I think there are two reasonable approaches:
> >
> >   1) A single generic driver that doesn't have any knowledge about the
> >   chipset registers; it uses run-time firmware interfaces to manage
> >   the bridge.  The ACPI pci_root.c driver is the best example so far
> >   and works very well.  It supports basically all x86 and ia64
> >   chipsets and requires no kernel work for new ones.
> >
> >   2) Native drivers specific to each chipset.  These may get
> >   configuration information from DT, but they do their own
> >   register-level programming of the device without run-time help from
> >   firmware.
> >
> > I think hisi is a native driver because it uses hip05/hip06 registers
> > to check link state and perform config operations.  And apparently you
> > rely on the ATU, BAR, class, and link width programming currently done
> > in dw_pcie_host_init().  But you want to rely on pre-boot firmware to
> > set up the link.  That doesn't make sense to me -- if the driver wants
> > to twiddle the registers, it should know how to do it all.  I don't
> > see how you can reasonably manage this half-way approach.
> >
> > > The patch I proposed above does the same job as the original patch
> > > proposed by Jisheng and also allows hisi driver to call the moved
> > > code.
> > >
> > > Do you see anything wrong with it?
> >
> > Only that it makes the structure more complicated and we haven't
> > identified a corresponding benefit yet.
> 
> Finally I have checked that assigning .host_init function pointer
> in our driver to call dw_pcie_setup_rc() will not affect the values
> already set by BIOS.
> 
> Also I agree with you that a hybrid approach is not ideal.

I also agree with Bjorn's opinion.
As far as I know, two approaches are reasonable.

In the case of using UEFI, how about using 'pci-host-generic.c'?
You may consult with Linaro guys for this issue.
Good luck.

Best regards,
Jingoo Han

> 
> So I will update the driver to call dw_pcie_setup_rc() from
> .host_init and ask the BIOS team to update the firmware for next
> releases (the driver will be backward compatible anyway).
> 
> Also during my investigation I have noticed that in dw_pcie_setup_rc()
> http://lxr.free-electrons.com/source/drivers/pci/host/pcie-designware.c#L762
> 
> we use pp->mem_base rather than pp->mem_bus_addr to setup
> memory base and memory limit in the Type1 header...I think this
> is wrong right?
> Also I do not see why this code is needed at all since we overwrite
> this register when we call pci_bus_assign_resources(bus) that
> will end up in calling pci_setup_bridge() and then
> pci_setup_bridge_mmio()...?
> 
> Many Thanks
> 
> Gab
> 
> >
> > Bjorn
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Jingoo Han]

On Tuesday, April 12, 2016 6:44 PM, Gabriele Paoloni wrote:
> 
> Hi Bjorn
> 
> [...]
> 
> > > >
> > > > What's the hisi plan for resuming after suspend-to-RAM?  How does
> > the
> > > > RC get reprogrammed after it loses all its state?
> > >
> > > PM is not part of the driver yet. This is planned for near
> > > future release so haven't made such considerations yet
> > > >
> > > > What would break if hisi did call dw_pcie_setup_rc()?  I know you
> > said
> > > > it would overwrite what the bootloader already did, which is true.
> > >
> > > I am try to figure this out now with our HW team.
> > >
> > > >
> > > > But hisi does call dw_pcie_host_init(), so it reads pp->mem (which
> > > > determines pp->mem_base) and pp->lanes from the DT.  Other drivers
> > > > then call dw_pcie_setup_rc() which programs the RC based on
> > > > pp->mem_base and pp->lanes.  So hisi assumes UEFI programmed the RC
> > to
> > > > match the DT, while the other drivers read the DT and program the
> > RC
> > > > to match.  The latter seems more robust because it enforces the
> > > > consistency rather than relying on it.
> > >
> > > Yes I agree with you, however we have preferred to move RC config to
> > > BIOS to have a single driver to support multiple versions of the
> > > same SoC.
> >
> > I think there are two reasonable approaches:
> >
> >   1) A single generic driver that doesn't have any knowledge about the
> >   chipset registers; it uses run-time firmware interfaces to manage
> >   the bridge.  The ACPI pci_root.c driver is the best example so far
> >   and works very well.  It supports basically all x86 and ia64
> >   chipsets and requires no kernel work for new ones.
> >
> >   2) Native drivers specific to each chipset.  These may get
> >   configuration information from DT, but they do their own
> >   register-level programming of the device without run-time help from
> >   firmware.
> >
> > I think hisi is a native driver because it uses hip05/hip06 registers
> > to check link state and perform config operations.  And apparently you
> > rely on the ATU, BAR, class, and link width programming currently done
> > in dw_pcie_host_init().  But you want to rely on pre-boot firmware to
> > set up the link.  That doesn't make sense to me -- if the driver wants
> > to twiddle the registers, it should know how to do it all.  I don't
> > see how you can reasonably manage this half-way approach.
> >
> > > The patch I proposed above does the same job as the original patch
> > > proposed by Jisheng and also allows hisi driver to call the moved
> > > code.
> > >
> > > Do you see anything wrong with it?
> >
> > Only that it makes the structure more complicated and we haven't
> > identified a corresponding benefit yet.
> 
> Finally I have checked that assigning .host_init function pointer
> in our driver to call dw_pcie_setup_rc() will not affect the values
> already set by BIOS.
> 
> Also I agree with you that a hybrid approach is not ideal.

I also agree with Bjorn's opinion.
As far as I know, two approaches are reasonable.

In the case of using UEFI, how about using 'pci-host-generic.c'?
You may consult with Linaro guys for this issue.
Good luck.

Best regards,
Jingoo Han

> 
> So I will update the driver to call dw_pcie_setup_rc() from
> .host_init and ask the BIOS team to update the firmware for next
> releases (the driver will be backward compatible anyway).
> 
> Also during my investigation I have noticed that in dw_pcie_setup_rc()
> http://lxr.free-electrons.com/source/drivers/pci/host/pcie-designware.c#L762
> 
> we use pp->mem_base rather than pp->mem_bus_addr to setup
> memory base and memory limit in the Type1 header...I think this
> is wrong right?
> Also I do not see why this code is needed at all since we overwrite
> this register when we call pci_bus_assign_resources(bus) that
> will end up in calling pci_setup_bridge() and then
> pci_setup_bridge_mmio()...?
> 
> Many Thanks
> 
> Gab
> 
> >
> > Bjorn
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> > the body of a message to majordomo at vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Gabriele Paoloni]

Hi Jingoo

> -----Original Message-----
> From: Jingoo Han [mailto:jingoohan1@gmail.com]
> Sent: 13 April 2016 06:52
> To: Gabriele Paoloni; 'Bjorn Helgaas'
> Cc: 'Jisheng Zhang'; pratyush.anand@gmail.com; bhelgaas@google.com;
> linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org; 'Jingoo Han'
> Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> to dw_pcie_setup_rc()
> 
> On Tuesday, April 12, 2016 6:44 PM, Gabriele Paoloni wrote:
> >
> > Hi Bjorn
> >
> > [...]
> >
> > > > >
> > > > > What's the hisi plan for resuming after suspend-to-RAM?  How
> does
> > > the
> > > > > RC get reprogrammed after it loses all its state?
> > > >
> > > > PM is not part of the driver yet. This is planned for near
> > > > future release so haven't made such considerations yet
> > > > >
> > > > > What would break if hisi did call dw_pcie_setup_rc()?  I know
> you
> > > said
> > > > > it would overwrite what the bootloader already did, which is
> true.
> > > >
> > > > I am try to figure this out now with our HW team.
> > > >
> > > > >
> > > > > But hisi does call dw_pcie_host_init(), so it reads pp->mem
> (which
> > > > > determines pp->mem_base) and pp->lanes from the DT.  Other
> drivers
> > > > > then call dw_pcie_setup_rc() which programs the RC based on
> > > > > pp->mem_base and pp->lanes.  So hisi assumes UEFI programmed
> the RC
> > > to
> > > > > match the DT, while the other drivers read the DT and program
> the
> > > RC
> > > > > to match.  The latter seems more robust because it enforces the
> > > > > consistency rather than relying on it.
> > > >
> > > > Yes I agree with you, however we have preferred to move RC config
> to
> > > > BIOS to have a single driver to support multiple versions of the
> > > > same SoC.
> > >
> > > I think there are two reasonable approaches:
> > >
> > >   1) A single generic driver that doesn't have any knowledge about
> the
> > >   chipset registers; it uses run-time firmware interfaces to manage
> > >   the bridge.  The ACPI pci_root.c driver is the best example so
> far
> > >   and works very well.  It supports basically all x86 and ia64
> > >   chipsets and requires no kernel work for new ones.
> > >
> > >   2) Native drivers specific to each chipset.  These may get
> > >   configuration information from DT, but they do their own
> > >   register-level programming of the device without run-time help
> from
> > >   firmware.
> > >
> > > I think hisi is a native driver because it uses hip05/hip06
> registers
> > > to check link state and perform config operations.  And apparently
> you
> > > rely on the ATU, BAR, class, and link width programming currently
> done
> > > in dw_pcie_host_init().  But you want to rely on pre-boot firmware
> to
> > > set up the link.  That doesn't make sense to me -- if the driver
> wants
> > > to twiddle the registers, it should know how to do it all.  I don't
> > > see how you can reasonably manage this half-way approach.
> > >
> > > > The patch I proposed above does the same job as the original
> patch
> > > > proposed by Jisheng and also allows hisi driver to call the moved
> > > > code.
> > > >
> > > > Do you see anything wrong with it?
> > >
> > > Only that it makes the structure more complicated and we haven't
> > > identified a corresponding benefit yet.
> >
> > Finally I have checked that assigning .host_init function pointer
> > in our driver to call dw_pcie_setup_rc() will not affect the values
> > already set by BIOS.
> >
> > Also I agree with you that a hybrid approach is not ideal.
> 
> I also agree with Bjorn's opinion.
> As far as I know, two approaches are reasonable.
> 
> In the case of using UEFI, how about using 'pci-host-generic.c'?
> You may consult with Linaro guys for this issue.

Many thanks for your suggestion, I'll take it into account for next
releases

> Good luck.
> 
> Best regards,
> Jingoo Han
> 
> >
> > So I will update the driver to call dw_pcie_setup_rc() from
> > .host_init and ask the BIOS team to update the firmware for next
> > releases (the driver will be backward compatible anyway).
> >
> > Also during my investigation I have noticed that in
> dw_pcie_setup_rc()
> > http://lxr.free-electrons.com/source/drivers/pci/host/pcie-
> designware.c#L762
> >
> > we use pp->mem_base rather than pp->mem_bus_addr to setup
> > memory base and memory limit in the Type1 header...I think this
> > is wrong right?
> > Also I do not see why this code is needed at all since we overwrite
> > this register when we call pci_bus_assign_resources(bus) that
> > will end up in calling pci_setup_bridge() and then
> > pci_setup_bridge_mmio()...?

Do you have any comment on this issue above?

> >
> > Many Thanks
> >
> > Gab
> >
> > >
> > > Bjorn
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-
> pci" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Gabriele Paoloni]

Hi Jingoo

> -----Original Message-----
> From: Jingoo Han [mailto:jingoohan1@gmail.com]
> Sent: 13 April 2016 06:52
> To: Gabriele Paoloni; 'Bjorn Helgaas'
> Cc: 'Jisheng Zhang'; pratyush.anand@gmail.com; bhelgaas@google.com;
> linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org; 'Jingoo Han'
> Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> to dw_pcie_setup_rc()
> 
> On Tuesday, April 12, 2016 6:44 PM, Gabriele Paoloni wrote:
> >
> > Hi Bjorn
> >
> > [...]
> >
> > > > >
> > > > > What's the hisi plan for resuming after suspend-to-RAM?  How
> does
> > > the
> > > > > RC get reprogrammed after it loses all its state?
> > > >
> > > > PM is not part of the driver yet. This is planned for near
> > > > future release so haven't made such considerations yet
> > > > >
> > > > > What would break if hisi did call dw_pcie_setup_rc()?  I know
> you
> > > said
> > > > > it would overwrite what the bootloader already did, which is
> true.
> > > >
> > > > I am try to figure this out now with our HW team.
> > > >
> > > > >
> > > > > But hisi does call dw_pcie_host_init(), so it reads pp->mem
> (which
> > > > > determines pp->mem_base) and pp->lanes from the DT.  Other
> drivers
> > > > > then call dw_pcie_setup_rc() which programs the RC based on
> > > > > pp->mem_base and pp->lanes.  So hisi assumes UEFI programmed
> the RC
> > > to
> > > > > match the DT, while the other drivers read the DT and program
> the
> > > RC
> > > > > to match.  The latter seems more robust because it enforces the
> > > > > consistency rather than relying on it.
> > > >
> > > > Yes I agree with you, however we have preferred to move RC config
> to
> > > > BIOS to have a single driver to support multiple versions of the
> > > > same SoC.
> > >
> > > I think there are two reasonable approaches:
> > >
> > >   1) A single generic driver that doesn't have any knowledge about
> the
> > >   chipset registers; it uses run-time firmware interfaces to manage
> > >   the bridge.  The ACPI pci_root.c driver is the best example so
> far
> > >   and works very well.  It supports basically all x86 and ia64
> > >   chipsets and requires no kernel work for new ones.
> > >
> > >   2) Native drivers specific to each chipset.  These may get
> > >   configuration information from DT, but they do their own
> > >   register-level programming of the device without run-time help
> from
> > >   firmware.
> > >
> > > I think hisi is a native driver because it uses hip05/hip06
> registers
> > > to check link state and perform config operations.  And apparently
> you
> > > rely on the ATU, BAR, class, and link width programming currently
> done
> > > in dw_pcie_host_init().  But you want to rely on pre-boot firmware
> to
> > > set up the link.  That doesn't make sense to me -- if the driver
> wants
> > > to twiddle the registers, it should know how to do it all.  I don't
> > > see how you can reasonably manage this half-way approach.
> > >
> > > > The patch I proposed above does the same job as the original
> patch
> > > > proposed by Jisheng and also allows hisi driver to call the moved
> > > > code.
> > > >
> > > > Do you see anything wrong with it?
> > >
> > > Only that it makes the structure more complicated and we haven't
> > > identified a corresponding benefit yet.
> >
> > Finally I have checked that assigning .host_init function pointer
> > in our driver to call dw_pcie_setup_rc() will not affect the values
> > already set by BIOS.
> >
> > Also I agree with you that a hybrid approach is not ideal.
> 
> I also agree with Bjorn's opinion.
> As far as I know, two approaches are reasonable.
> 
> In the case of using UEFI, how about using 'pci-host-generic.c'?
> You may consult with Linaro guys for this issue.

Many thanks for your suggestion, I'll take it into account for next
releases

> Good luck.
> 
> Best regards,
> Jingoo Han
> 
> >
> > So I will update the driver to call dw_pcie_setup_rc() from
> > .host_init and ask the BIOS team to update the firmware for next
> > releases (the driver will be backward compatible anyway).
> >
> > Also during my investigation I have noticed that in
> dw_pcie_setup_rc()
> > http://lxr.free-electrons.com/source/drivers/pci/host/pcie-
> designware.c#L762
> >
> > we use pp->mem_base rather than pp->mem_bus_addr to setup
> > memory base and memory limit in the Type1 header...I think this
> > is wrong right?
> > Also I do not see why this code is needed at all since we overwrite
> > this register when we call pci_bus_assign_resources(bus) that
> > will end up in calling pci_setup_bridge() and then
> > pci_setup_bridge_mmio()...?

Do you have any comment on this issue above?

> >
> > Many Thanks
> >
> > Gab
> >
> > >
> > > Bjorn
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-
> pci" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Gabriele Paoloni]

Hi Jingoo

> -----Original Message-----
> From: Jingoo Han [mailto:jingoohan1 at gmail.com]
> Sent: 13 April 2016 06:52
> To: Gabriele Paoloni; 'Bjorn Helgaas'
> Cc: 'Jisheng Zhang'; pratyush.anand at gmail.com; bhelgaas at google.com;
> linux-pci at vger.kernel.org; linux-kernel at vger.kernel.org; linux-arm-
> kernel at lists.infradead.org; 'Jingoo Han'
> Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> to dw_pcie_setup_rc()
> 
> On Tuesday, April 12, 2016 6:44 PM, Gabriele Paoloni wrote:
> >
> > Hi Bjorn
> >
> > [...]
> >
> > > > >
> > > > > What's the hisi plan for resuming after suspend-to-RAM?  How
> does
> > > the
> > > > > RC get reprogrammed after it loses all its state?
> > > >
> > > > PM is not part of the driver yet. This is planned for near
> > > > future release so haven't made such considerations yet
> > > > >
> > > > > What would break if hisi did call dw_pcie_setup_rc()?  I know
> you
> > > said
> > > > > it would overwrite what the bootloader already did, which is
> true.
> > > >
> > > > I am try to figure this out now with our HW team.
> > > >
> > > > >
> > > > > But hisi does call dw_pcie_host_init(), so it reads pp->mem
> (which
> > > > > determines pp->mem_base) and pp->lanes from the DT.  Other
> drivers
> > > > > then call dw_pcie_setup_rc() which programs the RC based on
> > > > > pp->mem_base and pp->lanes.  So hisi assumes UEFI programmed
> the RC
> > > to
> > > > > match the DT, while the other drivers read the DT and program
> the
> > > RC
> > > > > to match.  The latter seems more robust because it enforces the
> > > > > consistency rather than relying on it.
> > > >
> > > > Yes I agree with you, however we have preferred to move RC config
> to
> > > > BIOS to have a single driver to support multiple versions of the
> > > > same SoC.
> > >
> > > I think there are two reasonable approaches:
> > >
> > >   1) A single generic driver that doesn't have any knowledge about
> the
> > >   chipset registers; it uses run-time firmware interfaces to manage
> > >   the bridge.  The ACPI pci_root.c driver is the best example so
> far
> > >   and works very well.  It supports basically all x86 and ia64
> > >   chipsets and requires no kernel work for new ones.
> > >
> > >   2) Native drivers specific to each chipset.  These may get
> > >   configuration information from DT, but they do their own
> > >   register-level programming of the device without run-time help
> from
> > >   firmware.
> > >
> > > I think hisi is a native driver because it uses hip05/hip06
> registers
> > > to check link state and perform config operations.  And apparently
> you
> > > rely on the ATU, BAR, class, and link width programming currently
> done
> > > in dw_pcie_host_init().  But you want to rely on pre-boot firmware
> to
> > > set up the link.  That doesn't make sense to me -- if the driver
> wants
> > > to twiddle the registers, it should know how to do it all.  I don't
> > > see how you can reasonably manage this half-way approach.
> > >
> > > > The patch I proposed above does the same job as the original
> patch
> > > > proposed by Jisheng and also allows hisi driver to call the moved
> > > > code.
> > > >
> > > > Do you see anything wrong with it?
> > >
> > > Only that it makes the structure more complicated and we haven't
> > > identified a corresponding benefit yet.
> >
> > Finally I have checked that assigning .host_init function pointer
> > in our driver to call dw_pcie_setup_rc() will not affect the values
> > already set by BIOS.
> >
> > Also I agree with you that a hybrid approach is not ideal.
> 
> I also agree with Bjorn's opinion.
> As far as I know, two approaches are reasonable.
> 
> In the case of using UEFI, how about using 'pci-host-generic.c'?
> You may consult with Linaro guys for this issue.

Many thanks for your suggestion, I'll take it into account for next
releases

> Good luck.
> 
> Best regards,
> Jingoo Han
> 
> >
> > So I will update the driver to call dw_pcie_setup_rc() from
> > .host_init and ask the BIOS team to update the firmware for next
> > releases (the driver will be backward compatible anyway).
> >
> > Also during my investigation I have noticed that in
> dw_pcie_setup_rc()
> > http://lxr.free-electrons.com/source/drivers/pci/host/pcie-
> designware.c#L762
> >
> > we use pp->mem_base rather than pp->mem_bus_addr to setup
> > memory base and memory limit in the Type1 header...I think this
> > is wrong right?
> > Also I do not see why this code is needed at all since we overwrite
> > this register when we call pci_bus_assign_resources(bus) that
> > will end up in calling pci_setup_bridge() and then
> > pci_setup_bridge_mmio()...?

Do you have any comment on this issue above?

> >
> > Many Thanks
> >
> > Gab
> >
> > >
> > > Bjorn
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-
> pci" in
> > > the body of a message to majordomo at vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Jingoo Han]

On Wednesday, April 13, 2016 4:58 PM, Gabriele Paoloni wrote:
> 
> Hi Jingoo
>
> On 13 April 2016 06:52, Jingoo Han wrote:
> > On Tuesday, April 12, 2016 6:44 PM, Gabriele Paoloni wrote:
> > >
> > > Hi Bjorn
> > >
> > > [...]
> > >
> > > > > >
> > > > > > What's the hisi plan for resuming after suspend-to-RAM?  How
> > does
> > > > the
> > > > > > RC get reprogrammed after it loses all its state?
> > > > >
> > > > > PM is not part of the driver yet. This is planned for near
> > > > > future release so haven't made such considerations yet
> > > > > >
> > > > > > What would break if hisi did call dw_pcie_setup_rc()?  I know
> > you
> > > > said
> > > > > > it would overwrite what the bootloader already did, which is
> > true.
> > > > >
> > > > > I am try to figure this out now with our HW team.
> > > > >
> > > > > >
> > > > > > But hisi does call dw_pcie_host_init(), so it reads pp->mem
> > (which
> > > > > > determines pp->mem_base) and pp->lanes from the DT.  Other
> > drivers
> > > > > > then call dw_pcie_setup_rc() which programs the RC based on
> > > > > > pp->mem_base and pp->lanes.  So hisi assumes UEFI programmed
> > the RC
> > > > to
> > > > > > match the DT, while the other drivers read the DT and program
> > the
> > > > RC
> > > > > > to match.  The latter seems more robust because it enforces the
> > > > > > consistency rather than relying on it.
> > > > >
> > > > > Yes I agree with you, however we have preferred to move RC config
> > to
> > > > > BIOS to have a single driver to support multiple versions of the
> > > > > same SoC.
> > > >
> > > > I think there are two reasonable approaches:
> > > >
> > > >   1) A single generic driver that doesn't have any knowledge about
> > the
> > > >   chipset registers; it uses run-time firmware interfaces to manage
> > > >   the bridge.  The ACPI pci_root.c driver is the best example so
> > far
> > > >   and works very well.  It supports basically all x86 and ia64
> > > >   chipsets and requires no kernel work for new ones.
> > > >
> > > >   2) Native drivers specific to each chipset.  These may get
> > > >   configuration information from DT, but they do their own
> > > >   register-level programming of the device without run-time help
> > from
> > > >   firmware.
> > > >
> > > > I think hisi is a native driver because it uses hip05/hip06
> > registers
> > > > to check link state and perform config operations.  And apparently
> > you
> > > > rely on the ATU, BAR, class, and link width programming currently
> > done
> > > > in dw_pcie_host_init().  But you want to rely on pre-boot firmware
> > to
> > > > set up the link.  That doesn't make sense to me -- if the driver
> > wants
> > > > to twiddle the registers, it should know how to do it all.  I don't
> > > > see how you can reasonably manage this half-way approach.
> > > >
> > > > > The patch I proposed above does the same job as the original
> > patch
> > > > > proposed by Jisheng and also allows hisi driver to call the moved
> > > > > code.
> > > > >
> > > > > Do you see anything wrong with it?
> > > >
> > > > Only that it makes the structure more complicated and we haven't
> > > > identified a corresponding benefit yet.
> > >
> > > Finally I have checked that assigning .host_init function pointer
> > > in our driver to call dw_pcie_setup_rc() will not affect the values
> > > already set by BIOS.
> > >
> > > Also I agree with you that a hybrid approach is not ideal.
> >
> > I also agree with Bjorn's opinion.
> > As far as I know, two approaches are reasonable.
> >
> > In the case of using UEFI, how about using 'pci-host-generic.c'?
> > You may consult with Linaro guys for this issue.
> 
> Many thanks for your suggestion, I'll take it into account for next
> releases
> 
> > Good luck.
> >
> > Best regards,
> > Jingoo Han
> >
> > >
> > > So I will update the driver to call dw_pcie_setup_rc() from
> > > .host_init and ask the BIOS team to update the firmware for next
> > > releases (the driver will be backward compatible anyway).
> > >
> > > Also during my investigation I have noticed that in
> > dw_pcie_setup_rc()
> > > http://lxr.free-electrons.com/source/drivers/pci/host/pcie-
> > designware.c#L762
> > >
> > > we use pp->mem_base rather than pp->mem_bus_addr to setup
> > > memory base and memory limit in the Type1 header...I think this
> > > is wrong right?
> > > Also I do not see why this code is needed at all since we overwrite
> > > this register when we call pci_bus_assign_resources(bus) that
> > > will end up in calling pci_setup_bridge() and then
> > > pci_setup_bridge_mmio()...?
> 
> Do you have any comment on this issue above?

Sorry, I am not sure.
However, there are some redundant codes like this.
At that time, I was not able to decide to remove these codes.

Maybe, Pratyush Anand or other guys would give opinions about this.

Best regards,
Jingoo Han

> 
> > >
> > > Many Thanks
> > >
> > > Gab
> > >
> > > >
> > > > Bjorn
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe linux-
> > pci" in
> > > > the body of a message to majordomo@vger.kernel.org
> > > > More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Jingoo Han]

On Wednesday, April 13, 2016 4:58 PM, Gabriele Paoloni wrote:
> 
> Hi Jingoo
>
> On 13 April 2016 06:52, Jingoo Han wrote:
> > On Tuesday, April 12, 2016 6:44 PM, Gabriele Paoloni wrote:
> > >
> > > Hi Bjorn
> > >
> > > [...]
> > >
> > > > > >
> > > > > > What's the hisi plan for resuming after suspend-to-RAM?  How
> > does
> > > > the
> > > > > > RC get reprogrammed after it loses all its state?
> > > > >
> > > > > PM is not part of the driver yet. This is planned for near
> > > > > future release so haven't made such considerations yet
> > > > > >
> > > > > > What would break if hisi did call dw_pcie_setup_rc()?  I know
> > you
> > > > said
> > > > > > it would overwrite what the bootloader already did, which is
> > true.
> > > > >
> > > > > I am try to figure this out now with our HW team.
> > > > >
> > > > > >
> > > > > > But hisi does call dw_pcie_host_init(), so it reads pp->mem
> > (which
> > > > > > determines pp->mem_base) and pp->lanes from the DT.  Other
> > drivers
> > > > > > then call dw_pcie_setup_rc() which programs the RC based on
> > > > > > pp->mem_base and pp->lanes.  So hisi assumes UEFI programmed
> > the RC
> > > > to
> > > > > > match the DT, while the other drivers read the DT and program
> > the
> > > > RC
> > > > > > to match.  The latter seems more robust because it enforces the
> > > > > > consistency rather than relying on it.
> > > > >
> > > > > Yes I agree with you, however we have preferred to move RC config
> > to
> > > > > BIOS to have a single driver to support multiple versions of the
> > > > > same SoC.
> > > >
> > > > I think there are two reasonable approaches:
> > > >
> > > >   1) A single generic driver that doesn't have any knowledge about
> > the
> > > >   chipset registers; it uses run-time firmware interfaces to manage
> > > >   the bridge.  The ACPI pci_root.c driver is the best example so
> > far
> > > >   and works very well.  It supports basically all x86 and ia64
> > > >   chipsets and requires no kernel work for new ones.
> > > >
> > > >   2) Native drivers specific to each chipset.  These may get
> > > >   configuration information from DT, but they do their own
> > > >   register-level programming of the device without run-time help
> > from
> > > >   firmware.
> > > >
> > > > I think hisi is a native driver because it uses hip05/hip06
> > registers
> > > > to check link state and perform config operations.  And apparently
> > you
> > > > rely on the ATU, BAR, class, and link width programming currently
> > done
> > > > in dw_pcie_host_init().  But you want to rely on pre-boot firmware
> > to
> > > > set up the link.  That doesn't make sense to me -- if the driver
> > wants
> > > > to twiddle the registers, it should know how to do it all.  I don't
> > > > see how you can reasonably manage this half-way approach.
> > > >
> > > > > The patch I proposed above does the same job as the original
> > patch
> > > > > proposed by Jisheng and also allows hisi driver to call the moved
> > > > > code.
> > > > >
> > > > > Do you see anything wrong with it?
> > > >
> > > > Only that it makes the structure more complicated and we haven't
> > > > identified a corresponding benefit yet.
> > >
> > > Finally I have checked that assigning .host_init function pointer
> > > in our driver to call dw_pcie_setup_rc() will not affect the values
> > > already set by BIOS.
> > >
> > > Also I agree with you that a hybrid approach is not ideal.
> >
> > I also agree with Bjorn's opinion.
> > As far as I know, two approaches are reasonable.
> >
> > In the case of using UEFI, how about using 'pci-host-generic.c'?
> > You may consult with Linaro guys for this issue.
> 
> Many thanks for your suggestion, I'll take it into account for next
> releases
> 
> > Good luck.
> >
> > Best regards,
> > Jingoo Han
> >
> > >
> > > So I will update the driver to call dw_pcie_setup_rc() from
> > > .host_init and ask the BIOS team to update the firmware for next
> > > releases (the driver will be backward compatible anyway).
> > >
> > > Also during my investigation I have noticed that in
> > dw_pcie_setup_rc()
> > > http://lxr.free-electrons.com/source/drivers/pci/host/pcie-
> > designware.c#L762
> > >
> > > we use pp->mem_base rather than pp->mem_bus_addr to setup
> > > memory base and memory limit in the Type1 header...I think this
> > > is wrong right?
> > > Also I do not see why this code is needed at all since we overwrite
> > > this register when we call pci_bus_assign_resources(bus) that
> > > will end up in calling pci_setup_bridge() and then
> > > pci_setup_bridge_mmio()...?
> 
> Do you have any comment on this issue above?

Sorry, I am not sure.
However, there are some redundant codes like this.
At that time, I was not able to decide to remove these codes.

Maybe, Pratyush Anand or other guys would give opinions about this.

Best regards,
Jingoo Han

> 
> > >
> > > Many Thanks
> > >
> > > Gab
> > >
> > > >
> > > > Bjorn
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe linux-
> > pci" in
> > > > the body of a message to majordomo at vger.kernel.org
> > > > More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Pratyush Anand]

Hi Gabriele,

On Thu, Apr 14, 2016 at 5:22 PM, Jingoo Han <jingoohan1@gmail.com> wrote:
> On Wednesday, April 13, 2016 4:58 PM, Gabriele Paoloni wrote:
>>
>> Hi Jingoo
>>
>> On 13 April 2016 06:52, Jingoo Han wrote:
>> > On Tuesday, April 12, 2016 6:44 PM, Gabriele Paoloni wrote:

[...]

>> > > So I will update the driver to call dw_pcie_setup_rc() from
>> > > .host_init and ask the BIOS team to update the firmware for next
>> > > releases (the driver will be backward compatible anyway).
>> > >
>> > > Also during my investigation I have noticed that in
>> > dw_pcie_setup_rc()
>> > > http://lxr.free-electrons.com/source/drivers/pci/host/pcie-
>> > designware.c#L762
>> > >
>> > > we use pp->mem_base rather than pp->mem_bus_addr to setup
>> > > memory base and memory limit in the Type1 header...I think this
>> > > is wrong right?

Yes. RC's "memory base" and "memory limit" should be governed by PCI
addresses and not CPU addresses. So, it should use pp->mem_bus_addr.

>> > > Also I do not see why this code is needed at all since we overwrite
>> > > this register when we call pci_bus_assign_resources(bus) that
>> > > will end up in calling pci_setup_bridge() and then
>> > > pci_setup_bridge_mmio()...?
>>
>> Do you have any comment on this issue above?

Probably thats why things are working.
Thanks for finding it. I think, /* setup memory base, memory limit */
hunk can be removed from dw_pcie_setup_rc.

~Pratyush

Re: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Pratyush Anand]

Hi Gabriele,

On Thu, Apr 14, 2016 at 5:22 PM, Jingoo Han <jingoohan1@gmail.com> wrote:
> On Wednesday, April 13, 2016 4:58 PM, Gabriele Paoloni wrote:
>>
>> Hi Jingoo
>>
>> On 13 April 2016 06:52, Jingoo Han wrote:
>> > On Tuesday, April 12, 2016 6:44 PM, Gabriele Paoloni wrote:

[...]

>> > > So I will update the driver to call dw_pcie_setup_rc() from
>> > > .host_init and ask the BIOS team to update the firmware for next
>> > > releases (the driver will be backward compatible anyway).
>> > >
>> > > Also during my investigation I have noticed that in
>> > dw_pcie_setup_rc()
>> > > http://lxr.free-electrons.com/source/drivers/pci/host/pcie-
>> > designware.c#L762
>> > >
>> > > we use pp->mem_base rather than pp->mem_bus_addr to setup
>> > > memory base and memory limit in the Type1 header...I think this
>> > > is wrong right?

Yes. RC's "memory base" and "memory limit" should be governed by PCI
addresses and not CPU addresses. So, it should use pp->mem_bus_addr.

>> > > Also I do not see why this code is needed at all since we overwrite
>> > > this register when we call pci_bus_assign_resources(bus) that
>> > > will end up in calling pci_setup_bridge() and then
>> > > pci_setup_bridge_mmio()...?
>>
>> Do you have any comment on this issue above?

Probably thats why things are working.
Thanks for finding it. I think, /* setup memory base, memory limit */
hunk can be removed from dw_pcie_setup_rc.

~Pratyush

[PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Pratyush Anand]

Hi Gabriele,

On Thu, Apr 14, 2016 at 5:22 PM, Jingoo Han <jingoohan1@gmail.com> wrote:
> On Wednesday, April 13, 2016 4:58 PM, Gabriele Paoloni wrote:
>>
>> Hi Jingoo
>>
>> On 13 April 2016 06:52, Jingoo Han wrote:
>> > On Tuesday, April 12, 2016 6:44 PM, Gabriele Paoloni wrote:

[...]

>> > > So I will update the driver to call dw_pcie_setup_rc() from
>> > > .host_init and ask the BIOS team to update the firmware for next
>> > > releases (the driver will be backward compatible anyway).
>> > >
>> > > Also during my investigation I have noticed that in
>> > dw_pcie_setup_rc()
>> > > http://lxr.free-electrons.com/source/drivers/pci/host/pcie-
>> > designware.c#L762
>> > >
>> > > we use pp->mem_base rather than pp->mem_bus_addr to setup
>> > > memory base and memory limit in the Type1 header...I think this
>> > > is wrong right?

Yes. RC's "memory base" and "memory limit" should be governed by PCI
addresses and not CPU addresses. So, it should use pp->mem_bus_addr.

>> > > Also I do not see why this code is needed at all since we overwrite
>> > > this register when we call pci_bus_assign_resources(bus) that
>> > > will end up in calling pci_setup_bridge() and then
>> > > pci_setup_bridge_mmio()...?
>>
>> Do you have any comment on this issue above?

Probably thats why things are working.
Thanks for finding it. I think, /* setup memory base, memory limit */
hunk can be removed from dw_pcie_setup_rc.

~Pratyush

RE: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Gabriele Paoloni]

Hi Pratyush 

thanks for you reply 

> -----Original Message-----
> From: Pratyush Anand [mailto:pratyush.anand@gmail.com]
> Sent: 14 April 2016 14:08
> To: Jingoo Han; Gabriele Paoloni
> Cc: Bjorn Helgaas; Jisheng Zhang; Bjorn Helgaas; linux-
> pci@vger.kernel.org; linux-kernel@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org
> Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> to dw_pcie_setup_rc()
> 
> Hi Gabriele,
> 
> On Thu, Apr 14, 2016 at 5:22 PM, Jingoo Han <jingoohan1@gmail.com>
> wrote:
> > On Wednesday, April 13, 2016 4:58 PM, Gabriele Paoloni wrote:
> >>
> >> Hi Jingoo
> >>
> >> On 13 April 2016 06:52, Jingoo Han wrote:
> >> > On Tuesday, April 12, 2016 6:44 PM, Gabriele Paoloni wrote:
> 
> [...]
> 
> >> > > So I will update the driver to call dw_pcie_setup_rc() from
> >> > > .host_init and ask the BIOS team to update the firmware for next
> >> > > releases (the driver will be backward compatible anyway).
> >> > >
> >> > > Also during my investigation I have noticed that in
> >> > dw_pcie_setup_rc()
> >> > > http://lxr.free-electrons.com/source/drivers/pci/host/pcie-
> >> > designware.c#L762
> >> > >
> >> > > we use pp->mem_base rather than pp->mem_bus_addr to setup
> >> > > memory base and memory limit in the Type1 header...I think this
> >> > > is wrong right?
> 
> Yes. RC's "memory base" and "memory limit" should be governed by PCI
> addresses and not CPU addresses. So, it should use pp->mem_bus_addr.
> 
> >> > > Also I do not see why this code is needed at all since we
> overwrite
> >> > > this register when we call pci_bus_assign_resources(bus) that
> >> > > will end up in calling pci_setup_bridge() and then
> >> > > pci_setup_bridge_mmio()...?
> >>
> >> Do you have any comment on this issue above?
> 
> Probably thats why things are working.
> Thanks for finding it. I think, /* setup memory base, memory limit */
> hunk can be removed from dw_pcie_setup_rc.

Great, I'll send out a patch to remove this

Thanks

Gab

> 
> ~Pratyush

RE: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Gabriele Paoloni]

SGkgUHJhdHl1c2ggDQoNCnRoYW5rcyBmb3IgeW91IHJlcGx5IA0KDQo+IC0tLS0tT3JpZ2luYWwg
TWVzc2FnZS0tLS0tDQo+IEZyb206IFByYXR5dXNoIEFuYW5kIFttYWlsdG86cHJhdHl1c2guYW5h
bmRAZ21haWwuY29tXQ0KPiBTZW50OiAxNCBBcHJpbCAyMDE2IDE0OjA4DQo+IFRvOiBKaW5nb28g
SGFuOyBHYWJyaWVsZSBQYW9sb25pDQo+IENjOiBCam9ybiBIZWxnYWFzOyBKaXNoZW5nIFpoYW5n
OyBCam9ybiBIZWxnYWFzOyBsaW51eC0NCj4gcGNpQHZnZXIua2VybmVsLm9yZzsgbGludXgta2Vy
bmVsQHZnZXIua2VybmVsLm9yZzsgbGludXgtYXJtLQ0KPiBrZXJuZWxAbGlzdHMuaW5mcmFkZWFk
Lm9yZw0KPiBTdWJqZWN0OiBSZTogW1BBVENIIHYyXSBQQ0k6IGRlc2lnbndhcmU6IG1vdmUgcmVt
YWluaW5nIHJjIHNldHVwIGNvZGUNCj4gdG8gZHdfcGNpZV9zZXR1cF9yYygpDQo+IA0KPiBIaSBH
YWJyaWVsZSwNCj4gDQo+IE9uIFRodSwgQXByIDE0LCAyMDE2IGF0IDU6MjIgUE0sIEppbmdvbyBI
YW4gPGppbmdvb2hhbjFAZ21haWwuY29tPg0KPiB3cm90ZToNCj4gPiBPbiBXZWRuZXNkYXksIEFw
cmlsIDEzLCAyMDE2IDQ6NTggUE0sIEdhYnJpZWxlIFBhb2xvbmkgd3JvdGU6DQo+ID4+DQo+ID4+
IEhpIEppbmdvbw0KPiA+Pg0KPiA+PiBPbiAxMyBBcHJpbCAyMDE2IDA2OjUyLCBKaW5nb28gSGFu
IHdyb3RlOg0KPiA+PiA+IE9uIFR1ZXNkYXksIEFwcmlsIDEyLCAyMDE2IDY6NDQgUE0sIEdhYnJp
ZWxlIFBhb2xvbmkgd3JvdGU6DQo+IA0KPiBbLi4uXQ0KPiANCj4gPj4gPiA+IFNvIEkgd2lsbCB1
cGRhdGUgdGhlIGRyaXZlciB0byBjYWxsIGR3X3BjaWVfc2V0dXBfcmMoKSBmcm9tDQo+ID4+ID4g
PiAuaG9zdF9pbml0IGFuZCBhc2sgdGhlIEJJT1MgdGVhbSB0byB1cGRhdGUgdGhlIGZpcm13YXJl
IGZvciBuZXh0DQo+ID4+ID4gPiByZWxlYXNlcyAodGhlIGRyaXZlciB3aWxsIGJlIGJhY2t3YXJk
IGNvbXBhdGlibGUgYW55d2F5KS4NCj4gPj4gPiA+DQo+ID4+ID4gPiBBbHNvIGR1cmluZyBteSBp
bnZlc3RpZ2F0aW9uIEkgaGF2ZSBub3RpY2VkIHRoYXQgaW4NCj4gPj4gPiBkd19wY2llX3NldHVw
X3JjKCkNCj4gPj4gPiA+IGh0dHA6Ly9seHIuZnJlZS1lbGVjdHJvbnMuY29tL3NvdXJjZS9kcml2
ZXJzL3BjaS9ob3N0L3BjaWUtDQo+ID4+ID4gZGVzaWdud2FyZS5jI0w3NjINCj4gPj4gPiA+DQo+
ID4+ID4gPiB3ZSB1c2UgcHAtPm1lbV9iYXNlIHJhdGhlciB0aGFuIHBwLT5tZW1fYnVzX2FkZHIg
dG8gc2V0dXANCj4gPj4gPiA+IG1lbW9yeSBiYXNlIGFuZCBtZW1vcnkgbGltaXQgaW4gdGhlIFR5
cGUxIGhlYWRlci4uLkkgdGhpbmsgdGhpcw0KPiA+PiA+ID4gaXMgd3JvbmcgcmlnaHQ/DQo+IA0K
PiBZZXMuIFJDJ3MgIm1lbW9yeSBiYXNlIiBhbmQgIm1lbW9yeSBsaW1pdCIgc2hvdWxkIGJlIGdv
dmVybmVkIGJ5IFBDSQ0KPiBhZGRyZXNzZXMgYW5kIG5vdCBDUFUgYWRkcmVzc2VzLiBTbywgaXQg
c2hvdWxkIHVzZSBwcC0+bWVtX2J1c19hZGRyLg0KPiANCj4gPj4gPiA+IEFsc28gSSBkbyBub3Qg
c2VlIHdoeSB0aGlzIGNvZGUgaXMgbmVlZGVkIGF0IGFsbCBzaW5jZSB3ZQ0KPiBvdmVyd3JpdGUN
Cj4gPj4gPiA+IHRoaXMgcmVnaXN0ZXIgd2hlbiB3ZSBjYWxsIHBjaV9idXNfYXNzaWduX3Jlc291
cmNlcyhidXMpIHRoYXQNCj4gPj4gPiA+IHdpbGwgZW5kIHVwIGluIGNhbGxpbmcgcGNpX3NldHVw
X2JyaWRnZSgpIGFuZCB0aGVuDQo+ID4+ID4gPiBwY2lfc2V0dXBfYnJpZGdlX21taW8oKS4uLj8N
Cj4gPj4NCj4gPj4gRG8geW91IGhhdmUgYW55IGNvbW1lbnQgb24gdGhpcyBpc3N1ZSBhYm92ZT8N
Cj4gDQo+IFByb2JhYmx5IHRoYXRzIHdoeSB0aGluZ3MgYXJlIHdvcmtpbmcuDQo+IFRoYW5rcyBm
b3IgZmluZGluZyBpdC4gSSB0aGluaywgLyogc2V0dXAgbWVtb3J5IGJhc2UsIG1lbW9yeSBsaW1p
dCAqLw0KPiBodW5rIGNhbiBiZSByZW1vdmVkIGZyb20gZHdfcGNpZV9zZXR1cF9yYy4NCg0KR3Jl
YXQsIEknbGwgc2VuZCBvdXQgYSBwYXRjaCB0byByZW1vdmUgdGhpcw0KDQpUaGFua3MNCg0KR2Fi
DQoNCj4gDQo+IH5QcmF0eXVzaA0K

[PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Gabriele Paoloni]

Hi Pratyush 

thanks for you reply 

> -----Original Message-----
> From: Pratyush Anand [mailto:pratyush.anand at gmail.com]
> Sent: 14 April 2016 14:08
> To: Jingoo Han; Gabriele Paoloni
> Cc: Bjorn Helgaas; Jisheng Zhang; Bjorn Helgaas; linux-
> pci at vger.kernel.org; linux-kernel at vger.kernel.org; linux-arm-
> kernel at lists.infradead.org
> Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> to dw_pcie_setup_rc()
> 
> Hi Gabriele,
> 
> On Thu, Apr 14, 2016 at 5:22 PM, Jingoo Han <jingoohan1@gmail.com>
> wrote:
> > On Wednesday, April 13, 2016 4:58 PM, Gabriele Paoloni wrote:
> >>
> >> Hi Jingoo
> >>
> >> On 13 April 2016 06:52, Jingoo Han wrote:
> >> > On Tuesday, April 12, 2016 6:44 PM, Gabriele Paoloni wrote:
> 
> [...]
> 
> >> > > So I will update the driver to call dw_pcie_setup_rc() from
> >> > > .host_init and ask the BIOS team to update the firmware for next
> >> > > releases (the driver will be backward compatible anyway).
> >> > >
> >> > > Also during my investigation I have noticed that in
> >> > dw_pcie_setup_rc()
> >> > > http://lxr.free-electrons.com/source/drivers/pci/host/pcie-
> >> > designware.c#L762
> >> > >
> >> > > we use pp->mem_base rather than pp->mem_bus_addr to setup
> >> > > memory base and memory limit in the Type1 header...I think this
> >> > > is wrong right?
> 
> Yes. RC's "memory base" and "memory limit" should be governed by PCI
> addresses and not CPU addresses. So, it should use pp->mem_bus_addr.
> 
> >> > > Also I do not see why this code is needed at all since we
> overwrite
> >> > > this register when we call pci_bus_assign_resources(bus) that
> >> > > will end up in calling pci_setup_bridge() and then
> >> > > pci_setup_bridge_mmio()...?
> >>
> >> Do you have any comment on this issue above?
> 
> Probably thats why things are working.
> Thanks for finding it. I think, /* setup memory base, memory limit */
> hunk can be removed from dw_pcie_setup_rc.

Great, I'll send out a patch to remove this

Thanks

Gab

> 
> ~Pratyush

Re: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Bjorn Helgaas]

On Tue, Apr 12, 2016 at 09:43:32AM +0000, Gabriele Paoloni wrote:
> Hi Bjorn
> 
> [...]
> 
> > > >
> > > > What's the hisi plan for resuming after suspend-to-RAM?  How does
> > the
> > > > RC get reprogrammed after it loses all its state?
> > >
> > > PM is not part of the driver yet. This is planned for near
> > > future release so haven't made such considerations yet
> > > >
> > > > What would break if hisi did call dw_pcie_setup_rc()?  I know you
> > said
> > > > it would overwrite what the bootloader already did, which is true.
> > >
> > > I am try to figure this out now with our HW team.
> > >
> > > >
> > > > But hisi does call dw_pcie_host_init(), so it reads pp->mem (which
> > > > determines pp->mem_base) and pp->lanes from the DT.  Other drivers
> > > > then call dw_pcie_setup_rc() which programs the RC based on
> > > > pp->mem_base and pp->lanes.  So hisi assumes UEFI programmed the RC
> > to
> > > > match the DT, while the other drivers read the DT and program the
> > RC
> > > > to match.  The latter seems more robust because it enforces the
> > > > consistency rather than relying on it.
> > >
> > > Yes I agree with you, however we have preferred to move RC config to
> > > BIOS to have a single driver to support multiple versions of the
> > > same SoC.
> > 
> > I think there are two reasonable approaches:
> > 
> >   1) A single generic driver that doesn't have any knowledge about the
> >   chipset registers; it uses run-time firmware interfaces to manage
> >   the bridge.  The ACPI pci_root.c driver is the best example so far
> >   and works very well.  It supports basically all x86 and ia64
> >   chipsets and requires no kernel work for new ones.
> > 
> >   2) Native drivers specific to each chipset.  These may get
> >   configuration information from DT, but they do their own
> >   register-level programming of the device without run-time help from
> >   firmware.
> > 
> > I think hisi is a native driver because it uses hip05/hip06 registers
> > to check link state and perform config operations.  And apparently you
> > rely on the ATU, BAR, class, and link width programming currently done
> > in dw_pcie_host_init().  But you want to rely on pre-boot firmware to
> > set up the link.  That doesn't make sense to me -- if the driver wants
> > to twiddle the registers, it should know how to do it all.  I don't
> > see how you can reasonably manage this half-way approach.
> > 
> > > The patch I proposed above does the same job as the original patch
> > > proposed by Jisheng and also allows hisi driver to call the moved
> > > code.
> > >
> > > Do you see anything wrong with it?
> > 
> > Only that it makes the structure more complicated and we haven't
> > identified a corresponding benefit yet.
> 
> Finally I have checked that assigning .host_init function pointer
> in our driver to call dw_pcie_setup_rc() will not affect the values
> already set by BIOS.
> 
> Also I agree with you that a hybrid approach is not ideal.
> 
> So I will update the driver to call dw_pcie_setup_rc() from
> .host_init and ask the BIOS team to update the firmware for next
> releases (the driver will be backward compatible anyway). 

Am I right in assuming that the patch currently in my tree:

https://git.kernel.org/cgit/linux/kernel/git/helgaas/pci.git/commit/?h=pci/host-designware&id=1488aefa37a4033080942c860294d13c613ec829

will work for you?  I'm going to assume so unless I hear otherwise.

Bjorn

Re: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Bjorn Helgaas]

On Tue, Apr 12, 2016 at 09:43:32AM +0000, Gabriele Paoloni wrote:
> Hi Bjorn
> 
> [...]
> 
> > > >
> > > > What's the hisi plan for resuming after suspend-to-RAM?  How does
> > the
> > > > RC get reprogrammed after it loses all its state?
> > >
> > > PM is not part of the driver yet. This is planned for near
> > > future release so haven't made such considerations yet
> > > >
> > > > What would break if hisi did call dw_pcie_setup_rc()?  I know you
> > said
> > > > it would overwrite what the bootloader already did, which is true.
> > >
> > > I am try to figure this out now with our HW team.
> > >
> > > >
> > > > But hisi does call dw_pcie_host_init(), so it reads pp->mem (which
> > > > determines pp->mem_base) and pp->lanes from the DT.  Other drivers
> > > > then call dw_pcie_setup_rc() which programs the RC based on
> > > > pp->mem_base and pp->lanes.  So hisi assumes UEFI programmed the RC
> > to
> > > > match the DT, while the other drivers read the DT and program the
> > RC
> > > > to match.  The latter seems more robust because it enforces the
> > > > consistency rather than relying on it.
> > >
> > > Yes I agree with you, however we have preferred to move RC config to
> > > BIOS to have a single driver to support multiple versions of the
> > > same SoC.
> > 
> > I think there are two reasonable approaches:
> > 
> >   1) A single generic driver that doesn't have any knowledge about the
> >   chipset registers; it uses run-time firmware interfaces to manage
> >   the bridge.  The ACPI pci_root.c driver is the best example so far
> >   and works very well.  It supports basically all x86 and ia64
> >   chipsets and requires no kernel work for new ones.
> > 
> >   2) Native drivers specific to each chipset.  These may get
> >   configuration information from DT, but they do their own
> >   register-level programming of the device without run-time help from
> >   firmware.
> > 
> > I think hisi is a native driver because it uses hip05/hip06 registers
> > to check link state and perform config operations.  And apparently you
> > rely on the ATU, BAR, class, and link width programming currently done
> > in dw_pcie_host_init().  But you want to rely on pre-boot firmware to
> > set up the link.  That doesn't make sense to me -- if the driver wants
> > to twiddle the registers, it should know how to do it all.  I don't
> > see how you can reasonably manage this half-way approach.
> > 
> > > The patch I proposed above does the same job as the original patch
> > > proposed by Jisheng and also allows hisi driver to call the moved
> > > code.
> > >
> > > Do you see anything wrong with it?
> > 
> > Only that it makes the structure more complicated and we haven't
> > identified a corresponding benefit yet.
> 
> Finally I have checked that assigning .host_init function pointer
> in our driver to call dw_pcie_setup_rc() will not affect the values
> already set by BIOS.
> 
> Also I agree with you that a hybrid approach is not ideal.
> 
> So I will update the driver to call dw_pcie_setup_rc() from
> .host_init and ask the BIOS team to update the firmware for next
> releases (the driver will be backward compatible anyway). 

Am I right in assuming that the patch currently in my tree:

https://git.kernel.org/cgit/linux/kernel/git/helgaas/pci.git/commit/?h=pci/host-designware&id=1488aefa37a4033080942c860294d13c613ec829

will work for you?  I'm going to assume so unless I hear otherwise.

Bjorn

[PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Bjorn Helgaas]

On Tue, Apr 12, 2016 at 09:43:32AM +0000, Gabriele Paoloni wrote:
> Hi Bjorn
> 
> [...]
> 
> > > >
> > > > What's the hisi plan for resuming after suspend-to-RAM?  How does
> > the
> > > > RC get reprogrammed after it loses all its state?
> > >
> > > PM is not part of the driver yet. This is planned for near
> > > future release so haven't made such considerations yet
> > > >
> > > > What would break if hisi did call dw_pcie_setup_rc()?  I know you
> > said
> > > > it would overwrite what the bootloader already did, which is true.
> > >
> > > I am try to figure this out now with our HW team.
> > >
> > > >
> > > > But hisi does call dw_pcie_host_init(), so it reads pp->mem (which
> > > > determines pp->mem_base) and pp->lanes from the DT.  Other drivers
> > > > then call dw_pcie_setup_rc() which programs the RC based on
> > > > pp->mem_base and pp->lanes.  So hisi assumes UEFI programmed the RC
> > to
> > > > match the DT, while the other drivers read the DT and program the
> > RC
> > > > to match.  The latter seems more robust because it enforces the
> > > > consistency rather than relying on it.
> > >
> > > Yes I agree with you, however we have preferred to move RC config to
> > > BIOS to have a single driver to support multiple versions of the
> > > same SoC.
> > 
> > I think there are two reasonable approaches:
> > 
> >   1) A single generic driver that doesn't have any knowledge about the
> >   chipset registers; it uses run-time firmware interfaces to manage
> >   the bridge.  The ACPI pci_root.c driver is the best example so far
> >   and works very well.  It supports basically all x86 and ia64
> >   chipsets and requires no kernel work for new ones.
> > 
> >   2) Native drivers specific to each chipset.  These may get
> >   configuration information from DT, but they do their own
> >   register-level programming of the device without run-time help from
> >   firmware.
> > 
> > I think hisi is a native driver because it uses hip05/hip06 registers
> > to check link state and perform config operations.  And apparently you
> > rely on the ATU, BAR, class, and link width programming currently done
> > in dw_pcie_host_init().  But you want to rely on pre-boot firmware to
> > set up the link.  That doesn't make sense to me -- if the driver wants
> > to twiddle the registers, it should know how to do it all.  I don't
> > see how you can reasonably manage this half-way approach.
> > 
> > > The patch I proposed above does the same job as the original patch
> > > proposed by Jisheng and also allows hisi driver to call the moved
> > > code.
> > >
> > > Do you see anything wrong with it?
> > 
> > Only that it makes the structure more complicated and we haven't
> > identified a corresponding benefit yet.
> 
> Finally I have checked that assigning .host_init function pointer
> in our driver to call dw_pcie_setup_rc() will not affect the values
> already set by BIOS.
> 
> Also I agree with you that a hybrid approach is not ideal.
> 
> So I will update the driver to call dw_pcie_setup_rc() from
> .host_init and ask the BIOS team to update the firmware for next
> releases (the driver will be backward compatible anyway). 

Am I right in assuming that the patch currently in my tree:

https://git.kernel.org/cgit/linux/kernel/git/helgaas/pci.git/commit/?h=pci/host-designware&id=1488aefa37a4033080942c860294d13c613ec829

will work for you?  I'm going to assume so unless I hear otherwise.

Bjorn

RE: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Gabriele Paoloni]

Hi Bjorn

> -----Original Message-----
> From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> Sent: 21 April 2016 16:49
> To: Gabriele Paoloni
> Cc: Jisheng Zhang; jingoohan1@gmail.com; pratyush.anand@gmail.com;
> bhelgaas@google.com; linux-pci@vger.kernel.org; linux-
> kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
> Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> to dw_pcie_setup_rc()
> 
> On Tue, Apr 12, 2016 at 09:43:32AM +0000, Gabriele Paoloni wrote:
> > Hi Bjorn
> >
> > [...]
> >
> > > > >
> > > > > What's the hisi plan for resuming after suspend-to-RAM?  How
> does
> > > the
> > > > > RC get reprogrammed after it loses all its state?
> > > >
> > > > PM is not part of the driver yet. This is planned for near
> > > > future release so haven't made such considerations yet
> > > > >
> > > > > What would break if hisi did call dw_pcie_setup_rc()?  I know
> you
> > > said
> > > > > it would overwrite what the bootloader already did, which is
> true.
> > > >
> > > > I am try to figure this out now with our HW team.
> > > >
> > > > >
> > > > > But hisi does call dw_pcie_host_init(), so it reads pp->mem
> (which
> > > > > determines pp->mem_base) and pp->lanes from the DT.  Other
> drivers
> > > > > then call dw_pcie_setup_rc() which programs the RC based on
> > > > > pp->mem_base and pp->lanes.  So hisi assumes UEFI programmed
> the RC
> > > to
> > > > > match the DT, while the other drivers read the DT and program
> the
> > > RC
> > > > > to match.  The latter seems more robust because it enforces the
> > > > > consistency rather than relying on it.
> > > >
> > > > Yes I agree with you, however we have preferred to move RC config
> to
> > > > BIOS to have a single driver to support multiple versions of the
> > > > same SoC.
> > >
> > > I think there are two reasonable approaches:
> > >
> > >   1) A single generic driver that doesn't have any knowledge about
> the
> > >   chipset registers; it uses run-time firmware interfaces to manage
> > >   the bridge.  The ACPI pci_root.c driver is the best example so
> far
> > >   and works very well.  It supports basically all x86 and ia64
> > >   chipsets and requires no kernel work for new ones.
> > >
> > >   2) Native drivers specific to each chipset.  These may get
> > >   configuration information from DT, but they do their own
> > >   register-level programming of the device without run-time help
> from
> > >   firmware.
> > >
> > > I think hisi is a native driver because it uses hip05/hip06
> registers
> > > to check link state and perform config operations.  And apparently
> you
> > > rely on the ATU, BAR, class, and link width programming currently
> done
> > > in dw_pcie_host_init().  But you want to rely on pre-boot firmware
> to
> > > set up the link.  That doesn't make sense to me -- if the driver
> wants
> > > to twiddle the registers, it should know how to do it all.  I don't
> > > see how you can reasonably manage this half-way approach.
> > >
> > > > The patch I proposed above does the same job as the original
> patch
> > > > proposed by Jisheng and also allows hisi driver to call the moved
> > > > code.
> > > >
> > > > Do you see anything wrong with it?
> > >
> > > Only that it makes the structure more complicated and we haven't
> > > identified a corresponding benefit yet.
> >
> > Finally I have checked that assigning .host_init function pointer
> > in our driver to call dw_pcie_setup_rc() will not affect the values
> > already set by BIOS.
> >
> > Also I agree with you that a hybrid approach is not ideal.
> >
> > So I will update the driver to call dw_pcie_setup_rc() from
> > .host_init and ask the BIOS team to update the firmware for next
> > releases (the driver will be backward compatible anyway).
> 
> Am I right in assuming that the patch currently in my tree:
> 
> https://git.kernel.org/cgit/linux/kernel/git/helgaas/pci.git/commit/?h=
> pci/host-designware&id=1488aefa37a4033080942c860294d13c613ec829
> 
> will work for you?  I'm going to assume so unless I hear otherwise.

Yes you are right.

I thought it was clear by the last conclusion.
Sorry if it was not explicit.

Many Thanks and Regards

Gab


> 
> Bjorn

RE: [PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Gabriele Paoloni]

Hi Bjorn

> -----Original Message-----
> From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> Sent: 21 April 2016 16:49
> To: Gabriele Paoloni
> Cc: Jisheng Zhang; jingoohan1@gmail.com; pratyush.anand@gmail.com;
> bhelgaas@google.com; linux-pci@vger.kernel.org; linux-
> kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
> Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> to dw_pcie_setup_rc()
> 
> On Tue, Apr 12, 2016 at 09:43:32AM +0000, Gabriele Paoloni wrote:
> > Hi Bjorn
> >
> > [...]
> >
> > > > >
> > > > > What's the hisi plan for resuming after suspend-to-RAM?  How
> does
> > > the
> > > > > RC get reprogrammed after it loses all its state?
> > > >
> > > > PM is not part of the driver yet. This is planned for near
> > > > future release so haven't made such considerations yet
> > > > >
> > > > > What would break if hisi did call dw_pcie_setup_rc()?  I know
> you
> > > said
> > > > > it would overwrite what the bootloader already did, which is
> true.
> > > >
> > > > I am try to figure this out now with our HW team.
> > > >
> > > > >
> > > > > But hisi does call dw_pcie_host_init(), so it reads pp->mem
> (which
> > > > > determines pp->mem_base) and pp->lanes from the DT.  Other
> drivers
> > > > > then call dw_pcie_setup_rc() which programs the RC based on
> > > > > pp->mem_base and pp->lanes.  So hisi assumes UEFI programmed
> the RC
> > > to
> > > > > match the DT, while the other drivers read the DT and program
> the
> > > RC
> > > > > to match.  The latter seems more robust because it enforces the
> > > > > consistency rather than relying on it.
> > > >
> > > > Yes I agree with you, however we have preferred to move RC config
> to
> > > > BIOS to have a single driver to support multiple versions of the
> > > > same SoC.
> > >
> > > I think there are two reasonable approaches:
> > >
> > >   1) A single generic driver that doesn't have any knowledge about
> the
> > >   chipset registers; it uses run-time firmware interfaces to manage
> > >   the bridge.  The ACPI pci_root.c driver is the best example so
> far
> > >   and works very well.  It supports basically all x86 and ia64
> > >   chipsets and requires no kernel work for new ones.
> > >
> > >   2) Native drivers specific to each chipset.  These may get
> > >   configuration information from DT, but they do their own
> > >   register-level programming of the device without run-time help
> from
> > >   firmware.
> > >
> > > I think hisi is a native driver because it uses hip05/hip06
> registers
> > > to check link state and perform config operations.  And apparently
> you
> > > rely on the ATU, BAR, class, and link width programming currently
> done
> > > in dw_pcie_host_init().  But you want to rely on pre-boot firmware
> to
> > > set up the link.  That doesn't make sense to me -- if the driver
> wants
> > > to twiddle the registers, it should know how to do it all.  I don't
> > > see how you can reasonably manage this half-way approach.
> > >
> > > > The patch I proposed above does the same job as the original
> patch
> > > > proposed by Jisheng and also allows hisi driver to call the moved
> > > > code.
> > > >
> > > > Do you see anything wrong with it?
> > >
> > > Only that it makes the structure more complicated and we haven't
> > > identified a corresponding benefit yet.
> >
> > Finally I have checked that assigning .host_init function pointer
> > in our driver to call dw_pcie_setup_rc() will not affect the values
> > already set by BIOS.
> >
> > Also I agree with you that a hybrid approach is not ideal.
> >
> > So I will update the driver to call dw_pcie_setup_rc() from
> > .host_init and ask the BIOS team to update the firmware for next
> > releases (the driver will be backward compatible anyway).
> 
> Am I right in assuming that the patch currently in my tree:
> 
> https://git.kernel.org/cgit/linux/kernel/git/helgaas/pci.git/commit/?h=
> pci/host-designware&id=1488aefa37a4033080942c860294d13c613ec829
> 
> will work for you?  I'm going to assume so unless I hear otherwise.

Yes you are right.

I thought it was clear by the last conclusion.
Sorry if it was not explicit.

Many Thanks and Regards

Gab


> 
> Bjorn

[PATCH v2] PCI: designware: move remaining rc setup code to dw_pcie_setup_rc()

[Gabriele Paoloni]

Hi Bjorn

> -----Original Message-----
> From: Bjorn Helgaas [mailto:helgaas at kernel.org]
> Sent: 21 April 2016 16:49
> To: Gabriele Paoloni
> Cc: Jisheng Zhang; jingoohan1 at gmail.com; pratyush.anand at gmail.com;
> bhelgaas at google.com; linux-pci at vger.kernel.org; linux-
> kernel at vger.kernel.org; linux-arm-kernel at lists.infradead.org
> Subject: Re: [PATCH v2] PCI: designware: move remaining rc setup code
> to dw_pcie_setup_rc()
> 
> On Tue, Apr 12, 2016 at 09:43:32AM +0000, Gabriele Paoloni wrote:
> > Hi Bjorn
> >
> > [...]
> >
> > > > >
> > > > > What's the hisi plan for resuming after suspend-to-RAM?  How
> does
> > > the
> > > > > RC get reprogrammed after it loses all its state?
> > > >
> > > > PM is not part of the driver yet. This is planned for near
> > > > future release so haven't made such considerations yet
> > > > >
> > > > > What would break if hisi did call dw_pcie_setup_rc()?  I know
> you
> > > said
> > > > > it would overwrite what the bootloader already did, which is
> true.
> > > >
> > > > I am try to figure this out now with our HW team.
> > > >
> > > > >
> > > > > But hisi does call dw_pcie_host_init(), so it reads pp->mem
> (which
> > > > > determines pp->mem_base) and pp->lanes from the DT.  Other
> drivers
> > > > > then call dw_pcie_setup_rc() which programs the RC based on
> > > > > pp->mem_base and pp->lanes.  So hisi assumes UEFI programmed
> the RC
> > > to
> > > > > match the DT, while the other drivers read the DT and program
> the
> > > RC
> > > > > to match.  The latter seems more robust because it enforces the
> > > > > consistency rather than relying on it.
> > > >
> > > > Yes I agree with you, however we have preferred to move RC config
> to
> > > > BIOS to have a single driver to support multiple versions of the
> > > > same SoC.
> > >
> > > I think there are two reasonable approaches:
> > >
> > >   1) A single generic driver that doesn't have any knowledge about
> the
> > >   chipset registers; it uses run-time firmware interfaces to manage
> > >   the bridge.  The ACPI pci_root.c driver is the best example so
> far
> > >   and works very well.  It supports basically all x86 and ia64
> > >   chipsets and requires no kernel work for new ones.
> > >
> > >   2) Native drivers specific to each chipset.  These may get
> > >   configuration information from DT, but they do their own
> > >   register-level programming of the device without run-time help
> from
> > >   firmware.
> > >
> > > I think hisi is a native driver because it uses hip05/hip06
> registers
> > > to check link state and perform config operations.  And apparently
> you
> > > rely on the ATU, BAR, class, and link width programming currently
> done
> > > in dw_pcie_host_init().  But you want to rely on pre-boot firmware
> to
> > > set up the link.  That doesn't make sense to me -- if the driver
> wants
> > > to twiddle the registers, it should know how to do it all.  I don't
> > > see how you can reasonably manage this half-way approach.
> > >
> > > > The patch I proposed above does the same job as the original
> patch
> > > > proposed by Jisheng and also allows hisi driver to call the moved
> > > > code.
> > > >
> > > > Do you see anything wrong with it?
> > >
> > > Only that it makes the structure more complicated and we haven't
> > > identified a corresponding benefit yet.
> >
> > Finally I have checked that assigning .host_init function pointer
> > in our driver to call dw_pcie_setup_rc() will not affect the values
> > already set by BIOS.
> >
> > Also I agree with you that a hybrid approach is not ideal.
> >
> > So I will update the driver to call dw_pcie_setup_rc() from
> > .host_init and ask the BIOS team to update the firmware for next
> > releases (the driver will be backward compatible anyway).
> 
> Am I right in assuming that the patch currently in my tree:
> 
> https://git.kernel.org/cgit/linux/kernel/git/helgaas/pci.git/commit/?h=
> pci/host-designware&id=1488aefa37a4033080942c860294d13c613ec829
> 
> will work for you?  I'm going to assume so unless I hear otherwise.

Yes you are right.

I thought it was clear by the last conclusion.
Sorry if it was not explicit.

Many Thanks and Regards

Gab


> 
> Bjorn