fu740 pcie driver fixes

fu740 pcie driver fixes

[Ben Dooks]

This pair of patches is currently necessary to allow any PCIe
probe on the SiFive Unmatched board without using U-Boot to
probe the PCIe bus (which does all the setup below). We found
this as we have been network booting kernels onto our boards
for testing.

[PATCH 1/2] PCI: fu740: fix finding GPIOs

[Ben Dooks]

The calls to devm_gpiod_get_optional() have the -gpios at the end of
the name. This means the pcie driver is not finding the necessary
reset or power GPOOs to allow the PCIe devices on the SiFive Unmatched
boards.

This has not been a noted bug as the PCIe probe from u-boot has been
required to get the PCIe working due to other issues with the system
setup. It could have been broken since the driver inclusion, and not
been noticed as it is not necessary for the driver to funciton.

Signed-off-by: Ben Dooks <ben.dooks@codethink.co.uk>
---
 drivers/pci/controller/dwc/pcie-fu740.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/controller/dwc/pcie-fu740.c b/drivers/pci/controller/dwc/pcie-fu740.c
index 00cde9a248b5..842b7202b96e 100644
--- a/drivers/pci/controller/dwc/pcie-fu740.c
+++ b/drivers/pci/controller/dwc/pcie-fu740.c
@@ -259,11 +259,11 @@ static int fu740_pcie_probe(struct platform_device *pdev)
 		return PTR_ERR(afp->mgmt_base);
 
 	/* Fetch GPIOs */
-	afp->reset = devm_gpiod_get_optional(dev, "reset-gpios", GPIOD_OUT_LOW);
+	afp->reset = devm_gpiod_get_optional(dev, "reset", GPIOD_OUT_LOW);
 	if (IS_ERR(afp->reset))
 		return dev_err_probe(dev, PTR_ERR(afp->reset), "unable to get reset-gpios\n");
 
-	afp->pwren = devm_gpiod_get_optional(dev, "pwren-gpios", GPIOD_OUT_LOW);
+	afp->pwren = devm_gpiod_get_optional(dev, "pwren", GPIOD_OUT_LOW);
 	if (IS_ERR(afp->pwren))
 		return dev_err_probe(dev, PTR_ERR(afp->pwren), "unable to get pwren-gpios\n");
 
-- 
2.34.1

[PATCH 2/2] PCI: fu740: Force gen1 for initial device probe

[Ben Dooks]

The fu740 dw pcie core does not probe devices without this fix from
U-boot. The fix claims to set the link-speed to gen1 to get the probe
to work. As this is a copy from U-boot, the commentary is assumed to
be correct.

Without this in, and without U-boot starting the PCIe the Unmatched
board does not show any PCIe devices after the DW root port.

Signed-off-by: Ben Dooks <ben.dooks@codethink.co.uk>
---
 drivers/pci/controller/dwc/pcie-fu740.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/drivers/pci/controller/dwc/pcie-fu740.c b/drivers/pci/controller/dwc/pcie-fu740.c
index 842b7202b96e..19501ec8c487 100644
--- a/drivers/pci/controller/dwc/pcie-fu740.c
+++ b/drivers/pci/controller/dwc/pcie-fu740.c
@@ -177,11 +177,30 @@ static void fu740_pcie_init_phy(struct fu740_pcie *afp)
 	fu740_phyregwrite(1, PCIEX8MGMT_PHY_LANE3_BASE, PCIEX8MGMT_PHY_INIT_VAL, afp);
 }
 
+/* This is copied from u-boot. Force system to gen1 otherwise nothing probes
+ * as found on the SiFive Unmatched board.
+ */
+static void fu740_pcie_force_gen1(struct dw_pcie *dw, struct fu740_pcie *afp )
+{
+	unsigned val;
+
+	dw_pcie_dbi_ro_wr_en(dw);
+
+	val = dw_pcie_readl_dbi(dw, 0x70 + PCI_EXP_LNKCAP);
+	pr_info("%s: link-cap was %08x\n", __func__, val);
+	dw_pcie_writel_dbi(dw, 0x70 + PCI_EXP_LNKCAP, val | 0xf);
+
+	dw_pcie_dbi_ro_wr_dis(dw);
+}
+
 static int fu740_pcie_start_link(struct dw_pcie *pci)
 {
 	struct device *dev = pci->dev;
 	struct fu740_pcie *afp = dev_get_drvdata(dev);
 
+	/* Force PCIe gen1 otherwise Unmatched board does not probe */
+	fu740_pcie_force_gen1(pci, afp);
+
 	/* Enable LTSSM */
 	writel_relaxed(0x1, afp->mgmt_base + PCIEX8MGMT_APP_LTSSM_ENABLE);
 	return 0;
-- 
2.34.1

Re: [PATCH 1/2] PCI: fu740: fix finding GPIOs

[Bjorn Helgaas]

[+cc Rob for possible DT/kernel match issue,
Lorenzo (native host bridge driver maintainer)]

s/fix finding/Fix finding/ (in subject)
Or even better, say something specific about the DT properties in
question, e.g., look for "reset" instead of "reset-gpios".

On Mon, Feb 14, 2022 at 08:21:43AM +0000, Ben Dooks wrote:
> The calls to devm_gpiod_get_optional() have the -gpios at the end of
> the name. This means the pcie driver is not finding the necessary
> reset or power GPOOs to allow the PCIe devices on the SiFive Unmatched
> boards.

s/pcie/PCIe/
s/GPOOs/GPIOs/
"to allow the PCIe devices ...?"  Something is missing from this
sentence.  "To allow the devices <to do what>"?  Or maybe the driver
needs these GPIOs to power up the PCIe devices?

I guess the implication is that the code looks for "reset-gpios" and
"pwren-gpios", but the DT contains "reset" and "pwren"?

But both Documentation/devicetree/bindings/pci/sifive,fu740-pcie.yaml
and arch/riscv/boot/dts/sifive/fu740-c000.dtsi actually do contain
"reset-gpios" and "pwren-gpios".

If we *do* want to change the code, please change the error messages
to match.

> This has not been a noted bug as the PCIe probe from u-boot has been
> required to get the PCIe working due to other issues with the system
> setup. It could have been broken since the driver inclusion, and not
> been noticed as it is not necessary for the driver to funciton.

s/u-boot/U-Boot/
s/funciton/function/

Please add a line about what the connection between U-Boot and this
issue is, e.g., maybe U-Boot powers up the devices, so we wouldn't
notice the kernel's inability to do so?

> Signed-off-by: Ben Dooks <ben.dooks@codethink.co.uk>
> ---
>  drivers/pci/controller/dwc/pcie-fu740.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/pci/controller/dwc/pcie-fu740.c b/drivers/pci/controller/dwc/pcie-fu740.c
> index 00cde9a248b5..842b7202b96e 100644
> --- a/drivers/pci/controller/dwc/pcie-fu740.c
> +++ b/drivers/pci/controller/dwc/pcie-fu740.c
> @@ -259,11 +259,11 @@ static int fu740_pcie_probe(struct platform_device *pdev)
>  		return PTR_ERR(afp->mgmt_base);
>  
>  	/* Fetch GPIOs */
> -	afp->reset = devm_gpiod_get_optional(dev, "reset-gpios", GPIOD_OUT_LOW);
> +	afp->reset = devm_gpiod_get_optional(dev, "reset", GPIOD_OUT_LOW);
>  	if (IS_ERR(afp->reset))
>  		return dev_err_probe(dev, PTR_ERR(afp->reset), "unable to get reset-gpios\n");
>  
> -	afp->pwren = devm_gpiod_get_optional(dev, "pwren-gpios", GPIOD_OUT_LOW);
> +	afp->pwren = devm_gpiod_get_optional(dev, "pwren", GPIOD_OUT_LOW);
>  	if (IS_ERR(afp->pwren))
>  		return dev_err_probe(dev, PTR_ERR(afp->pwren), "unable to get pwren-gpios\n");
>  
> -- 
> 2.34.1
> 

Re: [PATCH 2/2] PCI: fu740: Force gen1 for initial device probe

[Bjorn Helgaas]

On Mon, Feb 14, 2022 at 08:21:44AM +0000, Ben Dooks wrote:
> The fu740 dw pcie core does not probe devices without this fix from
> U-boot. The fix claims to set the link-speed to gen1 to get the probe
> to work. As this is a copy from U-boot, the commentary is assumed to
> be correct.

s/dw/DW/ (to match below)
s/pcie/PCIe/
s/U-boot/U-Boot/ (twice, and again below)

Is there a stable URL to the place in U-Boot where this is copied
from?

> Without this in, and without U-boot starting the PCIe the Unmatched
> board does not show any PCIe devices after the DW root port.
> 
> Signed-off-by: Ben Dooks <ben.dooks@codethink.co.uk>
> ---
>  drivers/pci/controller/dwc/pcie-fu740.c | 19 +++++++++++++++++++
>  1 file changed, 19 insertions(+)
> 
> diff --git a/drivers/pci/controller/dwc/pcie-fu740.c b/drivers/pci/controller/dwc/pcie-fu740.c
> index 842b7202b96e..19501ec8c487 100644
> --- a/drivers/pci/controller/dwc/pcie-fu740.c
> +++ b/drivers/pci/controller/dwc/pcie-fu740.c
> @@ -177,11 +177,30 @@ static void fu740_pcie_init_phy(struct fu740_pcie *afp)
>  	fu740_phyregwrite(1, PCIEX8MGMT_PHY_LANE3_BASE, PCIEX8MGMT_PHY_INIT_VAL, afp);
>  }
>  
> +/* This is copied from u-boot. Force system to gen1 otherwise nothing probes
> + * as found on the SiFive Unmatched board.
> + */

s/u-boot/U-Boot/

Use this comment style to match the rest of the file:

  /*
   * Comment...
   */

> +static void fu740_pcie_force_gen1(struct dw_pcie *dw, struct fu740_pcie *afp )
> +{
> +	unsigned val;
> +
> +	dw_pcie_dbi_ro_wr_en(dw);
> +
> +	val = dw_pcie_readl_dbi(dw, 0x70 + PCI_EXP_LNKCAP);
> +	pr_info("%s: link-cap was %08x\n", __func__, val);
> +	dw_pcie_writel_dbi(dw, 0x70 + PCI_EXP_LNKCAP, val | 0xf);
> +
> +	dw_pcie_dbi_ro_wr_dis(dw);
> +}
> +
>  static int fu740_pcie_start_link(struct dw_pcie *pci)
>  {
>  	struct device *dev = pci->dev;
>  	struct fu740_pcie *afp = dev_get_drvdata(dev);
>  
> +	/* Force PCIe gen1 otherwise Unmatched board does not probe */
> +	fu740_pcie_force_gen1(pci, afp);

Is Unmatched the *only* board with this controller, i.e., do we want
to do this for every single FU740 device?

If this is an FU740 defect that will affect anything that uses it, we
should say that, and we shouldn't call out "Unmatched" specifically.

> +
>  	/* Enable LTSSM */
>  	writel_relaxed(0x1, afp->mgmt_base + PCIEX8MGMT_APP_LTSSM_ENABLE);
>  	return 0;
> -- 
> 2.34.1
> 

Re: [PATCH 1/2] PCI: fu740: fix finding GPIOs

[Ben Dooks]

On 14/02/2022 16:05, Bjorn Helgaas wrote:
> [+cc Rob for possible DT/kernel match issue,
> Lorenzo (native host bridge driver maintainer)]
> 
> s/fix finding/Fix finding/ (in subject)
> Or even better, say something specific about the DT properties in
> question, e.g., look for "reset" instead of "reset-gpios".
> 
> On Mon, Feb 14, 2022 at 08:21:43AM +0000, Ben Dooks wrote:
>> The calls to devm_gpiod_get_optional() have the -gpios at the end of
>> the name. This means the pcie driver is not finding the necessary
>> reset or power GPOOs to allow the PCIe devices on the SiFive Unmatched
>> boards.
> 
> s/pcie/PCIe/
> s/GPOOs/GPIOs/

Will fix

> "to allow the PCIe devices ...?"  Something is missing from this
> sentence.  "To allow the devices <to do what>"?  Or maybe the driver
> needs these GPIOs to power up the PCIe devices?
> 
> I guess the implication is that the code looks for "reset-gpios" and
> "pwren-gpios", but the DT contains "reset" and "pwren"?
> 
> But both Documentation/devicetree/bindings/pci/sifive,fu740-pcie.yaml
> and arch/riscv/boot/dts/sifive/fu740-c000.dtsi actually do contain
> "reset-gpios" and "pwren-gpios".

Yes, the issue is the gpio core code adds either "-gpio" or "-gpios"
itself, thus the find code does not need this. The error messages and
DT are correct. I'll reword the initial paragraph to note that which
should also deal with the other comments:

The calls to devm_gpiod_get_optional() have the -gpios at the end of
the name but the GPIO core code is already adding the suffix during
the lookup. This means the PCIe driver is not finding the necessary
reset or power lines to allow initialisation of the PCIe.



> If we *do* want to change the code, please change the error messages
> to match.

See above, it is just devm_gpiod_get_optional() parameters at fault.

>> This has not been a noted bug as the PCIe probe from u-boot has been
>> required to get the PCIe working due to other issues with the system
>> setup. It could have been broken since the driver inclusion, and not
>> been noticed as it is not necessary for the driver to funciton.
> 
> s/u-boot/U-Boot/
> s/funciton/function/
> 
> Please add a line about what the connection between U-Boot and this
> issue is, e.g., maybe U-Boot powers up the devices, so we wouldn't
> notice the kernel's inability to do so?

Does a reword to the following make better sense:

This bug has not been noticed as if U-Boot has setup the GPIO lines
for the hardware when it does the PCIe initialisation (either by
booting from PCIe or user command to access PCIe) then the PCIe
will work in Linux. The U-Boot as supplied by SiFive does not by
default initialise any PCIe component.

>> Signed-off-by: Ben Dooks <ben.dooks@codethink.co.uk>
>> ---
>>   drivers/pci/controller/dwc/pcie-fu740.c | 4 ++--
>>   1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/drivers/pci/controller/dwc/pcie-fu740.c b/drivers/pci/controller/dwc/pcie-fu740.c
>> index 00cde9a248b5..842b7202b96e 100644
>> --- a/drivers/pci/controller/dwc/pcie-fu740.c
>> +++ b/drivers/pci/controller/dwc/pcie-fu740.c
>> @@ -259,11 +259,11 @@ static int fu740_pcie_probe(struct platform_device *pdev)
>>   		return PTR_ERR(afp->mgmt_base);
>>   
>>   	/* Fetch GPIOs */
>> -	afp->reset = devm_gpiod_get_optional(dev, "reset-gpios", GPIOD_OUT_LOW);
>> +	afp->reset = devm_gpiod_get_optional(dev, "reset", GPIOD_OUT_LOW);
>>   	if (IS_ERR(afp->reset))
>>   		return dev_err_probe(dev, PTR_ERR(afp->reset), "unable to get reset-gpios\n");
>>   
>> -	afp->pwren = devm_gpiod_get_optional(dev, "pwren-gpios", GPIOD_OUT_LOW);
>> +	afp->pwren = devm_gpiod_get_optional(dev, "pwren", GPIOD_OUT_LOW);
>>   	if (IS_ERR(afp->pwren))
>>   		return dev_err_probe(dev, PTR_ERR(afp->pwren), "unable to get pwren-gpios\n");
>>   
>> -- 
>> 2.34.1
>>
> 


-- 
Ben Dooks				http://www.codethink.co.uk/
Senior Engineer				Codethink - Providing Genius

https://www.codethink.co.uk/privacy.html

Re: [PATCH 1/2] PCI: fu740: fix finding GPIOs

[Bjorn Helgaas]

On Tue, Feb 15, 2022 at 11:21:03AM +0000, Ben Dooks wrote:
> On 14/02/2022 16:05, Bjorn Helgaas wrote:
> > [+cc Rob for possible DT/kernel match issue,
> > Lorenzo (native host bridge driver maintainer)]
> > 
> > s/fix finding/Fix finding/ (in subject)
> > Or even better, say something specific about the DT properties in
> > question, e.g., look for "reset" instead of "reset-gpios".
> > 
> > On Mon, Feb 14, 2022 at 08:21:43AM +0000, Ben Dooks wrote:
> > > The calls to devm_gpiod_get_optional() have the -gpios at the end of
> > > the name. This means the pcie driver is not finding the necessary
> > > reset or power GPOOs to allow the PCIe devices on the SiFive Unmatched
> > > boards.
> > 
> > "to allow the PCIe devices ...?"  Something is missing from this
> > sentence.  "To allow the devices <to do what>"?  Or maybe the driver
> > needs these GPIOs to power up the PCIe devices?
> > 
> > I guess the implication is that the code looks for "reset-gpios" and
> > "pwren-gpios", but the DT contains "reset" and "pwren"?
> > 
> > But both Documentation/devicetree/bindings/pci/sifive,fu740-pcie.yaml
> > and arch/riscv/boot/dts/sifive/fu740-c000.dtsi actually do contain
> > "reset-gpios" and "pwren-gpios".
> 
> Yes, the issue is the gpio core code adds either "-gpio" or "-gpios"
> itself, thus the find code does not need this. The error messages and
> DT are correct. I'll reword the initial paragraph to note that which
> should also deal with the other comments:
> 
> The calls to devm_gpiod_get_optional() have the -gpios at the end of
> the name but the GPIO core code is already adding the suffix during
> the lookup. This means the PCIe driver is not finding the necessary
> reset or power lines to allow initialisation of the PCIe.

Ah, that makes things make a LOT more sense!  Please add something
about "the '-gpios' suffix" for non-experts like me who don't know the
inner workings of the GPIO core.

The GPIO core seems slightly too helpful here -- we have to know about
that detail to write error messages that make sense.

> > If we *do* want to change the code, please change the error messages
> > to match.
> 
> See above, it is just devm_gpiod_get_optional() parameters at fault.
> 
> > > This has not been a noted bug as the PCIe probe from u-boot has been
> > > required to get the PCIe working due to other issues with the system
> > > setup. It could have been broken since the driver inclusion, and not
> > > been noticed as it is not necessary for the driver to funciton.
> > 
> > Please add a line about what the connection between U-Boot and this
> > issue is, e.g., maybe U-Boot powers up the devices, so we wouldn't
> > notice the kernel's inability to do so?
> 
> Does a reword to the following make better sense:
> 
> This bug has not been noticed as if U-Boot has setup the GPIO lines
> for the hardware when it does the PCIe initialisation (either by
> booting from PCIe or user command to access PCIe) then the PCIe
> will work in Linux. The U-Boot as supplied by SiFive does not by
> default initialise any PCIe component.

Sounds good, thanks.

Bjorn

Re: [PATCH 1/2] PCI: fu740: fix finding GPIOs

[Rob Herring]

On Mon, Feb 14, 2022 at 10:05 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> [+cc Rob for possible DT/kernel match issue,
> Lorenzo (native host bridge driver maintainer)]
>
> s/fix finding/Fix finding/ (in subject)
> Or even better, say something specific about the DT properties in
> question, e.g., look for "reset" instead of "reset-gpios".

This is purely a kernel issue. The gpio API appends '-gpios' (and a
fallback to '-gpio') to the name passed. So passing 'reset-gpios'
looks for 'reset-gpios-gpios' property. This happens enough, there
should probably be a check for this mistake.

>
> On Mon, Feb 14, 2022 at 08:21:43AM +0000, Ben Dooks wrote:
> > The calls to devm_gpiod_get_optional() have the -gpios at the end of
> > the name. This means the pcie driver is not finding the necessary
> > reset or power GPOOs to allow the PCIe devices on the SiFive Unmatched
> > boards.
>
> s/pcie/PCIe/
> s/GPOOs/GPIOs/
> "to allow the PCIe devices ...?"  Something is missing from this
> sentence.  "To allow the devices <to do what>"?  Or maybe the driver
> needs these GPIOs to power up the PCIe devices?
>
> I guess the implication is that the code looks for "reset-gpios" and
> "pwren-gpios", but the DT contains "reset" and "pwren"?
>
> But both Documentation/devicetree/bindings/pci/sifive,fu740-pcie.yaml
> and arch/riscv/boot/dts/sifive/fu740-c000.dtsi actually do contain
> "reset-gpios" and "pwren-gpios".
>
> If we *do* want to change the code, please change the error messages
> to match.
>
> > This has not been a noted bug as the PCIe probe from u-boot has been
> > required to get the PCIe working due to other issues with the system
> > setup. It could have been broken since the driver inclusion, and not
> > been noticed as it is not necessary for the driver to funciton.
>
> s/u-boot/U-Boot/
> s/funciton/function/
>
> Please add a line about what the connection between U-Boot and this
> issue is, e.g., maybe U-Boot powers up the devices, so we wouldn't
> notice the kernel's inability to do so?
>
> > Signed-off-by: Ben Dooks <ben.dooks@codethink.co.uk>
> > ---
> >  drivers/pci/controller/dwc/pcie-fu740.c | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/pci/controller/dwc/pcie-fu740.c b/drivers/pci/controller/dwc/pcie-fu740.c
> > index 00cde9a248b5..842b7202b96e 100644
> > --- a/drivers/pci/controller/dwc/pcie-fu740.c
> > +++ b/drivers/pci/controller/dwc/pcie-fu740.c
> > @@ -259,11 +259,11 @@ static int fu740_pcie_probe(struct platform_device *pdev)
> >               return PTR_ERR(afp->mgmt_base);
> >
> >       /* Fetch GPIOs */
> > -     afp->reset = devm_gpiod_get_optional(dev, "reset-gpios", GPIOD_OUT_LOW);
> > +     afp->reset = devm_gpiod_get_optional(dev, "reset", GPIOD_OUT_LOW);
> >       if (IS_ERR(afp->reset))
> >               return dev_err_probe(dev, PTR_ERR(afp->reset), "unable to get reset-gpios\n");
> >
> > -     afp->pwren = devm_gpiod_get_optional(dev, "pwren-gpios", GPIOD_OUT_LOW);
> > +     afp->pwren = devm_gpiod_get_optional(dev, "pwren", GPIOD_OUT_LOW);
> >       if (IS_ERR(afp->pwren))
> >               return dev_err_probe(dev, PTR_ERR(afp->pwren), "unable to get pwren-gpios\n");
> >
> > --
> > 2.34.1
> >