[PATCH] PCI: dwc: designware: don't sleep in atomic context

[PATCH] PCI: dwc: designware: don't sleep in atomic context

[Pankaj Dubey]

In pcie-designware.c many places we are calling "usleep_range" which
are in atomic context. This patch fixes these potential BUGs and
replaces "usleep_range" with mdelay calls.

Signed-off-by: Pankaj Dubey <pankaj.dubey@samsung.com>
---
 drivers/pci/dwc/pcie-designware.c | 8 ++++----
 drivers/pci/dwc/pcie-designware.h | 3 +--
 2 files changed, 5 insertions(+), 6 deletions(-)

diff --git a/drivers/pci/dwc/pcie-designware.c b/drivers/pci/dwc/pcie-designware.c
index 88abddd..35d19b9 100644
--- a/drivers/pci/dwc/pcie-designware.c
+++ b/drivers/pci/dwc/pcie-designware.c
@@ -138,7 +138,7 @@ static void dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, int index,
 		if (val & PCIE_ATU_ENABLE)
 			return;
 
-		usleep_range(LINK_WAIT_IATU_MIN, LINK_WAIT_IATU_MAX);
+		mdelay(LINK_WAIT_IATU_MIN);
 	}
 	dev_err(pci->dev, "outbound iATU is not being enabled\n");
 }
@@ -181,7 +181,7 @@ void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
 		if (val & PCIE_ATU_ENABLE)
 			return;
 
-		usleep_range(LINK_WAIT_IATU_MIN, LINK_WAIT_IATU_MAX);
+		mdelay(LINK_WAIT_IATU_MIN);
 	}
 	dev_err(pci->dev, "outbound iATU is not being enabled\n");
 }
@@ -239,7 +239,7 @@ static int dw_pcie_prog_inbound_atu_unroll(struct dw_pcie *pci, int index,
 		if (val & PCIE_ATU_ENABLE)
 			return 0;
 
-		usleep_range(LINK_WAIT_IATU_MIN, LINK_WAIT_IATU_MAX);
+		mdelay(LINK_WAIT_IATU_MIN);
 	}
 	dev_err(pci->dev, "inbound iATU is not being enabled\n");
 
@@ -285,7 +285,7 @@ int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, int index, int bar,
 		if (val & PCIE_ATU_ENABLE)
 			return 0;
 
-		usleep_range(LINK_WAIT_IATU_MIN, LINK_WAIT_IATU_MAX);
+		mdelay(LINK_WAIT_IATU_MIN);
 	}
 	dev_err(pci->dev, "inbound iATU is not being enabled\n");
 
diff --git a/drivers/pci/dwc/pcie-designware.h b/drivers/pci/dwc/pcie-designware.h
index e5d9d77..13c17c9 100644
--- a/drivers/pci/dwc/pcie-designware.h
+++ b/drivers/pci/dwc/pcie-designware.h
@@ -28,8 +28,7 @@
 
 /* Parameters for the waiting for iATU enabled routine */
 #define LINK_WAIT_MAX_IATU_RETRIES	5
-#define LINK_WAIT_IATU_MIN		9000
-#define LINK_WAIT_IATU_MAX		10000
+#define LINK_WAIT_IATU_MIN		9
 
 /* Synopsys-specific PCIe configuration registers */
 #define PCIE_PORT_LINK_CONTROL		0x710
-- 
2.7.4

RE: [PATCH] PCI: dwc: designware: don't sleep in atomic context

[David Laight]

From: Pankaj Dubey
> Sent: 12 October 2017 08:55
> In pcie-designware.c many places we are calling "usleep_range" which
> are in atomic context. This patch fixes these potential BUGs and
> replaces "usleep_range" with mdelay calls.
> 
> Signed-off-by: Pankaj Dubey <pankaj.dubey@samsung.com>
> ---
>  drivers/pci/dwc/pcie-designware.c | 8 ++++----
>  drivers/pci/dwc/pcie-designware.h | 3 +--
>  2 files changed, 5 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/pci/dwc/pcie-designware.c b/drivers/pci/dwc/pcie-designware.c
> index 88abddd..35d19b9 100644
> --- a/drivers/pci/dwc/pcie-designware.c
> +++ b/drivers/pci/dwc/pcie-designware.c
> @@ -138,7 +138,7 @@ static void dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, int index,
>  		if (val & PCIE_ATU_ENABLE)
>  			return;
> 
> -		usleep_range(LINK_WAIT_IATU_MIN, LINK_WAIT_IATU_MAX);
> +		mdelay(LINK_WAIT_IATU_MIN);
>  	}

Spinning for 9ms (possibly 10 times) isn't really a good idea.

	David

Re: [PATCH] PCI: dwc: designware: don't sleep in atomic context

[Pankaj Dubey]

On 10/12/2017 04:09 PM, David Laight wrote:
> From: Pankaj Dubey
>> Sent: 12 October 2017 08:55
>> In pcie-designware.c many places we are calling "usleep_range" which
>> are in atomic context. This patch fixes these potential BUGs and
>> replaces "usleep_range" with mdelay calls.
>>
>> Signed-off-by: Pankaj Dubey <pankaj.dubey@samsung.com>
>> ---
>>   drivers/pci/dwc/pcie-designware.c | 8 ++++----
>>   drivers/pci/dwc/pcie-designware.h | 3 +--
>>   2 files changed, 5 insertions(+), 6 deletions(-)
>>
>> diff --git a/drivers/pci/dwc/pcie-designware.c b/drivers/pci/dwc/pcie-designware.c
>> index 88abddd..35d19b9 100644
>> --- a/drivers/pci/dwc/pcie-designware.c
>> +++ b/drivers/pci/dwc/pcie-designware.c
>> @@ -138,7 +138,7 @@ static void dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, int index,
>>   		if (val & PCIE_ATU_ENABLE)
>>   			return;
>>
>> -		usleep_range(LINK_WAIT_IATU_MIN, LINK_WAIT_IATU_MAX);
>> +		mdelay(LINK_WAIT_IATU_MIN);
>>   	}
> Spinning for 9ms (possibly 10 times) isn't really a good idea.

Yes. It may not be a good idea, however in our experiment it never hit 
maximum retry count.
I just converted usleep_range to mdelay keeping min time limitation as 
it is, though I
am not sure, how do we arrived on these numbers in original code, may be 
Joao Pinto
from Synopsys have some idea, I will try to do few experiment and try to 
find out what
is sufficient minimum time in our hardware for these mdelay.

Thanks,
Pankaj Dubey
> 	David
>
>
>
>

Re: [PATCH] PCI: dwc: designware: don't sleep in atomic context

[Bjorn Helgaas]

On Fri, Oct 13, 2017 at 09:10:38AM +0530, Pankaj Dubey wrote:
> 
> 
> On 10/12/2017 04:09 PM, David Laight wrote:
> >From: Pankaj Dubey
> >>Sent: 12 October 2017 08:55
> >>In pcie-designware.c many places we are calling "usleep_range" which
> >>are in atomic context. This patch fixes these potential BUGs and
> >>replaces "usleep_range" with mdelay calls.
> >>
> >>Signed-off-by: Pankaj Dubey <pankaj.dubey@samsung.com>
> >>---
> >>  drivers/pci/dwc/pcie-designware.c | 8 ++++----
> >>  drivers/pci/dwc/pcie-designware.h | 3 +--
> >>  2 files changed, 5 insertions(+), 6 deletions(-)
> >>
> >>diff --git a/drivers/pci/dwc/pcie-designware.c b/drivers/pci/dwc/pcie-designware.c
> >>index 88abddd..35d19b9 100644
> >>--- a/drivers/pci/dwc/pcie-designware.c
> >>+++ b/drivers/pci/dwc/pcie-designware.c
> >>@@ -138,7 +138,7 @@ static void dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, int index,
> >>  		if (val & PCIE_ATU_ENABLE)
> >>  			return;
> >>
> >>-		usleep_range(LINK_WAIT_IATU_MIN, LINK_WAIT_IATU_MAX);
> >>+		mdelay(LINK_WAIT_IATU_MIN);
> >>  	}
> >Spinning for 9ms (possibly 10 times) isn't really a good idea.
> 
> Yes. It may not be a good idea, however in our experiment it never
> hit maximum retry count.  I just converted usleep_range to mdelay
> keeping min time limitation as it is, though I am not sure, how do
> we arrived on these numbers in original code, may be Joao Pinto from
> Synopsys have some idea, I will try to do few experiment and try to
> find out what is sufficient minimum time in our hardware for these
> mdelay.

Just based on the preceding comment, it looks like the wait is
essential because subsequent config and I/O accesses won't work
correctly until the ATU enable takes effect.

If we timeout here, I suspect it's because something is seriously
wrong in the hardware, so I doubt there's any point in trying to
minimize the timeout period.  If something is that broken, it doesn't
matter whether we wait 9ms or 900ms.

Maybe the message should be more strident or maybe we should even
return failure so the caller can do something, e.g., fail an access,
instead of just printing an error and continuing on.

I'm also looking for an ack from Joao and/or Jingoo.

Bjorn

RE: [PATCH] PCI: dwc: designware: don't sleep in atomic context

[David Laight]

From: Bjorn Helgaas
> Sent: 20 October 2017 19:19
> On Fri, Oct 13, 2017 at 09:10:38AM +0530, Pankaj Dubey wrote:
> >
> >
> > On 10/12/2017 04:09 PM, David Laight wrote:
> > >From: Pankaj Dubey
> > >>Sent: 12 October 2017 08:55
> > >>In pcie-designware.c many places we are calling "usleep_range" which
> > >>are in atomic context. This patch fixes these potential BUGs and
> > >>replaces "usleep_range" with mdelay calls.
> > >>
> > >>Signed-off-by: Pankaj Dubey <pankaj.dubey@samsung.com>
> > >>---
> > >>  drivers/pci/dwc/pcie-designware.c | 8 ++++----
> > >>  drivers/pci/dwc/pcie-designware.h | 3 +--
> > >>  2 files changed, 5 insertions(+), 6 deletions(-)
> > >>
> > >>diff --git a/drivers/pci/dwc/pcie-designware.c b/drivers/pci/dwc/pcie-designware.c
> > >>index 88abddd..35d19b9 100644
> > >>--- a/drivers/pci/dwc/pcie-designware.c
> > >>+++ b/drivers/pci/dwc/pcie-designware.c
> > >>@@ -138,7 +138,7 @@ static void dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, int index,
> > >>  		if (val & PCIE_ATU_ENABLE)
> > >>  			return;
> > >>
> > >>-		usleep_range(LINK_WAIT_IATU_MIN, LINK_WAIT_IATU_MAX);
> > >>+		mdelay(LINK_WAIT_IATU_MIN);
> > >>  	}
> > >Spinning for 9ms (possibly 10 times) isn't really a good idea.
> >
> > Yes. It may not be a good idea, however in our experiment it never
> > hit maximum retry count.  I just converted usleep_range to mdelay
> > keeping min time limitation as it is, though I am not sure, how do
> > we arrived on these numbers in original code, may be Joao Pinto from
> > Synopsys have some idea, I will try to do few experiment and try to
> > find out what is sufficient minimum time in our hardware for these
> > mdelay.
> 
> Just based on the preceding comment, it looks like the wait is
> essential because subsequent config and I/O accesses won't work
> correctly until the ATU enable takes effect.
> 
> If we timeout here, I suspect it's because something is seriously
> wrong in the hardware, so I doubt there's any point in trying to
> minimize the timeout period.  If something is that broken, it doesn't
> matter whether we wait 9ms or 900ms.

Even one 9ms wait is extreme.
How long does this usually take?
...

	David

RE: [PATCH] PCI: dwc: designware: don't sleep in atomic context

[David Laight]

From: Bjorn Helgaas
> Sent: 20 October 2017 19:19
> On Fri, Oct 13, 2017 at 09:10:38AM +0530, Pankaj Dubey wrote:
> >
> >
> > On 10/12/2017 04:09 PM, David Laight wrote:
> > >From: Pankaj Dubey
> > >>Sent: 12 October 2017 08:55
> > >>In pcie-designware.c many places we are calling "usleep_range" which
> > >>are in atomic context. This patch fixes these potential BUGs and
> > >>replaces "usleep_range" with mdelay calls.
> > >>
> > >>Signed-off-by: Pankaj Dubey <pankaj.dubey@samsung.com>
> > >>---
> > >>  drivers/pci/dwc/pcie-designware.c | 8 ++++----
> > >>  drivers/pci/dwc/pcie-designware.h | 3 +--
> > >>  2 files changed, 5 insertions(+), 6 deletions(-)
> > >>
> > >>diff --git a/drivers/pci/dwc/pcie-designware.c b/drivers/pci/dwc/pcie=
-designware.c
> > >>index 88abddd..35d19b9 100644
> > >>--- a/drivers/pci/dwc/pcie-designware.c
> > >>+++ b/drivers/pci/dwc/pcie-designware.c
> > >>@@ -138,7 +138,7 @@ static void dw_pcie_prog_outbound_atu_unroll(stru=
ct dw_pcie *pci, int index,
> > >>  		if (val & PCIE_ATU_ENABLE)
> > >>  			return;
> > >>
> > >>-		usleep_range(LINK_WAIT_IATU_MIN, LINK_WAIT_IATU_MAX);
> > >>+		mdelay(LINK_WAIT_IATU_MIN);
> > >>  	}
> > >Spinning for 9ms (possibly 10 times) isn't really a good idea.
> >
> > Yes. It may not be a good idea, however in our experiment it never
> > hit maximum retry count.  I just converted usleep_range to mdelay
> > keeping min time limitation as it is, though I am not sure, how do
> > we arrived on these numbers in original code, may be Joao Pinto from
> > Synopsys have some idea, I will try to do few experiment and try to
> > find out what is sufficient minimum time in our hardware for these
> > mdelay.
>=20
> Just based on the preceding comment, it looks like the wait is
> essential because subsequent config and I/O accesses won't work
> correctly until the ATU enable takes effect.
>=20
> If we timeout here, I suspect it's because something is seriously
> wrong in the hardware, so I doubt there's any point in trying to
> minimize the timeout period.  If something is that broken, it doesn't
> matter whether we wait 9ms or 900ms.

Even one 9ms wait is extreme.
How long does this usually take?
...

	David

Re: [PATCH] PCI: dwc: designware: don't sleep in atomic context

[Bjorn Helgaas]

On Fri, Oct 20, 2017 at 01:19:17PM -0500, Bjorn Helgaas wrote:
> On Fri, Oct 13, 2017 at 09:10:38AM +0530, Pankaj Dubey wrote:
> > 
> > 
> > On 10/12/2017 04:09 PM, David Laight wrote:
> > >From: Pankaj Dubey
> > >>Sent: 12 October 2017 08:55
> > >>In pcie-designware.c many places we are calling "usleep_range" which
> > >>are in atomic context. This patch fixes these potential BUGs and
> > >>replaces "usleep_range" with mdelay calls.
> > >>
> > >>Signed-off-by: Pankaj Dubey <pankaj.dubey@samsung.com>
> > >>---
> > >>  drivers/pci/dwc/pcie-designware.c | 8 ++++----
> > >>  drivers/pci/dwc/pcie-designware.h | 3 +--
> > >>  2 files changed, 5 insertions(+), 6 deletions(-)
> > >>
> > >>diff --git a/drivers/pci/dwc/pcie-designware.c b/drivers/pci/dwc/pcie-designware.c
> > >>index 88abddd..35d19b9 100644
> > >>--- a/drivers/pci/dwc/pcie-designware.c
> > >>+++ b/drivers/pci/dwc/pcie-designware.c
> > >>@@ -138,7 +138,7 @@ static void dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, int index,
> > >>  		if (val & PCIE_ATU_ENABLE)
> > >>  			return;
> > >>
> > >>-		usleep_range(LINK_WAIT_IATU_MIN, LINK_WAIT_IATU_MAX);
> > >>+		mdelay(LINK_WAIT_IATU_MIN);
> > >>  	}
> > >Spinning for 9ms (possibly 10 times) isn't really a good idea.
> > 
> > Yes. It may not be a good idea, however in our experiment it never
> > hit maximum retry count.  I just converted usleep_range to mdelay
> > keeping min time limitation as it is, though I am not sure, how do
> > we arrived on these numbers in original code, may be Joao Pinto from
> > Synopsys have some idea, I will try to do few experiment and try to
> > find out what is sufficient minimum time in our hardware for these
> > mdelay.
> 
> Just based on the preceding comment, it looks like the wait is
> essential because subsequent config and I/O accesses won't work
> correctly until the ATU enable takes effect.
> 
> If we timeout here, I suspect it's because something is seriously
> wrong in the hardware, so I doubt there's any point in trying to
> minimize the timeout period.  If something is that broken, it doesn't
> matter whether we wait 9ms or 900ms.
> 
> Maybe the message should be more strident or maybe we should even
> return failure so the caller can do something, e.g., fail an access,
> instead of just printing an error and continuing on.
> 
> I'm also looking for an ack from Joao and/or Jingoo.

Dropping for lack of ack.

Re: [PATCH] PCI: dwc: designware: don't sleep in atomic context

[Jingoo Han]

On Thursday, October 12, 2017 3:55 AM, Pankaj Dubey wrote:
> 
> In pcie-designware.c many places we are calling "usleep_range" which
> are in atomic context. This patch fixes these potential BUGs and
> replaces "usleep_range" with mdelay calls.
> 
> Signed-off-by: Pankaj Dubey <pankaj.dubey@samsung.com>

Sorry for my late response.

Acked-by: Jingoo Han <jingoohan1@gmail.com>

Best regards,
Jingoo Han

> ---
>  drivers/pci/dwc/pcie-designware.c | 8 ++++----
>  drivers/pci/dwc/pcie-designware.h | 3 +--
>  2 files changed, 5 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/pci/dwc/pcie-designware.c b/drivers/pci/dwc/pcie-
> designware.c
> index 88abddd..35d19b9 100644
> --- a/drivers/pci/dwc/pcie-designware.c
> +++ b/drivers/pci/dwc/pcie-designware.c
> @@ -138,7 +138,7 @@ static void dw_pcie_prog_outbound_atu_unroll(struct
> dw_pcie *pci, int index,
>  		if (val & PCIE_ATU_ENABLE)
>  			return;
> 
> -		usleep_range(LINK_WAIT_IATU_MIN, LINK_WAIT_IATU_MAX);
> +		mdelay(LINK_WAIT_IATU_MIN);
>  	}
>  	dev_err(pci->dev, "outbound iATU is not being enabled\n");
>  }
> @@ -181,7 +181,7 @@ void dw_pcie_prog_outbound_atu(struct dw_pcie *pci,
> int index, int type,
>  		if (val & PCIE_ATU_ENABLE)
>  			return;
> 
> -		usleep_range(LINK_WAIT_IATU_MIN, LINK_WAIT_IATU_MAX);
> +		mdelay(LINK_WAIT_IATU_MIN);
>  	}
>  	dev_err(pci->dev, "outbound iATU is not being enabled\n");
>  }
> @@ -239,7 +239,7 @@ static int dw_pcie_prog_inbound_atu_unroll(struct
> dw_pcie *pci, int index,
>  		if (val & PCIE_ATU_ENABLE)
>  			return 0;
> 
> -		usleep_range(LINK_WAIT_IATU_MIN, LINK_WAIT_IATU_MAX);
> +		mdelay(LINK_WAIT_IATU_MIN);
>  	}
>  	dev_err(pci->dev, "inbound iATU is not being enabled\n");
> 
> @@ -285,7 +285,7 @@ int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, int
> index, int bar,
>  		if (val & PCIE_ATU_ENABLE)
>  			return 0;
> 
> -		usleep_range(LINK_WAIT_IATU_MIN, LINK_WAIT_IATU_MAX);
> +		mdelay(LINK_WAIT_IATU_MIN);
>  	}
>  	dev_err(pci->dev, "inbound iATU is not being enabled\n");
> 
> diff --git a/drivers/pci/dwc/pcie-designware.h b/drivers/pci/dwc/pcie-
> designware.h
> index e5d9d77..13c17c9 100644
> --- a/drivers/pci/dwc/pcie-designware.h
> +++ b/drivers/pci/dwc/pcie-designware.h
> @@ -28,8 +28,7 @@
> 
>  /* Parameters for the waiting for iATU enabled routine */
>  #define LINK_WAIT_MAX_IATU_RETRIES	5
> -#define LINK_WAIT_IATU_MIN		9000
> -#define LINK_WAIT_IATU_MAX		10000
> +#define LINK_WAIT_IATU_MIN		9
> 
>  /* Synopsys-specific PCIe configuration registers */
>  #define PCIE_PORT_LINK_CONTROL		0x710
> --
> 2.7.4

Re: [PATCH] PCI: dwc: designware: don't sleep in atomic context

[Pankaj Dubey]

Hi Jingoo,


On 12/21/2017 10:09 PM, Jingoo Han wrote:
> On Thursday, October 12, 2017 3:55 AM, Pankaj Dubey wrote:
>> In pcie-designware.c many places we are calling "usleep_range" which
>> are in atomic context. This patch fixes these potential BUGs and
>> replaces "usleep_range" with mdelay calls.
>>
>> Signed-off-by: Pankaj Dubey <pankaj.dubey@samsung.com>
> Sorry for my late response.
>
> Acked-by: Jingoo Han <jingoohan1@gmail.com>
>
> Best regards,
> Jingoo Han

Thanks for review and Ack. Will resubmit this along with your ack.

Thanks,
Pankaj Dubey