[RESENDING RFC PATCH 0/4] Add logic to consolidate TRBs for Synopsys xHC

[RESENDING RFC PATCH 0/4] Add logic to consolidate TRBs for Synopsys xHC

[Tejas Joglekar]

The Synopsys xHC has an internal TRB cache of size TRB_CACHE_SIZE for
each endpoint. The default value for TRB_CACHE_SIZE is 16 for SS and 8
for HS. The controller loads and updates the TRB cache from the
transfer ring in system memory whenever the driver issues a start
transfer or update transfer command.

For chained TRBs, the Synopsys xHC requires that the total amount of
bytes for all TRBs loaded in the TRB cache be greater than or equal to
1 MPS. Or the chain ends within the TRB cache (with a last TRB).

If this requirement is not met, the controller will not be able to
send or receive a packet and it will hang causing a driver timeout and
error.

This patch set adds logic to the XHCI driver to detect and prevent this
from happening along with the quirk to enable this logic for Synopsys
HAPS platform.

Based on Mathias's feedback on previous implementation where consolidation
was done in TRB cache, with this patch series the implementation is done
during mapping of the URB by consolidating the SG list into a temporary
buffer if the SG list buffer sizes within TRB_CACHE_SIZE is less than MPS.


Tejas Joglekar (4):
  dt-bindings: usb: Add snps,consolidate-sgl & consolidate-sgl
  usb: xhci: Set quirk for XHCI_CONSOLIDATE_SG_LIST
  usb: dwc3: Add device property consolidate-sgl
  usb: xhci: Use temporary buffer to consolidate SG

Resending as 'umlaut' is not accepted by some servers in email.

 Documentation/devicetree/bindings/usb/dwc3.txt     |   3 +
 Documentation/devicetree/bindings/usb/usb-xhci.txt |   3 +
 drivers/usb/core/hcd.c                             |   8 ++
 drivers/usb/dwc3/core.c                            |   2 +
 drivers/usb/dwc3/core.h                            |   2 +
 drivers/usb/dwc3/dwc3-haps.c                       |   1 +
 drivers/usb/dwc3/host.c                            |   5 +-
 drivers/usb/host/xhci-pci.c                        |   3 +
 drivers/usb/host/xhci-plat.c                       |   3 +
 drivers/usb/host/xhci-ring.c                       |   2 +-
 drivers/usb/host/xhci.c                            | 128 +++++++++++++++++++++
 drivers/usb/host/xhci.h                            |   5 +
 12 files changed, 163 insertions(+), 2 deletions(-)

-- 
2.11.0

[RFC PATCH 1/4] dt-bindings: usb: Add snps,consolidate-sgl & consolidate-sgl

[Tejas Joglekar]

This commit adds the documentation for consolidate-sgl, and
snps,consolidate-sgl property. These when set enables the quirk for
XHCI driver for consolidation of sg list into a temporary buffer when small
buffer sizes are scattered over the sg list not making up to MPS or total
transfer size within TRB cache size with Synopsys xHC.

Signed-off-by: Tejas Joglekar <joglekar@synopsys.com>
---
 Documentation/devicetree/bindings/usb/dwc3.txt     | 3 +++
 Documentation/devicetree/bindings/usb/usb-xhci.txt | 3 +++
 2 files changed, 6 insertions(+)

diff --git a/Documentation/devicetree/bindings/usb/dwc3.txt b/Documentation/devicetree/bindings/usb/dwc3.txt
index 9946ff9ba735..292d1f7969e4 100644
--- a/Documentation/devicetree/bindings/usb/dwc3.txt
+++ b/Documentation/devicetree/bindings/usb/dwc3.txt
@@ -104,6 +104,9 @@ Optional properties:
 			this and tx-thr-num-pkt-prd to a valid, non-zero value
 			1-16 (DWC_usb31 programming guide section 1.2.3) to
 			enable periodic ESS TX threshold.
+ - snps,consolidate-sgl: enable sg list consolidation - host mode only. Set to use
+			SG buffers of at least MPS size by consolidating smaller SG
+			buffers list into a single buffer.
 
  - <DEPRECATED> tx-fifo-resize: determines if the FIFO *has* to be reallocated.
  - snps,incr-burst-type-adjustment: Value for INCR burst type of GSBUSCFG0
diff --git a/Documentation/devicetree/bindings/usb/usb-xhci.txt b/Documentation/devicetree/bindings/usb/usb-xhci.txt
index 3f378951d624..a90d853557ee 100644
--- a/Documentation/devicetree/bindings/usb/usb-xhci.txt
+++ b/Documentation/devicetree/bindings/usb/usb-xhci.txt
@@ -43,6 +43,9 @@ Optional properties:
   - quirk-broken-port-ped: set if the controller has broken port disable mechanism
   - imod-interval-ns: default interrupt moderation interval is 5000ns
   - phys : see usb-hcd.yaml in the current directory
+  - consolidate-sgl: indicate if you need to consolidate sg list into a
+    temporary buffer when small SG buffer sizes does not make upto MPS
+    size or total transfer size across the TRB cache size.
 
 additionally the properties from usb-hcd.yaml (in the current directory) are
 supported.
-- 
2.11.0

[RESENDING RFC PATCH 1/4] dt-bindings: usb: Add snps,consolidate-sgl & consolidate-sgl

[Tejas Joglekar]

This commit adds the documentation for consolidate-sgl, and
snps,consolidate-sgl property. These when set enables the quirk for
XHCI driver for consolidation of sg list into a temporary buffer when small
buffer sizes are scattered over the sg list not making up to MPS or total
transfer size within TRB cache size with Synopsys xHC.

Signed-off-by: Tejas Joglekar <joglekar@synopsys.com>
---
 Documentation/devicetree/bindings/usb/dwc3.txt     | 3 +++
 Documentation/devicetree/bindings/usb/usb-xhci.txt | 3 +++
 2 files changed, 6 insertions(+)

diff --git a/Documentation/devicetree/bindings/usb/dwc3.txt b/Documentation/devicetree/bindings/usb/dwc3.txt
index 9946ff9ba735..292d1f7969e4 100644
--- a/Documentation/devicetree/bindings/usb/dwc3.txt
+++ b/Documentation/devicetree/bindings/usb/dwc3.txt
@@ -104,6 +104,9 @@ Optional properties:
 			this and tx-thr-num-pkt-prd to a valid, non-zero value
 			1-16 (DWC_usb31 programming guide section 1.2.3) to
 			enable periodic ESS TX threshold.
+ - snps,consolidate-sgl: enable sg list consolidation - host mode only. Set to use
+			SG buffers of at least MPS size by consolidating smaller SG
+			buffers list into a single buffer.
 
  - <DEPRECATED> tx-fifo-resize: determines if the FIFO *has* to be reallocated.
  - snps,incr-burst-type-adjustment: Value for INCR burst type of GSBUSCFG0
diff --git a/Documentation/devicetree/bindings/usb/usb-xhci.txt b/Documentation/devicetree/bindings/usb/usb-xhci.txt
index 3f378951d624..a90d853557ee 100644
--- a/Documentation/devicetree/bindings/usb/usb-xhci.txt
+++ b/Documentation/devicetree/bindings/usb/usb-xhci.txt
@@ -43,6 +43,9 @@ Optional properties:
   - quirk-broken-port-ped: set if the controller has broken port disable mechanism
   - imod-interval-ns: default interrupt moderation interval is 5000ns
   - phys : see usb-hcd.yaml in the current directory
+  - consolidate-sgl: indicate if you need to consolidate sg list into a
+    temporary buffer when small SG buffer sizes does not make upto MPS
+    size or total transfer size across the TRB cache size.
 
 additionally the properties from usb-hcd.yaml (in the current directory) are
 supported.
-- 
2.11.0

Re: [RESENDING RFC PATCH 1/4] dt-bindings: usb: Add snps,consolidate-sgl & consolidate-sgl

[Greg KH]

On Fri, Mar 27, 2020 at 03:11:56PM +0530, Tejas Joglekar wrote:
> This commit adds the documentation for consolidate-sgl, and
> snps,consolidate-sgl property. These when set enables the quirk for
> XHCI driver for consolidation of sg list into a temporary buffer when small
> buffer sizes are scattered over the sg list not making up to MPS or total
> transfer size within TRB cache size with Synopsys xHC.
> 
> Signed-off-by: Tejas Joglekar <joglekar@synopsys.com>
> ---
>  Documentation/devicetree/bindings/usb/dwc3.txt     | 3 +++
>  Documentation/devicetree/bindings/usb/usb-xhci.txt | 3 +++
>  2 files changed, 6 insertions(+)
> 
> diff --git a/Documentation/devicetree/bindings/usb/dwc3.txt b/Documentation/devicetree/bindings/usb/dwc3.txt
> index 9946ff9ba735..292d1f7969e4 100644
> --- a/Documentation/devicetree/bindings/usb/dwc3.txt
> +++ b/Documentation/devicetree/bindings/usb/dwc3.txt
> @@ -104,6 +104,9 @@ Optional properties:
>  			this and tx-thr-num-pkt-prd to a valid, non-zero value
>  			1-16 (DWC_usb31 programming guide section 1.2.3) to
>  			enable periodic ESS TX threshold.
> + - snps,consolidate-sgl: enable sg list consolidation - host mode only. Set to use
> +			SG buffers of at least MPS size by consolidating smaller SG
> +			buffers list into a single buffer.
>  
>   - <DEPRECATED> tx-fifo-resize: determines if the FIFO *has* to be reallocated.
>   - snps,incr-burst-type-adjustment: Value for INCR burst type of GSBUSCFG0
> diff --git a/Documentation/devicetree/bindings/usb/usb-xhci.txt b/Documentation/devicetree/bindings/usb/usb-xhci.txt
> index 3f378951d624..a90d853557ee 100644
> --- a/Documentation/devicetree/bindings/usb/usb-xhci.txt
> +++ b/Documentation/devicetree/bindings/usb/usb-xhci.txt
> @@ -43,6 +43,9 @@ Optional properties:
>    - quirk-broken-port-ped: set if the controller has broken port disable mechanism
>    - imod-interval-ns: default interrupt moderation interval is 5000ns
>    - phys : see usb-hcd.yaml in the current directory
> +  - consolidate-sgl: indicate if you need to consolidate sg list into a
> +    temporary buffer when small SG buffer sizes does not make upto MPS
> +    size or total transfer size across the TRB cache size.

Shouldn't this refer to the fact that the hardware is broken?  Otherwise
why would anyone know if they should, or should not, enable this option?

thanks,

greg k-h

Re: [RESENDING RFC PATCH 1/4] dt-bindings: usb: Add snps,consolidate-sgl & consolidate-sgl

[Tejas Joglekar]

Hi,
On 3/27/2020 3:24 PM, Greg KH wrote:
> On Fri, Mar 27, 2020 at 03:11:56PM +0530, Tejas Joglekar wrote:
>> This commit adds the documentation for consolidate-sgl, and
>> snps,consolidate-sgl property. These when set enables the quirk for
>> XHCI driver for consolidation of sg list into a temporary buffer when small
>> buffer sizes are scattered over the sg list not making up to MPS or total
>> transfer size within TRB cache size with Synopsys xHC.
>>
>> Signed-off-by: Tejas Joglekar <joglekar@synopsys.com>
>> ---
>>  Documentation/devicetree/bindings/usb/dwc3.txt     | 3 +++
>>  Documentation/devicetree/bindings/usb/usb-xhci.txt | 3 +++
>>  2 files changed, 6 insertions(+)
>>
>> diff --git a/Documentation/devicetree/bindings/usb/dwc3.txt b/Documentation/devicetree/bindings/usb/dwc3.txt
>> index 9946ff9ba735..292d1f7969e4 100644
>> --- a/Documentation/devicetree/bindings/usb/dwc3.txt
>> +++ b/Documentation/devicetree/bindings/usb/dwc3.txt
>> @@ -104,6 +104,9 @@ Optional properties:
>>  			this and tx-thr-num-pkt-prd to a valid, non-zero value
>>  			1-16 (DWC_usb31 programming guide section 1.2.3) to
>>  			enable periodic ESS TX threshold.
>> + - snps,consolidate-sgl: enable sg list consolidation - host mode only. Set to use
>> +			SG buffers of at least MPS size by consolidating smaller SG
>> +			buffers list into a single buffer.
>>  
>>   - <DEPRECATED> tx-fifo-resize: determines if the FIFO *has* to be reallocated.
>>   - snps,incr-burst-type-adjustment: Value for INCR burst type of GSBUSCFG0
>> diff --git a/Documentation/devicetree/bindings/usb/usb-xhci.txt b/Documentation/devicetree/bindings/usb/usb-xhci.txt
>> index 3f378951d624..a90d853557ee 100644
>> --- a/Documentation/devicetree/bindings/usb/usb-xhci.txt
>> +++ b/Documentation/devicetree/bindings/usb/usb-xhci.txt
>> @@ -43,6 +43,9 @@ Optional properties:
>>    - quirk-broken-port-ped: set if the controller has broken port disable mechanism
>>    - imod-interval-ns: default interrupt moderation interval is 5000ns
>>    - phys : see usb-hcd.yaml in the current directory
>> +  - consolidate-sgl: indicate if you need to consolidate sg list into a
>> +    temporary buffer when small SG buffer sizes does not make upto MPS
>> +    size or total transfer size across the TRB cache size.
> 
> Shouldn't this refer to the fact that the hardware is broken?  Otherwise
> why would anyone know if they should, or should not, enable this option?
>
We have not seen issue with Linux environment for now. But with other OS with
SG list with very small buffer sizes the xHC controller hang was seen. So 
currently introducing the binding as optional one. One could enable this 
option when xHC halt happens due to small SG list sizes.  
 
> thanks,
> 
> greg k-h
> 

Thanks & Regards,
 Tejas Joglekar

Re: [RESENDING RFC PATCH 1/4] dt-bindings: usb: Add snps,consolidate-sgl & consolidate-sgl

[Greg KH]

On Fri, Mar 27, 2020 at 10:11:59AM +0000, Tejas Joglekar wrote:
> Hi,
> On 3/27/2020 3:24 PM, Greg KH wrote:
> > On Fri, Mar 27, 2020 at 03:11:56PM +0530, Tejas Joglekar wrote:
> >> This commit adds the documentation for consolidate-sgl, and
> >> snps,consolidate-sgl property. These when set enables the quirk for
> >> XHCI driver for consolidation of sg list into a temporary buffer when small
> >> buffer sizes are scattered over the sg list not making up to MPS or total
> >> transfer size within TRB cache size with Synopsys xHC.
> >>
> >> Signed-off-by: Tejas Joglekar <joglekar@synopsys.com>
> >> ---
> >>  Documentation/devicetree/bindings/usb/dwc3.txt     | 3 +++
> >>  Documentation/devicetree/bindings/usb/usb-xhci.txt | 3 +++
> >>  2 files changed, 6 insertions(+)
> >>
> >> diff --git a/Documentation/devicetree/bindings/usb/dwc3.txt b/Documentation/devicetree/bindings/usb/dwc3.txt
> >> index 9946ff9ba735..292d1f7969e4 100644
> >> --- a/Documentation/devicetree/bindings/usb/dwc3.txt
> >> +++ b/Documentation/devicetree/bindings/usb/dwc3.txt
> >> @@ -104,6 +104,9 @@ Optional properties:
> >>  			this and tx-thr-num-pkt-prd to a valid, non-zero value
> >>  			1-16 (DWC_usb31 programming guide section 1.2.3) to
> >>  			enable periodic ESS TX threshold.
> >> + - snps,consolidate-sgl: enable sg list consolidation - host mode only. Set to use
> >> +			SG buffers of at least MPS size by consolidating smaller SG
> >> +			buffers list into a single buffer.
> >>  
> >>   - <DEPRECATED> tx-fifo-resize: determines if the FIFO *has* to be reallocated.
> >>   - snps,incr-burst-type-adjustment: Value for INCR burst type of GSBUSCFG0
> >> diff --git a/Documentation/devicetree/bindings/usb/usb-xhci.txt b/Documentation/devicetree/bindings/usb/usb-xhci.txt
> >> index 3f378951d624..a90d853557ee 100644
> >> --- a/Documentation/devicetree/bindings/usb/usb-xhci.txt
> >> +++ b/Documentation/devicetree/bindings/usb/usb-xhci.txt
> >> @@ -43,6 +43,9 @@ Optional properties:
> >>    - quirk-broken-port-ped: set if the controller has broken port disable mechanism
> >>    - imod-interval-ns: default interrupt moderation interval is 5000ns
> >>    - phys : see usb-hcd.yaml in the current directory
> >> +  - consolidate-sgl: indicate if you need to consolidate sg list into a
> >> +    temporary buffer when small SG buffer sizes does not make upto MPS
> >> +    size or total transfer size across the TRB cache size.
> > 
> > Shouldn't this refer to the fact that the hardware is broken?  Otherwise
> > why would anyone know if they should, or should not, enable this option?
> >
> We have not seen issue with Linux environment for now. But with other OS with
> SG list with very small buffer sizes the xHC controller hang was seen. So 
> currently introducing the binding as optional one. One could enable this 
> option when xHC halt happens due to small SG list sizes.  

What I mean is this should be something like,
"quirk-broken-sg-list-handler" or something like that.  Otherwise how
does anyone know if this really is needed or not.  Reading this would
seem like everyone would like to do this, as consolidating links
sounds like a good optimization, when instead this really cause more
memory allocations, making this possibly worse performance.

thanks,

greg k-h

Re: [RESENDING RFC PATCH 1/4] dt-bindings: usb: Add snps,consolidate-sgl & consolidate-sgl

[Tejas Joglekar]

On 3/27/2020 3:54 PM, Greg KH wrote:
> On Fri, Mar 27, 2020 at 10:11:59AM +0000, Tejas Joglekar wrote:
>> Hi,
>> On 3/27/2020 3:24 PM, Greg KH wrote:
>>> On Fri, Mar 27, 2020 at 03:11:56PM +0530, Tejas Joglekar wrote:
>>>> This commit adds the documentation for consolidate-sgl, and
>>>> snps,consolidate-sgl property. These when set enables the quirk for
>>>> XHCI driver for consolidation of sg list into a temporary buffer when small
>>>> buffer sizes are scattered over the sg list not making up to MPS or total
>>>> transfer size within TRB cache size with Synopsys xHC.
>>>>
>>>> Signed-off-by: Tejas Joglekar <joglekar@synopsys.com>
>>>> ---
>>>>  Documentation/devicetree/bindings/usb/dwc3.txt     | 3 +++
>>>>  Documentation/devicetree/bindings/usb/usb-xhci.txt | 3 +++
>>>>  2 files changed, 6 insertions(+)
>>>>
>>>> diff --git a/Documentation/devicetree/bindings/usb/dwc3.txt b/Documentation/devicetree/bindings/usb/dwc3.txt
>>>> index 9946ff9ba735..292d1f7969e4 100644
>>>> --- a/Documentation/devicetree/bindings/usb/dwc3.txt
>>>> +++ b/Documentation/devicetree/bindings/usb/dwc3.txt
>>>> @@ -104,6 +104,9 @@ Optional properties:
>>>>  			this and tx-thr-num-pkt-prd to a valid, non-zero value
>>>>  			1-16 (DWC_usb31 programming guide section 1.2.3) to
>>>>  			enable periodic ESS TX threshold.
>>>> + - snps,consolidate-sgl: enable sg list consolidation - host mode only. Set to use
>>>> +			SG buffers of at least MPS size by consolidating smaller SG
>>>> +			buffers list into a single buffer.
>>>>  
>>>>   - <DEPRECATED> tx-fifo-resize: determines if the FIFO *has* to be reallocated.
>>>>   - snps,incr-burst-type-adjustment: Value for INCR burst type of GSBUSCFG0
>>>> diff --git a/Documentation/devicetree/bindings/usb/usb-xhci.txt b/Documentation/devicetree/bindings/usb/usb-xhci.txt
>>>> index 3f378951d624..a90d853557ee 100644
>>>> --- a/Documentation/devicetree/bindings/usb/usb-xhci.txt
>>>> +++ b/Documentation/devicetree/bindings/usb/usb-xhci.txt
>>>> @@ -43,6 +43,9 @@ Optional properties:
>>>>    - quirk-broken-port-ped: set if the controller has broken port disable mechanism
>>>>    - imod-interval-ns: default interrupt moderation interval is 5000ns
>>>>    - phys : see usb-hcd.yaml in the current directory
>>>> +  - consolidate-sgl: indicate if you need to consolidate sg list into a
>>>> +    temporary buffer when small SG buffer sizes does not make upto MPS
>>>> +    size or total transfer size across the TRB cache size.
>>>
>>> Shouldn't this refer to the fact that the hardware is broken?  Otherwise
>>> why would anyone know if they should, or should not, enable this option?
>>>
>> We have not seen issue with Linux environment for now. But with other OS with
>> SG list with very small buffer sizes the xHC controller hang was seen. So 
>> currently introducing the binding as optional one. One could enable this 
>> option when xHC halt happens due to small SG list sizes.  
> 
> What I mean is this should be something like,
> "quirk-broken-sg-list-handler" or something like that.  Otherwise how
> does anyone know if this really is needed or not.  Reading this would
> seem like everyone would like to do this, as consolidating links
> sounds like a good optimization, when instead this really cause more
> memory allocations, making this possibly worse performance.
> 
Sure, understood. I will rename this entry.
> thanks,
> 
> greg k-h
> 

Thanks & Regards,
 Tejas Joglekar

Re: [RESENDING RFC PATCH 1/4] dt-bindings: usb: Add snps,consolidate-sgl & consolidate-sgl

[Rob Herring]

On Fri, Mar 27, 2020 at 03:11:56PM +0530, Tejas Joglekar wrote:
> This commit adds the documentation for consolidate-sgl, and
> snps,consolidate-sgl property. These when set enables the quirk for
> XHCI driver for consolidation of sg list into a temporary buffer when small
> buffer sizes are scattered over the sg list not making up to MPS or total
> transfer size within TRB cache size with Synopsys xHC.
> 
> Signed-off-by: Tejas Joglekar <joglekar@synopsys.com>
> ---
>  Documentation/devicetree/bindings/usb/dwc3.txt     | 3 +++
>  Documentation/devicetree/bindings/usb/usb-xhci.txt | 3 +++
>  2 files changed, 6 insertions(+)
> 
> diff --git a/Documentation/devicetree/bindings/usb/dwc3.txt b/Documentation/devicetree/bindings/usb/dwc3.txt
> index 9946ff9ba735..292d1f7969e4 100644
> --- a/Documentation/devicetree/bindings/usb/dwc3.txt
> +++ b/Documentation/devicetree/bindings/usb/dwc3.txt
> @@ -104,6 +104,9 @@ Optional properties:
>  			this and tx-thr-num-pkt-prd to a valid, non-zero value
>  			1-16 (DWC_usb31 programming guide section 1.2.3) to
>  			enable periodic ESS TX threshold.
> + - snps,consolidate-sgl: enable sg list consolidation - host mode only. Set to use
> +			SG buffers of at least MPS size by consolidating smaller SG
> +			buffers list into a single buffer.

The preference is not to keep adding properties for every single quirk 
or feature. These should be implied by specific compatibles. As Synopsys 
knows what quirks/errata/features are in each version of IP, the 
compatible strings should reflect those versions. (And yes, I'm sure 
there's customer ECO fixes that aren't reflected in the version, but 
that's why we have SoC specific compatibles too.) This is the only way 
we can fix quirks in the OS without doing DT updates. For comparison, do 
you want to have to update your PC BIOS so an OS can work-around issues?

Rob

Re: [RESENDING RFC PATCH 1/4] dt-bindings: usb: Add snps,consolidate-sgl & consolidate-sgl

[Tejas Joglekar]

Hi,
On 4/5/2020 7:15 AM, Rob Herring wrote:
> On Fri, Mar 27, 2020 at 03:11:56PM +0530, Tejas Joglekar wrote:
>> This commit adds the documentation for consolidate-sgl, and
>> snps,consolidate-sgl property. These when set enables the quirk for
>> XHCI driver for consolidation of sg list into a temporary buffer when small
>> buffer sizes are scattered over the sg list not making up to MPS or total
>> transfer size within TRB cache size with Synopsys xHC.
>>
>> Signed-off-by: Tejas Joglekar <joglekar@synopsys.com>
>> ---
>>  Documentation/devicetree/bindings/usb/dwc3.txt     | 3 +++
>>  Documentation/devicetree/bindings/usb/usb-xhci.txt | 3 +++
>>  2 files changed, 6 insertions(+)
>>
>> diff --git a/Documentation/devicetree/bindings/usb/dwc3.txt b/Documentation/devicetree/bindings/usb/dwc3.txt
>> index 9946ff9ba735..292d1f7969e4 100644
>> --- a/Documentation/devicetree/bindings/usb/dwc3.txt
>> +++ b/Documentation/devicetree/bindings/usb/dwc3.txt
>> @@ -104,6 +104,9 @@ Optional properties:
>>  			this and tx-thr-num-pkt-prd to a valid, non-zero value
>>  			1-16 (DWC_usb31 programming guide section 1.2.3) to
>>  			enable periodic ESS TX threshold.
>> + - snps,consolidate-sgl: enable sg list consolidation - host mode only. Set to use
>> +			SG buffers of at least MPS size by consolidating smaller SG
>> +			buffers list into a single buffer.
> 
> The preference is not to keep adding properties for every single quirk 
> or feature. These should be implied by specific compatibles. As Synopsys 
> knows what quirks/errata/features are in each version of IP, the 
> compatible strings should reflect those versions. (And yes, I'm sure 
> there's customer ECO fixes that aren't reflected in the version, but 
> that's why we have SoC specific compatibles too.) This is the only way 
> we can fix quirks in the OS without doing DT updates. For comparison, do 
> you want to have to update your PC BIOS so an OS can work-around issues?
>
Yes, I understand what you want to say here. But I think this compatible string
does not work with the platform drivers with PCI based systems. So based on the vendor 
id and device id then I need to set the quirk required. 

@Felipe: What do you suggest for setting up quirk without the DT update for dwc3?
 
> Rob
> 

Thanks & Regards,
 Tejas Joglekar

Re: [RESENDING RFC PATCH 1/4] dt-bindings: usb: Add snps,consolidate-sgl & consolidate-sgl

[Felipe Balbi]

[-- Attachment #1: Type: text/plain, Size: 2694 bytes --]


Hi,

Tejas Joglekar <Tejas.Joglekar@synopsys.com> writes:
> Hi,
> On 4/5/2020 7:15 AM, Rob Herring wrote:
>> On Fri, Mar 27, 2020 at 03:11:56PM +0530, Tejas Joglekar wrote:
>>> This commit adds the documentation for consolidate-sgl, and
>>> snps,consolidate-sgl property. These when set enables the quirk for
>>> XHCI driver for consolidation of sg list into a temporary buffer when small
>>> buffer sizes are scattered over the sg list not making up to MPS or total
>>> transfer size within TRB cache size with Synopsys xHC.
>>>
>>> Signed-off-by: Tejas Joglekar <joglekar@synopsys.com>
>>> ---
>>>  Documentation/devicetree/bindings/usb/dwc3.txt     | 3 +++
>>>  Documentation/devicetree/bindings/usb/usb-xhci.txt | 3 +++
>>>  2 files changed, 6 insertions(+)
>>>
>>> diff --git a/Documentation/devicetree/bindings/usb/dwc3.txt b/Documentation/devicetree/bindings/usb/dwc3.txt
>>> index 9946ff9ba735..292d1f7969e4 100644
>>> --- a/Documentation/devicetree/bindings/usb/dwc3.txt
>>> +++ b/Documentation/devicetree/bindings/usb/dwc3.txt
>>> @@ -104,6 +104,9 @@ Optional properties:
>>>  			this and tx-thr-num-pkt-prd to a valid, non-zero value
>>>  			1-16 (DWC_usb31 programming guide section 1.2.3) to
>>>  			enable periodic ESS TX threshold.
>>> + - snps,consolidate-sgl: enable sg list consolidation - host mode only. Set to use
>>> +			SG buffers of at least MPS size by consolidating smaller SG
>>> +			buffers list into a single buffer.
>> 
>> The preference is not to keep adding properties for every single quirk 
>> or feature. These should be implied by specific compatibles. As Synopsys 
>> knows what quirks/errata/features are in each version of IP, the 
>> compatible strings should reflect those versions. (And yes, I'm sure 
>> there's customer ECO fixes that aren't reflected in the version, but 
>> that's why we have SoC specific compatibles too.) This is the only way 
>> we can fix quirks in the OS without doing DT updates. For comparison, do 
>> you want to have to update your PC BIOS so an OS can work-around issues?
>>
> Yes, I understand what you want to say here. But I think this compatible string
> does not work with the platform drivers with PCI based systems. So based on the vendor 
> id and device id then I need to set the quirk required. 
>
> @Felipe: What do you suggest for setting up quirk without the DT update for dwc3?

We have been using Synopsys controller's revision register, but that's
not visible to xhci driver and we don't have a separate compatible for
each synopsys version on the xhci driver side. One option would be to
add "snps,xhci-foo-bar" to xhci-plat and use that.

-- 
balbi

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: [RESENDING RFC PATCH 1/4] dt-bindings: usb: Add snps,consolidate-sgl & consolidate-sgl

[Tejas Joglekar]

Hi,
On 4/16/2020 1:52 PM, Felipe Balbi wrote:
> 
> Hi,
> 
> Tejas Joglekar <Tejas.Joglekar@synopsys.com> writes:
>> Hi,
>> On 4/5/2020 7:15 AM, Rob Herring wrote:
>>> On Fri, Mar 27, 2020 at 03:11:56PM +0530, Tejas Joglekar wrote:
>>>> This commit adds the documentation for consolidate-sgl, and
>>>> snps,consolidate-sgl property. These when set enables the quirk for
>>>> XHCI driver for consolidation of sg list into a temporary buffer when small
>>>> buffer sizes are scattered over the sg list not making up to MPS or total
>>>> transfer size within TRB cache size with Synopsys xHC.
>>>>
>>>> Signed-off-by: Tejas Joglekar <joglekar@synopsys.com>
>>>> ---
>>>>  Documentation/devicetree/bindings/usb/dwc3.txt     | 3 +++
>>>>  Documentation/devicetree/bindings/usb/usb-xhci.txt | 3 +++
>>>>  2 files changed, 6 insertions(+)
>>>>
>>>> diff --git a/Documentation/devicetree/bindings/usb/dwc3.txt b/Documentation/devicetree/bindings/usb/dwc3.txt
>>>> index 9946ff9ba735..292d1f7969e4 100644
>>>> --- a/Documentation/devicetree/bindings/usb/dwc3.txt
>>>> +++ b/Documentation/devicetree/bindings/usb/dwc3.txt
>>>> @@ -104,6 +104,9 @@ Optional properties:
>>>>  			this and tx-thr-num-pkt-prd to a valid, non-zero value
>>>>  			1-16 (DWC_usb31 programming guide section 1.2.3) to
>>>>  			enable periodic ESS TX threshold.
>>>> + - snps,consolidate-sgl: enable sg list consolidation - host mode only. Set to use
>>>> +			SG buffers of at least MPS size by consolidating smaller SG
>>>> +			buffers list into a single buffer.
>>>
>>> The preference is not to keep adding properties for every single quirk 
>>> or feature. These should be implied by specific compatibles. As Synopsys 
>>> knows what quirks/errata/features are in each version of IP, the 
>>> compatible strings should reflect those versions. (And yes, I'm sure 
>>> there's customer ECO fixes that aren't reflected in the version, but 
>>> that's why we have SoC specific compatibles too.) This is the only way 
>>> we can fix quirks in the OS without doing DT updates. For comparison, do 
>>> you want to have to update your PC BIOS so an OS can work-around issues?
>>>
>> Yes, I understand what you want to say here. But I think this compatible string
>> does not work with the platform drivers with PCI based systems. So based on the vendor 
>> id and device id then I need to set the quirk required. 
>>
>> @Felipe: What do you suggest for setting up quirk without the DT update for dwc3?
> 
> We have been using Synopsys controller's revision register, but that's
> not visible to xhci driver and we don't have a separate compatible for
> each synopsys version on the xhci driver side. One option would be to
> add "snps,xhci-foo-bar" to xhci-plat and use that.
> 
@Felipe: Thanks for suggestion

@Mathias: Do you prefer parameter passed this way? I can send the v2 patch set,
after your input.


Thanks & Regards,
 Tejas Joglekar 

Re: [RESENDING RFC PATCH 1/4] dt-bindings: usb: Add snps,consolidate-sgl & consolidate-sgl

[Mathias Nyman]

On 16.4.2020 19.37, Tejas Joglekar wrote:
> Hi,
> On 4/16/2020 1:52 PM, Felipe Balbi wrote:
>>
>> Hi,
>>
>> Tejas Joglekar <Tejas.Joglekar@synopsys.com> writes:
>>> Hi,
>>> On 4/5/2020 7:15 AM, Rob Herring wrote:
>>>> On Fri, Mar 27, 2020 at 03:11:56PM +0530, Tejas Joglekar wrote:
>>>>> This commit adds the documentation for consolidate-sgl, and
>>>>> snps,consolidate-sgl property. These when set enables the quirk for
>>>>> XHCI driver for consolidation of sg list into a temporary buffer when small
>>>>> buffer sizes are scattered over the sg list not making up to MPS or total
>>>>> transfer size within TRB cache size with Synopsys xHC.
>>>>>
>>>>> Signed-off-by: Tejas Joglekar <joglekar@synopsys.com>
>>>>> ---
>>>>>  Documentation/devicetree/bindings/usb/dwc3.txt     | 3 +++
>>>>>  Documentation/devicetree/bindings/usb/usb-xhci.txt | 3 +++
>>>>>  2 files changed, 6 insertions(+)
>>>>>
>>>>> diff --git a/Documentation/devicetree/bindings/usb/dwc3.txt b/Documentation/devicetree/bindings/usb/dwc3.txt
>>>>> index 9946ff9ba735..292d1f7969e4 100644
>>>>> --- a/Documentation/devicetree/bindings/usb/dwc3.txt
>>>>> +++ b/Documentation/devicetree/bindings/usb/dwc3.txt
>>>>> @@ -104,6 +104,9 @@ Optional properties:
>>>>>  			this and tx-thr-num-pkt-prd to a valid, non-zero value
>>>>>  			1-16 (DWC_usb31 programming guide section 1.2.3) to
>>>>>  			enable periodic ESS TX threshold.
>>>>> + - snps,consolidate-sgl: enable sg list consolidation - host mode only. Set to use
>>>>> +			SG buffers of at least MPS size by consolidating smaller SG
>>>>> +			buffers list into a single buffer.
>>>>
>>>> The preference is not to keep adding properties for every single quirk 
>>>> or feature. These should be implied by specific compatibles. As Synopsys 
>>>> knows what quirks/errata/features are in each version of IP, the 
>>>> compatible strings should reflect those versions. (And yes, I'm sure 
>>>> there's customer ECO fixes that aren't reflected in the version, but 
>>>> that's why we have SoC specific compatibles too.) This is the only way 
>>>> we can fix quirks in the OS without doing DT updates. For comparison, do 
>>>> you want to have to update your PC BIOS so an OS can work-around issues?
>>>>
>>> Yes, I understand what you want to say here. But I think this compatible string
>>> does not work with the platform drivers with PCI based systems. So based on the vendor 
>>> id and device id then I need to set the quirk required. 
>>>
>>> @Felipe: What do you suggest for setting up quirk without the DT update for dwc3?
>>
>> We have been using Synopsys controller's revision register, but that's
>> not visible to xhci driver and we don't have a separate compatible for
>> each synopsys version on the xhci driver side. One option would be to
>> add "snps,xhci-foo-bar" to xhci-plat and use that.
>>
> @Felipe: Thanks for suggestion
> 
> @Mathias: Do you prefer parameter passed this way? I can send the v2 patch set,
> after your input.

should work, I don't see any issues with that.

-Mathias
 

[RFC PATCH 2/4] usb: xhci: Set quirk for XHCI_CONSOLIDATE_SG_LIST

[Tejas Joglekar]

This commit enables the quirk when the consolidate_trbs property is set
for the Synopsys xHC. This patch fixes the SNPS xHC hang issue
when the data is scattered across small buffers which does not make
atleast MPS size for given TRB cache size of SNPS xHC.

Signed-off-by: Tejas Joglekar <joglekar@synopsys.com>
---
 drivers/usb/host/xhci-pci.c  | 3 +++
 drivers/usb/host/xhci-plat.c | 3 +++
 drivers/usb/host/xhci.h      | 1 +
 3 files changed, 7 insertions(+)

diff --git a/drivers/usb/host/xhci-pci.c b/drivers/usb/host/xhci-pci.c
index 766b74723e64..cdda8e2de1c2 100644
--- a/drivers/usb/host/xhci-pci.c
+++ b/drivers/usb/host/xhci-pci.c
@@ -268,6 +268,9 @@ static void xhci_pci_quirks(struct device *dev, struct xhci_hcd *xhci)
 	     pdev->device == 0x9026)
 		xhci->quirks |= XHCI_RESET_PLL_ON_DISCONNECT;
 
+	if (pdev->vendor == PCI_VENDOR_ID_SYNOPSYS)
+		xhci->quirks |= XHCI_CONSOLIDATE_SG_LIST;
+
 	if (xhci->quirks & XHCI_RESET_ON_RESUME)
 		xhci_dbg_trace(xhci, trace_xhci_dbg_quirks,
 				"QUIRK: Resetting on resume");
diff --git a/drivers/usb/host/xhci-plat.c b/drivers/usb/host/xhci-plat.c
index 315b4552693c..8333c78dcf03 100644
--- a/drivers/usb/host/xhci-plat.c
+++ b/drivers/usb/host/xhci-plat.c
@@ -289,6 +289,9 @@ static int xhci_plat_probe(struct platform_device *pdev)
 		if (device_property_read_bool(tmpdev, "quirk-broken-port-ped"))
 			xhci->quirks |= XHCI_BROKEN_PORT_PED;
 
+		if (device_property_read_bool(tmpdev, "consolidate-sgl"))
+			xhci->quirks |= XHCI_CONSOLIDATE_SG_LIST;
+
 		device_property_read_u32(tmpdev, "imod-interval-ns",
 					 &xhci->imod_interval);
 	}
diff --git a/drivers/usb/host/xhci.h b/drivers/usb/host/xhci.h
index 3289bb516201..a093eeaec70e 100644
--- a/drivers/usb/host/xhci.h
+++ b/drivers/usb/host/xhci.h
@@ -1873,6 +1873,7 @@ struct xhci_hcd {
 #define XHCI_DEFAULT_PM_RUNTIME_ALLOW	BIT_ULL(33)
 #define XHCI_RESET_PLL_ON_DISCONNECT	BIT_ULL(34)
 #define XHCI_SNPS_BROKEN_SUSPEND    BIT_ULL(35)
+#define XHCI_CONSOLIDATE_SG_LIST    BIT_ULL(36)
 
 	unsigned int		num_active_eps;
 	unsigned int		limit_active_eps;
-- 
2.11.0

[RESENDING RFC PATCH 2/4] usb: xhci: Set quirk for XHCI_CONSOLIDATE_SG_LIST

[Tejas Joglekar]

This commit enables the quirk when the consolidate_trbs property is set
for the Synopsys xHC. This patch fixes the SNPS xHC hang issue
when the data is scattered across small buffers which does not make
atleast MPS size for given TRB cache size of SNPS xHC.

Signed-off-by: Tejas Joglekar <joglekar@synopsys.com>
---
 drivers/usb/host/xhci-pci.c  | 3 +++
 drivers/usb/host/xhci-plat.c | 3 +++
 drivers/usb/host/xhci.h      | 1 +
 3 files changed, 7 insertions(+)

diff --git a/drivers/usb/host/xhci-pci.c b/drivers/usb/host/xhci-pci.c
index 766b74723e64..cdda8e2de1c2 100644
--- a/drivers/usb/host/xhci-pci.c
+++ b/drivers/usb/host/xhci-pci.c
@@ -268,6 +268,9 @@ static void xhci_pci_quirks(struct device *dev, struct xhci_hcd *xhci)
 	     pdev->device == 0x9026)
 		xhci->quirks |= XHCI_RESET_PLL_ON_DISCONNECT;
 
+	if (pdev->vendor == PCI_VENDOR_ID_SYNOPSYS)
+		xhci->quirks |= XHCI_CONSOLIDATE_SG_LIST;
+
 	if (xhci->quirks & XHCI_RESET_ON_RESUME)
 		xhci_dbg_trace(xhci, trace_xhci_dbg_quirks,
 				"QUIRK: Resetting on resume");
diff --git a/drivers/usb/host/xhci-plat.c b/drivers/usb/host/xhci-plat.c
index 315b4552693c..8333c78dcf03 100644
--- a/drivers/usb/host/xhci-plat.c
+++ b/drivers/usb/host/xhci-plat.c
@@ -289,6 +289,9 @@ static int xhci_plat_probe(struct platform_device *pdev)
 		if (device_property_read_bool(tmpdev, "quirk-broken-port-ped"))
 			xhci->quirks |= XHCI_BROKEN_PORT_PED;
 
+		if (device_property_read_bool(tmpdev, "consolidate-sgl"))
+			xhci->quirks |= XHCI_CONSOLIDATE_SG_LIST;
+
 		device_property_read_u32(tmpdev, "imod-interval-ns",
 					 &xhci->imod_interval);
 	}
diff --git a/drivers/usb/host/xhci.h b/drivers/usb/host/xhci.h
index 3289bb516201..a093eeaec70e 100644
--- a/drivers/usb/host/xhci.h
+++ b/drivers/usb/host/xhci.h
@@ -1873,6 +1873,7 @@ struct xhci_hcd {
 #define XHCI_DEFAULT_PM_RUNTIME_ALLOW	BIT_ULL(33)
 #define XHCI_RESET_PLL_ON_DISCONNECT	BIT_ULL(34)
 #define XHCI_SNPS_BROKEN_SUSPEND    BIT_ULL(35)
+#define XHCI_CONSOLIDATE_SG_LIST    BIT_ULL(36)
 
 	unsigned int		num_active_eps;
 	unsigned int		limit_active_eps;
-- 
2.11.0

[RFC PATCH 3/4] usb: dwc3: Add device property consolidate-sgl

[Tejas Joglekar]

This commit adds the consolidate-sgl property to enable quirk for the
XHCI driver with Synopsys xHC. This property is enabled as initial
property for the dwc3-haps driver.

Signed-off-by: Tejas Joglekar <joglekar@synopsys.com>
---
 drivers/usb/dwc3/core.c      | 2 ++
 drivers/usb/dwc3/core.h      | 2 ++
 drivers/usb/dwc3/dwc3-haps.c | 1 +
 drivers/usb/dwc3/host.c      | 5 ++++-
 4 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/drivers/usb/dwc3/core.c b/drivers/usb/dwc3/core.c
index edc17155cb2b..de1747c78cd6 100644
--- a/drivers/usb/dwc3/core.c
+++ b/drivers/usb/dwc3/core.c
@@ -1296,6 +1296,8 @@ static void dwc3_get_properties(struct dwc3 *dwc)
 				"snps,usb3_lpm_capable");
 	dwc->usb2_lpm_disable = device_property_read_bool(dev,
 				"snps,usb2-lpm-disable");
+	dwc->consolidate_sgl  = device_property_read_bool(dev,
+				"snps,consolidate-sgl");
 	device_property_read_u8(dev, "snps,rx-thr-num-pkt-prd",
 				&rx_thr_num_pkt_prd);
 	device_property_read_u8(dev, "snps,rx-max-burst-prd",
diff --git a/drivers/usb/dwc3/core.h b/drivers/usb/dwc3/core.h
index 6846eb0cba13..d6e15f75809e 100644
--- a/drivers/usb/dwc3/core.h
+++ b/drivers/usb/dwc3/core.h
@@ -1007,6 +1007,7 @@ struct dwc3_scratchpad_array {
  *			not needed for DWC_usb31 version 1.70a-ea06 and below
  * @usb3_lpm_capable: set if hadrware supports Link Power Management
  * @usb2_lpm_disable: set to disable usb2 lpm
+ * @consolidate_sgl: set to enable the SG list consolidation
  * @disable_scramble_quirk: set if we enable the disable scramble quirk
  * @u2exit_lfps_quirk: set if we enable u2exit lfps quirk
  * @u2ss_inp3_quirk: set if we enable P3 OK for U2/SS Inactive quirk
@@ -1206,6 +1207,7 @@ struct dwc3 {
 	unsigned		dis_start_transfer_quirk:1;
 	unsigned		usb3_lpm_capable:1;
 	unsigned		usb2_lpm_disable:1;
+	unsigned		consolidate_sgl:1;
 
 	unsigned		disable_scramble_quirk:1;
 	unsigned		u2exit_lfps_quirk:1;
diff --git a/drivers/usb/dwc3/dwc3-haps.c b/drivers/usb/dwc3/dwc3-haps.c
index 3cecbf169452..495815081576 100644
--- a/drivers/usb/dwc3/dwc3-haps.c
+++ b/drivers/usb/dwc3/dwc3-haps.c
@@ -29,6 +29,7 @@ static const struct property_entry initial_properties[] = {
 	PROPERTY_ENTRY_BOOL("snps,usb3_lpm_capable"),
 	PROPERTY_ENTRY_BOOL("snps,has-lpm-erratum"),
 	PROPERTY_ENTRY_BOOL("snps,dis_enblslpm_quirk"),
+	PROPERTY_ENTRY_BOOL("snps,consolidate-sgl"),
 	PROPERTY_ENTRY_BOOL("linux,sysdev_is_parent"),
 	{ },
 };
diff --git a/drivers/usb/dwc3/host.c b/drivers/usb/dwc3/host.c
index 86dbd012b984..2de8f3f74b93 100644
--- a/drivers/usb/dwc3/host.c
+++ b/drivers/usb/dwc3/host.c
@@ -44,7 +44,7 @@ static int dwc3_host_get_irq(struct dwc3 *dwc)
 
 int dwc3_host_init(struct dwc3 *dwc)
 {
-	struct property_entry	props[4];
+	struct property_entry	props[5];
 	struct platform_device	*xhci;
 	int			ret, irq;
 	struct resource		*res;
@@ -95,6 +95,9 @@ int dwc3_host_init(struct dwc3 *dwc)
 	if (dwc->usb2_lpm_disable)
 		props[prop_idx++] = PROPERTY_ENTRY_BOOL("usb2-lpm-disable");
 
+	if (dwc->consolidate_sgl)
+		props[prop_idx++] = PROPERTY_ENTRY_BOOL("consolidate-sgl");
+
 	/**
 	 * WORKAROUND: dwc3 revisions <=3.00a have a limitation
 	 * where Port Disable command doesn't work.
-- 
2.11.0

[RESENDING RFC PATCH 3/4] usb: dwc3: Add device property consolidate-sgl

[Tejas Joglekar]

This commit adds the consolidate-sgl property to enable quirk for the
XHCI driver with Synopsys xHC. This property is enabled as initial
property for the dwc3-haps driver.

Signed-off-by: Tejas Joglekar <joglekar@synopsys.com>
---
 drivers/usb/dwc3/core.c      | 2 ++
 drivers/usb/dwc3/core.h      | 2 ++
 drivers/usb/dwc3/dwc3-haps.c | 1 +
 drivers/usb/dwc3/host.c      | 5 ++++-
 4 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/drivers/usb/dwc3/core.c b/drivers/usb/dwc3/core.c
index edc17155cb2b..de1747c78cd6 100644
--- a/drivers/usb/dwc3/core.c
+++ b/drivers/usb/dwc3/core.c
@@ -1296,6 +1296,8 @@ static void dwc3_get_properties(struct dwc3 *dwc)
 				"snps,usb3_lpm_capable");
 	dwc->usb2_lpm_disable = device_property_read_bool(dev,
 				"snps,usb2-lpm-disable");
+	dwc->consolidate_sgl  = device_property_read_bool(dev,
+				"snps,consolidate-sgl");
 	device_property_read_u8(dev, "snps,rx-thr-num-pkt-prd",
 				&rx_thr_num_pkt_prd);
 	device_property_read_u8(dev, "snps,rx-max-burst-prd",
diff --git a/drivers/usb/dwc3/core.h b/drivers/usb/dwc3/core.h
index 6846eb0cba13..d6e15f75809e 100644
--- a/drivers/usb/dwc3/core.h
+++ b/drivers/usb/dwc3/core.h
@@ -1007,6 +1007,7 @@ struct dwc3_scratchpad_array {
  *			not needed for DWC_usb31 version 1.70a-ea06 and below
  * @usb3_lpm_capable: set if hadrware supports Link Power Management
  * @usb2_lpm_disable: set to disable usb2 lpm
+ * @consolidate_sgl: set to enable the SG list consolidation
  * @disable_scramble_quirk: set if we enable the disable scramble quirk
  * @u2exit_lfps_quirk: set if we enable u2exit lfps quirk
  * @u2ss_inp3_quirk: set if we enable P3 OK for U2/SS Inactive quirk
@@ -1206,6 +1207,7 @@ struct dwc3 {
 	unsigned		dis_start_transfer_quirk:1;
 	unsigned		usb3_lpm_capable:1;
 	unsigned		usb2_lpm_disable:1;
+	unsigned		consolidate_sgl:1;
 
 	unsigned		disable_scramble_quirk:1;
 	unsigned		u2exit_lfps_quirk:1;
diff --git a/drivers/usb/dwc3/dwc3-haps.c b/drivers/usb/dwc3/dwc3-haps.c
index 3cecbf169452..495815081576 100644
--- a/drivers/usb/dwc3/dwc3-haps.c
+++ b/drivers/usb/dwc3/dwc3-haps.c
@@ -29,6 +29,7 @@ static const struct property_entry initial_properties[] = {
 	PROPERTY_ENTRY_BOOL("snps,usb3_lpm_capable"),
 	PROPERTY_ENTRY_BOOL("snps,has-lpm-erratum"),
 	PROPERTY_ENTRY_BOOL("snps,dis_enblslpm_quirk"),
+	PROPERTY_ENTRY_BOOL("snps,consolidate-sgl"),
 	PROPERTY_ENTRY_BOOL("linux,sysdev_is_parent"),
 	{ },
 };
diff --git a/drivers/usb/dwc3/host.c b/drivers/usb/dwc3/host.c
index 86dbd012b984..2de8f3f74b93 100644
--- a/drivers/usb/dwc3/host.c
+++ b/drivers/usb/dwc3/host.c
@@ -44,7 +44,7 @@ static int dwc3_host_get_irq(struct dwc3 *dwc)
 
 int dwc3_host_init(struct dwc3 *dwc)
 {
-	struct property_entry	props[4];
+	struct property_entry	props[5];
 	struct platform_device	*xhci;
 	int			ret, irq;
 	struct resource		*res;
@@ -95,6 +95,9 @@ int dwc3_host_init(struct dwc3 *dwc)
 	if (dwc->usb2_lpm_disable)
 		props[prop_idx++] = PROPERTY_ENTRY_BOOL("usb2-lpm-disable");
 
+	if (dwc->consolidate_sgl)
+		props[prop_idx++] = PROPERTY_ENTRY_BOOL("consolidate-sgl");
+
 	/**
 	 * WORKAROUND: dwc3 revisions <=3.00a have a limitation
 	 * where Port Disable command doesn't work.
-- 
2.11.0

[RESENDING RFC PATCH 4/4] usb: xhci: Use temporary buffer to consolidate SG

[Tejas Joglekar]

The Synopsys xHC has an internal TRB cache of size TRB_CACHE_SIZE for
each endpoint. The default value for TRB_CACHE_SIZE is 16 for SS and 8
for HS. The controller loads and updates the TRB cache from the transfer
ring in system memory whenever the driver issues a start transfer or
update transfer command.

For chained TRBs, the Synopsys xHC requires that the total amount of
bytes for all TRBs loaded in the TRB cache be greater than or equal to 1
MPS. Or the chain ends within the TRB cache (with a last TRB).

If this requirement is not met, the controller will not be able to send
or receive a packet and it will hang causing a driver timeout and error.

This can be a problem if a class driver queues SG requests with many
small-buffer entries. The XHCI driver will create a chained TRB for each
entry which may trigger this issue.

This patch adds logic to the XHCI driver to detect and prevent this from
happening.

For every (TRB_CACHE_SIZE - 2), we check the total buffer size of
the SG list and if the last window of (TRB_CACHE_SIZE - 2) SG list length
and we don't make up at least 1 MPS, we create a temporary buffer to
consolidate full SG list into the buffer.

We check at (TRB_CACHE_SIZE - 2) window because it is possible that there
would be a link and/or event data TRB that take up to 2 of the cache
entries.

We discovered this issue with devices on other platforms but have not
yet come across any device that triggers this on Linux. But it could be
a real problem now or in the future. All it takes is N number of small
chained TRBs. And other instances of the Synopsys IP may have smaller
values for the TRB_CACHE_SIZE which would exacerbate the problem.

Signed-off-by: Tejas Joglekar <joglekar@synopsys.com>
---

Resending as 'umlaut' in email are not accepted by some servers.

 drivers/usb/core/hcd.c       |   8 +++
 drivers/usb/host/xhci-ring.c |   2 +-
 drivers/usb/host/xhci.c      | 128 +++++++++++++++++++++++++++++++++++++++++++
 drivers/usb/host/xhci.h      |   4 ++
 4 files changed, 141 insertions(+), 1 deletion(-)

diff --git a/drivers/usb/core/hcd.c b/drivers/usb/core/hcd.c
index aa45840d8273..fdd257a2b8a6 100644
--- a/drivers/usb/core/hcd.c
+++ b/drivers/usb/core/hcd.c
@@ -1459,6 +1459,14 @@ int usb_hcd_map_urb_for_dma(struct usb_hcd *hcd, struct urb *urb,
 					return -EINVAL;
 				}
 
+				/*
+				 * If SG is consolidate into single buffer
+				 * return early
+				 */
+				if ((urb->transfer_flags &
+				     URB_DMA_MAP_SINGLE))
+					return ret;
+
 				n = dma_map_sg(
 						hcd->self.sysdev,
 						urb->sg,
diff --git a/drivers/usb/host/xhci-ring.c b/drivers/usb/host/xhci-ring.c
index a78787bb5133..2fad9474912a 100644
--- a/drivers/usb/host/xhci-ring.c
+++ b/drivers/usb/host/xhci-ring.c
@@ -3291,7 +3291,7 @@ int xhci_queue_bulk_tx(struct xhci_hcd *xhci, gfp_t mem_flags,
 
 	full_len = urb->transfer_buffer_length;
 	/* If we have scatter/gather list, we use it. */
-	if (urb->num_sgs) {
+	if (urb->num_sgs && !(urb->transfer_flags & URB_DMA_MAP_SINGLE)) {
 		num_sgs = urb->num_mapped_sgs;
 		sg = urb->sg;
 		addr = (u64) sg_dma_address(sg);
diff --git a/drivers/usb/host/xhci.c b/drivers/usb/host/xhci.c
index fe38275363e0..94fddbd06179 100644
--- a/drivers/usb/host/xhci.c
+++ b/drivers/usb/host/xhci.c
@@ -1256,6 +1256,109 @@ EXPORT_SYMBOL_GPL(xhci_resume);
 
 /*-------------------------------------------------------------------------*/
 
+static int xhci_map_temp_buffer(struct usb_hcd *hcd, struct urb *urb)
+{
+	void *temp;
+	int ret = 0;
+	unsigned int len;
+	unsigned int buf_len;
+	enum dma_data_direction dir;
+	struct xhci_hcd *xhci;
+
+	xhci = hcd_to_xhci(hcd);
+	dir = usb_urb_dir_in(urb) ? DMA_FROM_DEVICE : DMA_TO_DEVICE;
+	buf_len = urb->transfer_buffer_length;
+
+	temp = kzalloc_node(buf_len, GFP_ATOMIC,
+			    dev_to_node(hcd->self.sysdev));
+	if (!temp) {
+		xhci_warn(xhci, "Failed to create temp buffer, HC may fail\n");
+		return -ENOMEM;
+	}
+
+	if (usb_urb_dir_out(urb)) {
+		len = sg_pcopy_to_buffer(urb->sg, urb->num_sgs,
+					 temp, buf_len, 0);
+		if (len != buf_len)
+			xhci_warn(xhci, "Wrong temp buffer write length\n");
+	}
+
+	urb->transfer_buffer = temp;
+	urb->transfer_dma = dma_map_single(hcd->self.sysdev,
+					   urb->transfer_buffer,
+					   urb->transfer_buffer_length,
+					   dir);
+	if (dma_mapping_error(hcd->self.sysdev,
+			      urb->transfer_dma)) {
+		xhci_err(xhci, "dma mapping error\n");
+		ret = -EAGAIN;
+		kfree(temp);
+	} else {
+		urb->transfer_flags |= URB_DMA_MAP_SINGLE;
+	}
+
+	return ret;
+}
+
+static bool xhci_urb_temp_buffer_required(struct usb_hcd *hcd,
+					  struct urb *urb)
+{
+	bool ret = false;
+	unsigned int i;
+	unsigned int len = 0;
+	unsigned int buf_len;
+	unsigned int trb_size;
+	unsigned int max_pkt;
+	struct scatterlist *sg;
+	struct scatterlist *tail_sg;
+
+	sg = urb->sg;
+	tail_sg = urb->sg;
+	buf_len = urb->transfer_buffer_length;
+	max_pkt = usb_endpoint_maxp(&urb->ep->desc);
+
+	if (urb->dev->speed >= USB_SPEED_SUPER)
+		trb_size = TRB_CACHE_SIZE_SS;
+	else
+		trb_size = TRB_CACHE_SIZE_HS;
+
+	for_each_sg(urb->sg, sg, urb->num_sgs, i) {
+		len = len + sg->length;
+		if (i > trb_size - 2) {
+			len = len - tail_sg->length;
+			if (len < max_pkt) {
+				ret = true;
+				break;
+			}
+
+			tail_sg = sg_next(tail_sg);
+		}
+	}
+	return ret;
+}
+
+static void xhci_unmap_temp_buf(struct urb *urb)
+{
+	struct scatterlist *sg;
+	unsigned int len;
+	unsigned int buf_len;
+
+	sg = urb->sg;
+	buf_len = urb->transfer_buffer_length;
+
+	if (usb_urb_dir_in(urb)) {
+		len = sg_pcopy_from_buffer(urb->sg, urb->num_sgs,
+					   urb->transfer_buffer,
+					   buf_len,
+					   0);
+		if (len != buf_len)
+			dev_err(&urb->dev->dev, "Wrong length for unmap\n");
+	}
+
+	kfree(urb->transfer_buffer);
+	urb->transfer_buffer = NULL;
+}
+
 /*
  * Bypass the DMA mapping if URB is suitable for Immediate Transfer (IDT),
  * we'll copy the actual data into the TRB address register. This is limited to
@@ -1265,12 +1368,36 @@ EXPORT_SYMBOL_GPL(xhci_resume);
 static int xhci_map_urb_for_dma(struct usb_hcd *hcd, struct urb *urb,
 				gfp_t mem_flags)
 {
+	struct xhci_hcd *xhci;
+
+	xhci = hcd_to_xhci(hcd);
+
 	if (xhci_urb_suitable_for_idt(urb))
 		return 0;
 
+	if (xhci->quirks & XHCI_CONSOLIDATE_SG_LIST) {
+		if (xhci_urb_temp_buffer_required(hcd, urb))
+			xhci_map_temp_buffer(hcd, urb);
+	}
 	return usb_hcd_map_urb_for_dma(hcd, urb, mem_flags);
 }
 
+static void xhci_unmap_urb_for_dma(struct usb_hcd *hcd, struct urb *urb)
+{
+	struct xhci_hcd *xhci;
+	bool unmap_temp_buf = false;
+
+	xhci = hcd_to_xhci(hcd);
+
+	if (urb->num_sgs && (urb->transfer_flags & URB_DMA_MAP_SINGLE))
+		unmap_temp_buf = true;
+
+	usb_hcd_unmap_urb_for_dma(hcd, urb);
+
+	if ((xhci->quirks & XHCI_CONSOLIDATE_SG_LIST) && unmap_temp_buf)
+		xhci_unmap_temp_buf(urb);
+}
+
 /**
  * xhci_get_endpoint_index - Used for passing endpoint bitmasks between the core and
  * HCDs.  Find the index for an endpoint given its descriptor.  Use the return
@@ -5315,6 +5442,7 @@ static const struct hc_driver xhci_hc_driver = {
 	 * managing i/o requests and associated device resources
 	 */
 	.map_urb_for_dma =      xhci_map_urb_for_dma,
+	.unmap_urb_for_dma =    xhci_unmap_urb_for_dma,
 	.urb_enqueue =		xhci_urb_enqueue,
 	.urb_dequeue =		xhci_urb_dequeue,
 	.alloc_dev =		xhci_alloc_dev,
diff --git a/drivers/usb/host/xhci.h b/drivers/usb/host/xhci.h
index a093eeaec70e..341d1dfbe689 100644
--- a/drivers/usb/host/xhci.h
+++ b/drivers/usb/host/xhci.h
@@ -1330,6 +1330,10 @@ enum xhci_setup_dev {
 #define TRB_SIA			(1<<31)
 #define TRB_FRAME_ID(p)		(((p) & 0x7ff) << 20)
 
+/* TRB cache size for xHC with TRB cache */
+#define TRB_CACHE_SIZE_HS	8
+#define TRB_CACHE_SIZE_SS	16
+
 struct xhci_generic_trb {
 	__le32 field[4];
 };
-- 
2.11.0

Re: [RESENDING RFC PATCH 4/4] usb: xhci: Use temporary buffer to consolidate SG

[Greg KH]

On Fri, Mar 27, 2020 at 02:23:46PM +0530, Tejas Joglekar wrote:
> The Synopsys xHC has an internal TRB cache of size TRB_CACHE_SIZE for
> each endpoint. The default value for TRB_CACHE_SIZE is 16 for SS and 8
> for HS. The controller loads and updates the TRB cache from the transfer
> ring in system memory whenever the driver issues a start transfer or
> update transfer command.
> 
> For chained TRBs, the Synopsys xHC requires that the total amount of
> bytes for all TRBs loaded in the TRB cache be greater than or equal to 1
> MPS. Or the chain ends within the TRB cache (with a last TRB).
> 
> If this requirement is not met, the controller will not be able to send
> or receive a packet and it will hang causing a driver timeout and error.

Sounds like broken hardware, or is this requirement in the xhci spec?

> 
> This can be a problem if a class driver queues SG requests with many
> small-buffer entries. The XHCI driver will create a chained TRB for each
> entry which may trigger this issue.
> 
> This patch adds logic to the XHCI driver to detect and prevent this from
> happening.
> 
> For every (TRB_CACHE_SIZE - 2), we check the total buffer size of
> the SG list and if the last window of (TRB_CACHE_SIZE - 2) SG list length
> and we don't make up at least 1 MPS, we create a temporary buffer to
> consolidate full SG list into the buffer.
> 
> We check at (TRB_CACHE_SIZE - 2) window because it is possible that there
> would be a link and/or event data TRB that take up to 2 of the cache
> entries.
> 
> We discovered this issue with devices on other platforms but have not
> yet come across any device that triggers this on Linux. But it could be
> a real problem now or in the future. All it takes is N number of small
> chained TRBs. And other instances of the Synopsys IP may have smaller
> values for the TRB_CACHE_SIZE which would exacerbate the problem.
> 
> Signed-off-by: Tejas Joglekar <joglekar@synopsys.com>
> ---
> 
> Resending as 'umlaut' in email are not accepted by some servers.
> 
>  drivers/usb/core/hcd.c       |   8 +++
>  drivers/usb/host/xhci-ring.c |   2 +-
>  drivers/usb/host/xhci.c      | 128 +++++++++++++++++++++++++++++++++++++++++++
>  drivers/usb/host/xhci.h      |   4 ++
>  4 files changed, 141 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/usb/core/hcd.c b/drivers/usb/core/hcd.c
> index aa45840d8273..fdd257a2b8a6 100644
> --- a/drivers/usb/core/hcd.c
> +++ b/drivers/usb/core/hcd.c
> @@ -1459,6 +1459,14 @@ int usb_hcd_map_urb_for_dma(struct usb_hcd *hcd, struct urb *urb,
>  					return -EINVAL;
>  				}
>  
> +				/*
> +				 * If SG is consolidate into single buffer
> +				 * return early

I do not understand this comment.

> +				 */
> +				if ((urb->transfer_flags &
> +				     URB_DMA_MAP_SINGLE))
> +					return ret;

Why?  Isn't this now going to affect other host controllers (like all of
them?)

> +
>  				n = dma_map_sg(
>  						hcd->self.sysdev,
>  						urb->sg,
> diff --git a/drivers/usb/host/xhci-ring.c b/drivers/usb/host/xhci-ring.c
> index a78787bb5133..2fad9474912a 100644
> --- a/drivers/usb/host/xhci-ring.c
> +++ b/drivers/usb/host/xhci-ring.c
> @@ -3291,7 +3291,7 @@ int xhci_queue_bulk_tx(struct xhci_hcd *xhci, gfp_t mem_flags,
>  
>  	full_len = urb->transfer_buffer_length;
>  	/* If we have scatter/gather list, we use it. */
> -	if (urb->num_sgs) {
> +	if (urb->num_sgs && !(urb->transfer_flags & URB_DMA_MAP_SINGLE)) {
>  		num_sgs = urb->num_mapped_sgs;
>  		sg = urb->sg;
>  		addr = (u64) sg_dma_address(sg);
> diff --git a/drivers/usb/host/xhci.c b/drivers/usb/host/xhci.c
> index fe38275363e0..94fddbd06179 100644
> --- a/drivers/usb/host/xhci.c
> +++ b/drivers/usb/host/xhci.c
> @@ -1256,6 +1256,109 @@ EXPORT_SYMBOL_GPL(xhci_resume);
>  
>  /*-------------------------------------------------------------------------*/
>  
> +static int xhci_map_temp_buffer(struct usb_hcd *hcd, struct urb *urb)
> +{
> +	void *temp;
> +	int ret = 0;
> +	unsigned int len;
> +	unsigned int buf_len;
> +	enum dma_data_direction dir;
> +	struct xhci_hcd *xhci;
> +
> +	xhci = hcd_to_xhci(hcd);
> +	dir = usb_urb_dir_in(urb) ? DMA_FROM_DEVICE : DMA_TO_DEVICE;
> +	buf_len = urb->transfer_buffer_length;
> +
> +	temp = kzalloc_node(buf_len, GFP_ATOMIC,
> +			    dev_to_node(hcd->self.sysdev));
> +	if (!temp) {
> +		xhci_warn(xhci, "Failed to create temp buffer, HC may fail\n");

Didn't kzalloc just warn before this?

And isn't this whole thing going to cause a lot more memory allocations
per submission than before?

> +		return -ENOMEM;
> +	}
> +
> +	if (usb_urb_dir_out(urb)) {
> +		len = sg_pcopy_to_buffer(urb->sg, urb->num_sgs,
> +					 temp, buf_len, 0);
> +		if (len != buf_len)
> +			xhci_warn(xhci, "Wrong temp buffer write length\n");

How could this happen?  And if it does, why spam the kernel log about it
and yet not return an error?

> +	}
> +
> +	urb->transfer_buffer = temp;
> +	urb->transfer_dma = dma_map_single(hcd->self.sysdev,
> +					   urb->transfer_buffer,
> +					   urb->transfer_buffer_length,
> +					   dir);
> +	if (dma_mapping_error(hcd->self.sysdev,
> +			      urb->transfer_dma)) {
> +		xhci_err(xhci, "dma mapping error\n");

Again, didn't dma_mapping_error() spit out a message?

> +		ret = -EAGAIN;
> +		kfree(temp);
> +	} else {
> +		urb->transfer_flags |= URB_DMA_MAP_SINGLE;
> +	}
> +
> +	return ret;
> +}

thanks,

greg k-h

Re: [RESENDING RFC PATCH 4/4] usb: xhci: Use temporary buffer to consolidate SG

[Tejas Joglekar]

Hi,
On 3/27/2020 2:57 PM, Greg KH wrote:
> On Fri, Mar 27, 2020 at 02:23:46PM +0530, Tejas Joglekar wrote:
>> The Synopsys xHC has an internal TRB cache of size TRB_CACHE_SIZE for
>> each endpoint. The default value for TRB_CACHE_SIZE is 16 for SS and 8
>> for HS. The controller loads and updates the TRB cache from the transfer
>> ring in system memory whenever the driver issues a start transfer or
>> update transfer command.
>>
>> For chained TRBs, the Synopsys xHC requires that the total amount of
>> bytes for all TRBs loaded in the TRB cache be greater than or equal to 1
>> MPS. Or the chain ends within the TRB cache (with a last TRB).
>>
>> If this requirement is not met, the controller will not be able to send
>> or receive a packet and it will hang causing a driver timeout and error.
> 
> Sounds like broken hardware, or is this requirement in the xhci spec?
> 
Not a xhci spec requirement, but behavior of Synopsys xHC. We have not seen
actual failure on Linux yet but it is possible in future if SG list with
very small buffer size is given for transfer. 
>>
>> This can be a problem if a class driver queues SG requests with many
>> small-buffer entries. The XHCI driver will create a chained TRB for each
>> entry which may trigger this issue.
>>
>> This patch adds logic to the XHCI driver to detect and prevent this from
>> happening.
>>
>> For every (TRB_CACHE_SIZE - 2), we check the total buffer size of
>> the SG list and if the last window of (TRB_CACHE_SIZE - 2) SG list length
>> and we don't make up at least 1 MPS, we create a temporary buffer to
>> consolidate full SG list into the buffer.
>>
>> We check at (TRB_CACHE_SIZE - 2) window because it is possible that there
>> would be a link and/or event data TRB that take up to 2 of the cache
>> entries.
>>
>> We discovered this issue with devices on other platforms but have not
>> yet come across any device that triggers this on Linux. But it could be
>> a real problem now or in the future. All it takes is N number of small
>> chained TRBs. And other instances of the Synopsys IP may have smaller
>> values for the TRB_CACHE_SIZE which would exacerbate the problem.
>>
>> Signed-off-by: Tejas Joglekar <joglekar@synopsys.com>
>> ---
>>
>> Resending as 'umlaut' in email are not accepted by some servers.
>>
>>  drivers/usb/core/hcd.c       |   8 +++
>>  drivers/usb/host/xhci-ring.c |   2 +-
>>  drivers/usb/host/xhci.c      | 128 +++++++++++++++++++++++++++++++++++++++++++
>>  drivers/usb/host/xhci.h      |   4 ++
>>  4 files changed, 141 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/usb/core/hcd.c b/drivers/usb/core/hcd.c
>> index aa45840d8273..fdd257a2b8a6 100644
>> --- a/drivers/usb/core/hcd.c
>> +++ b/drivers/usb/core/hcd.c
>> @@ -1459,6 +1459,14 @@ int usb_hcd_map_urb_for_dma(struct usb_hcd *hcd, struct urb *urb,
>>  					return -EINVAL;
>>  				}
>>  
>> +				/*
>> +				 * If SG is consolidate into single buffer
>> +				 * return early
> 
> I do not understand this comment.

The SG list is copied to a temporary buffer, and buffer is DMA mapped so we should 
not map the SG list again, and return without any mapping here. 
> 
>> +				 */
>> +				if ((urb->transfer_flags &
>> +				     URB_DMA_MAP_SINGLE))
>> +					return ret;
> 
> Why?  Isn't this now going to affect other host controllers (like all of
> them?)
> 
When urb->num_sgs is greater than 0, other than my quirk dma_map function no one
will set the DMA transfer flag as URB_DMA_MAP_SINGLE. So it would not be called by
all HC's. Even when the SG list does not have very small buffer sizes this quirk will
not set the URB_DMA_MAP_SINGLE transfer flag.
>> +
>>  				n = dma_map_sg(
>>  						hcd->self.sysdev,
>>  						urb->sg,
>> diff --git a/drivers/usb/host/xhci-ring.c b/drivers/usb/host/xhci-ring.c
>> index a78787bb5133..2fad9474912a 100644
>> --- a/drivers/usb/host/xhci-ring.c
>> +++ b/drivers/usb/host/xhci-ring.c
>> @@ -3291,7 +3291,7 @@ int xhci_queue_bulk_tx(struct xhci_hcd *xhci, gfp_t mem_flags,
>>  
>>  	full_len = urb->transfer_buffer_length;
>>  	/* If we have scatter/gather list, we use it. */
>> -	if (urb->num_sgs) {
>> +	if (urb->num_sgs && !(urb->transfer_flags & URB_DMA_MAP_SINGLE)) {
>>  		num_sgs = urb->num_mapped_sgs;
>>  		sg = urb->sg;
>>  		addr = (u64) sg_dma_address(sg);
>> diff --git a/drivers/usb/host/xhci.c b/drivers/usb/host/xhci.c
>> index fe38275363e0..94fddbd06179 100644
>> --- a/drivers/usb/host/xhci.c
>> +++ b/drivers/usb/host/xhci.c
>> @@ -1256,6 +1256,109 @@ EXPORT_SYMBOL_GPL(xhci_resume);
>>  
>>  /*-------------------------------------------------------------------------*/
>>  
>> +static int xhci_map_temp_buffer(struct usb_hcd *hcd, struct urb *urb)
>> +{
>> +	void *temp;
>> +	int ret = 0;
>> +	unsigned int len;
>> +	unsigned int buf_len;
>> +	enum dma_data_direction dir;
>> +	struct xhci_hcd *xhci;
>> +
>> +	xhci = hcd_to_xhci(hcd);
>> +	dir = usb_urb_dir_in(urb) ? DMA_FROM_DEVICE : DMA_TO_DEVICE;
>> +	buf_len = urb->transfer_buffer_length;
>> +
>> +	temp = kzalloc_node(buf_len, GFP_ATOMIC,
>> +			    dev_to_node(hcd->self.sysdev));
>> +	if (!temp) {
>> +		xhci_warn(xhci, "Failed to create temp buffer, HC may fail\n");
> 
> Didn't kzalloc just warn before this?
> 
Yes, It should.
> And isn't this whole thing going to cause a lot more memory allocations
> per submission than before?
> 
If buffer sizes for SG list are very small (less than MPS size per TRB_CACHE_SIZE)
yes, it will have more memory allocations. 

>> +		return -ENOMEM;
>> +	}
>> +
>> +	if (usb_urb_dir_out(urb)) {
>> +		len = sg_pcopy_to_buffer(urb->sg, urb->num_sgs,
>> +					 temp, buf_len, 0);
>> +		if (len != buf_len)
>> +			xhci_warn(xhci, "Wrong temp buffer write length\n");
> 
> How could this happen?  And if it does, why spam the kernel log about it
> and yet not return an error?
>
Logic similar to bounce buffer allocation, but I agree should not log this as
we don't want to return with error here.
 
>> +	}
>> +
>> +	urb->transfer_buffer = temp;
>> +	urb->transfer_dma = dma_map_single(hcd->self.sysdev,
>> +					   urb->transfer_buffer,
>> +					   urb->transfer_buffer_length,
>> +					   dir);
>> +	if (dma_mapping_error(hcd->self.sysdev,
>> +			      urb->transfer_dma)) {
>> +		xhci_err(xhci, "dma mapping error\n");
> 
> Again, didn't dma_mapping_error() spit out a message?
>
Yes, should remove this too.
 
>> +		ret = -EAGAIN;
>> +		kfree(temp);
>> +	} else {
>> +		urb->transfer_flags |= URB_DMA_MAP_SINGLE;
>> +	}
>> +
>> +	return ret;
>> +}
> 
> thanks,
> 
> greg k-h
> 

Thanks & Regards,
Tejas Joglekar

Re: [RESENDING RFC PATCH 4/4] usb: xhci: Use temporary buffer to consolidate SG

[Greg KH]

On Fri, Mar 27, 2020 at 10:05:21AM +0000, Tejas Joglekar wrote:
> Hi,
> On 3/27/2020 2:57 PM, Greg KH wrote:
> > On Fri, Mar 27, 2020 at 02:23:46PM +0530, Tejas Joglekar wrote:
> >> The Synopsys xHC has an internal TRB cache of size TRB_CACHE_SIZE for
> >> each endpoint. The default value for TRB_CACHE_SIZE is 16 for SS and 8
> >> for HS. The controller loads and updates the TRB cache from the transfer
> >> ring in system memory whenever the driver issues a start transfer or
> >> update transfer command.
> >>
> >> For chained TRBs, the Synopsys xHC requires that the total amount of
> >> bytes for all TRBs loaded in the TRB cache be greater than or equal to 1
> >> MPS. Or the chain ends within the TRB cache (with a last TRB).
> >>
> >> If this requirement is not met, the controller will not be able to send
> >> or receive a packet and it will hang causing a driver timeout and error.
> > 
> > Sounds like broken hardware, or is this requirement in the xhci spec?
> > 
> Not a xhci spec requirement, but behavior of Synopsys xHC. We have not seen
> actual failure on Linux yet but it is possible in future if SG list with
> very small buffer size is given for transfer. 

So this is something required that is outside of the spec, meaning that
the hardware is imposing additional requirements, which implies it's a
hardware bug, or "quirk", right?

> >> This can be a problem if a class driver queues SG requests with many
> >> small-buffer entries. The XHCI driver will create a chained TRB for each
> >> entry which may trigger this issue.
> >>
> >> This patch adds logic to the XHCI driver to detect and prevent this from
> >> happening.
> >>
> >> For every (TRB_CACHE_SIZE - 2), we check the total buffer size of
> >> the SG list and if the last window of (TRB_CACHE_SIZE - 2) SG list length
> >> and we don't make up at least 1 MPS, we create a temporary buffer to
> >> consolidate full SG list into the buffer.
> >>
> >> We check at (TRB_CACHE_SIZE - 2) window because it is possible that there
> >> would be a link and/or event data TRB that take up to 2 of the cache
> >> entries.
> >>
> >> We discovered this issue with devices on other platforms but have not
> >> yet come across any device that triggers this on Linux. But it could be
> >> a real problem now or in the future. All it takes is N number of small
> >> chained TRBs. And other instances of the Synopsys IP may have smaller
> >> values for the TRB_CACHE_SIZE which would exacerbate the problem.
> >>
> >> Signed-off-by: Tejas Joglekar <joglekar@synopsys.com>
> >> ---
> >>
> >> Resending as 'umlaut' in email are not accepted by some servers.
> >>
> >>  drivers/usb/core/hcd.c       |   8 +++
> >>  drivers/usb/host/xhci-ring.c |   2 +-
> >>  drivers/usb/host/xhci.c      | 128 +++++++++++++++++++++++++++++++++++++++++++
> >>  drivers/usb/host/xhci.h      |   4 ++
> >>  4 files changed, 141 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/drivers/usb/core/hcd.c b/drivers/usb/core/hcd.c
> >> index aa45840d8273..fdd257a2b8a6 100644
> >> --- a/drivers/usb/core/hcd.c
> >> +++ b/drivers/usb/core/hcd.c
> >> @@ -1459,6 +1459,14 @@ int usb_hcd_map_urb_for_dma(struct usb_hcd *hcd, struct urb *urb,
> >>  					return -EINVAL;
> >>  				}
> >>  
> >> +				/*
> >> +				 * If SG is consolidate into single buffer
> >> +				 * return early
> > 
> > I do not understand this comment.
> 
> The SG list is copied to a temporary buffer, and buffer is DMA mapped so we should 
> not map the SG list again, and return without any mapping here. 

Please write this all out a lot more to make it more obvious.


> > 
> >> +				 */
> >> +				if ((urb->transfer_flags &
> >> +				     URB_DMA_MAP_SINGLE))
> >> +					return ret;
> > 
> > Why?  Isn't this now going to affect other host controllers (like all of
> > them?)
> > 
> When urb->num_sgs is greater than 0, other than my quirk dma_map function no one
> will set the DMA transfer flag as URB_DMA_MAP_SINGLE. So it would not be called by
> all HC's. Even when the SG list does not have very small buffer sizes this quirk will
> not set the URB_DMA_MAP_SINGLE transfer flag.

Are you sure?  :)

> >> +
> >>  				n = dma_map_sg(
> >>  						hcd->self.sysdev,
> >>  						urb->sg,
> >> diff --git a/drivers/usb/host/xhci-ring.c b/drivers/usb/host/xhci-ring.c
> >> index a78787bb5133..2fad9474912a 100644
> >> --- a/drivers/usb/host/xhci-ring.c
> >> +++ b/drivers/usb/host/xhci-ring.c
> >> @@ -3291,7 +3291,7 @@ int xhci_queue_bulk_tx(struct xhci_hcd *xhci, gfp_t mem_flags,
> >>  
> >>  	full_len = urb->transfer_buffer_length;
> >>  	/* If we have scatter/gather list, we use it. */
> >> -	if (urb->num_sgs) {
> >> +	if (urb->num_sgs && !(urb->transfer_flags & URB_DMA_MAP_SINGLE)) {
> >>  		num_sgs = urb->num_mapped_sgs;
> >>  		sg = urb->sg;
> >>  		addr = (u64) sg_dma_address(sg);
> >> diff --git a/drivers/usb/host/xhci.c b/drivers/usb/host/xhci.c
> >> index fe38275363e0..94fddbd06179 100644
> >> --- a/drivers/usb/host/xhci.c
> >> +++ b/drivers/usb/host/xhci.c
> >> @@ -1256,6 +1256,109 @@ EXPORT_SYMBOL_GPL(xhci_resume);
> >>  
> >>  /*-------------------------------------------------------------------------*/
> >>  
> >> +static int xhci_map_temp_buffer(struct usb_hcd *hcd, struct urb *urb)
> >> +{
> >> +	void *temp;
> >> +	int ret = 0;
> >> +	unsigned int len;
> >> +	unsigned int buf_len;
> >> +	enum dma_data_direction dir;
> >> +	struct xhci_hcd *xhci;
> >> +
> >> +	xhci = hcd_to_xhci(hcd);
> >> +	dir = usb_urb_dir_in(urb) ? DMA_FROM_DEVICE : DMA_TO_DEVICE;
> >> +	buf_len = urb->transfer_buffer_length;
> >> +
> >> +	temp = kzalloc_node(buf_len, GFP_ATOMIC,
> >> +			    dev_to_node(hcd->self.sysdev));
> >> +	if (!temp) {
> >> +		xhci_warn(xhci, "Failed to create temp buffer, HC may fail\n");
> > 
> > Didn't kzalloc just warn before this?
> > 
> Yes, It should.

Then do not spit out another message please.

> > And isn't this whole thing going to cause a lot more memory allocations
> > per submission than before?
> > 
> If buffer sizes for SG list are very small (less than MPS size per TRB_CACHE_SIZE)
> yes, it will have more memory allocations. 

That's not good :(

greg k-h

Re: [RESENDING RFC PATCH 4/4] usb: xhci: Use temporary buffer to consolidate SG

[Mathias Nyman]

>>>> +static int xhci_map_temp_buffer(struct usb_hcd *hcd, struct urb *urb)
>>>> +{
>>>> +	void *temp;
>>>> +	int ret = 0;
>>>> +	unsigned int len;
>>>> +	unsigned int buf_len;
>>>> +	enum dma_data_direction dir;
>>>> +	struct xhci_hcd *xhci;
>>>> +
>>>> +	xhci = hcd_to_xhci(hcd);
>>>> +	dir = usb_urb_dir_in(urb) ? DMA_FROM_DEVICE : DMA_TO_DEVICE;
>>>> +	buf_len = urb->transfer_buffer_length;
>>>> +
>>>> +	temp = kzalloc_node(buf_len, GFP_ATOMIC,
>>>> +			    dev_to_node(hcd->self.sysdev));
>>>> +	if (!temp) {
>>>> +		xhci_warn(xhci, "Failed to create temp buffer, HC may fail\n");
>>>
>>> Didn't kzalloc just warn before this?
>>>
>> Yes, It should.
> 
> Then do not spit out another message please.
> 
>>> And isn't this whole thing going to cause a lot more memory allocations
>>> per submission than before?
>>>
>> If buffer sizes for SG list are very small (less than MPS size per TRB_CACHE_SIZE)
>> yes, it will have more memory allocations. 
> 
> That's not good :(
> 

I actually recommended this after looking at the real numbers.

It was explained to me that the Synopsis xHC has a flaw that it will hang if its
TRB cache contains 16 chained TRBs (no ending TRB), and the combined TRB size
is _less_ than 1024 bytes.

So this only happens if a URB has a sg list with more than 16 entries whose
total size is less than 1024 bytes.

I was told this has not been seen ever in real life usage.

So in the unlikey case this is ever triggered we will end up allocating a small
bounce buffer, probably around 1024 bytes, and copy the data over from the sg-list.

(for HS speeds limits are 8 chained TRBs and 512 bytes)

- Mathias

Re: [RESENDING RFC PATCH 4/4] usb: xhci: Use temporary buffer to consolidate SG

[Raul Rangel]

Not sure if I'm hijacking the thread, or if this patch is related, but
I've been seeing xHC hangs and timeouts with the dwc_3_1 controller.
Protocol Trace: https://0paste.com/59611 or
https://photos.app.goo.gl/54DGJJuH4kQa9Psy6
kernel trace: https://0paste.com/59613
This hang happens on transactions larger than 16K regardless of the
number of TRBs. I only see this on SS and not 2.0.

Re: [RESENDING RFC PATCH 0/4] Add logic to consolidate TRBs for Synopsys xHC

[Greg KH]

On Fri, Mar 27, 2020 at 02:22:23PM +0530, Tejas Joglekar wrote:
> The Synopsys xHC has an internal TRB cache of size TRB_CACHE_SIZE for
> each endpoint. The default value for TRB_CACHE_SIZE is 16 for SS and 8
> for HS. The controller loads and updates the TRB cache from the
> transfer ring in system memory whenever the driver issues a start
> transfer or update transfer command.
> 
> For chained TRBs, the Synopsys xHC requires that the total amount of
> bytes for all TRBs loaded in the TRB cache be greater than or equal to
> 1 MPS. Or the chain ends within the TRB cache (with a last TRB).
> 
> If this requirement is not met, the controller will not be able to
> send or receive a packet and it will hang causing a driver timeout and
> error.
> 
> This patch set adds logic to the XHCI driver to detect and prevent this
> from happening along with the quirk to enable this logic for Synopsys
> HAPS platform.
> 
> Based on Mathias's feedback on previous implementation where consolidation
> was done in TRB cache, with this patch series the implementation is done
> during mapping of the URB by consolidating the SG list into a temporary
> buffer if the SG list buffer sizes within TRB_CACHE_SIZE is less than MPS.
> 
> 
> Tejas Joglekar (4):
>   dt-bindings: usb: Add snps,consolidate-sgl & consolidate-sgl
>   usb: xhci: Set quirk for XHCI_CONSOLIDATE_SG_LIST
>   usb: dwc3: Add device property consolidate-sgl
>   usb: xhci: Use temporary buffer to consolidate SG
> 
> Resending as 'umlaut' is not accepted by some servers in email.

Only patch 4/4 seems to have gone through :(

Re: [RESENDING RFC PATCH 0/4] Add logic to consolidate TRBs for Synopsys xHC

[Tejas Joglekar]

On 3/27/2020 2:58 PM, Greg KH wrote:
> On Fri, Mar 27, 2020 at 02:22:23PM +0530, Tejas Joglekar wrote:
>> The Synopsys xHC has an internal TRB cache of size TRB_CACHE_SIZE for
>> each endpoint. The default value for TRB_CACHE_SIZE is 16 for SS and 8
>> for HS. The controller loads and updates the TRB cache from the
>> transfer ring in system memory whenever the driver issues a start
>> transfer or update transfer command.
>>
>> For chained TRBs, the Synopsys xHC requires that the total amount of
>> bytes for all TRBs loaded in the TRB cache be greater than or equal to
>> 1 MPS. Or the chain ends within the TRB cache (with a last TRB).
>>
>> If this requirement is not met, the controller will not be able to
>> send or receive a packet and it will hang causing a driver timeout and
>> error.
>>
>> This patch set adds logic to the XHCI driver to detect and prevent this
>> from happening along with the quirk to enable this logic for Synopsys
>> HAPS platform.
>>
>> Based on Mathias's feedback on previous implementation where consolidation
>> was done in TRB cache, with this patch series the implementation is done
>> during mapping of the URB by consolidating the SG list into a temporary
>> buffer if the SG list buffer sizes within TRB_CACHE_SIZE is less than MPS.
>>
>>
>> Tejas Joglekar (4):
>>   dt-bindings: usb: Add snps,consolidate-sgl & consolidate-sgl
>>   usb: xhci: Set quirk for XHCI_CONSOLIDATE_SG_LIST
>>   usb: dwc3: Add device property consolidate-sgl
>>   usb: xhci: Use temporary buffer to consolidate SG
>>
>> Resending as 'umlaut' is not accepted by some servers in email.
> 
> Only patch 4/4 seems to have gone through :(
> 

Resent the missing patches