[PATCH v3 0/4] Add logic to consolidate TRBs for Synopsys xHC

[PATCH v3 0/4] Add logic to consolidate TRBs for Synopsys xHC

[Tejas Joglekar]

The Synopsys xHC has an internal TRB cache of size TRB_CACHE_SIZE for
each endpoint. The default value for TRB_CACHE_SIZE is 16 for SS and 8
for HS. The controller loads and updates the TRB cache from the
transfer ring in system memory whenever the driver issues a start
transfer or update transfer command.

For chained TRBs, the Synopsys xHC requires that the total amount of
bytes for all TRBs loaded in the TRB cache be greater than or equal to
1 MPS. Or the chain ends within the TRB cache (with a last TRB).

If this requirement is not met, the controller will not be able to
send or receive a packet and it will hang causing a driver timeout and
error.

This patch set adds logic to the XHCI driver to detect and prevent this
from happening along with the quirk to enable this logic for Synopsys
HAPS platform.

Based on Mathias's feedback on previous implementation where consolidation
was done in TRB cache, with this patch series the implementation is done
during mapping of the URB by consolidating the SG list into a temporary
buffer if the SG list buffer sizes within TRB_CACHE_SIZE is less than MPS.

Changes since v2:
 - Modified the xhci_unmap_temp_buffer function to unmap dma and clear
   the dma flag
 - Removed RFC tag

Changes since v1:
 - Comments from Greg are addressed on [PATCH 4/4] and [PATCH 1/4]
 - Renamed the property and quirk as in other patches based on [PATCH 1/4]

Tejas Joglekar (4):
  dt-bindings: usb: Add documentation for SG trb cache size quirk
  usb: xhci: Set quirk for XHCI_SG_TRB_CACHE_SIZE_QUIRK
  usb: dwc3: Add device property sgl-trb-cache-size-quirk
  usb: xhci: Use temporary buffer to consolidate SG

 Documentation/devicetree/bindings/usb/dwc3.txt     |   4 +
 Documentation/devicetree/bindings/usb/usb-xhci.txt |   3 +
 drivers/usb/dwc3/core.c                            |   2 +
 drivers/usb/dwc3/core.h                            |   2 +
 drivers/usb/dwc3/dwc3-haps.c                       |   1 +
 drivers/usb/dwc3/host.c                            |   6 +-
 drivers/usb/host/xhci-pci.c                        |   3 +
 drivers/usb/host/xhci-plat.c                       |   4 +
 drivers/usb/host/xhci-ring.c                       |   2 +-
 drivers/usb/host/xhci.c                            | 135 +++++++++++++++++++++
 drivers/usb/host/xhci.h                            |   5 +
 11 files changed, 165 insertions(+), 2 deletions(-)

-- 
2.11.0

[PATCH v3 1/4] dt-bindings: usb: Add documentation for SG trb cache size quirk

[Tejas Joglekar]

This commit adds the documentation for sgl-trb-cache-size-quirk, and
snps,sgl-trb-cache-size-quirk property. These when set enables the
quirk for XHCI driver for consolidation of sg list into a temporary
buffer when small buffer sizes are scattered over the sg list not
making up to MPS or total transfer size within TRB cache size with
Synopsys xHC.

Signed-off-by: Tejas Joglekar <joglekar@synopsys.com>
---
 Documentation/devicetree/bindings/usb/dwc3.txt     | 4 ++++
 Documentation/devicetree/bindings/usb/usb-xhci.txt | 3 +++
 2 files changed, 7 insertions(+)

diff --git a/Documentation/devicetree/bindings/usb/dwc3.txt b/Documentation/devicetree/bindings/usb/dwc3.txt
index d03edf9d3935..0fcbaa51f66e 100644
--- a/Documentation/devicetree/bindings/usb/dwc3.txt
+++ b/Documentation/devicetree/bindings/usb/dwc3.txt
@@ -102,6 +102,10 @@ Optional properties:
 			this and tx-thr-num-pkt-prd to a valid, non-zero value
 			1-16 (DWC_usb31 programming guide section 1.2.3) to
 			enable periodic ESS TX threshold.
+ - snps,sgl-trb-cache-size-quirk: enable sg list consolidation - host mode
+			only. Set to use SG buffers of at least MPS size
+			by consolidating smaller SG buffers list into a
+			single buffer.
 
  - <DEPRECATED> tx-fifo-resize: determines if the FIFO *has* to be reallocated.
  - snps,incr-burst-type-adjustment: Value for INCR burst type of GSBUSCFG0
diff --git a/Documentation/devicetree/bindings/usb/usb-xhci.txt b/Documentation/devicetree/bindings/usb/usb-xhci.txt
index dc025f126d71..c53eb19ae67e 100644
--- a/Documentation/devicetree/bindings/usb/usb-xhci.txt
+++ b/Documentation/devicetree/bindings/usb/usb-xhci.txt
@@ -44,6 +44,9 @@ Optional properties:
   - quirk-broken-port-ped: set if the controller has broken port disable mechanism
   - imod-interval-ns: default interrupt moderation interval is 5000ns
   - phys : see usb-hcd.yaml in the current directory
+  - sgl-trb-cache-size-quirk: set if you need to consolidate sg list into a
+    temporary buffer when small SG buffer sizes does not make upto MPS
+    size or total transfer size across the TRB cache size.
 
 additionally the properties from usb-hcd.yaml (in the current directory) are
 supported.
-- 
2.11.0

Re: [PATCH v3 1/4] dt-bindings: usb: Add documentation for SG trb cache size quirk

[Rob Herring]

On Wed, May 27, 2020 at 04:10:55PM +0530, Tejas Joglekar wrote:
> This commit adds the documentation for sgl-trb-cache-size-quirk, and
> snps,sgl-trb-cache-size-quirk property. These when set enables the
> quirk for XHCI driver for consolidation of sg list into a temporary
> buffer when small buffer sizes are scattered over the sg list not
> making up to MPS or total transfer size within TRB cache size with
> Synopsys xHC.
> 
> Signed-off-by: Tejas Joglekar <joglekar@synopsys.com>
> ---
>  Documentation/devicetree/bindings/usb/dwc3.txt     | 4 ++++
>  Documentation/devicetree/bindings/usb/usb-xhci.txt | 3 +++
>  2 files changed, 7 insertions(+)
> 
> diff --git a/Documentation/devicetree/bindings/usb/dwc3.txt b/Documentation/devicetree/bindings/usb/dwc3.txt
> index d03edf9d3935..0fcbaa51f66e 100644
> --- a/Documentation/devicetree/bindings/usb/dwc3.txt
> +++ b/Documentation/devicetree/bindings/usb/dwc3.txt
> @@ -102,6 +102,10 @@ Optional properties:
>  			this and tx-thr-num-pkt-prd to a valid, non-zero value
>  			1-16 (DWC_usb31 programming guide section 1.2.3) to
>  			enable periodic ESS TX threshold.
> + - snps,sgl-trb-cache-size-quirk: enable sg list consolidation - host mode
> +			only. Set to use SG buffers of at least MPS size
> +			by consolidating smaller SG buffers list into a
> +			single buffer.
>  
>   - <DEPRECATED> tx-fifo-resize: determines if the FIFO *has* to be reallocated.
>   - snps,incr-burst-type-adjustment: Value for INCR burst type of GSBUSCFG0
> diff --git a/Documentation/devicetree/bindings/usb/usb-xhci.txt b/Documentation/devicetree/bindings/usb/usb-xhci.txt
> index dc025f126d71..c53eb19ae67e 100644
> --- a/Documentation/devicetree/bindings/usb/usb-xhci.txt
> +++ b/Documentation/devicetree/bindings/usb/usb-xhci.txt
> @@ -44,6 +44,9 @@ Optional properties:
>    - quirk-broken-port-ped: set if the controller has broken port disable mechanism
>    - imod-interval-ns: default interrupt moderation interval is 5000ns
>    - phys : see usb-hcd.yaml in the current directory
> +  - sgl-trb-cache-size-quirk: set if you need to consolidate sg list into a
> +    temporary buffer when small SG buffer sizes does not make upto MPS
> +    size or total transfer size across the TRB cache size.

Still don't understand why you have 2 properties? Is this a generic 
issue for multiple XHCI controllers? If yes, you don't need the first 
one. If no, then you don't need the second one.

Really, I'd prefer neither, and this should be implied by a specific 
compatible string. Having a separate property doesn't work if you find 
this issue later on after already adding XHCI support. IOW, don't make 
users update their DT to handle a quirk.

Rob

Re: [PATCH v3 0/4] Add logic to consolidate TRBs for Synopsys xHC

[Tejas Joglekar]

Hi Mathias,
   Will this be added to your next branch ?
On 5/27/2020 4:10 PM, Tejas Joglekar wrote:
> The Synopsys xHC has an internal TRB cache of size TRB_CACHE_SIZE for
> each endpoint. The default value for TRB_CACHE_SIZE is 16 for SS and 8
> for HS. The controller loads and updates the TRB cache from the
> transfer ring in system memory whenever the driver issues a start
> transfer or update transfer command.
> 
> For chained TRBs, the Synopsys xHC requires that the total amount of
> bytes for all TRBs loaded in the TRB cache be greater than or equal to
> 1 MPS. Or the chain ends within the TRB cache (with a last TRB).
> 
> If this requirement is not met, the controller will not be able to
> send or receive a packet and it will hang causing a driver timeout and
> error.
> 
> This patch set adds logic to the XHCI driver to detect and prevent this
> from happening along with the quirk to enable this logic for Synopsys
> HAPS platform.
> 
> Based on Mathias's feedback on previous implementation where consolidation
> was done in TRB cache, with this patch series the implementation is done
> during mapping of the URB by consolidating the SG list into a temporary
> buffer if the SG list buffer sizes within TRB_CACHE_SIZE is less than MPS.
> 
> Changes since v2:
>  - Modified the xhci_unmap_temp_buffer function to unmap dma and clear
>    the dma flag
>  - Removed RFC tag
> 
> Changes since v1:
>  - Comments from Greg are addressed on [PATCH 4/4] and [PATCH 1/4]
>  - Renamed the property and quirk as in other patches based on [PATCH 1/4]
> 
> Tejas Joglekar (4):
>   dt-bindings: usb: Add documentation for SG trb cache size quirk
>   usb: xhci: Set quirk for XHCI_SG_TRB_CACHE_SIZE_QUIRK
>   usb: dwc3: Add device property sgl-trb-cache-size-quirk
>   usb: xhci: Use temporary buffer to consolidate SG
> 
>  Documentation/devicetree/bindings/usb/dwc3.txt     |   4 +
>  Documentation/devicetree/bindings/usb/usb-xhci.txt |   3 +
>  drivers/usb/dwc3/core.c                            |   2 +
>  drivers/usb/dwc3/core.h                            |   2 +
>  drivers/usb/dwc3/dwc3-haps.c                       |   1 +
>  drivers/usb/dwc3/host.c                            |   6 +-
>  drivers/usb/host/xhci-pci.c                        |   3 +
>  drivers/usb/host/xhci-plat.c                       |   4 +
>  drivers/usb/host/xhci-ring.c                       |   2 +-
>  drivers/usb/host/xhci.c                            | 135 +++++++++++++++++++++
>  drivers/usb/host/xhci.h                            |   5 +
>  11 files changed, 165 insertions(+), 2 deletions(-)
> 

Thanks and Regards,
 Tejas Joglekar

Re: [PATCH v3 0/4] Add logic to consolidate TRBs for Synopsys xHC

[Mathias Nyman]

On 8.6.2020 7.32, Tejas Joglekar wrote:
> Hi Mathias,
>    Will this be added to your next branch ?

There are still some opens Rob Herring pointed out regarding devicetree.
Adding a compatible string for the synopsys xhci and setting quirk based
on that sounds good to me.

Not sure how that works in cases where the xhci device is created by the DWC3 driver.
Once we have a solution that Felipe and Rob agrees with I can take the whole
series.  

The xhci parts and PCI case looks good to me.

-Mathias

Re: [PATCH v3 0/4] Add logic to consolidate TRBs for Synopsys xHC

[Tejas Joglekar]

Hi,
On 6/9/2020 2:27 PM, Mathias Nyman wrote:
> On 8.6.2020 7.32, Tejas Joglekar wrote:
>> Hi Mathias,
>>    Will this be added to your next branch ?
> 
> There are still some opens Rob Herring pointed out regarding devicetree.
> Adding a compatible string for the synopsys xhci and setting quirk based
> on that sounds good to me.
> 
> Not sure how that works in cases where the xhci device is created by the DWC3 driver.
> Once we have a solution that Felipe and Rob agrees with I can take the whole
> series.  
> 
@Felipe: Can you please have a look and see if the current implementation is ok or anything to be changed?

> The xhci parts and PCI case looks good to me.
> 
> -Mathias
> 
> 
Thanks & Regards,
 Tejas Joglekar

Re: [PATCH v3 0/4] Add logic to consolidate TRBs for Synopsys xHC

[Tejas Joglekar]

Hi Felipe,
On 6/11/2020 11:37 PM, Tejas Joglekar wrote:
> Hi,
> On 6/9/2020 2:27 PM, Mathias Nyman wrote:
>> On 8.6.2020 7.32, Tejas Joglekar wrote:
>>> Hi Mathias,
>>>    Will this be added to your next branch ?
>>
>> There are still some opens Rob Herring pointed out regarding devicetree.
>> Adding a compatible string for the synopsys xhci and setting quirk based
>> on that sounds good to me.
>>
>> Not sure how that works in cases where the xhci device is created by the DWC3 driver.
>> Once we have a solution that Felipe and Rob agrees with I can take the whole
>> series.  
>>
> @Felipe: Can you please have a look and see if the current implementation is ok or anything to be changed?
> 
Can you please have review my patch and suggest the changes required, so Mathias can accept the series?

>> The xhci parts and PCI case looks good to me.
>>
>> -Mathias
>>
>>
> Thanks & Regards,
>  Tejas Joglekar
> 

Thanks & Regards,
  Tejas Joglekar

Re: [PATCH v3 3/4] usb: dwc3: Add device property sgl-trb-cache-size-quirk

[Tejas Joglekar]

Hi Rob,

On 7/6/2020 12:13 PM, Felipe Balbi wrote:
> 
> Hi,
> 
> Tejas Joglekar <Tejas.Joglekar@synopsys.com> writes:
>>> @@ -95,6 +95,10 @@ int dwc3_host_init(struct dwc3 *dwc)
>>>  	if (dwc->usb2_lpm_disable)
>>>  		props[prop_idx++] = PROPERTY_ENTRY_BOOL("usb2-lpm-disable");
>>>  
>>> +	if (dwc->sgl_trb_cache_size_quirk)
>>> +		props[prop_idx++] =
>>> +			PROPERTY_ENTRY_BOOL("sgl-trb-cache-size-quirk");
>>> +
>>>  	/**
>>>  	 * WORKAROUND: dwc3 revisions <=3.00a have a limitation
>>>  	 * where Port Disable command doesn't work.
>>>
>> Does this implementation looks good to you? Rob has some concerned over the DT entries,
>> you suggested using compatible string with this quirk addition.
>> Can you please brief about how you would like to have this quirk implemented?
>> I can send the updated patch. My patch series is pending for merge just because of the
>> DT and quirk issue. Can you please help?
> 
> Yeah, you need to get into an agreement with Rob :-) I don't mind having
> extra DT flags for things which can't be detected in runtime, Rob
> disagrees.
> 
The compatible string is not suitable option as it does not work with platform drivers
with PCI based system. Also Synopsys controllers IP version register is not visible to xhci
driver and hence we don't have separate compatible string for each Synopsys version on the
xhci driver side. 
Due to which I depend on DT flag addition for the quirk. Can we add these DT flags and quirk?


Thanks & Regards,
 Tejas Joglekar

Re: [PATCH v3 3/4] usb: dwc3: Add device property sgl-trb-cache-size-quirk

[Felipe Balbi]

[-- Attachment #1: Type: text/plain, Size: 1934 bytes --]

Tejas Joglekar <Tejas.Joglekar@synopsys.com> writes:

> Hi Rob,
>
> On 7/6/2020 12:13 PM, Felipe Balbi wrote:
>> 
>> Hi,
>> 
>> Tejas Joglekar <Tejas.Joglekar@synopsys.com> writes:
>>>> @@ -95,6 +95,10 @@ int dwc3_host_init(struct dwc3 *dwc)
>>>>  	if (dwc->usb2_lpm_disable)
>>>>  		props[prop_idx++] = PROPERTY_ENTRY_BOOL("usb2-lpm-disable");
>>>>  
>>>> +	if (dwc->sgl_trb_cache_size_quirk)
>>>> +		props[prop_idx++] =
>>>> +			PROPERTY_ENTRY_BOOL("sgl-trb-cache-size-quirk");
>>>> +
>>>>  	/**
>>>>  	 * WORKAROUND: dwc3 revisions <=3.00a have a limitation
>>>>  	 * where Port Disable command doesn't work.
>>>>
>>> Does this implementation looks good to you? Rob has some concerned over the DT entries,
>>> you suggested using compatible string with this quirk addition.
>>> Can you please brief about how you would like to have this quirk implemented?
>>> I can send the updated patch. My patch series is pending for merge just because of the
>>> DT and quirk issue. Can you please help?
>> 
>> Yeah, you need to get into an agreement with Rob :-) I don't mind having
>> extra DT flags for things which can't be detected in runtime, Rob
>> disagrees.
>> 
> The compatible string is not suitable option as it does not work with platform drivers
> with PCI based system. Also Synopsys controllers IP version register is not visible to xhci
> driver and hence we don't have separate compatible string for each Synopsys version on the
> xhci driver side. 
> Due to which I depend on DT flag addition for the quirk. Can we add these DT flags and quirk?

As I said, I'm well aware of the situation regarding usage of compatible
strings and the fact that dwc3 must work on PCI and non-PCI systems (I
wrote the thing as it is after all). The person blocking new quirk flags
is Rob, not me. You need to convince Rob that this is the way to go.

Rob, ball's in your court. Sorry.

-- 
balbi

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: [PATCH v3 3/4] usb: dwc3: Add device property sgl-trb-cache-size-quirk

[Tejas Joglekar]

Hello,

On 7/21/2020 3:17 PM, Felipe Balbi wrote:
> Tejas Joglekar <Tejas.Joglekar@synopsys.com> writes:
> 
>> Hi Rob,
>>
>> On 7/6/2020 12:13 PM, Felipe Balbi wrote:
>>>
>>> Hi,
>>>
>>> Tejas Joglekar <Tejas.Joglekar@synopsys.com> writes:
>>>>> @@ -95,6 +95,10 @@ int dwc3_host_init(struct dwc3 *dwc)
>>>>>  	if (dwc->usb2_lpm_disable)
>>>>>  		props[prop_idx++] = PROPERTY_ENTRY_BOOL("usb2-lpm-disable");
>>>>>  
>>>>> +	if (dwc->sgl_trb_cache_size_quirk)
>>>>> +		props[prop_idx++] =
>>>>> +			PROPERTY_ENTRY_BOOL("sgl-trb-cache-size-quirk");
>>>>> +
>>>>>  	/**
>>>>>  	 * WORKAROUND: dwc3 revisions <=3.00a have a limitation
>>>>>  	 * where Port Disable command doesn't work.
>>>>>
>>>> Does this implementation looks good to you? Rob has some concerned over the DT entries,
>>>> you suggested using compatible string with this quirk addition.
>>>> Can you please brief about how you would like to have this quirk implemented?
>>>> I can send the updated patch. My patch series is pending for merge just because of the
>>>> DT and quirk issue. Can you please help?
>>>
>>> Yeah, you need to get into an agreement with Rob :-) I don't mind having
>>> extra DT flags for things which can't be detected in runtime, Rob
>>> disagrees.
>>>
>> The compatible string is not suitable option as it does not work with platform drivers
>> with PCI based system. Also Synopsys controllers IP version register is not visible to xhci
>> driver and hence we don't have separate compatible string for each Synopsys version on the
>> xhci driver side. 
>> Due to which I depend on DT flag addition for the quirk. Can we add these DT flags and quirk?
> 
> As I said, I'm well aware of the situation regarding usage of compatible
> strings and the fact that dwc3 must work on PCI and non-PCI systems (I
> wrote the thing as it is after all). The person blocking new quirk flags
> is Rob, not me. You need to convince Rob that this is the way to go.
> 
@Felipe: Sorry for confusion if any, previous mail was intended for Rob asking about his approval.

> Rob, ball's in your court. Sorry.> 
@Rob: As I and Felipe have mentioned before, it is very much necessary to have quirk flags
for the current changes as compatible string would not be a solution for PCI and non-PCI
systems. Can you please approve this change ? If you have any concern about naming or any
other thing, please let us know.

Thanks & Regards,
 Tejas Joglekar

Re: [PATCH v3 0/4] Add logic to consolidate TRBs for Synopsys xHC

[Jun Li]

Tejas Joglekar <Tejas.Joglekar@synopsys.com> 于2020年5月27日周三 下午7:54写道：
>
> The Synopsys xHC has an internal TRB cache of size TRB_CACHE_SIZE for
> each endpoint. The default value for TRB_CACHE_SIZE is 16 for SS and 8
> for HS. The controller loads and updates the TRB cache from the
> transfer ring in system memory whenever the driver issues a start
> transfer or update transfer command.
>
> For chained TRBs, the Synopsys xHC requires that the total amount of
> bytes for all TRBs loaded in the TRB cache be greater than or equal to
> 1 MPS. Or the chain ends within the TRB cache (with a last TRB).
>
> If this requirement is not met, the controller will not be able to
> send or receive a packet and it will hang causing a driver timeout and
> error.

Hi Tejas Joglekar

I am debugging  a similar issue on Synipsys XHC, it's not the same case
but I am wondering if it also linked to this HW limitation.

My Synopsys XHC based host enable UAS, when enumerates a UAS
HDD, one BULK-IN EP with stream enabled will not generate event for
trb(with stream ID 1) after a 16/4096 bytes(with stream ID 2) finished in
previous trb.

If I change the last OK urb/trb's buffer length from 4096 to 512, the issue
will gone.

following is the sequence of the question EP-IN:

<idle>-0     [000] d.h1   154.961710: xhci_urb_giveback: ep3in-bulk:
urb ffff0001775f6f00 pipe 3221324672 slot 1 length 36/36 sgs 1/1
stream 1 flags 00040200
<idle>-0     [000] d.h1   154.962023: xhci_urb_giveback: ep3in-bulk:
urb ffff000177d00400 pipe 3221324672 slot 1 length 96/96 sgs 1/1
stream 1 flags 00040200
<idle>-0     [000] d.h1   154.970395: xhci_urb_giveback: ep3in-bulk:
urb ffff000177d00400 pipe 3221324672 slot 1 length 11/255 sgs 1/1
stream 1 flags 00040200
<idle>-0     [000] d.h1   154.970562: xhci_urb_giveback: ep3in-bulk:
urb ffff000177d00400 pipe 3221324672 slot 1 length 20/255 sgs 1/1
stream 1 flags 00040200
<idle>-0     [000] d.h1   154.970786: xhci_urb_giveback: ep3in-bulk:
urb ffff000177d00400 pipe 3221324672 slot 1 length 60/255 sgs 1/1
stream 1 flags 00040200
<idle>-0     [000] d.h1   155.851600: xhci_urb_giveback: ep3in-bulk:
urb ffff000177d00200 pipe 3221324672 slot 1 length 16/4096 sgs 1/1
stream 2 flags 00040200

/* then the next ep3-in trb will not generate event and stopped, so
driver timeout in the end */
kworker/u8:2-349   [003] d..3   155.851987: xhci_urb_enqueue:
ep3in-bulk: urb ffff000170492400 pipe 3221324672 slot 1 length 0/32
sgs 1/1 stream 1 flags 00040200
kworker/u8:2-349   [003] d..4   155.851989: xhci_queue_trb: STREAM:
Buffer 00000000c19cf000 length 32 TD size 0 intr 0 type 'Normal' flags
b:i:I:c:s:I:e:c
kworker/u8:2-349   [003] d..4   155.851991: xhci_inc_enq: STREAM
ffff000177f86f80: enq 0x00000000be0eb060(0x00000000be0eb000) deq
0x00000000be0eb050(0x00000000be0eb000) segs 2 stream 1 free_trbs 508
bounce 1024 cycle 1

Do you have any ideas?

thanks
Li Jun
>
> This patch set adds logic to the XHCI driver to detect and prevent this
> from happening along with the quirk to enable this logic for Synopsys
> HAPS platform.
>
> Based on Mathias's feedback on previous implementation where consolidation
> was done in TRB cache, with this patch series the implementation is done
> during mapping of the URB by consolidating the SG list into a temporary
> buffer if the SG list buffer sizes within TRB_CACHE_SIZE is less than MPS.
>
> Changes since v2:
>  - Modified the xhci_unmap_temp_buffer function to unmap dma and clear
>    the dma flag
>  - Removed RFC tag
>
> Changes since v1:
>  - Comments from Greg are addressed on [PATCH 4/4] and [PATCH 1/4]
>  - Renamed the property and quirk as in other patches based on [PATCH 1/4]
>
> Tejas Joglekar (4):
>   dt-bindings: usb: Add documentation for SG trb cache size quirk
>   usb: xhci: Set quirk for XHCI_SG_TRB_CACHE_SIZE_QUIRK
>   usb: dwc3: Add device property sgl-trb-cache-size-quirk
>   usb: xhci: Use temporary buffer to consolidate SG
>
>  Documentation/devicetree/bindings/usb/dwc3.txt     |   4 +
>  Documentation/devicetree/bindings/usb/usb-xhci.txt |   3 +
>  drivers/usb/dwc3/core.c                            |   2 +
>  drivers/usb/dwc3/core.h                            |   2 +
>  drivers/usb/dwc3/dwc3-haps.c                       |   1 +
>  drivers/usb/dwc3/host.c                            |   6 +-
>  drivers/usb/host/xhci-pci.c                        |   3 +
>  drivers/usb/host/xhci-plat.c                       |   4 +
>  drivers/usb/host/xhci-ring.c                       |   2 +-
>  drivers/usb/host/xhci.c                            | 135 +++++++++++++++++++++
>  drivers/usb/host/xhci.h                            |   5 +
>  11 files changed, 165 insertions(+), 2 deletions(-)
>
> --
> 2.11.0
>

Re: [PATCH v3 0/4] Add logic to consolidate TRBs for Synopsys xHC

[Tejas Joglekar]

Hello,
On 7/23/2020 4:05 PM, Jun Li wrote:
> Tejas Joglekar <Tejas.Joglekar@synopsys.com> 于2020年5月27日周三 下午7:54写道：
>>
>> The Synopsys xHC has an internal TRB cache of size TRB_CACHE_SIZE for
>> each endpoint. The default value for TRB_CACHE_SIZE is 16 for SS and 8
>> for HS. The controller loads and updates the TRB cache from the
>> transfer ring in system memory whenever the driver issues a start
>> transfer or update transfer command.
>>
>> For chained TRBs, the Synopsys xHC requires that the total amount of
>> bytes for all TRBs loaded in the TRB cache be greater than or equal to
>> 1 MPS. Or the chain ends within the TRB cache (with a last TRB).
>>
>> If this requirement is not met, the controller will not be able to
>> send or receive a packet and it will hang causing a driver timeout and
>> error.
> 
> Hi Tejas Joglekar
> 
> I am debugging  a similar issue on Synipsys XHC, it's not the same case
> but I am wondering if it also linked to this HW limitation.
> 
> My Synopsys XHC based host enable UAS, when enumerates a UAS
> HDD, one BULK-IN EP with stream enabled will not generate event for
> trb(with stream ID 1) after a 16/4096 bytes(with stream ID 2) finished in
> previous trb.
> 
> If I change the last OK urb/trb's buffer length from 4096 to 512, the issue
> will gone.
> 
> following is the sequence of the question EP-IN:
> 
> <idle>-0     [000] d.h1   154.961710: xhci_urb_giveback: ep3in-bulk:
> urb ffff0001775f6f00 pipe 3221324672 slot 1 length 36/36 sgs 1/1
> stream 1 flags 00040200
> <idle>-0     [000] d.h1   154.962023: xhci_urb_giveback: ep3in-bulk:
> urb ffff000177d00400 pipe 3221324672 slot 1 length 96/96 sgs 1/1
> stream 1 flags 00040200
> <idle>-0     [000] d.h1   154.970395: xhci_urb_giveback: ep3in-bulk:
> urb ffff000177d00400 pipe 3221324672 slot 1 length 11/255 sgs 1/1
> stream 1 flags 00040200
> <idle>-0     [000] d.h1   154.970562: xhci_urb_giveback: ep3in-bulk:
> urb ffff000177d00400 pipe 3221324672 slot 1 length 20/255 sgs 1/1
> stream 1 flags 00040200
> <idle>-0     [000] d.h1   154.970786: xhci_urb_giveback: ep3in-bulk:
> urb ffff000177d00400 pipe 3221324672 slot 1 length 60/255 sgs 1/1
> stream 1 flags 00040200
> <idle>-0     [000] d.h1   155.851600: xhci_urb_giveback: ep3in-bulk:
> urb ffff000177d00200 pipe 3221324672 slot 1 length 16/4096 sgs 1/1
> stream 2 flags 00040200
> 
> /* then the next ep3-in trb will not generate event and stopped, so
> driver timeout in the end */
> kworker/u8:2-349   [003] d..3   155.851987: xhci_urb_enqueue:
> ep3in-bulk: urb ffff000170492400 pipe 3221324672 slot 1 length 0/32
> sgs 1/1 stream 1 flags 00040200
> kworker/u8:2-349   [003] d..4   155.851989: xhci_queue_trb: STREAM:
> Buffer 00000000c19cf000 length 32 TD size 0 intr 0 type 'Normal' flags
> b:i:I:c:s:I:e:c
> kworker/u8:2-349   [003] d..4   155.851991: xhci_inc_enq: STREAM
> ffff000177f86f80: enq 0x00000000be0eb060(0x00000000be0eb000) deq
> 0x00000000be0eb050(0x00000000be0eb000) segs 2 stream 1 free_trbs 508
> bounce 1024 cycle 1
> 
> Do you have any ideas?
> 
From initial observation of your issue it seems to be unrelated with the
changes in my patch. But could you please provide, USB analyzer trace logs
for the working and non-working case? Also it would be helpful if you can
provide device details which you are testing.

> thanks
> Li Jun

Thanks & Regards,
 Tejas Joglekar

Re: [PATCH v3 3/4] usb: dwc3: Add device property sgl-trb-cache-size-quirk

[Tejas Joglekar]

Hello Rob,
On 7/21/2020 10:27 PM, Tejas Joglekar wrote:
> Hello,
> 
> On 7/21/2020 3:17 PM, Felipe Balbi wrote:
>> Tejas Joglekar <Tejas.Joglekar@synopsys.com> writes:
>>
>>> Hi Rob,
>>>
>>> On 7/6/2020 12:13 PM, Felipe Balbi wrote:
>>>>
>>>> Hi,
>>>>
>>>> Tejas Joglekar <Tejas.Joglekar@synopsys.com> writes:
>>>>>> @@ -95,6 +95,10 @@ int dwc3_host_init(struct dwc3 *dwc)
>>>>>>  	if (dwc->usb2_lpm_disable)
>>>>>>  		props[prop_idx++] = PROPERTY_ENTRY_BOOL("usb2-lpm-disable");
>>>>>>  
>>>>>> +	if (dwc->sgl_trb_cache_size_quirk)
>>>>>> +		props[prop_idx++] =
>>>>>> +			PROPERTY_ENTRY_BOOL("sgl-trb-cache-size-quirk");
>>>>>> +
>>>>>>  	/**
>>>>>>  	 * WORKAROUND: dwc3 revisions <=3.00a have a limitation
>>>>>>  	 * where Port Disable command doesn't work.
>>>>>>
>>>>> Does this implementation looks good to you? Rob has some concerned over the DT entries,
>>>>> you suggested using compatible string with this quirk addition.
>>>>> Can you please brief about how you would like to have this quirk implemented?
>>>>> I can send the updated patch. My patch series is pending for merge just because of the
>>>>> DT and quirk issue. Can you please help?
>>>>
>>>> Yeah, you need to get into an agreement with Rob :-) I don't mind having
>>>> extra DT flags for things which can't be detected in runtime, Rob
>>>> disagrees.
>>>>
>>> The compatible string is not suitable option as it does not work with platform drivers
>>> with PCI based system. Also Synopsys controllers IP version register is not visible to xhci
>>> driver and hence we don't have separate compatible string for each Synopsys version on the
>>> xhci driver side. 
>>> Due to which I depend on DT flag addition for the quirk. Can we add these DT flags and quirk?
>>
>> As I said, I'm well aware of the situation regarding usage of compatible
>> strings and the fact that dwc3 must work on PCI and non-PCI systems (I
>> wrote the thing as it is after all). The person blocking new quirk flags
>> is Rob, not me. You need to convince Rob that this is the way to go.
>>
> @Felipe: Sorry for confusion if any, previous mail was intended for Rob asking about his approval.
> 
>> Rob, ball's in your court. Sorry.> 
> @Rob: As I and Felipe have mentioned before, it is very much necessary to have quirk flags
> for the current changes as compatible string would not be a solution for PCI and non-PCI
> systems. Can you please approve this change ? If you have any concern about naming or any
> other thing, please let us know.
> 
Can you please comment?

Thanks & Regards,
Tejas Joglekar
 

Re: [PATCH v3 3/4] usb: dwc3: Add device property sgl-trb-cache-size-quirk

[Thinh Nguyen]

Hi Rob,

Tejas Joglekar wrote:
> Hello Rob,
> On 7/21/2020 10:27 PM, Tejas Joglekar wrote:
>> Hello,
>>
>> On 7/21/2020 3:17 PM, Felipe Balbi wrote:
>>> Tejas Joglekar <Tejas.Joglekar@synopsys.com> writes:
>>>
>>>> Hi Rob,
>>>>
>>>> On 7/6/2020 12:13 PM, Felipe Balbi wrote:
>>>>> Hi,
>>>>>
>>>>> Tejas Joglekar <Tejas.Joglekar@synopsys.com> writes:
>>>>>>> @@ -95,6 +95,10 @@ int dwc3_host_init(struct dwc3 *dwc)
>>>>>>>  	if (dwc->usb2_lpm_disable)
>>>>>>>  		props[prop_idx++] = PROPERTY_ENTRY_BOOL("usb2-lpm-disable");
>>>>>>>  
>>>>>>> +	if (dwc->sgl_trb_cache_size_quirk)
>>>>>>> +		props[prop_idx++] =
>>>>>>> +			PROPERTY_ENTRY_BOOL("sgl-trb-cache-size-quirk");
>>>>>>> +
>>>>>>>  	/**
>>>>>>>  	 * WORKAROUND: dwc3 revisions <=3.00a have a limitation
>>>>>>>  	 * where Port Disable command doesn't work.
>>>>>>>
>>>>>> Does this implementation looks good to you? Rob has some concerned over the DT entries,
>>>>>> you suggested using compatible string with this quirk addition.
>>>>>> Can you please brief about how you would like to have this quirk implemented?
>>>>>> I can send the updated patch. My patch series is pending for merge just because of the
>>>>>> DT and quirk issue. Can you please help?
>>>>> Yeah, you need to get into an agreement with Rob :-) I don't mind having
>>>>> extra DT flags for things which can't be detected in runtime, Rob
>>>>> disagrees.
>>>>>
>>>> The compatible string is not suitable option as it does not work with platform drivers
>>>> with PCI based system. Also Synopsys controllers IP version register is not visible to xhci
>>>> driver and hence we don't have separate compatible string for each Synopsys version on the
>>>> xhci driver side. 
>>>> Due to which I depend on DT flag addition for the quirk. Can we add these DT flags and quirk?
>>> As I said, I'm well aware of the situation regarding usage of compatible
>>> strings and the fact that dwc3 must work on PCI and non-PCI systems (I
>>> wrote the thing as it is after all). The person blocking new quirk flags
>>> is Rob, not me. You need to convince Rob that this is the way to go.
>>>
>> @Felipe: Sorry for confusion if any, previous mail was intended for Rob asking about his approval.
>>
>>> Rob, ball's in your court. Sorry.> 
>> @Rob: As I and Felipe have mentioned before, it is very much necessary to have quirk flags
>> for the current changes as compatible string would not be a solution for PCI and non-PCI
>> systems. Can you please approve this change ? If you have any concern about naming or any
>> other thing, please let us know.
>>
> Can you please comment?
>
> Thanks & Regards,
> Tejas Joglekar
>  
>

Can you help provide pointers and changes that Tejas can make to help
bring this issue to conclusion?

Thanks,
Thinh

Re: [PATCH v3 3/4] usb: dwc3: Add device property sgl-trb-cache-size-quirk

[Jun Li]

Thinh Nguyen <Thinh.Nguyen@synopsys.com> 于2020年8月4日周二 上午8:46写道：
>
> Hi Rob,
>
> Tejas Joglekar wrote:
> > Hello Rob,
> > On 7/21/2020 10:27 PM, Tejas Joglekar wrote:
> >> Hello,
> >>
> >> On 7/21/2020 3:17 PM, Felipe Balbi wrote:
> >>> Tejas Joglekar <Tejas.Joglekar@synopsys.com> writes:
> >>>
> >>>> Hi Rob,
> >>>>
> >>>> On 7/6/2020 12:13 PM, Felipe Balbi wrote:
> >>>>> Hi,
> >>>>>
> >>>>> Tejas Joglekar <Tejas.Joglekar@synopsys.com> writes:
> >>>>>>> @@ -95,6 +95,10 @@ int dwc3_host_init(struct dwc3 *dwc)
> >>>>>>>         if (dwc->usb2_lpm_disable)
> >>>>>>>                 props[prop_idx++] = PROPERTY_ENTRY_BOOL("usb2-lpm-disable");
> >>>>>>>
> >>>>>>> +       if (dwc->sgl_trb_cache_size_quirk)
> >>>>>>> +               props[prop_idx++] =
> >>>>>>> +                       PROPERTY_ENTRY_BOOL("sgl-trb-cache-size-quirk");
> >>>>>>> +
> >>>>>>>         /**
> >>>>>>>          * WORKAROUND: dwc3 revisions <=3.00a have a limitation
> >>>>>>>          * where Port Disable command doesn't work.
> >>>>>>>
> >>>>>> Does this implementation looks good to you? Rob has some concerned over the DT entries,
> >>>>>> you suggested using compatible string with this quirk addition.
> >>>>>> Can you please brief about how you would like to have this quirk implemented?
> >>>>>> I can send the updated patch. My patch series is pending for merge just because of the
> >>>>>> DT and quirk issue. Can you please help?
> >>>>> Yeah, you need to get into an agreement with Rob :-) I don't mind having
> >>>>> extra DT flags for things which can't be detected in runtime, Rob
> >>>>> disagrees.
> >>>>>
> >>>> The compatible string is not suitable option as it does not work with platform drivers
> >>>> with PCI based system. Also Synopsys controllers IP version register is not visible to xhci
> >>>> driver and hence we don't have separate compatible string for each Synopsys version on the
> >>>> xhci driver side.
> >>>> Due to which I depend on DT flag addition for the quirk. Can we add these DT flags and quirk?
> >>> As I said, I'm well aware of the situation regarding usage of compatible
> >>> strings and the fact that dwc3 must work on PCI and non-PCI systems (I
> >>> wrote the thing as it is after all). The person blocking new quirk flags
> >>> is Rob, not me. You need to convince Rob that this is the way to go.
> >>>
> >> @Felipe: Sorry for confusion if any, previous mail was intended for Rob asking about his approval.
> >>
> >>> Rob, ball's in your court. Sorry.>
> >> @Rob: As I and Felipe have mentioned before, it is very much necessary to have quirk flags
> >> for the current changes as compatible string would not be a solution for PCI and non-PCI
> >> systems. Can you please approve this change ? If you have any concern about naming or any
> >> other thing, please let us know.
> >>
> > Can you please comment?
> >
> > Thanks & Regards,
> > Tejas Joglekar
> >
> >
>
> Can you help provide pointers and changes that Tejas can make to help
> bring this issue to conclusion?

We really need a direction to handle growing dwc3 quirks/flags as dwc3
is widly used,
I have a patchset also pending there after tried both property[1] and
platform data[2].

[1] https://www.spinics.net/lists/linux-usb/msg196055.html
[2] https://www.spinics.net/lists/arm-kernel/msg824995.html

thanks
Li Jun
>
> Thanks,
> Thinh