Re: [PATCH v9 0/5] Re-introduce TX FIFO resize for larger EP bursting

[Wesley Cheng <wcheng@codeaurora.org>]

Hi Felipe,

On 6/10/2021 2:20 AM, Felipe Balbi wrote:
> 
> Hi,
> 
> Wesley Cheng <wcheng@codeaurora.org> writes:
> 
>> Hi Greg/Felipe,
>>
>> On 6/4/2021 7:36 AM, Greg KH wrote:
>>> On Fri, Jun 04, 2021 at 05:18:16PM +0300, Felipe Balbi wrote:
>>>>
>>>> Hi,
>>>>
>>>> Greg KH <gregkh@linuxfoundation.org> writes:
>>>>> On Wed, May 19, 2021 at 12:49:16AM -0700, Wesley Cheng wrote:
>>>>>> Changes in V9:
>>>>>>  - Fixed incorrect patch in series.  Removed changes in DTSI, as dwc3-qcom will
>>>>>>    add the property by default from the kernel.
>>>>>
>>>>> This patch series has one build failure and one warning added:
>>>>>
>>>>> drivers/usb/dwc3/gadget.c: In function ‘dwc3_gadget_calc_tx_fifo_size’:
>>>>> drivers/usb/dwc3/gadget.c:653:45: warning: passing argument 1 of ‘dwc3_mdwidth’ makes pointer from integer without a cast [-Wint-conversion]
>>>>>   653 |         mdwidth = dwc3_mdwidth(dwc->hwparams.hwparams0);
>>>>>       |                                ~~~~~~~~~~~~~^~~~~~~~~~
>>>>>       |                                             |
>>>>>       |                                             u32 {aka unsigned int}
>>>>> In file included from drivers/usb/dwc3/debug.h:14,
>>>>>                  from drivers/usb/dwc3/gadget.c:25:
>>>>> drivers/usb/dwc3/core.h:1493:45: note: expected ‘struct dwc3 *’ but argument is of type ‘u32’ {aka ‘unsigned int’}
>>>>>  1493 | static inline u32 dwc3_mdwidth(struct dwc3 *dwc)
>>>>>       |                                ~~~~~~~~~~~~~^~~
>>>>>
>>>>>
>>>>> drivers/usb/dwc3/dwc3-qcom.c: In function ‘dwc3_qcom_of_register_core’:
>>>>> drivers/usb/dwc3/dwc3-qcom.c:660:23: error: implicit declaration of function ‘of_add_property’; did you mean ‘of_get_property’? [-Werror=implicit-function-declaration]
>>>>>   660 |                 ret = of_add_property(dwc3_np, prop);
>>>>>       |                       ^~~~~~~~~~~~~~~
>>>>>       |                       of_get_property
>>>>>
>>>>>
>>>>> How did you test these?
>>
>> I ran these changes on our internal branches, which were probably
>> missing some of the recent changes done to the DWC3 drivers.  Will fix
>> the above compile errors and re-submit.
>>
>> In regards to how much these changes have been tested, we've been
>> maintaining the TX FIFO resize logic downstream for a few years already,
>> so its being used in end products.  We also verify this with our
>> internal testing, which has certain benchmarks we need to meet.
> 
> the problem with that is that you *know* which gadget is running
> there. You know everyone of those is going to run the android
> gadget. In a sense, all those multiple products are testing the same
> exact use case :-)
> 

Mmmm, the USB gadget has changed from since we've implemented it, such
as going from Android gadget to Configfs.  Don't forget, we do have
other business segments that use this feature in other configurations as
well :).

>>>> to be honest, I don't think these should go in (apart from the build
>>>> failure) because it's likely to break instantiations of the core with
>>>> differing FIFO sizes. Some instantiations even have some endpoints with
>>>> dedicated functionality that requires the default FIFO size configured
>>>> during coreConsultant instantiation. I know of at OMAP5 and some Intel
>>>> implementations which have dedicated endpoints for processor tracing.
>>>>
>>>> With OMAP5, these endpoints are configured at the top of the available
>>>> endpoints, which means that if a gadget driver gets loaded and takes
>>>> over most of the FIFO space because of this resizing, processor tracing
>>>> will have a hard time running. That being said, processor tracing isn't
>>>> supported in upstream at this moment.
>>>>
>>
>> I agree that the application of this logic may differ between vendors,
>> hence why I wanted to keep this controllable by the DT property, so that
>> for those which do not support this use case can leave it disabled.  The
>> logic is there to ensure that for a given USB configuration, for each EP
>> it would have at least 1 TX FIFO.  For USB configurations which don't
>> utilize all available IN EPs, it would allow re-allocation of internal
>> memory to EPs which will actually be in use.
> 
> The feature ends up being all-or-nothing, then :-) It sounds like we can
> be a little nicer in this regard.
> 

Don't get me wrong, I think once those features become available
upstream, we can improve the logic.  From what I remember when looking
at Andy Shevchenko's Github, the Intel tracer downstream changes were
just to remove physical EP1 and 2 from the DWC3 endpoint list.  If that
was the change which ended up upstream for the Intel tracer then we
could improve the logic to avoid re-sizing those particular EPs.
However, I'm not sure how the changes would look like in the end, so I
would like to wait later down the line to include that :).

>>>> I still think this may cause other places to break down. The promise the
>>>> databook makes is that increasing the FIFO size over 2x wMaxPacketSize
>>>> should bring little to no benefit, if we're not maintaining that, I
>>>> wonder if the problem is with some of the BUSCFG registers instead,
>>>> where we configure interconnect bursting and the like.
>>>
>>
>> I've been referring mainly to the DWC3 programming guide for
>> recommendations on how to improve USB performance in:
>> Section 3.3.5 System Bus Features to Improve USB Performance
> 
> dwc3 or dwc3.1? Either way, since I left Intel I don't have access to
> the databook anymore. I have to trust what you guys are telling me and,
> based on the description so far, I don't think we're doing the right
> thing (yet).
> 

Ah, I see.  DWC3.1 and DWC3 both have that USB performance section.  I
can explain some of the points I made with a bit more detail.  I thought
you still had access to it.

> It would be nice if other users would test this patchset with different
> gadget drivers and different platforms to have some confidence that
> we're limiting possible regressions.
> 
> I would like for Thinh to comment from Synopsys side here.
> 
>> At least when I ran the initial profiling, adjusting the RX/TX
>> thresholds brought little to no benefits.  Even in some of the examples,
> 
> right, the FIFO sizes shouldn't help much. At least that's what Paul
> told me several years ago. Thinh, has the recommendation changed?
> 

So when I mention the RX/TX thresholds, this is different than the FIFO
resize.  The RX/TX threshold is used by the controller to determine when
to send or receive data based on the number of available FIFOs.  So for
the TX case, if we set the TX threshold, the controller will not start
transmitting data over the link after X amount of packets are copied to
the TXFIFO.  So for example, a TXFIFO size of 6 w/ a TX threshold of 3,
means that the controller will wait for 3 FIFO slots to be filled before
it sends the data.  So as you can see, with our configuration of TX FIFO
size of 2 and TX threshold of 1, this would really be not beneficial to
us, because we can only change the TX threshold to 2 at max, and at
least in my observations, once we have to go out to system memory to
fetch the next data packet, that latency takes enough time for the
controller to end the current burst.

>> they have diagrams showing a TXFIFO size of 6 max packets (Figure 3-5).
>>  I think its difficult to say that the TX fifo resizing won't help in
>> systems with limited, or shared resources where the bus latencies would
>> be somewhat larger.  By adjusting the TX FIFO size, the controller would
>> be able to fetch more data from system memory into the memory within the
>> controller, leading to less frequent end of bursts, etc... as data is
>> readily available.
>>
>> In terms of adjusting the AXI/AHB bursting, I would think the bandwidth
>> increase would eventually be constrained based on your system's design.
>>  We don't touch the GSBUSCFG registers, and leave them as is based off
>> the recommendations from the HW designers.
> 
> Right, I want to touch those as little as possible too :-) However, to
> illustrate, the only reason I implemented FIFO resizing was because
> OMAP5 ES1 had TX FIFOs that were smaller than a full USB3 packet. HW
> Designer's recommendation can be bogus too ;-)
> 

Haha...true, we question their designs only when there's something
clearly wrong, but the AXI/AHB settings look good.  :)

>>> Good points.
>>>
>>> Wesley, what kind of testing have you done on this on different devices?
>>>
>>
>> As mentioned above, these changes are currently present on end user
>> devices for the past few years, so its been through a lot of testing :).
> 
> all with the same gadget driver. Also, who uses USB on android devices
> these days? Most of the data transfer goes via WiFi or Bluetooth, anyway
> :-)
> 
> I guess only developers are using USB during development to flash dev
> images heh.
> 

I used to be a customer facing engineer, so honestly I did see some
really interesting and crazy designs.  Again, we do have non-Android
products that use the same code, and it has been working in there for a
few years as well.  The TXFIFO sizing really has helped with multimedia
use cases, which use isoc endpoints, since esp. in those lower end CPU
chips where latencies across the system are much larger, and a missed
ISOC interval leads to a pop in your ear.

Thanks
Wesley Cheng

-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project