Re: USB network gadget / DWC3 issue

[Ferry Toth <fntoth@gmail.com>]

Hi,

Op 03-04-2021 om 13:25 schreef Ferry Toth:
> Hi,
> 
> Op 03-04-2021 om 04:02 schreef Thinh Nguyen:
>> Ferry Toth wrote:
>>> Hi,
>>>
>>> Op 02-04-2021 om 22:16 schreef Thinh Nguyen:
>>>> Ferry Toth wrote:
>>>>> Hi
>>>>>
>>>>> Op 30-03-2021 om 23:57 schreef Ferry Toth:
>>>>>> Hi
>>>>>>
>>>>>> Op 30-03-2021 om 22:26 schreef Ferry Toth:
>>>>>>> Hi,
>>>>>>>
>>>>>>> Op 30-03-2021 om 18:17 schreef Felipe Balbi:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Andy Shevchenko <andy.shevchenko@gmail.com> writes:
>>>>>>>>> Hi!
>>>>>>>>>
>>>>>>>>> I have a platform with DWC3 in Dual Role mode. Currently I'm
>>>>>>>>> experimenting on v5.12-rc5 with a few patches (mostly 
>>>>>>>>> configuration)
>>>>>>>>> applied [1]. I'm using Debian Unstable on the host machine and
>>>>>>>>> BuildRoot with the above mentioned kernel on the target.
>>>>>>>>>
>>>>>>>>> **So, scenario 0:
>>>>>>>>> 1. Run iperf3 -s on target
>>>>>>>>> 2. Run iperf3 -c ... -t 0 on the host
>>>>>>>>> 3.  0.00-10.36  sec   237 MBytes   192 Mbits/sec
>>>>>>>>> receiver
>>>>>>>>>
>>>>>>>>> **Scenario 1:
>>>>>>>>> 1. Now, detach USB cable, wait for several seconds, attach it 
>>>>>>>>> back,
>>>>>>>>> repeat above:
>>>>>>>>> 0.00-9.94   sec   209 MBytes   176 Mbits/sec receiver
>>>>>>>>>
>>>>>>>>> Note the bandwidth drop (177 vs. 192).
>>>>>>>>>
>>>>>>>>> (Repeating scenario 1 will give now the same result)
>>>>>>>>>
>>>>>>>>> **Scenario 2.
>>>>>>>>> 1. Detach USB cable, attach a device, for example USB stick,
>>>>>>>>> 2. See it being enumerated and detach it.
>>>>>>>>> 3. Attach cable from host
>>>>>>>>> 4 .   0.00-19.36  sec   315 MBytes   136 Mbits/sec
>>>>>>>>> receiver
>>>>>>>>>
>>>>>>>>> Note even more bandwidth drop!
>>>>>>>>>
>>>>>>>>> (Repeating scenario 1 keeps the same lower bandwidth)
>>>>>>>>>
>>>>>>>>> NOTE, sometimes on this scenario after several seconds the target
>>>>>>>>> simply reboots (w/o any logs [from kernel] printed)!
>>>>>>>>>
>>>>>>>>> So, any pointers on how to debug and what can be a smoking gun 
>>>>>>>>> here?
>>>>>>>>>
>>>>>>>>> Ferry reported this in [2]. There are different kernel versions 
>>>>>>>>> and
>>>>>>>>> tools to establish the connection (like connman vs. none in my
>>>>>>>>> case).
>>>>>>>>>
>>>>>>>>> [1]:
>>>>>>>>> https://urldefense.com/v3/__https://github.com/andy-shev/linux/__;!!A4F2R9G_pg!KpQnudHIK6XgK6HbPaqtbVgipDmkNBWewo-euAIuBlGdtSiaQiJ8jLn9OoMEppG6qq-d$ 
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [2]:
>>>>>>>>> https://urldefense.com/v3/__https://github.com/andy-shev/linux/issues/31__;!!A4F2R9G_pg!KpQnudHIK6XgK6HbPaqtbVgipDmkNBWewo-euAIuBlGdtSiaQiJ8jLn9OoMEptMCrp-F$ 
>>>>>>>>>
>>>>>>>>>
>>>>>>>> dwc3 tracepoints should give some initial hints. Look at packets
>>>>>>>> sizes
>>>>>>>> and period of transmission. From dwc3 side, I can't think of
>>>>>>>> anything we
>>>>>>>> would do to throttle the transmission, but tracepoints should 
>>>>>>>> tell a
>>>>>>>> clearer story.
>>>>>>>>
>>>>>>> My testing (but yes, with difference kernel and network managed by
>>>>>>> connman) shows:
>>>>>>>
>>>>>>> 1) on cold boot eem network gadget works fine
>>>>>>>
>>>>>>> 2) after unplug or warm reboot (which is also an unplug) it's 
>>>>>>> broken,
>>>>>>> speed is lost (|12.0 Mbits/sec from 200Mb/s normally)|, packets 
>>>>>>> lost,
>>>>>>> no configuration received from dhcp, occasional reboot, only way to
>>>>>>> fix is cold boot
>>>>>>>
>>>>>>> 3) if before unplug `connmanctl disable gadget`, on replugging and
>>>>>>> enabling it works fine
>>>>>>>
>>>>>>> My theory is that some HW register is disturbed on a surprise 
>>>>>>> unplug,
>>>>>>> but not reset on plug or warm boot. But on cold boot is cleared.
>>>>>>> Maybe that can help to narrow down tracepoints?
>>>>>>>
>>>>>> I captured a plug after warm and after cold boot. This includes
>>>>>> network setup (dhcp). You can find it in [2] or directly link here:
>>>>>> https://urldefense.com/v3/__https://github.com/andy-shev/linux/files/6232410/boot.zip 
>>>>>>
>>>>>>
>>>>>
>>>>> While the above traces in boot.zip allow compare which regs not
>>>>> correctly initialized on warm boot, I have now captured traces of
>>>>> unplug/plug.
>>>>>
>>>>> Here kernel is 5.10.27 (LTS), cold booted with USB cable plugged 
>>>>> and the
>>>>> eem gadget network setup (dhcp). Then trace unplug. Then trace plug.
>>>>>
>>>>> After plug the eem connection is again broken.
>>>>>
>>>>> This might allow figuring out what goes wrong on unplug. Traces here:
>>>>> https://urldefense.com/v3/__https://github.com/andy-shev/linux/files/6250924/plug-unplug.zip 
>>>>>
>>>>>
>>>>> **
>>>>>
>>>> Hi,
>>>>
>>>> Were you able to narrow down the issue to only DWC3 device? (i.e. you
>>>> tested with different hosts and different device controllers to confirm
>>>> this)
>>> I haven't tried with other devices. I have been forced to replace my
>>> host mobo and nothing changed. But I didn't pay attention to the
>>> particular host controller.
>>>
>> It'd be better if we can narrow down the culprit as this seems to me
>> like a synchronization issue at the upper layer between the host and 
>> device.
>>
>>>> Did you see this issue previously? If not, is it possible to do git
>>>> bisection?
>>> This is with Intel Edison where main line usb gadget support appeared
>>> around 4.19 iirc. I believed the problem appeared between 5.4 and 5.7
>>> and tried to bisect but failed.
>>>
>>> I realize only now that I failed because:
>>> 1) 5.4 already has this issue as I recently retested
>> I'm confused, why do you believe the problem is between 5.4 and 5.7 if
>> 5.4 already has this issue? So when did you start seeing this problem?
> 
> Because at the time of 5.4 I didn't notice the issue as I normally did 
> cold boots due to other problems on warm boot (i.e. sdhc inaccessible).
> 
> I never new that on a cold boot it works. Even during bisecting I didn't 
> know until the end, and then I found 5.4 has the same problem as all the 
> later kernels (tested up to 5.11)
> 
>> Also, these kernel versions are really old, there's been a lot of
>> updates/fixes to dwc3 since then. Can we run tests on the latest kernel?
> 
> I have tested 5.10.27, 5.11.0 and 5.11.4-rt11.
> 
> But of course I am completely prepared to run Andy's latest (v5.12-rc5) 
> on the device.
> 
>>> 2) I didn't use a reproducible criterion. After warm reboot the eem
>>> gadget fails, but you can flip the host/gadget switch back and forth and
>>> have the illusion that the connection restored.
>>>
>>> The scenario described here is reproducible: leaving the switch in
>>> gadget mode eem works after cold boot only. And it likely breaks on 
>>> unplug.
>>>
>>> A 2nd hint is that disabling gadget (I used `connmanctl disable gadget`
>>> but I believe that has the same effect as `iw link set dev usb0 down`)
>>> before unplug prevents messing up the driver, so you can replug and
>>> enable again.
>> These data points are good. However, we'd need to know where to look
>> first. The issue isn't obvious from the DWC3 controller or the DWC3 
>> driver.
>>
>> Can you check a few things:
>> 1) Any error/timeout messages from the host's dmesg? Or device side?
> 
> I'll add log from the host side.
> 
> For now I only see (on a warm plug):
> 
> kernel: usb 1-11: can't set config #1, error -110
> 
>> 2) What kernel version is your host using? Can you use the latest for
>> both host and device?
> 
> The host is ubuntu's amd64 5.8.0-48-generic.
> 
> I will test with v5.12-rc5  from ubuntu kernel ppa on the host. And 
> Andy's latest (v5.12-rc5) on the device.

I upgraded host kernel, but not yet device and captured relevant host 
journal messages and device traces. Something did change: after cold 
boot I don't a eem until after I unplug/replug. I then traced a iperf 
transfer. Then after again unplug/replug I get the throttled connection, 
which I also traced.

See https://github.com/andy-shev/linux/files/6253414/transfer.zip


> I am expecting results this evening.
> 
>> 3) Snapshot of dwc3 tracepoints of active transfers between the normal
>> vs throttled of the latest kernel
> 
> I don't know if the problem I see is really throttling.
> 
> I can trace an active transfer, but that does actually throttle from 
> 200Mb/s down to 139MB/s and produces a trace of 53MB. (2x1sec of iperf3).
> 
>> BR,
>> Thinh