MUSB-HDRC Gadget driving VBUS inappropriately

MUSB-HDRC Gadget driving VBUS inappropriately

[Ian Coolidge]

Hi,

We're using Linux 3.3 on DM3730 with TPS6595xx PMIC for an embedded product.
For a particular board, we have musb-hdrc (RTL 1.4) configured in peripheral mode.
We use the Ethernet gadget configured for cdc_eem to use Ethernet emulation over USB for this link, and the robustness of the link is mission-critical. To assure this, we have performed some massive reboot testing to ensure that this usb link reliably comes up.

After working through an issue which required pulling in the following patch,
-----
commit b1b552a69b8805e7e338074a9e8b670b4a795218
Author: Michael Grzeschik <m.grzeschik-bIcnvbaLZ9MEGnE8C9+IrQ@public.gmane.org>
Date:   Wed Aug 8 11:48:10 2012 +0200

    usb: gadget: u_ether: fix kworker 100% CPU issue with still used interfaces in eth_stop
-----

we find that with about a 2% chance, the gadget comes up in some kind of firmware failed state, driving VBUS.
In this condition, we found that MUSB_DEVCTL_SESSION bit was set.
I understand this to be indicative of MUSB thinking it is in host mode, which agrees with the driven VBUS.
Further, in this state, I found that usually, there was one interrupt from twl4030_usb, but NO interrupts from musb-hdrc.

So, as a hack, I cleared MUSB_DEVCTL_SESSION in omap2430.c in the notifier block from the VBUS interrupt.
This appears to reduce the frequency of the problem -- I now always at least one instance of the musb-hdrc interrupt.
However, sometimes, it looks like the MUSB PHY dies shortly thereafter.
So, I don't think this hack is fully effective.

I reviewed commits that post-date 3.3 in omap2430.c, musb_core.c, twl4030-usb.c, musb_gadget.c, and I couldn't find anything interesting -- it looks like mostly reorganization has taken place.

We have also engaged TI to try to get some silicon errata from Mentor Graphics, and maybe a register manual for the MUSB HDRC IP, but this is slow going, and that has little guarantee of paying off anyways.

Are there any ideas, or patches that anyone might suggest?

Thanks!

Ian--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: MUSB-HDRC Gadget driving VBUS inappropriately

[Felipe Balbi]

[-- Attachment #1: Type: text/plain, Size: 2528 bytes --]

Hi,

On Fri, Dec 14, 2012 at 01:53:07AM -0800, Ian Coolidge wrote:
> Hi,
> 
> We're using Linux 3.3 on DM3730 with TPS6595xx PMIC for an embedded
> product.
> For a particular board, we have musb-hdrc (RTL 1.4) configured in peripheral mode.
> We use the Ethernet gadget configured for cdc_eem to use Ethernet
> emulation over USB for this link, and the robustness of the link is
> mission-critical. To assure this, we have performed some massive
> reboot testing to ensure that this usb link reliably comes up.
> 
> After working through an issue which required pulling in the following
> patch,
> -----
> commit b1b552a69b8805e7e338074a9e8b670b4a795218
> Author: Michael Grzeschik <m.grzeschik-bIcnvbaLZ9MEGnE8C9+IrQ@public.gmane.org>
> Date:   Wed Aug 8 11:48:10 2012 +0200
> 
>     usb: gadget: u_ether: fix kworker 100% CPU issue with still used interfaces in eth_stop
> -----
> 
> we find that with about a 2% chance, the gadget comes up in some kind
> of firmware failed state, driving VBUS.
> In this condition, we found that MUSB_DEVCTL_SESSION bit was set.
> I understand this to be indicative of MUSB thinking it is in host
> mode, which agrees with the driven VBUS.
> Further, in this state, I found that usually, there was one interrupt
> from twl4030_usb, but NO interrupts from musb-hdrc.
> 
> So, as a hack, I cleared MUSB_DEVCTL_SESSION in omap2430.c in the
> notifier block from the VBUS interrupt.
> This appears to reduce the frequency of the problem -- I now always at
> least one instance of the musb-hdrc interrupt.
> However, sometimes, it looks like the MUSB PHY dies shortly
> thereafter.
> So, I don't think this hack is fully effective.
> 
> I reviewed commits that post-date 3.3 in omap2430.c, musb_core.c,
> twl4030-usb.c, musb_gadget.c, and I couldn't find anything interesting
> -- it looks like mostly reorganization has taken place.
> 
> We have also engaged TI to try to get some silicon errata from Mentor
> Graphics, and maybe a register manual for the MUSB HDRC IP, but this
> is slow going, and that has little guarantee of paying off anyways.
> 
> Are there any ideas, or patches that anyone might suggest?

I don't think it's a silicon errata. Looks like a driver bug to me. But
since you're using such an old kernel, your TI support channel is the
only way forward, unless you can test with a more recent kernel like
v3.7.

From what you describe, however, it sounds like it's a problem with some
->set_vbus() call.

-- 
balbi

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: MUSB-HDRC Gadget driving VBUS inappropriately

[Grazvydas Ignotas]

On Fri, Dec 14, 2012 at 11:53 AM, Ian Coolidge <iancoolidge@gmail.com> wrote:
> we find that with about a 2% chance, the gadget comes up in some kind of firmware failed state, driving VBUS.
> In this condition, we found that MUSB_DEVCTL_SESSION bit was set.
> I understand this to be indicative of MUSB thinking it is in host mode, which agrees with the driven VBUS.
> Further, in this state, I found that usually, there was one interrupt from twl4030_usb, but NO interrupts from musb-hdrc.

I'm also also often seeing this and don't know any workaround except
powercycling the board :(
This was kind or relieved for me after I changed it to deinit musb on
shutdown/reset (3.3 should have that patch merged), however if you
randomly reset the board, there is still something like 30-50% chance
musb will come up dead, and on proper reset it's still something like
5% chance like you reported.


-- 
Gražvydas
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: MUSB-HDRC Gadget driving VBUS inappropriately

[Felipe Balbi]

[-- Attachment #1: Type: text/plain, Size: 1448 bytes --]

Hi,

On Fri, Dec 14, 2012 at 02:13:07PM +0200, Grazvydas Ignotas wrote:
> On Fri, Dec 14, 2012 at 11:53 AM, Ian Coolidge <iancoolidge@gmail.com> wrote:
> > we find that with about a 2% chance, the gadget comes up in some kind of firmware failed state, driving VBUS.
> > In this condition, we found that MUSB_DEVCTL_SESSION bit was set.
> > I understand this to be indicative of MUSB thinking it is in host mode, which agrees with the driven VBUS.
> > Further, in this state, I found that usually, there was one interrupt from twl4030_usb, but NO interrupts from musb-hdrc.
> 
> I'm also also often seeing this and don't know any workaround except
> powercycling the board :(
> This was kind or relieved for me after I changed it to deinit musb on
> shutdown/reset (3.3 should have that patch merged), however if you
> randomly reset the board, there is still something like 30-50% chance
> musb will come up dead, and on proper reset it's still something like
> 5% chance like you reported.

hehe, then you should've reported earlier :-)

Anyway, I really think this has something to do with some bogus
set_vbus() calls.

Likely that looking at probe() path for checks for MUSB_DEVCTL_VBUS will
probably show you something which is wrong. Maybe the driver is assuming
that if VBUS bitfield is zero, then it kicks host side, or something.

Go over that part of the code and you probably will find something.

-- 
balbi

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: MUSB-HDRC Gadget driving VBUS inappropriately

[Tony Lindgren]

* Ian Coolidge <iancoolidge@gmail.com> [121214 01:55]:
> Hi,
> 
> We're using Linux 3.3 on DM3730 with TPS6595xx PMIC for an embedded product.
> For a particular board, we have musb-hdrc (RTL 1.4) configured in peripheral mode.
> We use the Ethernet gadget configured for cdc_eem to use Ethernet emulation over USB for this link, and the robustness of the link is mission-critical. To assure this, we have performed some massive reboot testing to ensure that this usb link reliably comes up.

Hehehe I've never seen MUSB and mission critical in the same
sentence before :)
 
> After working through an issue which required pulling in the following patch,
> -----
> commit b1b552a69b8805e7e338074a9e8b670b4a795218
> Author: Michael Grzeschik <m.grzeschik@pengutronix.de>
> Date:   Wed Aug 8 11:48:10 2012 +0200
> 
>     usb: gadget: u_ether: fix kworker 100% CPU issue with still used interfaces in eth_stop
> -----
> 
> we find that with about a 2% chance, the gadget comes up in some kind of firmware failed state, driving VBUS.
> In this condition, we found that MUSB_DEVCTL_SESSION bit was set.
> I understand this to be indicative of MUSB thinking it is in host mode, which agrees with the driven VBUS.
> Further, in this state, I found that usually, there was one interrupt from twl4030_usb, but NO interrupts from musb-hdrc.
> 
> So, as a hack, I cleared MUSB_DEVCTL_SESSION in omap2430.c in the notifier block from the VBUS interrupt.
> This appears to reduce the frequency of the problem -- I now always at least one instance of the musb-hdrc interrupt.
> However, sometimes, it looks like the MUSB PHY dies shortly thereafter.
> So, I don't think this hack is fully effective.
> 
> I reviewed commits that post-date 3.3 in omap2430.c, musb_core.c, twl4030-usb.c, musb_gadget.c, and I couldn't find anything interesting -- it looks like mostly reorganization has taken place.
> 
> We have also engaged TI to try to get some silicon errata from Mentor Graphics, and maybe a register manual for the MUSB HDRC IP, but this is slow going, and that has little guarantee of paying off anyways.
> 
> Are there any ideas, or patches that anyone might suggest?

Don't know if this is related, but it might.

I've noticed and issu with MUSB host mode where MUSB with
USB-A cable plugged in fails to come up properly in host mode.

It seems to require replugging the USB cable to get it to
work in host mode. To reproduce this issue:

1. Boot system with USB-A cable connected and some devices

2. Load g_ether or some gadget module and notice how the
   USB devices are not discovered, and reloading g_ether
   does not help.

3. Unplug the USB-A cable, plug in USB-B cable, then
   plug in USB-A cable again, and the devices are discovered.

This one is annoying as it means that trying to use Panda
with USB HID devices connected to the MUSB port does not
discover them. And the reason to do that is to keep the
EHCI ports free for other use naturally..

Regards,

Tony

Re: MUSB-HDRC Gadget driving VBUS inappropriately

[Felipe Balbi]

[-- Attachment #1: Type: text/plain, Size: 2869 bytes --]

On Fri, Dec 14, 2012 at 05:38:10PM -0800, ian coolidge wrote:
> Felipe, Tony, Grazvydas, Thanks for the responses,
> 
> On Fri, Dec 14, 2012 at 4:13 AM, Felipe Balbi <balbi@ti.com> wrote:
> 
> > Hi,
> >
> > On Fri, Dec 14, 2012 at 02:13:07PM +0200, Grazvydas Ignotas wrote:
> > > On Fri, Dec 14, 2012 at 11:53 AM, Ian Coolidge <iancoolidge@gmail.com>
> > wrote:
> > > > we find that with about a 2% chance, the gadget comes up in some kind
> > of firmware failed state, driving VBUS.
> > > > In this condition, we found that MUSB_DEVCTL_SESSION bit was set.
> > > > I understand this to be indicative of MUSB thinking it is in host
> > mode, which agrees with the driven VBUS.
> > > > Further, in this state, I found that usually, there was one interrupt
> > from twl4030_usb, but NO interrupts from musb-hdrc.
> > >
> > > I'm also also often seeing this and don't know any workaround except
> > > powercycling the board :(
> > > This was kind or relieved for me after I changed it to deinit musb on
> > > shutdown/reset (3.3 should have that patch merged), however if you
> > > randomly reset the board, there is still something like 30-50% chance
> > > musb will come up dead, and on proper reset it's still something like
> > > 5% chance like you reported.
> >
> > hehe, then you should've reported earlier :-)
> >
> > Anyway, I really think this has something to do with some bogus
> > set_vbus() calls.
> >
> > Likely that looking at probe() path for checks for MUSB_DEVCTL_VBUS will
> > probably show you something which is wrong. Maybe the driver is assuming
> > that if VBUS bitfield is zero, then it kicks host side, or something.
> >
> > Go over that part of the code and you probably will find something.
> >
> > --
> > balbi
> >
> 
> So, I did some more testing, just printing out MUSB_DEVCTL each time. At
> omap2430.c "init",
> musb_probe()->musb_init_controller()->omap2430_musb_init(),
> I always saw 0x80. This corresponds to MUSB_DEVCTL_BDEVICE.
> It appears, then, that the MUSB is initialized correctly, but becomes bad
> somehow. I'll continue to dig into this, but it would be nice to have some
> simple abstract description of what the SESSION bit actually does here.
> It's used as both an input and an output throughout the MUSB Linux driver,
> and judging by comments, appears to have different behavior in different
> configurations. Felipe?

Session bit, really means a session, a USB session. It has different
meanings (to some extent) when working with Host or Gadget. For Gadget
mode, the session bit also triggers SRP, on host mode, maybe it's used
to start sourcing VBUS, who knows.

Also look at the usage of musb->is_active. That's a flag use for host
mode. IIRC, it tells other parts of the driver to connect the vbus
charge pump, but my memory fails now :-s

-- 
balbi

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]